Consider a linear 2nd-order ODE system, $$\dot{\mathbf{y}} = B \mathbf{y}$$ It may have one or a line of critical points, because unless $B = 0$, $B \text{y} =0$ has a unique solution or solutions forming a one-dimensional vector space. We assume $\det(B) \ne 0$ in the following discussion.
If $B$ has real (nonzero) eigenvalues $\lambda, \mu$, for example $B = \begin{pmatrix} \lambda & 0 \\ 0 & \mu \end{pmatrix}$, then $\begin{cases} y_1 = y_1(0) e^{\lambda t} \\ y_2 = y_2(0) e^{\mu t} \end{cases}$.
If $B$ has a pair of complex conjugate eigenvalues $a \pm ib$, for example $B = \begin{pmatrix} a & b \\ -b & a \end{pmatrix}$ , then $\begin{cases} y_1 = A e^{at} \sin(bt+\phi) \\ y_2 = A e^{at} \cos(bt+\phi) \end{cases}$
For linear systems, closed orbits exist around a center and have the same period. For a nonlinear system, a center found in its linearized form may not be an actual center; but symmetry of the original system about $y$-axis suffice to preserve this property.
For a nonlinear 2nd-order ODE system: $$\frac{\text{d}}{\text{d}t} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} f(x,y) \\ g(x,y) \end{pmatrix}$$
To get the phase plane near a critical point, we just need a coordinate transformation, which does not change the eigenvalues of $A$ and the type of the critical point.
If $\mathbf{v}$ is an eigenvector corresponding to eigenvalue $\lambda$ of the linearized system, along the line of the eigenvector we have $\dot{x} = \lambda x$. This means along the eigenvectors at the critical point, trajectories go directly into or away from the critical point, approximately in a straight line.
Trajectories in the Duffing equation
A form of Duffing equation is $$u'' + u - u^3 = 0$$ Or, in the form of ODE systems, $$\begin{cases} u' = v \\ v' = -u+u^3 \end{cases}$$
We find three critical points: $(0,0)$ and $(\pm 1,0)$. Furthermore, the first one is a center, and the other two are saddles.
Homoclinic orbits
A trajectory may connect two critical points, or forms a loop from and toward the same critical point. The former one is called a heteroclinic orbit, and the latter a homoclinic orbit.
We've already seen heteroclinic orbits in duffing equation, now we illustrate a homoclinic oribt by the following system.
$$\begin{cases} x' = y + y(1-x^2)[y^2-x^2(1-\frac{x^2}{2})] \\ y' = x(1-x^2) - y[y^2-x^2(1-\frac{x^2}{2})] \end{cases}$$
There are also three critical points: $(0,0)$ and $(\pm 1,0)$. The origin is a saddle, while the other two are unstable focus (foci).
Note: Period of homoclinic or heteroclinic orbit
Van der Pol equation has an attracting limit cycle.
The Poincare-Bendixson theorem and limit cycle
A limit cycle is a closed orbit that any trajectory nearby either goes towards it, or moves away. It is another important type of closed orbits.
Poincare-Bendixson Theorem: For a planar dynamical system, if a ring-shaped region between two simple closed curves has no critical point within, and any trajectory that intersects with the two boundary curves enters the region, then the region has at least one limit cycle. The conclusion holds if the inner boundary curve shrinks into an unstable node.
Hierarchy of local stabilities in nonlinear dynamical systems:
Notice that Lyapunov stability is distinct from local attractiveness.
An equilibrium is globally asymptotically stable if it is Lyapunov stable and globally attractive, that is, all trajectories converge to it.
A Lyapunov function $V(\mathbf{x})$ for an autonomous dynamical system $\dot{\mathbf{x}} = \mathbf{g}(\mathbf{x})$ with an equilibrium point at the origin is a locally positive-definite and continuously differentiable function such that its time derivative is locally negative-semidefinite:
An equilibrium is Lyapunov stable if and only if the dynamical system has a Lyapunov function. An equilibrium is locally asymptotically stable if and only if the dynamical system has a Lyapunov function whose time derivative is locally negative-definite. An equilibrium is globally asymptotically stable if the dynamical system has a radially unbounded global Lyapunov function whose time derivative is globally negative-definite.
Lyapunov function is analogous to the energy of a dissipative physical system, while not unique and thus easier to find.
A subset of the phase space is an attractor if it is the smallest subset of itself that is forward invariant and has a basin of attraction: $\Phi(A, t) \subseteq A, \forall t > 0$; and $\exists B \in \tau(X), B \supseteq A$, $\forall b \in B, \forall \varepsilon > 0, \exists T > 0: \Phi(b, t) \in N_\varepsilon(A), \forall t > T$.
Types of attractors:
The ω-limit set of a trajectory $\Phi(x, t)$ of a dynamical system is the set of limit points of all convergent subsequences of the forward trajectory: $\Omega_x = \{ \omega \mid \exists \{t_k\}, \lim_{k \to \infty} t_k = +\infty, \lim_{k \to \infty} \Phi(x, t_k) = \omega \}$.