Notes on Inner Product & Hilbert Space
Inner product $\langle \cdot, \cdot \rangle$ on a finite-dimensional vector space is a positive definite bilinear form: $x \ne 0$ then $\langle x, x \rangle > 0$. Inner product on an infinite-dimensional vector space is a positive definite bilinear functional. Inner product space $(V, (+, \cdot_\mathbb{F}), \langle \cdot, \cdot \rangle)$ is a vector space endowed with a specific inner product. Inner product specifies the geometry of a vector space, e.g. length and angle. Inner product space has norm $\|x\| = \sqrt{\langle x, x \rangle}$. Inner product space may have a finite or infinite number of dimensions. Given a basis of a finite-dimensional inner product space, the coordinate representation of its inner product is a positive definite matrix.
Euclidean space of dimension $n$ or Euclidean n-space $(\mathbb{R}^n, +, \cdot_{\mathbb{R}}, (\cdot, \cdot))$ is the product space of real numbers to the power of $n$, endowed with the inner product $(x,y) = \sum_{i=1}^n x_i y_i$. Euclidean spaces generalize two- and three-dimensional spaces of Euclidean geometry to arbitrary finite-dimensional spaces. It is the abstraction of a geometric object into a topological and algebraic structure. Many concepts in algebra and analysis are developed by analogy with geometry, where Euclidean space is among the first few.
Linear isometry $T: V \mapsto W$ between inner product spaces is a vector space isomorphism that takes the inner product of the domain to that of the codomain: $\forall v, w \in V, \langle Tv, Tw \rangle = \langle v, w \rangle$. Any linear operator between $n$-dimensional inner product spaces that maps an orthonormal basis to an orthonormal basis is a linear isometry. Any $n$-dimensional inner product space over the real numbers is linearly isometric with the Euclidean $n$-space: $V \cong \mathbb{R}^n$.
Hilbert Space is a complete inner product space. Hilbert spaces typically arise as infinite-dimensional function spaces, e.g. the $l^2$ space of square summable infinite sequences, $L^2$ spaces of square-integrable functions, $H^s$ Sobolev spaces of twice square-integrably weakly-differentiable functions, and Hardy spaces $H^2(U)$ and Bergman spaces $L^2_a(G)$ of holomorphic functions.
Orthogonal vectors of an inner product space are two vectors whose inner product is zero: $\langle v, w \rangle = 0$. Orthogonal subset of an inner product space is a set of vectors that are pairwise orthogonal: $v, w \in S, v \ne w$, then $\langle v, w \rangle = 0$. Orthonormal subset of an inner product space is an orthogonal subset consisting of unit-length vectors: $\langle v_i, v_j \rangle = \delta^i_j$. Orthogonal complement $S^\perp$ of a linear subspace of an inner product space is the linear subspace consisting of all vectors in the inner product space that are orthogonal to the subspace: $S^\perp = \{v \in V: \forall w \in S, \langle v, w \rangle = 0\}$. Orthogonal projection $\pi: V \mapsto S$ from an inner product space onto a subspace is the projection whose kernel is the orthogonal complement of the subspace: $\pi(v + w) = v$ where $v \in S, w \in S^\perp$.
Maximal/complete orthogonal set. Orthonormal basis. Every finite-dimensional inner product space has an orthonormal basis. Gram-Schmidt orthogonalization/orthonormalization, QR decomposition. Orthogonal transformation: rotation, reflection; improper rotation.
Approximation of a vector in a vector space on a subspace is a vector on the subspace that is closest to the point. Approximation in Banach and Hilbert spaces: For a Banach space, an approximation may not exist. For a Hilbert space, the approximation of any point on any subspace exists and is unique, which is the orthogonal projection of that point.
Riesz Representation Theorem...
Lax-Milgram Theorem...
For a Hilbert space $(H, \langle \cdot , \cdot \rangle)$ of scalar functions on $X$, a scalar function $K(x,y)$ on $X \times X$ is a reproducing kernel of the Hilbert space if: let $K_y(x) \equiv K(x,y)$,
Reproducing kernels are symmetric and positive definite.
Reproducing kernel Hilbert space (RKHS) is a Hilbert space with a reproducing kernel [@Aronszajn1950].
Theorem (equivalent definition of RKHS): A Hilbert space of functions on a set $X$ is a reproducing kernel Hilbert space iff $f(y) \le c(y) \|f\|, \forall y \in X$, where $c(y) \equiv \|K_y\|$.
A reproducing kernel Hilbert space uniquely defines a reproducing kernel which is symmetric and positive definite.
Theorem: For a symmetric positive definite kernel $K$ on a set $X$, there is a unique Hilbert space of functions on $X$ for which $K$ is a reproducing kernel.
Therefore, reproducing kernel Hilbert spaces of functions on a domain are in one-to-one correspondence with positive definite kernels on the domain. We can denote any reproducing kernel Hilbert space as $H_K$, where $K$ is the unique reproducing kernel of the Hilbert space and $H_K$ is the unique Hilbert space generated by the symmetric positive definite kernel $K$.
If the space $H_K$ is sufficiently rich, then the reproducing kernel $K$ is positive definite.
Let $X$ has measure $\mu$, which defines a Hilbert space $(L^2_\mu, \langle \cdot , \cdot \rangle_\mu)$. Define integral operator $L_K$ on $H_K$ as $(L_K f)(x) = \langle K_x, f \rangle_\mu$. If $X$ is compact, then $L_K$ is compact and self-adjoint w.r.t $L^2_\mu$, so its eigenfunctions $\{e_i\}_{i \in \mathbb{N}}$ form an orthonormal basis of $L^2_\mu$, its eigenvalues $\{\lambda_i\}_{i \in \mathbb{N}}$ have finite multiplicities and converge to zero, and $K(x,y) = \sum_{i \in \mathbb{N}} \lambda_i e_i(x) e_i(y)$. If $f(x) = \sum_{i \in \mathbb{N}} a_i e_i(x)$, then $L_K f = \sum_{i \in \mathbb{N}} \lambda_i a_i e_i(x)$. It can be shown that the eigenfunctions are in $H_K$, so $\langle e_i, e_j \rangle = \delta_{ij} / \lambda_i$, and $f(x) = \sum_{i \in \mathbb{N}} a_i e_i(x) \in H_K$ iff $\sum_{i \in \mathbb{N}} a_i^2 / \lambda_i < \infty$. Let $L_K^{1/2}$ be the only positive definite self-adjoint operator that satifies $L_K^{1/2} \circ L_K^{1/2} = L_K$, then $L_K^{1/2}$ is an isomorphism from $L^2_\mu$ to $H_K$.