Notes on Inner Product & Hilbert Space
Inner product $\langle \cdot, \cdot \rangle$ on a finite-dimensional vector space is a positive definite bilinear form: $x \ne 0$ then $\langle x, x \rangle > 0$. Inner product on a vector space is a positive definite bilinear functional. Inner product space $(V, (+, \cdot_\mathbb{F}), \langle \cdot, \cdot \rangle)$ is a vector space endowed with a specific inner product. Inner product specifies the geometry of a vector space, e.g. length and angle. Inner product space has norm $\|x\| = \sqrt{\langle x, x \rangle}$. Inner product space may have a finite or infinite number of dimensions. Given a basis of a finite-dimensional inner product space, the coordinate representation of its inner product is a positive definite matrix.
Euclidean space of dimension $n$ or Euclidean n-space $(\mathbb{R}^n, +, \cdot_{\mathbb{R}}, (\cdot, \cdot))$ is the product space of real numbers to the power of $n$, endowed with the inner product $(x,y) = \sum_{i=1}^n x_i y_i$. Euclidean spaces generalize two- and three-dimensional spaces of Euclidean geometry to arbitrary finite-dimensional spaces. It is the abstraction of a geometric object into a topological and algebraic structure. Many concepts in algebra and analysis are developed by analogy with geometry, where Euclidean space is among the first few.
Linear isometry $T: V \mapsto W$ between inner product spaces is a vector space isomorphism that takes the inner product of the domain to that of the codomain: $\forall v, w \in V, \langle Tv, Tw \rangle = \langle v, w \rangle$. Any linear operator between n-dimensional inner product spaces that maps an orthonormal basis to an orthonormal basis is a linear isometry. Any n-dimensional inner product space over the real numbers is linearly isometric with the Euclidean n-space: $V \cong \mathbb{R}^n$.
Hilbert Space $(H, (+, \cdot_\mathbb{F}), \langle \cdot, \cdot \rangle)$ is a complete inner product space. Here, completeness is in regard to the metric induced by the inner product. Hilbert spaces typically arise as infinite-dimensional function spaces, e.g. the $l^2$ space of square summable infinite sequences, $L^2$ spaces of square-integrable functions, $H^s$ Sobolev spaces of twice square-integrably weakly-differentiable functions, and Hardy spaces $H^2(U)$ and Bergman spaces $L^2_a(G)$ of holomorphic functions.
Orthogonal vectors of an inner product space are two vectors whose inner product is zero: $\langle v, w \rangle = 0$. Orthogonal subset of an inner product space is a set of vectors that are pairwise orthogonal: $v, w \in S, v \ne w$, then $\langle v, w \rangle = 0$. Orthonormal subset of an inner product space is an orthogonal subset consisting of unit-length vectors: $\langle v_i, v_j \rangle = \delta^i_j$. Orthogonal complement $S^\perp$ of a linear subspace of an inner product space is the linear subspace consisting of all vectors in the inner product space that are orthogonal to the subspace: $S^\perp = \{v \in V: \forall w \in S, \langle v, w \rangle = 0\}$. Orthogonal projection $\pi: V \mapsto S$ from an inner product space onto a subspace is the projection whose kernel is the orthogonal complement of the subspace: $\pi(v + w) = v$ where $v \in S, w \in S^\perp$.
Maximal/complete orthogonal set. Orthonormal basis. Every finite-dimensional inner product space has an orthonormal basis. Gram-Schmidt orthogonalization/orthonormalization, QR decomposition. Orthogonal transformation: rotation, reflection; improper rotation.
Approximation of a vector in a vector space on a subspace is a vector on the subspace that is closest to the point. Approximation in Banach and Hilbert spaces: For a Banach space, an approximation may not exist. For a Hilbert space, the approximation of any point on any subspace exists and is unique, which is the orthogonal projection of that point.
Riesz Representation Theorem...
Lax-Milgram Theorem...
Reproducing kernel $K(x, y)$ of a Hilbert space of scalar-valued functions is a bivariate scalar-valued function such that with one variable fixed at any value (1) it is an element of the space, and (2) the inner product with any element of the space equals the value of that function at the fixed variable: $(H, (+, \cdot_\mathbb{F}), \langle \cdot, \cdot \rangle)$, $H \subset \mathbb{F}^X$, let $K_y(x) = K(x, y)$, then $\{K_y : y \in X\} \subset H$ and $\forall f \in H$, $\forall y \in X$, $\langle f, K_y \rangle = f(y)$. The second property satisfied by a reproducing kernel is called the reproducing property, because it defines an operator that "reproduces" any function in the Hilbert space: $K: H \mapsto H$; $\forall f \in H$, $\forall y \in X$, $(K f)(y) = \langle f, K_y \rangle$, $K f = f$. But the identity map is not a reproducing kernel unless the Hilbert space contains all indicator functions of singletons.
Reproducing kernels are self-adjoint and positive semi-definite: $K(x, y) = \bar{K}(y, x)$;.
Reproducing kernel Hilbert space (RKHS) is a Hilbert space of scalar-valued functions with a reproducing kernel [@Aronszajn1950]. A Hilbert space of scalar-valued functions has a reproducing kernel if and only if it is uniformly bounded: $\forall f \in H$, $\forall y \in X$, $f(y) \le c(y) \|f\|$. For a reproducing kernel Hilbert space, the bound $c(y) = \|K_y\|$ is tight.
A reproducing kernel Hilbert space uniquely defines a reproducing kernel which is symmetric and positive definite.
Theorem: For a symmetric positive definite kernel $K$ on a set $X$, there is a unique Hilbert space of functions on $X$ for which $K$ is a reproducing kernel.
Therefore, reproducing kernel Hilbert spaces of functions on a domain are in one-to-one correspondence with positive definite kernels on the domain. We can denote any reproducing kernel Hilbert space as $H_K$, where $K$ is the unique reproducing kernel of the Hilbert space and $H_K$ is the unique Hilbert space generated by the symmetric positive definite kernel $K$.
If the space $H_K$ is sufficiently rich, then the reproducing kernel $K$ is positive definite.
Let $X$ has measure $\mu$, which defines a Hilbert space $(L^2_\mu, \langle \cdot , \cdot \rangle_\mu)$. Define integral operator $L_K$ on $H_K$ as $(L_K f)(x) = \langle K_x, f \rangle_\mu$. If $X$ is compact, then $L_K$ is compact and self-adjoint w.r.t $L^2_\mu$, so its eigenfunctions $\{e_i\}_{i \in \mathbb{N}}$ form an orthonormal basis of $L^2_\mu$, its eigenvalues $\{\lambda_i\}_{i \in \mathbb{N}}$ have finite multiplicities and converge to zero, and $K(x,y) = \sum_{i \in \mathbb{N}} \lambda_i e_i(x) e_i(y)$. If $f(x) = \sum_{i \in \mathbb{N}} a_i e_i(x)$, then $L_K f = \sum_{i \in \mathbb{N}} \lambda_i a_i e_i(x)$. It can be shown that the eigenfunctions are in $H_K$, so $\langle e_i, e_j \rangle = \delta_{ij} / \lambda_i$, and $f(x) = \sum_{i \in \mathbb{N}} a_i e_i(x) \in H_K$ iff $\sum_{i \in \mathbb{N}} a_i^2 / \lambda_i < \infty$. Let $L_K^{1/2}$ be the only positive definite self-adjoint operator that satifies $L_K^{1/2} \circ L_K^{1/2} = L_K$, then $L_K^{1/2}$ is an isomorphism from $L^2_\mu$ to $H_K$.