In functional analysis, the theories of inner product space, Hilbert space, and linear operators on such spaces are simpler and richer than those of the more general normed space and Banach space.
Notes on Inner Product & Hilbert Space
Sesquilinear form (半双线性形式) or sesquilinear functional $h: V \times W \mapsto \mathbb{F}$ on the Cartesian product of two vector spaces over the same field is a scalar-valued function that is linear in the first variable and conjugate linear in the second, $h(a v, w) = a h(v, w)$, $h(v, a w) = \bar a h(v, w)$; $h(u + v, w) = h(u, w) + h(v, w)$, $h(u, v + w) = h(u, v) + h(u, w)$. Sesquilinear form $h: V^2 \mapsto \mathbb{F}$ on a vector space refers to a sesquilinear form on the Cartesian square of the vector space. Bilinear form (双线性形式) $q: V \times W \mapsto \mathbb{R}$ is a sesquilinear form where the scalar field is the real numbers. Note that bilinear forms and covariant 2-tensors are identical. Norm $\|h\|$ of a sesquilinear form on two normed spaces over the same field is the supremum of its normalized absoute value, if exists: $\|h\| = \sup_{v, w \ne 0} \frac{|h(v, w)|}{\|v\| \|w\|}$, or equivalently, $\|h\| = \sup_{\|v\|, \|w\| = 1} |h(v, w)|$. Bounded sesquilinear form on two normed spaces over the same field is a sesquilinear form with a finite norm: $\|h\| < \infty$. Space of bounded sesquilinear forms $B(V \times W)$ on two normed spaces over the same field is the normed space of all bounded sesquilinear forms on the two spaces, endowed with the norm of sesquilinear forms. Coercive sesquilinear form $h: V^2 \mapsto \mathbb{F}$ on a normed space is a sesquilinear form which as a functional has a positive absolute lower bound: $\inf_{v \ne 0} \frac{|h(v, v)|}{\|v\|^2} > 0$.
Inner product $\langle \cdot, \cdot \rangle$ on a vector space over real/complex numbers is a (symmetric/Hermitian) positive definite bilinear/sesquilinear form on the vector space: $\langle v, w \rangle = \overline{\langle w, v \rangle}$; $v \ne 0$ then $\langle v, v \rangle > 0$. Every inner product defines a norm as the square root of the inner product of a vector with itself: $\|x\| = \sqrt{\langle x, x \rangle}$. Schwarz inequality: Normalized inner product lies between -1 and 1; $|\langle v, w \rangle| \le \|v\| \|w\|$. Inner product space $(V, (+, \cdot_\mathbb{F}), \langle \cdot, \cdot \rangle)$ is a vector space endowed with a specific inner product. Inner product specifies the geometry of a vector space, e.g. length and angle. Every inner product space is a normed space. The inner product in any inner product space is a continuous map: $\langle \cdot, \cdot \rangle \in C(V^2, \mathbb{F})$. Inner product space may have a finite or infinite dimension. For a finite-dimensional inner product space, the coordinate representation of its inner product w.r.t. any basis is a positive definite matrix. Gram determinant $G(v_i)_{i=1}^n$ of an n-tuple of vectors in an inner product space is the determinant of the matrix of their inner products: $G(v_i)_{i=1}^n = \det(\langle v_i, v_j \rangle)^{i \in n}_{j \in n}$. The Gram determinant of n vectors equals the squared n-volume of the parallelotope they determine: $G(v_i)_{i=1}^n = \det([V]^T [V]) = \det(V)^2$, where $[V]$ is the matrix representation of the vectors in any n orthonormal vectors whose span contain the n vectors. The Gram determinant operator $G: V^n \mapsto \mathbb{R}_{\ge 0}$ is a nonnegative symmetric tensor. A finite subset of vectors in an inner product space is linearly independent if and only if its Gram determinant is nonzero.
Euclidean space of dimension $n$ or Euclidean n-space $(\mathbb{R}^n, +, \cdot_{\mathbb{R}}, (\cdot, \cdot))$ is the product space of real numbers to the power of $n$, endowed with the inner product $(x,y) = \sum_{i=1}^n x_i y_i$. Euclidean spaces generalize two- and three-dimensional spaces of Euclidean geometry to arbitrary finite-dimensional spaces. It is the abstraction of a geometric object into a topological and algebraic structure. Many concepts in algebra and analysis are developed by analogy with geometry, where Euclidean space is among the first few.
Hilbert space $H$ is an inner product space that is complete in the metric induced by the inner product. Every Hilbert space is a strictly convex Banach space. Hilbert spaces typically arise as infinite-dimensional function spaces, e.g. the $l^2$ space of square summable infinite sequences, $L^2$ spaces of square-integrable functions, $H^s$ Sobolev spaces of twice square-integrably weakly-differentiable functions, and Hardy spaces $H^2(U)$ and Bergman spaces $L^2_a(G)$ of holomorphic functions.
Subspace $(S, (+, \cdot_\mathbb{F}), \langle \cdot, \cdot \rangle)$ of an inner product space is an inner product space consisting of a linear subspace and the inner product restricted to the Cartesian square of the subspace. A subspace of a Hilbert space is a Hilbert space if and only if it is a closed subset.
Linear isometry or inner product space isomorphism $T: V \mapsto W$ between inner product spaces is a vector space isomorphism that takes the inner product of the domain to that of the codomain: $\forall v, v' \in V, \langle Tv, Tv' \rangle = \langle v, v' \rangle$. Any linear operator between n-dimensional inner product spaces that maps an orthonormal basis to an orthonormal basis is a linear isometry. Any n-dimensional inner product space over the real numbers is linearly isometric with the Euclidean n-space: $V \cong \mathbb{R}^n$. Completion of inner product space: Every inner product space is linearly isometric to a dense subspace of a unique Hilbert space up to linear isometries. Completion $\widehat V$ of an inner product space is the Hilbert space with a dense subspace isomorphic to the inner product space.
Orthogonal vectors $v \perp w$ of an inner product space are two vectors whose inner product is zero: $\langle v, w \rangle = 0$. A vector and a subset of an inner product space are orthogonal $v \perp S$ if the vector is orthogonal to all vectors in the subset: $\forall w \in S$, $v \perp w$. Annihilator $S^\perp$ of a subset $S$ of an inner product space is the set of all vectors orthogonal to the subset: $S^\perp = \{v : v \in V, v \perp S\}$. The annihilator of any subset of an inner product space is a closed subspace: $\overline{S^\perp} = S^\perp$. The second annihilator of any subset of an inner product space includes the subset: $S \subset S^{\perp\perp}$. Orthogonal complement $S^\perp$ of a subspace of an inner product space is the annihilator of the subspace. Every Hilbert space is the direct sum of any closed subspace and its orthogonal complement: $H = \overline{S} \oplus \overline{S}^\perp$. A subset of a Hilbert space is total (see Normed Space) if and only if there is no nonzero vector orthogonal to it: $\overline{\text{Span}(S)} = H$ iff $S^\perp = \{0\}$. The second orthogonal complement of any closed subspace of a Hilbert space is the subspace: $S \subset H$, $\bar S = S$, $\text{Span}(S) = S$ then $S = S^{\perp\perp}$. Orthogonal projection $\pi: V \mapsto S$ from an inner product space onto a subspace is the projection whose kernel is the orthogonal complement of the subspace: $\pi(v + w) = v$ where $v \in S, w \in S^\perp$. Every orthogonal projection is a bounded linear operator that is surjective and idenpotent: $\pi \in B(V, S)$, $\pi(V) = S$, $\pi^2 = \pi$.
Orthogonal subset of an inner product space is a set of vectors that are pairwise orthogonal: $v, w \in S, v \ne w$, then $\langle v, w \rangle = 0$. An orthogonal subset is linearly independent if it does not include the zero vector. Orthonormal subset $(e_\alpha)_{\alpha \in A}$ of an inner product space is an orthogonal subset consisting of unit-length vectors: $\langle e_\alpha, e_\beta \rangle = \delta^\alpha_\beta$. Gram-Schmidt Algorithm for linearly independent sequences: An orthonormal sequence in an inner product space can be contructed from a linearly independent sequence such that their principal subtuples of the same size span to the same subspace. The sum of an orthogonal sequence in a Hilbert space converges if and only if The sum of their square norms converges: $\sum_{i=1}^\infty a_i e_i \in H$ iff $\sum_{i=1}^\infty |a_i|^2 < \infty$. Fourier coefficient $\langle x, e_i \rangle$ of a vector in an inner product space w.r.t. an orthonormal sequence $(e_i)_{i=1}^\infty$ is its inner product with the i-th term of the sequence. The Fourier coefficients of the limit of a convergent orthogonal sequence in a Hilbert space equals the norms of terms of the sequence: $\sum_{i=1}^\infty a_i e_i = x$ then $\langle x, e_i \rangle = a_i$. Bessel Inequality: Given an orthonormal sequence in an inner product space, the sum of square Fourier coefficients of any vector in the space is no greater than its square norm; $\sum_{i=1}^\infty |\langle x, e_i \rangle|^2 \le \|x\|^2$. Any vector in an inner product space can have at most countably many nonzero Fourier coefficients w.r.t. an orthonormal subset: $\overline{\{\alpha: \alpha \in A, \langle x, e_\alpha \rangle \ne 0\}} \le \omega$. Total orthonormal subset of an inner product space is an orthonormal subset that is a total subset: $\overline{\text{Span}(e_\alpha)_{\alpha \in A}} = X$, $\langle e_\alpha, e_\beta \rangle = \delta^\alpha_\beta$. A total orthonormal subset is a basis if and only if it is finite. Every Hilbert space has a total orthonormal subset. An orthonormal subset in a Hilbert space is total if and only if every vector in the space can be written as the sum of squares of all nonzero Fourier coefficients w.r.t. the subset, which is well-defined because it is at most a countable sequence: $\sum_{\alpha \in A} |\langle x, e_\alpha \rangle|^2 = \|x\|^2$. The total orthonormal subsets of a Hilbert space have the same cardinality. Hilbert dimension of a Hilbert space is the cardinality of any of its total orthonormal subsets: $\dim H = |A|$. A Hilbert space is separable if and only if it has a total orthonormal sequence: $\dim H \le \aleph_0$. Hilbert spaces over the same field are isomorphic if and only if they have the same Hilbert dimension: $H \cong H'$ iff $\dim H = \dim H'$. Thus, every Hilbert space is isomorphic to every Hilbert space of its scalar field to the power of the cardinality of a total orthonormal set: $\widehat{\text{Span}}(E) \cong \widehat{\mathbb{F}^{|E|}}$. In particular, every Hilbert space of all scalar-valued functions on a set is isomorphic to every Hilbert space of the scalar field to the power of the cardinality of the set: $\widehat{\mathbb{F}^X} \cong \widehat{\mathbb{F}^{|X|}}$. (Although this generalizes the result for isomorphic finite-dimensional vector spaces, we are almost never interested in arbitary functions on infinite sets.)
Because every Hilbert space is the direct sum of any Hilbert subspace and its orthogonal complement, every vector in a Hilbert space has a unique best approximation in every closed subspace, which is the orthogonal projection of that vector to the subspace: $H_0 \subset H$ then $\forall x \in H$, $P x = \arg\min_{y \in H_0} d(x, y)$. The distance between a vector and its best approximation on a finite-dimensional subspace equals the ratio of the volumes of two parallelotopes, one determined by the vector and any basis of the subspace, the other by the basis only; writing volume as the square root of a Gram determinant, we have: $d(x, Px) = \sqrt{\frac{G(x, y_i)_{i=1}^n}{G(y_i)_{i=1}^n}}$; with an orthonormal basis, we have $d(x, Px) = \sqrt{G(x, e_i)_{i=1}^n}$, which gives $d(x, Px) = \sqrt{\|x\|^2 - \sum_{i=1}^n |\langle x, e_i \rangle|^2}$.
The space $L^2(\mathbb{R})$ of Lebesgue square integrable real functions is separable.
Riesz Representation Theorem of Covectors: Every bounded linear functional on a Hilbert space can be written uniquely as the inner product with a vector in the space: $\forall \omega \in H^∗$, $\exists! w \in H$: $\|w\| = \|\omega\|$, $\omega = \langle \cdot, w \rangle$.
Lax-Milgram Lemma: The Riesz representation theorem still holds if the inner product is replaced by any bounded coercive sesquilinear form. $h \in B(H^2)$, $\inf_{\|v\| = 1} |h(v, v)| > 0$, then $\forall \omega \in H^∗$, $\exists! w \in H$: $\|w\| = \|\omega\|$, $\omega = h(\cdot, w)$.
Riesz Representation Theorem of Sesquilinear Forms: Every bounded sesquilinear form on two Hilbert spaces over the same field can be written uniquely as the inner product of one space where vectors in the other space are mapped into the space by a bounded linear operator: $\forall h \in B(H_1 \times H_2)$, $\exists! T \in B(H_1, H_2)$: $\|T\| = \|h\|$, $h = \langle T \cdot, \cdot \rangle_2$.
Zero Linear Operator: A bounded linear operator between inner product spaces is zero if and only if it represents the zero sesquilinear form: $T \in B(V, W)$, then $T = 0$ iff $\langle T \cdot, \cdot \rangle_W = 0$. A bounded linear transformation on a complex inner product space is zero if the inner product of any vector and its value is zero: $T \in B(V, V)$, $\{\langle T v, v \rangle : v \in V\} = \{0\}$, then $T = 0$.
Hilbert-adjoint operator $T^∗: H_2 \mapsto H_1$ of a bounded linear operator $T: H_1 \mapsto H_2$ between Hilbert spaces is the bounded linear operator that represents the same bounded sesquilinear form: $\forall v \in H_1$, $\forall w \in H_2$, $\langle T v, w \rangle_2 = \langle v, T^∗ w \rangle_1$. Every bounded linear operator and its Hilbert-adjoint between two given Hilbert spaces have the same operator norm: $\forall T \in \mathcal{L}(H_1, H_2) \cap C(H_1, H_2)$, $\|T\| = \|T^∗\|$. The conjugation map $∗$ is an involution that is conjugate linear and composable, and the composite transformation squares the operator norm: $T^{∗∗} = T$, $(a S + b T)^∗ = \bar{a} S^∗ + \bar{b} T^∗$, $(S T)^∗ = T^∗ S^∗$, $\|T^∗ T\| = \|T\|^2$.
Self-adjoint operator or Hermitian operator on a Hilbert space is a bounded linear operator that equals its adjoint operator: $T \in \mathcal{L}(H, H) \cap C(H, H)$, $T = T^∗$. A bounded linear operator on a complex Hilbert space is self-adjoint if and only if the inner product of any vector and its value is real given $T \in \mathcal{L}(H, H) \cap C(H, H)$, $T = T^∗$ iff $\forall v \in H$, $\langle T v, v \rangle \in \mathbb{R}$. The limit operator of a convergent sequence of self-adjoint operators on a Hilbert space is also self-adjoint, i.e. the set of all self-adjoint operators on a Hilbert space is a closed subset of the bounded linear operators on the space: $\forall n \in \mathbb{N}$, $T_n = T_n^∗$, $\exists T \in B(H, H)$: $\lim_{n \to \infty} |T_n - T| = 0$, then $T = T^∗$. Boundedness of Self-adjoint Operators [@Hellinger and Toeplitz, 1910]: Every self-adjoint linear operator on a Hilbert space is bounded: $T \in \mathcal{L}(H, H)$, $\langle T v, w \rangle = \overline{\langle T w, v \rangle}$, then $T \in B(H, H)$.
Unitary operator on a Hilbert space is an invertible bounded linear operator whose inverse equals its adjoint: $U^{-1} = U^∗$. A bounded linear operator on a complex Hilbert space is unitary if and only if it is isometric and surjective. Every unitary operator on a Hilbert space is a linear isometry, has unit operator norm, and has unitary adjoint: $\langle U v, U w \rangle = \langle v, w \rangle$; $\|U\| = 1$; $(U^∗)^{-1} = (U^∗)^∗$. Note that an isometric operator need not be unitary since it may not be surjective. The composition of two unitary operators on a Hilbert space is also unitary: $(U V)^{-1} = (U V)^∗$.
Normal operator on a Hilbert space is a bounded linear operator that commutes with its adjoint: $T^∗ T = T^∗ T$. A bounded linear operator on a Hilbert space is normal if it is self-adjoint or unitary.
The spectrum of a self-adjoint operator is real, and lies within the infimum and the supremum the inner products of unit vectors and their values. Eigenspaces corresponding to different eigenvalues are orthogonal. ...
Every self-adjoint operator can be represented by an integral determined by its spectral family. Spectral family or decomposition of unity is a family of projection operators having certain properties.
Harmonic analysis... Laplace–Beltrami operator
...
Evaluation functional $E_x$ on a function space is a functional that maps every function in the space to its value at a given point: $V \subset \mathbb{F}^X$, $\forall f \in V$, $E_x f = f(x)$. Reproducing kernel Hilbert space (RKHS) is a Hilbert space of scalar-valued functions where every evaluation functional is bounded: $H \subset \mathbb{F}^X$, $\{E_x : x \in X\} \subset H^\#$. By the Riesz representation theorem, every evaluation functional in an RKHS determines a unique vector in the space: $\forall x \in X$, $\exists! k_x \in H$: $E_x = \langle \cdot, k_x \rangle$.
Reproducing kernel $k: X^2 \mapsto \mathbb{F}$ of a Hilbert space of scalar-valued functions is a bivariate scalar-valued function such that with one variable fixed at any value (1) it is an element of the space, and (2) the inner product with any element of the space equals the value of that function at the fixed variable: $(H, (+, \cdot_\mathbb{F}), \langle \cdot, \cdot \rangle)$, $H \subset \mathbb{F}^X$, $k_y(x) := k(x, y)$, then $\{k_y : y \in X\} \subset H$ and $\forall f \in H$, $\forall y \in X$, $\langle f, k_y \rangle = f(y)$. The second property satisfied by a reproducing kernel is called the reproducing property, because it defines an operator $K: H \mapsto H$ that "reproduces" any function in the Hilbert space: $\forall f \in H$, $\forall y \in X$, $(K f)(y) := \langle f, k_y \rangle$, hence $K f = f$; but the identity map is not a reproducing kernel unless the Hilbert space contains all indicator functions of singletons.
Positive (semi-)definite kernel, or simply kernel, is a bivariate scalar-valued function that is self-adjoint and, for any finite subset of the underlying space, its values at all pairs of the subset form a positive semi-definite matrix: $k(x, y) = \bar{k}(y, x)$; $\forall n \in \mathbb{N}_+$, $\forall I \in X^n$, $[k]_{I,I} \succeq 0$. Real-valued positive semi-definite kernels form a function algebra over non-negative real numbers: $(\mathcal{K}_+, (+, \cdot_{\mathbb{R}_{\ge 0}}, \times))$. Every reproducing kernel is positive (semi-)definite.
Reproducing kernel Hilbert space (RKHS) $H_K$ is a Hilbert space of scalar-valued functions with a reproducing kernel [@Aronszajn1950]. A Hilbert space of scalar-valued functions has a reproducing kernel if and only if it is uniformly bounded: $\forall f \in H$, $\forall y \in X$, $f(y) \le c(y) \|f\|$. For an RKHS, the bound $c(y) = \|k_y\|$ is tight. Every RKHS has a unique reproducing kernel. Moore–Aronszajn theorem: For any positive semi-definite bivariate function on a set, there is a unique Hilbert space of scalar-valued functions on the set such that the bivariate function is a reproducing kernel. Therefore, RKHSs of scalar-valued functions on a set are in one-to-one correspondence with positive semi-definite bivariate functions on the set. ... Given an orthonormal basis $(\psi_\alpha)_\alpha$ for the RKHS, its reproducing kernel can be written as: $k(x, y) = \sum_{\alpha} \psi_\alpha(x) \bar{\psi}_\alpha(y)$.
If an RKHS is sufficiently rich, then its reproducing kernel is positive definite, which defines another inner product on the Hilbert space. ...
Let $X$ has measure $\mu$, which defines a Hilbert space $(L^2_\mu, \langle \cdot , \cdot \rangle_\mu)$. Define integral operator $L_K$ on $H_K$ as $(L_K f)(x) = \langle k_x, f \rangle_\mu$. If $X$ is compact, then $L_K$ is compact and self-adjoint w.r.t $L^2_\mu$, so its eigenfunctions $\{e_i\}_{i \in \mathbb{N}}$ form an orthonormal basis of $L^2_\mu$, its eigenvalues $\{\lambda_i\}_{i \in \mathbb{N}}$ have finite multiplicities and converge to zero, and $k(x,y) = \sum_{i \in \mathbb{N}} \lambda_i e_i(x) e_i(y)$. If $f(x) = \sum_{i \in \mathbb{N}} a_i e_i(x)$, then $L_K f = \sum_{i \in \mathbb{N}} \lambda_i a_i e_i(x)$. It can be shown that the eigenfunctions are in $H_K$, so $\langle e_i, e_j \rangle = \delta_{ij} / \lambda_i$, and $f(x) = \sum_{i \in \mathbb{N}} a_i e_i(x) \in H_K$ iff $\sum_{i \in \mathbb{N}} a_i^2 / \lambda_i < \infty$. Let $L_K^{1/2}$ be the only positive definite self-adjoint operator that satifies $L_K^{1/2} \circ L_K^{1/2} = L_K$, then $L_K^{1/2}$ is an isomorphism from $L^2_\mu$ to $H_K$.
The distance function to any vector in an inner product space has a unqiue minimum on any complete convex subset: $\forall v \in V$, $\forall C \subset V$, $C = \widehat C$, $C = \text{conv}(C)$, $\exists! v' \in C$: $d(v, v') = \inf_{w \in C} d(v, w)$.