In functional analysis, the theories of inner product space, Hilbert space, and linear operators on such spaces are simpler and richer than those of the more general normed space and Banach space.

Notes on Inner Product & Hilbert Space

**Sesquilinear form** (半双线性形式) or **sesquilinear functional** $h: V \times W \mapsto \mathbb{F}$
on the Cartesian product of two vector spaces over the same field
is a scalar-valued function that is linear in the first variable and conjugate linear in the second,
$h(a v, w) = a h(v, w)$, $h(v, a w) = \bar a h(v, w)$;
$h(u + v, w) = h(u, w) + h(v, w)$, $h(u, v + w) = h(u, v) + h(u, w)$.
**Sesquilinear form** $h: V^2 \mapsto \mathbb{F}$ on a vector space
refers to a sesquilinear form on the Cartesian square of the vector space.
**Bilinear form** (双线性形式) $q: V \times W \mapsto \mathbb{R}$
is a sesquilinear form where the scalar field is the real numbers.
Note that bilinear forms and covariant 2-tensors are identical.
**Norm** $\|h\|$ of a sesquilinear form on two normed spaces over the same field
is the supremum of its normalized absoute value, if exists:
$\|h\| = \sup_{v, w \ne 0} \frac{|h(v, w)|}{\|v\| \|w\|}$, or equivalently,
$\|h\| = \sup_{\|v\|, \|w\| = 1} |h(v, w)|$.
**Bounded sesquilinear form** on two normed spaces over the same field
is a sesquilinear form with a finite norm: $\|h\| < \infty$.
**Space of bounded sesquilinear forms** $B(V \times W)$ on two normed spaces over the same field
is the normed space of all bounded sesquilinear forms on the two spaces,
endowed with the norm of sesquilinear forms.
**Coercive sesquilinear form** $h: V^2 \mapsto \mathbb{F}$ on a normed space
is a sesquilinear form which as a functional has a positive absolute lower bound:
$\inf_{v \ne 0} \frac{|h(v, v)|}{\|v\|^2} > 0$.

**Inner product** $\langle \cdot, \cdot \rangle$ on a vector space over real/complex numbers
is a (symmetric/Hermitian) positive definite bilinear/sesquilinear form on the vector space:
$\langle v, w \rangle = \overline{\langle w, v \rangle}$;
$v \ne 0$ then $\langle v, v \rangle > 0$.
Every inner product defines a norm as the square root of the inner product of a vector with itself:
$\|x\| = \sqrt{\langle x, x \rangle}$.
*Schwarz inequality*: Normalized inner product lies between -1 and 1;
$|\langle v, w \rangle| \le \|v\| \|w\|$.
**Inner product space** $(V, (+, \cdot_\mathbb{F}), \langle \cdot, \cdot \rangle)$
is a vector space endowed with a specific inner product.
Inner product specifies the geometry of a vector space, e.g. length and angle.
Every inner product space is a normed space.
The inner product in any inner product space is a continuous map:
$\langle \cdot, \cdot \rangle \in C(V^2, \mathbb{F})$.
Inner product space may have a finite or infinite dimension.
For a finite-dimensional inner product space,
the coordinate representation of its inner product w.r.t. any basis is a positive definite matrix.
**Gram determinant** $G(v_i)_{i=1}^n$ of an n-tuple of vectors in an inner product space
is the determinant of the matrix of their inner products:
$G(v_i)_{i=1}^n = \det(\langle v_i, v_j \rangle)^{i \in n}_{j \in n}$.
The Gram determinant of n vectors equals the squared n-volume of the parallelotope they determine:
$G(v_i)_{i=1}^n = \det([V]^T [V]) = \det(V)^2$, where $[V]$ is the matrix representation
of the vectors in any n orthonormal vectors whose span contain the n vectors.
The Gram determinant operator $G: V^n \mapsto \mathbb{R}_{\ge 0}$ is a nonnegative symmetric tensor.
A finite subset of vectors in an inner product space is linearly independent if and only if
its Gram determinant is nonzero.

**Euclidean space** of dimension $n$ or **Euclidean n-space**
$(\mathbb{R}^n, +, \cdot_{\mathbb{R}}, (\cdot, \cdot))$
is the product space of real numbers to the power of $n$,
endowed with the inner product $(x,y) = \sum_{i=1}^n x_i y_i$.
Euclidean spaces generalize two- and three-dimensional spaces of Euclidean geometry
to arbitrary finite-dimensional spaces.
It is the abstraction of a geometric object into a topological and algebraic structure.
Many concepts in algebra and analysis are developed by analogy with geometry,
where Euclidean space is among the first few.

**Hilbert space** $H$ is an inner product space
that is complete in the metric induced by the inner product.
Every Hilbert space is a strictly convex Banach space.
Hilbert spaces typically arise as infinite-dimensional function spaces,
e.g. the $l^2$ space of square summable infinite sequences,
$L^2$ spaces of square-integrable functions,
$H^s$ Sobolev spaces of twice square-integrably weakly-differentiable functions,
and Hardy spaces $H^2(U)$ and Bergman spaces $L^2_a(G)$ of holomorphic functions.

**Subspace** $(S, (+, \cdot_\mathbb{F}), \langle \cdot, \cdot \rangle)$ of an inner product space
is an inner product space consisting of a linear subspace and
the inner product restricted to the Cartesian square of the subspace.
A subspace of a Hilbert space is a Hilbert space if and only if it is a closed subset.

**Linear isometry** or **inner product space isomorphism** $T: V \mapsto W$
between inner product spaces is a vector space isomorphism
that takes the inner product of the domain to that of the codomain:
$\forall v, v' \in V, \langle Tv, Tv' \rangle = \langle v, v' \rangle$.
Any linear operator between n-dimensional inner product spaces
that maps an orthonormal basis to an orthonormal basis is a linear isometry.
Any n-dimensional inner product space over the real numbers
is linearly isometric with the Euclidean n-space: $V \cong \mathbb{R}^n$.
*Completion of inner product space*:
Every inner product space is linearly isometric to a dense subspace of a unique Hilbert space
up to linear isometries.
**Completion** $\widehat V$ of an inner product space
is the Hilbert space with a dense subspace isomorphic to the inner product space.

**Orthogonal vectors** $v \perp w$ of an inner product space
are two vectors whose inner product is zero: $\langle v, w \rangle = 0$.
A vector and a subset of an inner product space are **orthogonal** $v \perp S$
if the vector is orthogonal to all vectors in the subset: $\forall w \in S$, $v \perp w$.
**Annihilator** $S^\perp$ of a subset $S$ of an inner product space
is the set of all vectors orthogonal to the subset: $S^\perp = \{v : v \in V, v \perp S\}$.
The annihilator of any subset of an inner product space is a closed subspace:
$\overline{S^\perp} = S^\perp$.
The second annihilator of any subset of an inner product space includes the subset:
$S \subset S^{\perp\perp}$.
**Orthogonal complement** $S^\perp$ of a subspace of an inner product space
is the annihilator of the subspace.
Every Hilbert space is the direct sum of any closed subspace and its orthogonal complement:
$H = \overline{S} \oplus \overline{S}^\perp$.
A subset of a Hilbert space is total (see Normed Space)
if and only if there is no nonzero vector orthogonal to it:
$\overline{\text{Span}(S)} = H$ iff $S^\perp = \{0\}$.
The second orthogonal complement of any closed subspace of a Hilbert space is the subspace:
$S \subset H$, $\bar S = S$, $\text{Span}(S) = S$ then $S = S^{\perp\perp}$.
**Orthogonal projection** $\pi: V \mapsto S$ from an inner product space onto a subspace
is the projection whose kernel is the orthogonal complement of the subspace:
$\pi(v + w) = v$ where $v \in S, w \in S^\perp$.
Every orthogonal projection is a bounded linear operator that is surjective and idenpotent:
$\pi \in B(V, S)$, $\pi(V) = S$, $\pi^2 = \pi$.

**Orthogonal subset** of an inner product space is a set of vectors that are pairwise orthogonal:
$v, w \in S, v \ne w$, then $\langle v, w \rangle = 0$.
An orthogonal subset is linearly independent if it does not include the zero vector.
**Orthonormal subset** $(e_\alpha)_{\alpha \in A}$ of an inner product space
is an orthogonal subset consisting of unit-length vectors:
$\langle e_\alpha, e_\beta \rangle = \delta^\alpha_\beta$.
**Gram-Schmidt Algorithm** for linearly independent sequences:
An orthonormal sequence in an inner product space
can be contructed from a linearly independent sequence
such that their principal subtuples of the same size span to the same subspace.
The sum of an orthogonal sequence in a Hilbert space converges if and only if
The sum of their square norms converges:
$\sum_{i=1}^\infty a_i e_i \in H$ iff $\sum_{i=1}^\infty |a_i|^2 < \infty$.
**Fourier coefficient** $\langle x, e_i \rangle$ of a vector in an inner product space
w.r.t. an orthonormal sequence $(e_i)_{i=1}^\infty$
is its inner product with the i-th term of the sequence.
The Fourier coefficients of the limit of a convergent orthogonal sequence in a Hilbert space
equals the norms of terms of the sequence:
$\sum_{i=1}^\infty a_i e_i = x$ then $\langle x, e_i \rangle = a_i$.
*Bessel Inequality*:
Given an orthonormal sequence in an inner product space,
the sum of square Fourier coefficients of any vector in the space is no greater than its square norm;
$\sum_{i=1}^\infty |\langle x, e_i \rangle|^2 \le \|x\|^2$.
Any vector in an inner product space can have at most
countably many nonzero Fourier coefficients w.r.t. an orthonormal subset:
$\overline{\{\alpha: \alpha \in A, \langle x, e_\alpha \rangle \ne 0\}} \le \omega$.
**Total orthonormal subset** of an inner product space
is an orthonormal subset that is a total subset:
$\overline{\text{Span}(e_\alpha)_{\alpha \in A}} = X$,
$\langle e_\alpha, e_\beta \rangle = \delta^\alpha_\beta$.
A total orthonormal subset is a basis if and only if it is finite.
Every Hilbert space has a total orthonormal subset.
An orthonormal subset in a Hilbert space is total if and only if
every vector in the space can be written as the sum of squares of
all nonzero Fourier coefficients w.r.t. the subset,
which is well-defined because it is at most a countable sequence:
$\sum_{\alpha \in A} |\langle x, e_\alpha \rangle|^2 = \|x\|^2$.
The total orthonormal subsets of a Hilbert space have the same cardinality.
**Hilbert dimension** of a Hilbert space is the cardinality of any of its total orthonormal subsets:
$\dim H = |A|$.
A Hilbert space is separable if and only if it has a total orthonormal sequence:
$\dim H \le \aleph_0$.
Hilbert spaces over the same field are isomorphic if and only if
they have the same Hilbert dimension: $H \cong H'$ iff $\dim H = \dim H'$.
Thus, every Hilbert space is isomorphic to every Hilbert space of
its scalar field to the power of the cardinality of a total orthonormal set:
$\widehat{\text{Span}}(E) \cong \widehat{\mathbb{F}^{|E|}}$.
In particular, every Hilbert space of all scalar-valued functions on a set is isomorphic to
every Hilbert space of the scalar field to the power of the cardinality of the set:
$\widehat{\mathbb{F}^X} \cong \widehat{\mathbb{F}^{|X|}}$.
(Although this generalizes the result for isomorphic finite-dimensional vector spaces,
we are almost never interested in arbitary functions on infinite sets.)

Because every Hilbert space is the direct sum of any Hilbert subspace and its orthogonal complement, every vector in a Hilbert space has a unique best approximation in every closed subspace, which is the orthogonal projection of that vector to the subspace: $H_0 \subset H$ then $\forall x \in H$, $P x = \arg\min_{y \in H_0} d(x, y)$. The distance between a vector and its best approximation on a finite-dimensional subspace equals the ratio of the volumes of two parallelotopes, one determined by the vector and any basis of the subspace, the other by the basis only; writing volume as the square root of a Gram determinant, we have: $d(x, Px) = \sqrt{\frac{G(x, y_i)_{i=1}^n}{G(y_i)_{i=1}^n}}$; with an orthonormal basis, we have $d(x, Px) = \sqrt{G(x, e_i)_{i=1}^n}$, which gives $d(x, Px) = \sqrt{\|x\|^2 - \sum_{i=1}^n |\langle x, e_i \rangle|^2}$.

The space $L^2(\mathbb{R})$ of Lebesgue square integrable real functions is separable.

*Riesz Representation Theorem* of Covectors:
Every bounded linear functional on a Hilbert space
can be written uniquely as the inner product with a vector in the space:
$\forall \omega \in H^∗$, $\exists! w \in H$:
$\|w\| = \|\omega\|$, $\omega = \langle \cdot, w \rangle$.

*Lax-Milgram Lemma*:
The Riesz representation theorem still holds
if the inner product is replaced by any bounded coercive sesquilinear form.
$h \in B(H^2)$, $\inf_{\|v\| = 1} |h(v, v)| > 0$,
then $\forall \omega \in H^∗$, $\exists! w \in H$:
$\|w\| = \|\omega\|$, $\omega = h(\cdot, w)$.

*Riesz Representation Theorem of Sesquilinear Forms*:
Every bounded sesquilinear form on two Hilbert spaces over the same field
can be written uniquely as the inner product of one space
where vectors in the other space are mapped into the space by a bounded linear operator:
$\forall h \in B(H_1 \times H_2)$, $\exists! T \in B(H_1, H_2)$:
$\|T\| = \|h\|$, $h = \langle T \cdot, \cdot \rangle_2$.

*Zero Linear Operator*:
A bounded linear operator between inner product spaces is zero if and only if
it represents the zero sesquilinear form:
$T \in B(V, W)$, then $T = 0$ iff $\langle T \cdot, \cdot \rangle_W = 0$.
A bounded linear transformation on a complex inner product space is zero
if the inner product of any vector and its value is zero:
$T \in B(V, V)$, $\{\langle T v, v \rangle : v \in V\} = \{0\}$, then $T = 0$.

**Hilbert-adjoint operator** $T^∗: H_2 \mapsto H_1$
of a bounded linear operator $T: H_1 \mapsto H_2$ between Hilbert spaces
is the bounded linear operator that represents the same bounded sesquilinear form:
$\forall v \in H_1$, $\forall w \in H_2$, $\langle T v, w \rangle_2 = \langle v, T^∗ w \rangle_1$.
Every bounded linear operator and its Hilbert-adjoint between two given Hilbert spaces
have the same operator norm:
$\forall T \in \mathcal{L}(H_1, H_2) \cap C(H_1, H_2)$, $\|T\| = \|T^∗\|$.
The conjugation map $∗$ is an involution that is conjugate linear and composable,
and the composite transformation squares the operator norm:
$T^{∗∗} = T$, $(a S + b T)^∗ = \bar{a} S^∗ + \bar{b} T^∗$,
$(S T)^∗ = T^∗ S^∗$, $\|T^∗ T\| = \|T\|^2$.

**Self-adjoint operator** or **Hermitian operator** on a Hilbert space
is a bounded linear operator that equals its adjoint operator:
$T \in \mathcal{L}(H, H) \cap C(H, H)$, $T = T^∗$.
A bounded linear operator on a complex Hilbert space is self-adjoint
if and only if the inner product of any vector and its value is real
given $T \in \mathcal{L}(H, H) \cap C(H, H)$,
$T = T^∗$ iff $\forall v \in H$, $\langle T v, v \rangle \in \mathbb{R}$.
The limit operator of a convergent sequence of self-adjoint operators on a Hilbert space
is also self-adjoint, i.e. the set of all self-adjoint operators on a Hilbert space
is a closed subset of the bounded linear operators on the space:
$\forall n \in \mathbb{N}$, $T_n = T_n^∗$,
$\exists T \in B(H, H)$: $\lim_{n \to \infty} |T_n - T| = 0$, then $T = T^∗$.
*Boundedness of Self-adjoint Operators* [@Hellinger and Toeplitz, 1910]:
Every self-adjoint linear operator on a Hilbert space is bounded:
$T \in \mathcal{L}(H, H)$, $\langle T v, w \rangle = \overline{\langle T w, v \rangle}$,
then $T \in B(H, H)$.

**Unitary operator** on a Hilbert space
is an invertible bounded linear operator whose inverse equals its adjoint: $U^{-1} = U^∗$.
A bounded linear operator on a complex Hilbert space is unitary
if and only if it is isometric and surjective.
Every unitary operator on a Hilbert space
is a linear isometry, has unit operator norm, and has unitary adjoint:
$\langle U v, U w \rangle = \langle v, w \rangle$; $\|U\| = 1$; $(U^∗)^{-1} = (U^∗)^∗$.
Note that an isometric operator need not be unitary since it may not be surjective.
The composition of two unitary operators on a Hilbert space is also unitary: $(U V)^{-1} = (U V)^∗$.

**Normal operator** on a Hilbert space
is a bounded linear operator that commutes with its adjoint: $T^∗ T = T^∗ T$.
A bounded linear operator on a Hilbert space is normal if it is self-adjoint or unitary.

The spectrum of a self-adjoint operator is real, and lies within the infimum and the supremum the inner products of unit vectors and their values. Eigenspaces corresponding to different eigenvalues are orthogonal. ...

Every self-adjoint operator can be represented by an integral determined by its spectral family.
**Spectral family** or **decomposition of unity**
is a family of projection operators having certain properties.

Harmonic analysis... Laplace–Beltrami operator

...

**Evaluation functional** $e_x$ on a function space
is a functional that maps every function in the space to its value at a given point:
$V \subset \mathbb{F}^X$, $\forall f \in V$, $e_x f = f(x)$.
**Reproducing kernel Hilbert space** (RKHS)
is a Hilbert space of scalar-valued functions where every evaluation functional is bounded:
$\mathcal{H} \subset \mathbb{F}^X$, $\{e_x : x \in X\} \subset H^\#$.
By the Riesz representation theorem, every evaluation functional in an RKHS
determines a unique function in the space:
$\forall x \in X$, $\exists! k_x \in \mathcal{H}$: $e_x = \langle \cdot, k_x \rangle$.

**Reproducing kernel** $k: X^2 \mapsto \mathbb{F}$ of a Hilbert space of scalar-valued functions
is a bivariate scalar-valued function such that with one variable fixed at any value
(1) it is an element of the space, and (2) the inner product with any element of the space
equals the value of that function at the fixed variable:
$\mathcal{H} \subset \mathbb{F}^X$,
$(\mathcal{H}, (+, \cdot_\mathbb{F}), \langle \cdot, \cdot \rangle)$,
$k_y(x) := k(x, y)$, then (1) $\{k_y : y \in X\} \subset \mathcal{H}$, and (2)
$\forall f \in \mathcal{H}$, $\forall y \in X$, $\langle f, k_y \rangle = f(y)$.
The second property satisfied by a reproducing kernel is called the **reproducing property**,
because it defines an operator $K: \mathcal{H} \mapsto \mathcal{H}$
that "reproduces" any function in the Hilbert space:
$\forall f \in \mathcal{H}$, $\forall y \in X$, $(K f)(y) := \langle f, k_y \rangle$, hence $K f = f$;
but the Kronecker delta $\delta_{xy} = 1(x = y)$ is not a reproducing kernel
unless the Hilbert space contains all indicator functions of singletons, $1(x = x_0)$.

**Positive (semi-)definite kernel**, or simply **kernel**, is a bivariate scalar-valued function
that is self-adjoint and, for any finite subset of the underlying space,
its values at all pairs of the subset form a positive semi-definite matrix:
$k(x, y) = \bar{k}(y, x)$; $\forall n \in \mathbb{N}_+$, $\forall I \in X^n$, $[k]_{I,I} \succeq 0$.
Real-valued positive semi-definite kernels form a function algebra over non-negative real numbers:
$(\mathcal{K}_+, (+, \cdot_{\mathbb{R}_{\ge 0}}, \times))$.
Every reproducing kernel is a positive (semi-)definite kernel.

As an equivalent definition, a **reproducing kernel Hilbert space** (RKHS) $\mathcal{H}_k$
is a Hilbert space of scalar-valued functions with a reproducing kernel $k$ [@Aronszajn1950].
This can be established as follows.
A Hilbert space of scalar-valued functions has a reproducing kernel if and only if
it is uniformly bounded: $\forall f \in \mathcal{H}$, $\forall y \in X$, $f(y) \le c(y) \|f\|$.
For an RKHS, the bound $c(y) = \|k_y\|$ is tight.
Every RKHS has a unique reproducing kernel.
*Moore–Aronszajn theorem*:
For any positive semi-definite bivariate function on a set,
there is a unique Hilbert space of scalar-valued functions on the set
such that the bivariate function is a reproducing kernel.
Therefore, RKHSs of scalar-valued functions on a set
are in one-to-one correspondence with positive semi-definite bivariate functions on the set.
...
Given an orthonormal basis $(\psi_\alpha)_\alpha$ for the RKHS,
its reproducing kernel can be written as:
$k(x, y) = \sum_{\alpha} \psi_\alpha(x) \bar{\psi}_\alpha(y)$.

If an RKHS is sufficiently rich, then its reproducing kernel is positive definite, which defines another inner product on the Hilbert space. ...

*A simple example*.
Let X = {1, ..., n}, then every function $f: X \mapsto \mathbb{R}$ is a real vector of length n.
Every RKHS $\mathcal{H} \subset \mathbb{R}^n$ can be defined as:
$\mathcal{H} = \text{range}(H)$ and $\langle f, g \rangle = f^T H g$,
where $H \in \mathcal{S}_{\ge 0}(n)$; there is a one-to-one mapping between $\mathcal{H}$ and H.
The RKHS corresponds to the reproducing kernel $k: X^2 \mapsto \mathbb{R}$
identified by the order-n matrix $K = H^\dagger$.
To verify this, let $H = V \Lambda V^T$ be a compact EVD, then $K = V \Lambda^{-1} V^T$.
For any $j \in \{1, \cdots, n\}$ let $k_j := K e_j$, we have:
(1) $k_j \in \text{range}(V) = \text{range}(H) = \mathcal{H}$;
(2) $\langle k_j, f \rangle = k_j^T H f = e_j^T H^\dagger H f = e_j^T f = f_j$.
In summary, every RKHS that is a subspace of a Euclidean space
is uniquely identified by a PSD matrix.

Let $X$ has measure $\mu$, which defines a Hilbert space $(L^2_\mu, \langle \cdot , \cdot \rangle_\mu)$. Define a kernel integral operator $L_k$ on $L^2_\mu$ as $(L_k f)(x) = \langle k_x, f \rangle_\mu$. If $X$ is compact, then $L_k$ is compact and self-adjoint w.r.t $L^2_\mu$, so its eigenfunctions $\{\phi_i\}_{i \in \mathbb{N}}$ form an orthonormal basis of $L^2_\mu$, its eigenvalues $\{\lambda_i\}_{i \in \mathbb{N}}$ have finite multiplicities and converge to zero, and $k(x,y) = \sum_{i \in \mathbb{N}} \lambda_i \phi_i(x) \phi_i(y)$. If $f(x) = \sum_{i \in \mathbb{N}} a_i \phi_i(x)$, then $L_k f = \sum_{i \in \mathbb{N}} \lambda_i a_i \phi_i(x)$. It can be shown that the eigenfunctions are in $\mathcal{H}_k$, so $\langle \phi_i, \phi_j \rangle = \delta_{ij} / \lambda_i$, and $f(x) = \sum_{i \in \mathbb{N}} a_i \phi_i(x) \in \mathcal{H}_k$ iff $\sum_{i \in \mathbb{N}} a_i^2 / \lambda_i < \infty$. Let $L_k^{1/2}$ be the only positive definite self-adjoint operator that satifies $L_k^{1/2} \circ L_k^{1/2} = L_k$, then $L_k^{1/2}$ is an isomorphism from $L^2_\mu$ to $\mathcal{H}_k$.

The distance function to any vector in an inner product space has a unqiue minimum on any complete convex subset: $\forall v \in V$, $\forall C \subset V$, $C = \widehat C$, $C = \text{conv}(C)$, $\exists! v' \in C$: $d(v, v') = \inf_{w \in C} d(v, w)$.

- [@Paulsen2016] An Introduction to the Theory of Reproducing Kernel Hilbert Spaces.