Notes on Inner Product & Hilbert Space

An **inner product** is a positive definite bilinear form (双线性形式) on a vector space:
for a vector space $X$ with underlying scalar field $\mathbb{F}$,
a map $\langle \cdot, \cdot \rangle: X \times X \to \mathbb{F}$ that satisfies

- Additivity: $\langle x + y, z \rangle = \langle x, z \rangle + \langle y, z \rangle$;
- Homogeneity: $\langle a x, y \rangle = a \langle x, y \rangle$, $\forall a \in \mathbb{F}$;
- Symmetry: $\langle y, x \rangle = \overline{\langle x, y \rangle}$;
- Positive definiteness: $\langle x, x \rangle > 0$, if $x \ne 0$;

An **inner product space** is a set with an inner product, $(X, \langle \cdot, \cdot \rangle)$.
Inner product specifies the *geometry* of a vector space.
Inner product space has norm $|x| = \sqrt{\langle x, x \rangle}$.
Inner product space may have a finite or infinite number of dimensions.

A **Hilbert Space** is a complete inner product space.
Hilbert spaces typically arise as infinite-dimensional function spaces,
e.g. the $l^2$ space of square summable infinite sequences,
$L^2$ spaces of square-integrable functions,
$H^s$ spaces of twice square-integrably weakly-differentiable functions,
and Hardy spaces $H^2(U)$ and Bergman spaces $L^2_a(G)$ of holomorphic functions.

A **Euclidean space** is a finite-dimensional real vector space $\mathbb{R}^n$
with an inner product $(x,y) = \sum_{i=1}^n x_i y_i$.

Two elements of an inner product space are **orthogonal** if their inner product is zero:
$\langle x, y \rangle = 0$.

Orthogonal complement

Orthogonal projection

Orthogonal set; Orthogonal set; Maximal/complete orthogonal set; Orthonormal basis

Gram-Schmidt orthogonalization process

For a vector space,
an **approximation** of a point on a (closed) subspace is
a point on the subspace that is closest to the point.

Theorem (Approximation in Banach and Hilbert spaces): For a Banach space, an approximation may not exist. For a Hilbert space, the approximation of any point on any subspace exists and is unique, which is the orthogonal projection of that point.

Theorem (Riesz representation):

Theorem (Lax-Milgram):

For a Hilbert space $(H, \langle \cdot , \cdot \rangle)$ of scalar functions on $X$,
a scalar function $K(x,y)$ on $X \times X$ is a **reproducing kernel** of the Hilbert space if:
let $K_y(x) \equiv K(x,y)$,

- $K_y(x) \in H$, $\forall y \in X$;
- Reproducing property: $\langle f, K_y \rangle = f(y)$, $\forall f \in H$.

Reproducing kernels are symmetric and positive definite.

**Reproducing kernel Hilbert space** (RKHS) is a Hilbert space with a reproducing kernel
[@Aronszajn1950].

Theorem (equivalent definition of RKHS): A Hilbert space of functions on a set $X$ is a reproducing kernel Hilbert space iff $f(y) \le c(y) |f|, \forall y \in X$, where $c(y) \equiv |K_y|$.

A reproducing kernel Hilbert space uniquely defines a reproducing kernel which is symmetric and positive definite.

Theorem: For a symmetric positive definite kernel $K$ on a set $X$, there is a unique Hilbert space of functions on $X$ for which $K$ is a reproducing kernel.

Therefore, reproducing kernel Hilbert spaces of functions on a domain are in one-to-one correspondence with positive definite kernels on the domain. We can denote any reproducing kernel Hilbert space as $H_K$, where $K$ is the unique reproducing kernel of the Hilbert space and $H_K$ is the unique Hilbert space generated by the symmetric positive definite kernel $K$.

If the space $H_K$ is sufficiently rich, then the reproducing kernel $K$ is positive definite.

Let $X$ has measure $\mu$, which defines a Hilbert space $(L^2_\mu, \langle \cdot , \cdot \rangle_\mu)$. Define integral operator $L_K$ on $H_K$ as $(L_K f)(x) = \langle K_x, f \rangle_\mu$. If $X$ is compact, then $L_K$ is compact and self-adjoint w.r.t $L^2_\mu$, so its eigenfunctions $\{e_i\}_{i \in \mathbb{N}}$ form an orthonormal basis of $L^2_\mu$, its eigenvalues $\{\lambda_i\}_{i \in \mathbb{N}}$ have finite multiplicities and converge to zero, and $K(x,y) = \sum_{i \in \mathbb{N}} \lambda_i e_i(x) e_i(y)$. If $f(x) = \sum_{i \in \mathbb{N}} a_i e_i(x)$, then $L_K f = \sum_{i \in \mathbb{N}} \lambda_i a_i e_i(x)$. It can be shown that the eigenfunctions are in $H_K$, so $\langle e_i, e_j \rangle = \delta_{ij} / \lambda_i$, and $f(x) = \sum_{i \in \mathbb{N}} a_i e_i(x) \in H_K$ iff $\sum_{i \in \mathbb{N}} a_i^2 / \lambda_i < \infty$. Let $L_K^{1/2}$ be the only positive definite self-adjoint operator that satifies $L_K^{1/2} \circ L_K^{1/2} = L_K$, then $L_K^{1/2}$ is an isomorphism from $L^2_\mu$ to $H_K$.