Notes on Inner Product & Hilbert Space

Basic Concepts

An inner product is a positive definite bilinear form (双线性形式) on a vector space: for a vector space $X$ with underlying scalar field $\mathbb{F}$, a map $\langle \cdot, \cdot \rangle: X \times X \to \mathbb{F}$ that satisfies

  • Additivity: $\langle x + y, z \rangle = \langle x, z \rangle + \langle y, z \rangle$;
  • Homogeneity: $\langle a x, y \rangle = a \langle x, y \rangle$, $\forall a \in \mathbb{F}$;
  • Symmetry: $\langle y, x \rangle = \overline{\langle x, y \rangle}$;
  • Positive definiteness: $\langle x, x \rangle > 0$, if $x \ne 0$;

An inner product space is a set with an inner product, $(X, \langle \cdot, \cdot \rangle)$. Inner product specifies the geometry of a vector space. Inner product space has norm $|x| = \sqrt{\langle x, x \rangle}$. Inner product space may have a finite or infinite number of dimensions.

A Hilbert Space is a complete inner product space. Hilbert spaces typically arise as infinite-dimensional function spaces, e.g. the $l^2$ space of square summable infinite sequences, $L^2$ spaces of square-integrable functions, $H^s$ spaces of twice square-integrably weakly-differentiable functions, and Hardy spaces $H^2(U)$ and Bergman spaces $L^2_a(G)$ of holomorphic functions.

A Euclidean space is a finite-dimensional real vector space $\mathbb{R}^n$ with an inner product $(x,y) = \sum_{i=1}^n x_i y_i$.

Orthogonality

Two elements of an inner product space are orthogonal if their inner product is zero: $\langle x, y \rangle = 0$.

Orthogonal complement

Orthogonal projection

Orthogonal set; Orthogonal set; Maximal/complete orthogonal set; Orthonormal basis

Gram-Schmidt orthogonalization process

Approximation and Fourier Series

For a vector space, an approximation of a point on a (closed) subspace is a point on the subspace that is closest to the point.

Theorem (Approximation in Banach and Hilbert spaces): For a Banach space, an approximation may not exist. For a Hilbert space, the approximation of any point on any subspace exists and is unique, which is the orthogonal projection of that point.

Theorem (Riesz representation):

Theorem (Lax-Milgram):

Reproducing Kernel

For a Hilbert space $H$ of scalar functions on $E$, a scalar function $K(x,y)$ on $E \times E$ is a reproducing kernel of the Hilbert space if: let $K_y(x) \equiv K(x,y)$,

  1. $K_y(x) \in H$, $\forall y \in E$;
  2. Reproducing property: $\langle f, K_y \rangle = f(y)$, $\forall f \in H$.

Reproducing kernels are symmetric and positive definite.

Reproducing kernel Hilbert space (RKHS) is a Hilbert space with a reproducing kernel. [@Aronszajn1950]

Theorem (equivalent definition of RKHS): A Hilbert space of functions on a set $E$ is a reproducing-kernel Hilbert space iff $f(y) \le c(y) |f|, \forall y \in E$, where $c(y) \equiv |K_y|$.

A reproducing kernel Hilbert space defines a reproducing kernel which is symmetric and positive definite.

Theorem (A RKHS is uniquely defined by its reproducing kernel.): For a symmetric positive definite kernel $K$ on a set $X$, there is a unique Hilbert space of functions on $X$ for which $K$ is a reproducing kernel.

$H_K$ denotes the reproducing-kernel Hilbert space generated by reproducing kernel $K$.

🏷 Category=Analysis