Inner product space and Hilbert space

Keywords: Inner product space and Hilbert space 细节梳理

1. Vector space

A vector space (also called a linear space) is a set of objects called vectors, which may be added together and multiplied(“scaled”) by numbers, called scalars.

Remark: 所谓空间，就是具备某些性质的元素的集合。大多数空间的定义是先射箭，再画靶的典型。

2. Inner product space

A complex(real) vector space $\mathcal{H}$ is said to be an inner-product space if for each pair of elements $x$ and $y$ in $\mathcal{H}$, there is a complex number $(x, y)$, called the inner product of $x$ and $y$, such that:

$(a) (x, y) = \bar{(y, x)}$

$(b) (x+y, z) = (x, z) + (y, z)$

$(c) (\alpha x, y) = \alpha (x, y)$ $\alpha \in \mathcal{C}$

$(d) (x, x) \ge 0, \forall x \in \mathcal{H}$

$(e) (x, x) = 0$ if and only if $x=0$

The norm of $x$ is defined as $|x| = \sqrt{(x, x)}$

2.1 Cauchy-Schwarz inequality

If $\mathcal{H}$ is an inner-product space, then

$|(x, y)| \le |x||y|$

and the equality holds iff $x=y(x, y)/(y, y)$

Proof:

Let $a = |y|^2$, $b = |(x, y)|$ and $c=|x|^2$. The polar representation of $$ is then

$(x, y) = b e^{i\theta}$ for some $\theta \in (-\pi, \pi])$

Note that $\forall r \in \mathbb{R}$,

$(x - re^{i\theta}y, x - re^{i\theta}y) = (x, x) - re^{i\theta}(y, x) - re^{-i\theta}(x, y) + r^2(y, y)$ (Note that $(x, by) = \bar{(by, x)} = \bar{b} \bar{(y, x)} = \bar{b} (x, y)$)

Note that $\bar{z_1 z_2} = \bar{z_1}\bar{z_2}$, $(y, x) = e^{-i \theta}$

$(x - re^{i\theta}y, x - re^{-i\theta}y) = c - 2rb + r^2a \ge 0$ (by definition of inner product)

The inequality holds for any $a, b, c, r$. Therefore $\min(c - 2rb + r^2a) = c - \frac{b^2}{a} \ge 0$, i.e. $b^2 \le ac$

The minimum value is achieved when $r = \frac{b}{a}$, i.e. $x = y e^{i\theta}b/a = y(x, y)/(y, y)$

2.2 Convergence and continuity

Definition: Convergence in Norm: A sequence $\{x_n, n = 1, 2, \dots\}$ of elements of an inner-product space $\mathcal{H}$ is said to converge in norm to $x \in \mathcal{H}$ if $|x_n - x|\xrightarrow{}0$ as $n \xrightarrow{} 0$

Proposition: Continuity of the Inner Product: If ${x_n}$ and ${y_n}$ are sequence of elements of the inner-product space $\mathcal{H}$ such that $|x_n - x|\xrightarrow{}0$ and $|y_n - y|\xrightarrow{}0$:

(a) $|x_n| \xrightarrow{} x$ (易证)

(b) $(x_n, y_n) \xrightarrow{} (x, y)$

3. Hilbert space

An inner product space with completeness is called a Hilbert space.

3.1 Cauchy Sequence

A sequence $\{x_n, n=1, 2, \dots\}$ of elements of an inner-product space is said to be a Cauchy sequence if

$|x_n - x_m| \xrightarrow{} 0$ as $m, n \xrightarrow{} \infty$

i.e. if for every $\epsilon > 0$ there exists a positive integer $N(\epsilon)$ such that

$|x_n - x_m| < \epsilon$ for all $m, n > N(\epsilon)$

3.2 Definition of Hilbert space

A Hilbert space $\mathcal{H}$ is an inner-product space which is complete, i.e. an inner-product space in which every Cauchy sequence ${x_n}$ converges in norm to some element $x \in \mathcal{H}$.

Remark: 收敛以及连续实际上定义了一种逼近过程，即不断靠近某个值，但是不一定能够等于这个值（我们把等于元素a定义为找不到更加接近a的元素，再收敛中的序列中，我们总能根据$\epsilon-\delta$找到更接近的，因此这是一种动态的感觉，某种程度上类似于$a.s.$和依概率收敛的感觉）。完全的意义在于，这个空间里面不存在一个“空”。我们可以轻松地用Euclidean space中挖去一些点来构造一个不完备的内积空间。Hilbert某种程度上避免了不知逼近到何处的尴尬。

Norm convergence and the Cauchy criterion: If $\{x_n\}$ is a sequence of elements belonging to a Hilbert space $\mathcal{H}$, then $\{x_n\}$ converges in norm iff $|x_n - x_m| \xrightarrow{} 0$ as $m, n \xrightarrow{} 0$.

Note that Cauchy series to norm convergence is the definition of Hilbert space.

We need to prove norm converge has a corresponding Cauchy series. It is obvious if we consider the following fact:

if $|x_n - x|\xrightarrow{}0$ $|x_n - x_m| \le |x_n - x| + |x_m - x|$

4 A review of probability space-from the perspective of Inner-product space

Consider a probability space $(\Omega, \mathcal{F}, P)$ and the collection $C$ of all random variables X defined on $\Omega$ and satisfying the condition,

$\mathcal{E}(X^2) = \int_{\Omega} X(\omega)^2P(d \omega) < +\infty$

Let’s define in the $L2$ space that $(X, Y) = E(XY)$ (注意这是随机变量，即从概率空间映射到向量空间的一个函数，的距离。)

Note that if $E(X^2) = 0$, then $X(\cdot) \xrightarrow{a.s.} 0$, i.e. $\textbf{Pr}(X = 0) = 1$

In the L2 space, norm convergence of a sequence $\{x_n\}$ can be written as:
$|X_n - X|^2 = \mathbb{E}|X_n - X|^2 \xrightarrow{} 0$. It is also called mean-square convergence and is also written as $X_n \xrightarrow{m.s.} X$