Orthonormal Set and Bessel's Inequality

Keywords: Orthonormal set, Bessel’s inequality, Parseval’s identity, Fourier Coefficients

1. Definitions

Orthonormal Set: A set ${e_t, t \in T}$ of elements of an inner-product space is said to be orthonormal if for every $s, t \in T$,

$(e_s, e_t) = 1$ when $s = t$; otherwise $0$.

Closed span: The closed span ($\bar{sp}(x_t)$) of any subset $\{x_t, t \in T\}$ of a Hilbert space $\mathcal{H}$ is defined to be the smallest closed subspace of $\mathcal{H}$ which contains each element $x_t, t \in T$. (The set of all linear combinations. 这是因为Hilbert 空间是向量空间，及线性空间，对数乘封闭；另外在更加General的形况下考虑$\sigma$-algebra)

Remark for Closed Span in Hilbert space: Note that if $M = \bar{sp}(x_t, t \in T)$, then for any $x \in \mathcal{H}$, $P_{\mathcal{M}}x = \alpha_1 x_1 + \dots + \alpha_t x_t$

2. Fourier Coefficients

If $\{e_1, \dots, e_k\}$ is an orthonormal subset of the Hilbert space $\mathcal{H}$ and $\mathcal{M} = \bar{sp}\{e_1, \dots, e_k\}$

Then:

$P_{\mathcal{M}} x = \sum_{i = 1}^k (x, e_i)e_i$ for all $x \in \mathcal{H}$

Proof: According to prediction equation, $(x - P_{\mathcal{M}}x, y) = 0$, i.e. $(x, y) = (P_{\mathcal{M}}x, y)$, where $y$ is any element in $\mathcal{M}$. It is clear that the prediction equation holds.

Based on the property of projection mapping, we can further get more property of orthonormal set.

Comment: Prediction equation真好用！注意这都是内积空间的性质（当然也需要线性空间），不依赖于其他结构。

Definition: The numbers $(x, e_i)$ are sometimes called the Fourier Coefficients.

Bessel inequality: $\sum_{i = 1}^k |(x, e_i)|^2 \le |x|^2$ (这个不等式在证明Fourier级数收敛的时候还挺重要的)

Proof of Bessel inequality

From prediction equation, we know that $P_{\mathcal{M]}} x = \sum_{i = 1}^k (x, e_i) e_i$. Therefore, $|P_{\mathcal{M]}} x|^2 = \sum_{i = 1}^k |(x, e_i)|^2 \le |P_{\mathcal{M]}} x|^2 + |(I - P_{\mathcal{M]}}) x|^2 = |x|^2$

3. Parseval’s Identity

Complete Orthonormal Set: $e_i$ such that $\mathcal{H} \in \bar{sp}\{e_i\}$ (sometimes complete Orthonormal Set is also call orthonormal basis 标准正交基)

Separable Hilbert Space: The orthonormal basis is finite or countable.

Parseval’s Identity can be claimed as:

If $\mathcal{H}$ is the separable Hilbert space $\mathcal{H} = \bar{sp}\{e_1, e_2, \}$ where $\{e_i\}$ is an orthonormal set, then

(1) For each $x \in \mathcal{H}$ and $\epsilon > 0$, there exists a positive integer k and constants $c_1,\dots ,c_k$, such that

$|x - \sum_{i =1}^k c_i e_i| < \epsilon$ (可数可以被有限任意逼近, 也算是一种直观的体现, 可以找一列递增并且逼近正交基的集合序列证明)

(2) $x = \sum_{i = 1}^{\infty} (x, e_i)e_i$ (Bessel inequality 和 内积空间的连续性可以证得)

(3) $(x, y) = \sum_{i = 1}^{\infty} (x, e_i)(e_i, y)$

(4) $x = 0$ iff $(x, e_i) = 0$

Most of the aforementioned properties are obvious. The proof of (3) can be written as:

$\lim\limits_{n \xrightarrow \infty} \sum_{i = 1}^n (x, e_i) e_i \xrightarrow x$

$\lim\limits_{n \xrightarrow \infty} \sum_{i = 1}^n (y, e_i) e_i \xrightarrow y$

According to the continuity of inner-product space, we can prove that:

$(x, y) = \lim\limits_{n \xrightarrow \infty} (\sum_{i = 1}^n (x, e_i) e_i, \sum_{i = 1}^n (y, e_i) e_i ) = \sum_{i = 1}^{\infty} (x, e_i)(e_i, y)$

Comment: 这一部分可以说是傅里叶分析为代表的信号分解的基石。如果我们只考虑离散时间的时间序列(大多是时候都是)信号分解的前提是有符合内积定义的内积，以及一套标准正交基。在连续傅里叶变换中，我们可以理解为这一些列都在某个函数的基下进行，本质上其实是不变的(Recall: Dirac function should be considered in a space whose density is generated by a Lebesgue integrable function)