Keywords: Closed Subspace, Orthogonal Complement, The Projection Theorem
1. Definitions
Closed Subspace A linear subspace $\mathcal{M}$ of a Hilbert space $\mathcal{H}$ is said to be a closed subspace of $\mathcal{H}$ if $\mathcal{M}$ contains all of its limit points (i.e. if $x_n \in \mathcal{M}$ and $|x_n - x| \xrightarrow{} 0$ imply that $x \in \mathcal{M}$).
Orthogonal Complement: The orthogonal complement of a subset $\mathcal{M}$ of $\mathcal{H}$ is defined to be the set $\mathcal{M}^{\perp}$ of all elements of $\mathcal{H}$ which are orthogonal to every element of $\mathcal{M}$. Thus
$x \in M^{\perp}$ if and only if $(x, y) = 0$ for all $y \in \mathcal{M}$
Proposition: If $\mathcal{M}$ is any subset of a Hilbert space $\mathcal{H}$, then its orthogonal complement is a closed subspace of $\mathcal{H}$.
Proof: For any Cauchy sequence $x_n \in \mathcal{M}^{\perp}$, $(x_n, y) = 0 \forall y \in \mathcal{M}$. According to the continuity of inner product (triangle inequality), if $|x_n - x| \xrightarrow{} 0$, we can conclude that $(x, y) = 0$. Therefore, the $\mathcal{H}^{\perp}$ must be complete. QED.
Lemma Parallelogram law: If $\mathcal{\Hilbert}$ is an inner-product space, then
$|x + y|^2 + |x - y|^2 = 2 |x|^2 + 2 |y|^2$ (显然)
Projection Theorem : If $\mathcal{M}$ is a closed subspace of the Hilbert space $\mathcal{H}$ and $x \in \mathcal{H}$, then
(1) There is a unique element $\hat{x} \in \mathcal{M}$ such that $|x - \hat{x}| = \inf\limits_{y \in \mathcal{M}} |x - y|$
(2) $\hat{x} \in \mathcal{M}$ and $|x - \hat{x}| = \inf\limits_{y \in \mathcal{M}} |x - y|$ if and only if $\hat{x} \in \mathcal{M}$ and $(x - \hat{x}) \in \mathcal{M}^{\perp}$
Proof:
(1) By the definition of closed subspace, there exist a Cauchy sequence $y_n \in \mathcal{M}, |y_n| \xrightarrow{} y$, such that $|y_n - x| \xrightarrow{} \inf\limits_{y \in \mathcal{M}} |y - x|$. Therefore, by Cauchy criterion there exist $\hat{x} \in \mathcal{H}$, such that $|x - \hat{x}|^2 = \lim\limits_{n \xrightarrow{} \infty} |x - y_n|^2 = d$. Existence proved.
To prove the uniqueness, we need to prove that for any $\hat{y} \in \mathcal{M}$ and that $|x - \hat{y}| = |x - \hat{x}| = d$, $|\hat{x} -\hat{y}| = 0$. Apply the parallelogram law:
$$
$0 \le|\hat{x} - y|^2 = -4|(\hat{x} + \hat{y})/2 - x|^2 + 2(|\hat{x} - x|^2 + |\hat{y} - x|^2) \le -4d + 4d = 0$
Remark: 三角不等式一下会把上面的式子的右边放得太大，在证明唯一性的时候一定要优先使用严格等号。
Hence $\hat{y} \overset{a.s.}{=} \hat{x}$
(2)
step 1: $\hat{x} \in M$ and $(x - \hat{x}) \in \mathcal{M}^{\perp} \xrightarrow{} |x - \hat{x}| = \inf |x-y|$:
(易证)
step 2: $\hat{x} \in M$ and $(x - \hat{x}) \in \mathcal{M}^{\perp} \xleftarrow{} |x - \hat{x}| = \inf |x-y|$:
Prove it by contradiction. (是构造性证明，技巧上类似Cauchy-Schwarz)
Remark: 映射的存在性和唯一性都是基于Closed subspace上的性质成立的。
2. Projection mapping
If $\mathcal{M}$ is a closed subspace of the Hilbert space $\mathcal{H}$ and $I$ is the identity mapping on $\mathcal{H}$, then there is a unique mapping $P_{\mathcal{M}}$ of $\mathcal{H}$ onto $\mathcal{M}$ such that $I - P_{\mathcal{M}}$ maps $\mathcal{H}$ on to $\mathcal{M}^{\perp}$.
Proof:
By projection theorem, for each $x \in \mathcal{H}$ there is a unique $\hat{x} \in \mathcal{M}$ such that $x-\hat{x} \in \mathcal{M}^{\perp}$. The required mapping is therefore $P_{\mathcal{M}}x = \hat{x}$
Remark: 这里的映射的定义来源于距离该元素最近的点是唯一的。映射定理也可以因此表述为，一定存在一个映射，使得Hilbert空间中任何的元素都能够被映射到某个Closed subspace中，并且$I - P_{\mathcal{M}}$可以把这个元素映射到Closed subspace的正交补空间上。
Properties
(1) $P_{\mathcal{M}}(\alpha x + \beta y) = \alpha P_{\mathcal{M}}(x) + \beta P_{\mathcal{M}}(y)$
(2) $|x|^2 = |P_{\mathcal{M}}x|^2 + |(I - P_{\mathcal{M}})x|^2$
(3) $x = P_{\mathcal{M}}x + (I - P_{\mathcal{M}})x$
(4) If $|x_n - x| \xrightarrow{} 0$, $P_{\mathcal{M}}x_n \xrightarrow{} P_{\mathcal{M}}x$
(5) $x \in \mathcal{M}$ iff $P_{\mathcal{M}} x=x $
(6) (5) $x \in \mathcal{M}^{\perp}$ iff $P_{\mathcal{M} x=0 $
Remark: 在内积空间的连续性中，$|x_n - x| \xrightarrow{} 0$可以推出$|x_n| \xrightarrow |x|$, 反之不成立，除非$x_n$可以被$x$线性表示。
Prediction Equation: Given a Hilbert space $\mathcal{H}$, a closed subspace $\mathcal{M}$, and an element $x \in \mathcal{H}$. The element of $\mathcal{M}$ closest to x, is the $\hat{x}$ such that
$(x - \hat{x}, y) = 0 \forall y \in \mathcal{M}$
Remark: Note that $P_{\mathcal{M}}(P_{\mathcal{M}}x) = P_{\mathcal{M}}x$. Therefore, the projection mapping must satisfy that $P_{\mathcal{M}}^n = P_{\mathcal{M}}$. Recall 线性回归中的一些投影矩阵，实际就是$\mathbb{R}^n$空间中的$P_{\mathcal{M}}$, 而$\mathcal{M}$ 是有observation 构成的closed span.
Comments