Keywords: Lehmann-Scheffe Theorem, Sufficient statistics, Minimal Sufficient statistics, Complete statistics, Lehmann Scheffe Theorem, Rao-Blackwell Theorem
1. A review of Sufficient statistics
1.1 The definitions of Sufficient statistics
Lemma: Let $t(x)$ denote some function of x. Then the following are equivalent:
(1) There exist functions $h(x)$ and $g(t; \theta)$ such that
$f(x; \theta) = h(x) g(t(x); \theta)$
(2) If $t(x) = t(x^\prime)$, for the likelihood ratio defined $lr(x) = \frac{f(x; \theta_1)}{f(x;\theta_2)}$, we can conclude that
$lr(x) = lr(x^\prime)$
Proof: (1) $\xrightarrow{}$ (2) is obvious.
(2) $\xrightarrow{}$ (1):
For some parameter $\theta_0 \in \Theta$, we can conclude that $f(x; \theta_1) = f(x; \theta_0) g^{*}(t(x), \theta_1, \theta_0)$. Let $h(x) = f(x; \theta_0)$ and $g(t(x), \theta_1) = g^{*}(t(x), \theta_1, \theta_0)$, which is exactly of the from of (1).
Definition: The statistic $T = T(x)$ is sufficient for $\theta$ is the distribution of $X$, conditional on $T(X) = t$, is independent of $\theta$. The two criteria for sufficiency are:
(a) Factorization Theorem: $t(X)$ is sufficient for $\theta$ if and only if Lemma.1 holds.
(b) Likelihood ratio Criterion: $T(x)$ is sufficient for $\theta$ if and only if Lemma.2 holds.
Note that the sufficient statistics may have different dimension of $\theta$.
1.2 The minimal sufficient statistics
We want to find a ‘minimal’ sufficient statistics that include all the sufficient statistics, i.e. there exist a function mapping from sufficient statistics to the minimal sufficient statistics.
Lemma: If T and S are minimal sufficient statistics, then there exist injective functions $g_1$ and $g_2$ such that $T = g_1(S)$ and $S = g_2(T)$.
The injective function means for any $y_1, y_2 \in Y$, for the function $X \xrightarrow{} Y$, if $y_1 = y_2$, then $x_1 = x_2$.
Proof: The definition of minimal sufficient statistics means that there exist functions $g_1$ and $g_2$, such that $T = g_1(S)$, $S = g_2(T)$. Consider $x_1, x_2$ s.t. $T(x_1) = g_1(S(x_1)) = g_2(S(x_2)) = T(x_2)$, therefore $S(x_1) = g_1(T(x_1)) = g_1(T(x_2)) = S(x_2)$.
Theorem: A necessary and sufficient condition for a statistic $T(X)$ to be minimal sufficient is that
$T(x) = T(x^\prime)$ if and only if $\Lambda_x(\theta_1, \theta_2) = \Lambda_{x^\prime}(\theta_1, \theta_2)$ for all $\theta_1$ and $\theta_2$.
This theorem shows that $x = x^{\prime}$ iff they define the same likelihood ratio for any $\theta_1, \theta_2 \in Theta$. Note that the conclusion only holds in a finite Euclidean space.
2. Completeness
A sufficient statistic $T(X)$ is complete if for any real measurable function $g$,
$\mathbb{E}_{\theta}[g(T)] = 0$ for all $\theta$.
implies
$\textbf{Pr}(g(T) = 0) = 1$ for all $\theta$.
3. Lehmann-Scheffe Theorem
Suppose $X$ has density $f(x; \theta)$ and $T(X)$ is sufficient and complete for $\theta$. Then $T$ is minimal sufficient.
Suppose $T(X)$ is a sufficient statistic for $\theta$, and let $f_{T}(t; \theta)$ be a complete family. If $\varphi$: $\mathbb{E}[\varphi(Y)] = \theta$ then $\varphi(Y)$ is the unique MVUE of $\theta$. (The optimality can be easily proved by Rao-Blackwell. The uniqueness is determined by the complete family. )
（对带估计参数无偏的充分完全统计量（充分完全统计量是最小充分统计量）是无偏估计中MSE最小的估计量；完全性之影响是否唯一性）
4. Convex loss function and Rao-Blackwell Theorem
Suppose we want to estimate a real-valued parameter $\theta$ with an estimator $d(X)$. Suppose the loss function is a convex function of $d$ for each $\theta$. Let $d_1(X)$ be an unbiased estimator for $\theta$ and suppose $T$ is a sufficient statistic. Then the estimator $\mathcal{X}(T) = \mathbb{E}\{d_1(X)|T\}$ (充分性保证了这里还是充分统计量的函数， T的结构不会被破坏) is also unbiased and is at least as good as $d_1$.
Proof: $\mathcal{X}(T)$ is unbiased from the tower law.
$\mathbb{E}(\mathcal{X}(T)) = \mathbb{E}\{d_1(X)|T\} = \mathbb{E}_{\theta} d_1(X) = \theta$.
For the risk function, we have
$R(\theta, d_1) = \mathbb{E}_{\theta}\{L(\theta, d_1(X))\} = \mathbb{E}_{\theta}\{\mathbb{E}[L(\theta, d_1(X))|T]\} \ge \mathbb{E}_{\theta}[L(\theta, \mathcal{X}(T))] = R(\theta, \mathcal{X}(T))$ （注意倒数第二步使用了Jenson 不等式和充分统计量的定义/性质）
Reference
Essentials of Statistics Inference
Comments