Keywords: Complex-valued ACF, Spectral Distribution, Herglotz’s Theorem, Isomorphism, Orthogonal-increment process
1. Definitions and Properties
Complex-valued stationary process: The process $\{X_n\}$ is a complex values stationary process if $\mathbb{E}|X_t|^2 < \infty$, $\mathbb{E}X_t$ is independent of $t$ and $\mathbb{E}(X_{t+h}\bar{X}_t)$ is independent of $t$.
Note that in the complex vector space, the inner product is defined as $(x, y) = \mathbb{E}(X\bar{Y})$, and the autocovariance function(ACF) $\gamma(\cdot)$ of a complex-valued stationary process $\{X_t\}$ is $\gamma(h) = (X_{t+h}, X_t) - (X_{t + h}, 1)(1, X_t)$, i.e. $\gamma(h) = \mathbb{E}(X_{t + h}\bar{X_t}) - \mathbb{E}(X_{t + h})\mathbb{E}(\bar{X_t})$ (复数域一定要小心顺序和复共轭的问题)
The properties of complex-valued ACF includes:
(1) $\gamma(0) \ge 0$
(2) $|\gamma(h)| \le \gamma(0)$ (Complex-valued Cauchy-Schwarz)
(3) $\gamma(\cdot)$ is a Hermitian function, i.e. $\gamma(h) = \bar{\gamma(-h)}$
Theorem 1: The relationship between ACF and Stochastic process
A function $K(\cdot)$ defined on the integers is the autocovariance function of a stationary time series if and only if $K(\cdot)$ is Hermitian and non-negative definite, i.e. if and only if $K(n) = \bar{K(-n)}$ and $\sum\limits_{i, j}^n a_i K(i - j) \bar{a_j} \ge 0$ for any integer $n$ and all vectors $a = (a_1, \dots, a_n)^\prime \in \mathbb{C}^n$ (还是用构造法设计一个符合Kolmogorov‘s Theorem的离散时间随机过程)
2. Fourier Series
Consider the complex Hilbert space $L^2[-\pi, \pi] = L^2([-\pi, \pi], \mathcal{B}, U)$, where $\mathcal{B}$ is the Borel set of $[-\pi, \pi]$, $U$ is the uniform probability measure $U(dx) = d(U(x)) = \frac{1}{2\pi} dx$. The inner product in the complex space is defined as usual by $(f, g) = Ef\bar{g} = \frac{1}{2\pi} \int_{-\pi}^{\pi} f(x)\bar{g}(x)dx$.
Consider $\{e_n(x) = e^{inx}\}, n \in \mathbb{Z}$.
Note that $(e_n(x), e_m(x)) = \frac{1}{2\pi} \int_{-\pi}^{\pi} e^{i(m - n)x}dx = 0$, therefore $e_i(x), i \in \mathbb{Z}$ is the orthonormal set of this complex $L^2[-\pi, \pi]$.
Fourier approximation: The nth order Fourier approximation to any function $f \in L^2[-\pi, \pi]$ is defined to be the projection of f onto $\bar{sp}(e_j)$, with the same logic of Parseval’s identity, we can write the nth order of Fourier approximation as: $S_n f = \sum_{i = -n}^{i = n} (f, e_i) e_i$. The Fourier coefficients can be written as $(f, e_j) = \frac{1}{2\pi} \int_{-\pi}^{\pi} f(x) e^{-ijx}dx$
Theorem $Sf = f$:
(a) The sequence $\{S_n f\}$ has a mean square limit as $n \xrightarrow{} \infty$. Let $Sf$ denote $\lim\limits_{n \xrightarrow{} \infty}S_nf = \lim\limits_{n \xrightarrow{} \infty} \sum_{|j| < n} (f, e^{ijt}) e^{ijt}$.
(b) $Sf = f$.
Proof:
(a) From Bessel inequality, we have $\sum_{|j| < n}|(f, e_j)|^2 \le |f|^2$. (Note that $(f, f) = \frac{1}{2\pi} \int_{-\pi}^{\pi}f(x)\bar{f}(x)dx$). Note that we are working in the $L^2$ space. Therefore, we have a converge Cauchy sequence.
(b) For $|j| < n$, $(S_n f, e_j) = (f, e_j)$ (by the definition of orthonormal set), $(Sf, e_j) = \lim\limits_{n \xrightarrow{} \infty} (S_nf, e_j) = (f, e_j)$ ($Sf \xrightarrow{} f$ and continuity of inner-product space)
3. The spectral distribution of a linear combination of sinusoids
Let’s consider a specific stochastic process in this section. Let’s assume $X_t$ can be written as:
$X_t = \sum_{j = 1}^n A(\lambda_j)e^{it\lambda_j}$,
where $-\pi < \lambda_1 < \lambda_2 < \dots < \lambda_n < \pi$, and $A(\lambda_i)$ are uncorrelated complex-valued random coefficients (may follow Dirac distribution) such that
$\mathbb{E}(A(\lambda_j)) = 0$, $j \in \mathbb{Z}$ and $\mathbb{E}(A(\lambda_j)\bar{A(\lambda_j)}) = \sigma_j^2$
Note that the autocorrelation function can be written via a distribution function, i.e.,
$\gamma(h) = \int_{(-\pi, \pi]} e^{ihv} d F(v)$, where $F(\lambda) = \sum\limits_{j:\lambda_j < \lambda} \sigma^2_j$ (Spectral distribution function)
The remarkable feature of this example is that every zero-mean stationary process has a representation which is a natural generalization $X_t = \int_{(-\pi, \pi]} e^{itv}dZ(v)$, which is a stochastic integral w.r.t. an orthogonal-increment process.
The ACF of a stationary process can be written as $\gamma_X(h) = \int_{(-\pi, \pi]} e^{ihv} dF(v)$, where $F(\cdot)$ is a distribution function with $F(-\pi) = 0$ and $F(\pi) = \mathbb{E}(X_t^2)$
Herglotz’s Theorem: A complex-valued function $\gamma(\cdot)$ defined on the integers is non-negative definite *if and only if *
$\gamma(h) = \int_{(-\pi,\pi]} e^{ihv} d F(v)$, where $F(\cdot)$ is a right-continuous, non-decreasing, bounded function on $[-\pi, \pi]$ and $F(-\pi) = 0$. (Function F is called the spectral distribution function of $\gamma$ and if $F(\lambda) = \int_{-\pi}^\lambda f(v) dv, -\pi < \lambda < \pi$, then f is called a spectral density function.)
结合Herglotz’s Theorem和Theorem 1， 我们得到了谱概率分布和时间序列之间的关系。
Inverse Transform: If $K(\cdot)$ is any complex-valued function on the integers such that
$\sum_{-\infty}^{\infty}|K(n)| < \infty$
Then $K(h) = \int_{-\pi}^{\pi} e^{ihv} f(v) dv$, where $f(v) = \sum_{-\infty}^{\infty} e^{-in\lambda}K(n)$
Proof: Use Dirac function
Corollary: An absolutely summable complex-valued function $\gamma(\cdot)$ defined on the integers is the autocovariance function of a stationary process iff $f(\lambda) = \frac{1}{2\pi} \sum_{-\infty}^{\infty} e^{-in\lambda}\gamma(n)$, where $f(\lambda)$ is the spectral density of the $\gamma(\cdot)$, i.e. $\gamma(h) = \int_{-\pi}^{\pi}e^{ihv}f(v)dv$
4. Corollaries for stationary time series
Theorem 4.1
If $\{Y_t\}$ is any zero-mean, possibly complex-valued stationary process with spectral distribution function $F_{Y}(\cdot)$, and $\{X_t\}$ is the process $$X_t = \sum_{j = -\infty}^{+\infty} \varphi_{j}Y_{t - j}$$, where $\sum_{j = -\infty}^{+\infty} |\varphi_{j}| < + \infty$ (绝对可积老条件了，一个充分条件是指数衰减，保证某个N之后的和有界即可) then $\{X_t\}$ is stationary with spectral distribution function $F_{X}(\lambda) = \int_{(-\pi, \lambda]} |\sum_{j = -\infty}^{\infty} \varphi_j e^{-ijv}|^2 dF_y(v)$
Proof:
$\mathbb{E}(X_{t + h} \bar{X}_{t}) = \mathbb{E}(\sum_{j = -\infty}^{j = \infty}\varphi_{t + h + j}Y_{t + h + j} \sum_{k = -\infty}^{k = \infty}\bar{\varphi}_{t + k}\bar{Y}_{t + k}) = \sum_{j, k}^{\infty}\varphi_{t + h + j}\bar{\varphi}_{t + k} \mathbb{E}(Y_{t + h + j}\bar{Y}_{t + k}) = \sum_{j, k}^{\infty}\varphi_{t + h + j}\bar{\varphi}_{t + k} \mathbb{E}(Y_{t + h + j}\bar{Y}_{t + k}) = \sum_{j, k}^{\infty}\varphi_{t + h + j}\bar{\varphi}_{t + k} \gamma_Y(h + j - k) = \gamma_X(h)$
$\gamma_X(h) = \sum_{j, k}^{\infty}\varphi_{t + h + j}\bar{\varphi}_{t + k} \gamma_Y(h + j - k) = \sum_{j, k}^{\infty}\varphi_{t + h + j}\bar{\varphi}_{t + k} \int_{(-\pi, \pi]} e^{i(h + j - k)v}dF_Y(v) = \int_{(-\pi, \pi]} |\sum_{j = -\infty}^{\infty} \varphi_j e^{-ijv}|^2 e^{ihv}dF_Y(v) = \int_{(-\pi, \pi]} e^{ihv}dF_X(v)$
Theorem 4.2 Spectral Density of an ARMA(p, q) Process
Let $\{X_t\}$ be an ARMA(p, q) process (not necessarily causal and invertible) satisfying $\phi(B) X_ t= \theta(B) Z_t$, $Z_t \sim WN(0, \sigma^2)$. $\phi(\cdot)$ has no zeroes on the unit circle. Then $\{X_t\}$ has spectral density $f_{X}(\lambda) = \frac{\sigma^2 |\theta(e^{-i\lambda})|^2}{2\pi |\phi(e^{-i\lambda})|^2}$
Proof:
$U(t) = \phi(B)X_t = \theta(B)Z_t$ (滞后算子值得借鉴). According to Theorem 4.1, we can conclude that:
$f_U(\lambda) = |\phi(e^{-iv})|^2 f_X(\lambda) = |\theta(e^{-iv})|^2 f_Z(\lambda)$
$f_X(\lambda) = \frac{|\theta(e^{-iv})|^2}{|\phi(e^{-iv})|^2}f_Z(\lambda)$
Note that $Z_t$ is white noise, the autocovariance function is $\gamma_Z(h) = \sigma^2\delta_0(h)$
According to the last corollary in the last section, we can conclude that
$f_z(\lambda) = \frac{1}{2\pi} \sum_{n = -\infty}^{+\infty} \gamma_z(h) e^{-in\lambda} = \frac{\sigma^2}{2\pi}$
$\therefore f_X(\lambda) = \frac{|\theta(e^{-iv})|^2}{|\phi(e^{-iv})|^2}\frac{\sigma^2}{2\pi}$
5. Spectral representation from a isomorphism perspective
Let $\{X_t\}$ be a zero mean stationary process with spectral distribution function $F$, i.e. $\mathbb{E}[X_{t + h}\bar{X_t}] = \int_{(-\pi, \pi]}e^{ihv}dF(v)$ (according to Herglotz’s theorem, such spectral distribution function $F$ is a sufficient and necessary condition for a non-negative definite function). We need to find a isomorphism between the subspace $\mathcal{H} = sp\{X_t, t \in \mathbb{Z}\} \subset L^2(\Omega, \mathcal{F}, P)$ and $\mathcal{K} = sp\{e^{it\cdot}, t\in \mathbb{Z}\} \subset L^2([-\pi, \pi], \mathcal{B}, F)$, which connected the random variables in the time domain and the functions in the frequency domain.
Note that the mapping
$T(\sum_{j = 1}^n a_j X_{tj}) = \sum_{j = 1}^n a_je^{it\cdot}$
defines an isomorphism between $\mathcal{H}$ and $\mathcal{K}$. This mapping T has the property that:
$(T(\sum_{j = 1}^n a_j X_{tj}), T(\sum_{k = 1}^m b_k X_{sk})) = (\sum_{j = 1}^n a_j e^{it_j \cdot}, \sum_{k = 1}^m b_k e^{is_k \cdot})_{L^2(F)} = \sum_{i, j = 1}^{n, m} a_j \bar{b}_k(e^{it_j \cdot}, e^{is_k \cdot})_{L^2(F)} = \sum_{i, j = 1}^{n, m} a_j \bar{b}_k \int_{(-\pi, \pi]}e^{iv(t_j - s_k)}dF(v) = \sum_{i, j = 1}^{n, m} a_j \bar{b}_k \mathbb{E}[X_{t_j}\bar{X}_{s_k}] = \sum_{i, j = 1}^{n, m} a_j \bar{b}_k (X_{t_j}, X_{s_k}) = (\sum_{j = 1}^n a_j X_{tj}, \sum_{k = 1}^m b_k X_{sk})$
If $(\sum_{j = 1}^n a_j X_{tj}, \sum_{k = 1}^m b_k X_{sk}) \xrightarrow{} 0$, $(T(\sum_{j = 1}^n a_j X_{tj}), T(\sum_{k = 1}^m b_k X_{sk}))$ also converge to zero. Therefore, this isomorphism is well-defined. We can further extend the mapping to a complete inner-product space.
Theorem 5.1: If $F$ is the spectral distribution function of the stationary process $\{X_t, t \in \mathbb{Z}\}$, then there is a unique isomorphism T of $\bar{sp}\{X_t, t\in \mathbb{Z}\}$ onto $L^2(F)$ such that $T(X_t) = e^{it\cdot}$ (证明略，这里的unique大概又是范数为零，a.s.搞过来的)
Comment: T是一个把随机变量映射(概率空间映射到状态空间的函数)到deterministic function的映射. 我们希望基于T找到一个正交增量过程，用它来表示平稳随机过程$X_t$.
Theorem 5.2 (The spectral Representation Theorem) If $\{X_t\}$ is a stationary sequence with mean-zero and spectral distribution function $F$, then there exists a right-continuous orthogonal-increment process $\{Z(\lambda), -\pi < \lambda < \pi\}$ such that:
(i) $\mathbb{E}|Z(\lambda) - Z(-\pi)|^2 = F(\lambda)$ (Recall: $F(\lambda) = \frac{1}{2\pi} \sum_{h = -\infty}^{\infty} e^{-ih\lambda} \gamma(h)$, where $\gamma(\cdot)$ is the autocovariance function)
(ii) $X_t = \int_{(-\pi, \pi]} e^{itv} dZ(v)$
Comment: 谱分解定理表明，我们总是可以找到一个定义在某概率空间上的正交增量过程，使得由改正交增量过程构成的最小闭子空间与$L^2([-\pi, \pi], \mathcal{B}, F)$同构。($\{Z_{\lambda}\}$)同样是定义在
Proof:
Lemma 5.2: If T is defined as in Theorem 5.1, then the process $\{Z(\lambda), -\pi \le \lambda \le \pi\}$ defined by $Z(\lambda) = T^{-1}(I_{(-\pi, \lambda]}(\cdot))$ ($I$ is the identification function), $-\pi\le \lambda \le \pi$ is an orthogonal increment process. Moreover the distribution function associated with $\{Z(\lambda)\}$ is exactly the spectral distribution function $F$ of $\{X_t\}$.
Proof of Lemma 5.2: Since $Z(\lambda) \in \bar{sp}\{X_t, t\in \mathbb{Z}\}$ there is a sequence $\{Y_n\}$ of elements of $\bar{sp}\{X_t, t\in \mathbb{Z}\}$ (Note: $\bar{\cdot}$ means closed subspace) such that $|Y_n - Z(\lambda)| \xrightarrow{} 0$ as $n \xrightarrow{} \infty$. By the continuity of inner-product, we can conclude that $(Z(\lambda), 1) = \lim\limits_{n \xrightarrow{} \infty}(Y_n, 1) = 0$ since each $\{Y_n\}$ has zero mean.
Finally, if $-\pi \le \lambda_1 \le \lambda_2 \le \lambda_3 \le \lambda_4 \le \pi$, $(Z(\lambda_4) - Z(\lambda_3), Z(\lambda_2) - Z(\lambda_1))_{L^2(\Omega, \mathcal{F}, P)} = (TZ(\lambda_4) - TZ(\lambda_3), TZ(\lambda_2) - Z(\lambda_1))_{L^2(F)} = (I_{(\lambda_3, \lambda_4]}(\cdot), I_{(\lambda_1, \lambda_2]}(\cdot))_{L^2(F)} = 0$. Therefore, $Z(\lambda)$ is an orthogonal increment process. Moreover, if we take $(Z(\mu) - Z(\lambda), Z(\mu) - Z(\lambda))_{L^2(\Omega, \mathcal{F}, P)} = (TZ(\mu) - TZ(\lambda), TZ(\mu) - Z(\lambda))_{L^2(F)} = \int_{(-\pi, \pi]}I_{(\lambda, \mu]}(v)I_{(\lambda, \mu]}(v)dF(v) = F(\mu) - F(\lambda)$ $\therefore Q.E.D$
Let $\{Z_t\}$ be the process defined in Lemma 5.2. and let $I$ (not the identification function we used in Lemma 5.1) be the isomorphism $I(f) = \int_{(-\pi, \pi]} f(v)dZ(v)$. In the blog ‘Orthogonal-increment process’, we know if $\mathcal{D} = L^2(F)$ onto $I(\mathcal{D}) \subset L^2(\Omega, \mathcal{F}, P)$. For a Borel measurable function $f \in \mathcal{D}$, we can represent f as:
$I(f) = \sum_{i = 0}^n f_{i}(Z(\lambda_{i + 1}) - Z(\lambda_{i})) = \sum_{i = 0}^n f_{i}T^{-1}I_{(\lambda_{i}, \lambda_{i + 1}]} = T^{-1}(f)$. Therefore, we can conclude that both stochastic integral operator $I(\cdot)$ and $T^{-1}$ are isomorphisms between $L^2(F)$ and $L^2(\Omega, \mathcal{F}, P)$. Theorem 5.2 Q.E.D.
Comment: 谱表示实际上是一个构造性的证明。这一整套来源于两个Hilbert空间上的两个同构。其中$\mathcal{K} = L^2(F)$是一个辅助空间, 对于概率空间$\mathcal{H} = L^2(\Omega, \mathcal{F}, P)$ $T: \mathcal{H} \xrightarrow{} \mathcal{K}$, 和$I: \mathcal{K} \xrightarrow{} \mathcal{H}$ 是两个同构，而且恰好 $TI(f) = f \quad \forall f \in \mathcal{K}$
Corollary: If $\{X_t\}$ is a zero-mean stationary sequence then there exists a right continuous orthogonal increment process $\{Z(\lambda), -\pi \le \lambda \le \pi\}$ such that $Z(-\pi) = 0$ and
$X_t = \int_{-\pi, \pi} e^{itv}dZ(v)$ with probability one. If $\{Y(\lambda)\}$ and $\{Z(\lambda)\}$ are two such processes then $P(Y(\lambda) = Z(\lambda)) = 1$
We can easily prove the above statement with the help of $L^2(F)$. The spectral presentation does not reveal how to construct $\{Z(\lambda)\}$. We need the inversed transformation to solve the $\{Z(\lambda)\}$
6. Inversion Formulae
With the same logic, we want to find a random variable which equals $Z(\lambda)$ almost surely (mean square convergence). Recall that $T(Z(w) - Z(v)) = I_{(v, w]}(\cdot)$ and $TX_t = e^{it\cdot}$. Therefore, if we can find $\sum_{|j| < n} \alpha_j e^{ij\cdot}\xrightarrow{L^2(F)} I_{(v, w]}(\cdot)$, then by the isomorphism, $\sum_{|j| \le n} \alpha_j X_j \xrightarrow{m.s.} Z(w) - Z(v)$.
Let $h_n(\lambda) = \sum_{|j| < n} \alpha_je^{ij\lambda} $ denote the nth-order Fourier series approximation to $\frac{1}{2\pi}\int_{(-\pi, \pi]}I_{(v, w]}(\lambda) e^{-ij\lambda} d \lambda = \frac{1}{2\pi}\int_{(v, w]}e^{-ij\lambda} d \lambda$
Theorem: If $\{X_n\}$ is a stationary sequence with autocovariance function $\gamma(\cdot)$, spectral distribution function F, and spectral representation $X_t = \int_{(-\pi, \pi]} e^{itv}dZ(v)$, and if v and w ($-\pi < v < w < \pi$) are continuity points of F, then as $n \xrightarrow{} \infty$,
$\frac{1}{2\pi}\sum_{|j| \le n} X_j (\int_v^w e^{-ijv}dv) \xrightarrow{m.s.} Z(w) - Z(\lambda)$ (Note it is the Fourier transformation of the indicator function, 易错)
$\frac{1}{2\pi}\sum_{|j| \le n} \gamma(j) (\int_v^w e^{-ijv}dv) \xrightarrow{} F(w) - F(\lambda)$
7. Prediction in Frequency Domain
The stochastic integral $I$ defined on the associated orthogonal increment process $\{Z(\lambda), -\pi < \lambda < \pi\}$ is an isomorphism of $L^2(F)$ on to the probability space, with the property that
$I(e^{it\cdot}) = X_t$
Therefore, we can find the best prediction on the $L^2(F)$. For example, the best linear predictor $P_{\mathcal{M_n}}X_{n + h}$ of $X_{n + h}$ in $\mathcal{M_n} = \bar{sp}\{X_t, -\infty \le t \le n\}$ can be written as:
$P_{\mathcal{M}_n}X_{n + h} = I(P_{\bar{sp}\{\exp(it\cdot)\}}e^{i(n + h)\cdot})$
Reference
Time Series: Theory and Methods
Comments