SDSC6012 Course 2-Stationarity and autoregressive models

#sdsc6012

English / 中文

Components and Decomposition of Time Series

A time series typically consists of three components:

1. Trend Component: Long-term direction of change

2. Seasonal Component: Fluctuations with fixed periods

3. Random Noise: Unexplained random fluctuations

Using Python’s Matplotlib and NumPy libraries, one can generate and visualize the combined effects of these components.

---

Key Statistics: Measuring Dependence

Mean Function

$$\mu_t=E\left(x_t\right)=\int_{-\infty}^{\infty} x f_t(x) d x$$

• Represents the average level of the time series at time $t$

• For continuous random variables, calculated by integrating with the probability density function $f(x)$

Autocovariance Function

$$\gamma(s, t)=\operatorname{Cov}\left(x_{s}, x_{t}\right)=E\left[\left(x_{s}-\mu_{s}\right)\left(x_{t}-\mu_{t}\right)\right]$$

• Measures the linear relationship between observations at two different time points $(s, t)$ in the same time series

• Difference from ordinary covariance: Autocovariance measures the relationship of the same variable at different time points

Calculation Example:
For vectors $X: [1, 2, 3, 4, 5]$ and $Y: [2, 4, 6, 8, 10]$, the covariance is calculated as:

$$\operatorname{Cov}(X, Y)=\frac{1}{N}\sum_{i=1}^{N}\left(x_{i}-\mu_{X}\right)\left(y_{i}-\mu_{Y}\right)=4$$

---

Detailed Calculation of Expected Value

Expected Value for Discrete Random Variables

$$E(X)=\Sigma\left[x_{i}* P\left(x_{i}\right)\right]$$

• $x_{i}$: The $i$-th possible value of random variable $X$

• $P\left(x_{i}\right)$: The probability corresponding to that value

Dice Expected Value Calculation:

$$E(X)=\frac{1}{6}(1+2+3+4+5+6)=3.5$$

Expected Value for Continuous Random Variables

$$E(X)=\int_{-\infty}^{\infty}x\cdot f(x)dx$$

• $f(x)$: Probability Density Function (PDF)

• For continuous variables, the probability at a single point is 0, only interval probabilities can be calculated

Bus Waiting Time Example (Uniform distribution $[0,10]$ minutes):

$$E(X)=\int_{0}^{10}x\cdot\frac{1}{10}dx=5$$

---

Stationarity

Strict Stationarity

• Requires that the entire probability distribution of the time series remains unchanged over time

• For any $k, t_{1},\ldots, t_{k}, h$, satisfy:

  $$P\left\{x_{t_{1}}\leq c_{1},\ldots, x_{t_{k}}\leq c_{k}\right\}=P\left\{x_{t_{1}+h}\leq c_{1},\ldots, x_{t_{k}+h}\leq c_{k}\right\}$$

• The autocovariance function satisfies: $\gamma(s,t)=\gamma(s+h,t+h)$

Weak Stationarity

• More lenient conditions, only need to satisfy:

  1. Constant Mean: $\mu_t=\mu$ (independent of time $t$)
  2. Constant Variance: $\gamma(0)$ is constant
  3. Autocovariance Depends Only on Time Lag: $\gamma(t+h, t)=\gamma(h, 0)=\gamma(h)$

Analogy for Understanding:

• Strict Stationarity: All aspects of an orchestra’s performance (melody, harmony, rhythm, volume, timbre) remain exactly the same
• Weak Stationarity: Only requires constant average volume, constant range of volume fluctuations, and stable rhythm

Properties of the Autocovariance Function

1. $\gamma(0)\geq 0$ (variance is non-negative)

2. $|\gamma(h)|\leq\gamma(0),\forall h$ (autocovariance does not exceed variance)

3. $\gamma(h)=\gamma(-h),\forall h$ (symmetry)

---

Autocorrelation Function (ACF)

$$\rho(h)=\frac{\gamma(h)}{\gamma(0)}=\frac{\operatorname{Cov}\left(x_{t+h}, x_{t}\right)}{\operatorname{Var}\left(x_{t}\right)}=\operatorname{Corr}\left(x_{t+h}, x_{t}\right)$$

• Range: $-1\leq\rho(h)\leq 1$

• Answers: “How correlated is today’s data with yesterday’s, the day before yesterday’s, etc.?”

---

Stationarity Example Analysis

White Noise Process

• $E\left(w_{t}\right)=0$ (constant mean)

• $$\gamma(s, t)=\operatorname{cov}\left(w_s, w_t\right)=\begin{cases}\sigma_w^2& s=t\\ 0& s\neq t\end{cases}$$

• Is a stationary process

Random Walk Process

• $x_{t}=\sum_{j=1}^{t} w_{j}$

• $\gamma(s, t)=\min\{s, t\}\sigma_{w}^{2}$ (depends on both $s$ and $t$)

• Is not a stationary process

First-Order Moving Average Process MA(1)

• $x_{t}=w_{t}+\theta w_{t-1}$

• $E\left(x_{t}\right)=0$ (constant mean)

• $$\gamma(s, t)=\begin{cases}\left(1+\theta^2\right)\sigma^2,& s=t\\ \theta\sigma^2,&|s-t|=1\\ 0,&|s-t|\geq 2\end{cases}$$

• Is a stationary process

---

Estimation of Correlation

Sample Mean

$$\bar{x}=\frac{1}{n}\sum_{t=1}^{n} x_{t}$$

• $E(\bar{x})=\mu$ (unbiased estimator)

• $\operatorname{Var}(\bar{x})=\frac{1}{n}\sum_{h=-n}^{n}\left(1-\frac{|h|}{n}\right)\gamma(h)$

Sample Autocovariance Function

$$\hat{\gamma}(h)=\frac{1}{n}\sum_{t=1}^{n-|h|}\left(x_{t+|h|}-\bar{x}\right)\left(x_{t}-\bar{x}\right),\quad\text{ for}-n<h<n$$

Sample Autocorrelation Function (Sample ACF)

$$\hat{\rho}(h)=\frac{\hat{\gamma}(h)}{\hat{\gamma}(0)},\quad\text{ for}-n<h<n$$

White Noise Test: If $\left\{w_{t}\right\}$ is white noise, then no more than 5% of sample ACF values satisfy:

$$|\hat{\rho}(h)|>\frac{1.96}{\sqrt{n}}$$

---

Differencing Operations

Backshift Operator

• $B x_{t}=x_{t-1}$

• $B^{k} x_{t}=x_{t-k}$

Difference Operator

• First-order difference: $\nabla x_{t}=x_{t}-x_{t-1}=(1-B) x_{t}$

• d-th order difference: $\nabla^{d}=(1-B)^{d}$

Differencing to Remove Trends

• First-order difference removes linear trend:

  $$\begin{align*}x_t&=\beta_0+\beta_1 t+y_t\\ \nabla x_t&=\beta_1+y_t-y_{t-1}\end{align*}$$

• Second-order difference removes quadratic trend:

  $$\begin{align*}x_t&=\beta_0+\beta_1 t+\beta_2 t^2+y_t\\ \nabla^2 x_t&=2\beta_2+y_t-2 y_{t-1}+y_{t-2}\end{align*}$$

---

Tool Libraries Introduction

Matplotlib

• Comprehensive visualization library for Python

• Creates static, dynamic, and interactive visualizations

• Supports high-quality publication graphics output

NumPy

• Fundamental package for scientific computing in Python

• Powerful N-dimensional array object

• Provides comprehensive mathematical functions, random number generators, etc.

SciPy

• Scientific computing library based on NumPy

• Provides algorithms for optimization, integration, interpolation, eigenvalue problems, etc.

• Underlying implementation uses highly optimized Fortran, C, and C++