AR, MA & ARIMA Models

The ACF and PACF told you what kind of memory a series has. Now we build the machines that capture it. The three letters AR, MA, and I — autoregressive, moving-average, and integrated — combine into the ARIMA(p,d,q) family, the workhorse of classical time-series forecasting. Each letter answers a different question: AR asks “how much does today depend on its own past values?”, MA asks “how much does today depend on past shocks?”, and I asks “how many times must we difference to make it stationary?” Master what each term contributes and you can read an ARIMA specification like a sentence.

AR: today depends on its own past

Analogy. An autoregressive process has inertia. Like room temperature drifting back toward the thermostat setting, today’s value is mostly an echo of recent values plus a fresh nudge. Yesterday’s level predicts today’s level.

Definition. An AR(p) (autoregressive of order $p$) model writes today as a weighted sum of the previous $p$ values plus a shock: $X_t = c + \phi_1 X_{t-1} + \phi_2 X_{t-2} + \cdots + \phi_p X_{t-p} + \varepsilon_t,$ where $\varepsilon_t$ is white noise. The simplest case, AR(1), is $X_t = c + \phi_1 X_{t-1} + \varepsilon_t$ — the same equation that gave us unit roots two lessons ago.

Stationarity condition. For AR(1), the process is stationary only if $|\phi_1| < 1$. At $\phi_1 = 1$ it’s a unit-root random walk; beyond, it explodes. The coefficient $\phi_1$ is the persistence: $\phi_1 = 0.9$ means a shock fades slowly, $\phi_1 = 0.2$ means it’s nearly forgotten by tomorrow.

Worked example — forecasting with AR(1). Suppose $X_t = 2 + 0.6 X_{t-1} + \varepsilon_t$ and today’s value is $X_t = 10$.

• One-step forecast: $\hat{X}_{t+1} = 2 + 0.6 \times 10 = 8$ (set the future shock to its mean, zero).
• Two-step: $\hat{X}_{t+2} = 2 + 0.6 \times 8 = 6.8$.
• Long-run mean: solve $\mu = 2 + 0.6\mu \Rightarrow \mu = 2/(1 - 0.6) = 5$. The forecasts decay geometrically toward 5 — the unconditional mean — which is exactly the mean-reversion an AR(1) with $|\phi| < 1$ produces.

The AR signature on the diagnostic plots: PACF cuts off at lag $p$, ACF decays gradually.

MA: today depends on past shocks

Analogy. A moving-average process has aftershocks. Imagine a calm pond: a stone (a shock) creates ripples that persist for a few moments and then vanish completely. Today’s value is the current shock plus an echo of the previous few shocks — not the previous values. The memory is in the surprises, not the levels, and it’s strictly finite.

Definition. An MA(q) (moving-average of order $q$) model writes today as the current shock plus a weighted sum of the previous $q$ shocks: $X_t = \mu + \varepsilon_t + \theta_1 \varepsilon_{t-1} + \theta_2 \varepsilon_{t-2} + \cdots + \theta_q \varepsilon_{t-q}.$ The key difference from AR: an MA(q) has a finite memory of exactly $q$ steps. A shock today affects the next $q$ observations and then is completely gone — unlike AR, where a shock’s influence decays forever but never fully disappears.

Worked example — MA(1). Suppose $X_t = \varepsilon_t + 0.8\varepsilon_{t-1}$ (zero mean). Say last period’s shock was $\varepsilon_{t-1} = 3$ and this period’s is $\varepsilon_t = -1$. Then $X_t = -1 + 0.8 \times 3 = 1.4$. Next period, the influence of that $\varepsilon_{t-1} = 3$ is gone entirely — only $\varepsilon_t$ and the new $\varepsilon_{t+1}$ matter. That’s the abrupt cutoff.

The MA signature, the mirror image of AR: ACF cuts off at lag $q$, PACF decays gradually.

ARMA: combining both

Many real series carry both kinds of memory — inertia in the levels and lingering shock effects. ARMA(p,q) simply stacks the two: $X_t = c + \sum_{i=1}^{p}\phi_i X_{t-i} + \varepsilon_t + \sum_{j=1}^{q}\theta_j \varepsilon_{t-j}.$ It needs the series to be stationary already. The diagnostic signature is the giveaway that you need a blend: both the ACF and PACF decay gradually (neither cleanly cuts off), because the AR part muddies the MA’s cutoff and vice versa. ARMA is more parsimonious than a long pure-AR or pure-MA — a low-order ARMA can capture what would otherwise need many AR or MA terms.

The “I”: integration and ARIMA(p,d,q)

Pure ARMA needs a stationary series — but most financial series (prices!) aren’t. The I (“integrated”) handles that by folding in the differencing from lesson 1. ARIMA(p,d,q) says:

1. Difference the series $d$ times to make it stationary (the $d$),
2. then fit an ARMA(p,q) to the differenced series.

So the three numbers are:

• $p$ = autoregressive order (lags of the differenced series),
• $d$ = number of differences needed for stationarity,
• $q$ = moving-average order (lags of the shocks).

Reading specifications like sentences:

• ARIMA(0,0,0) = white noise (no structure).
• ARIMA(1,0,0) = AR(1) on an already-stationary series.
• ARIMA(0,1,0) = a random walk (difference once, model the difference as pure noise) — the canonical price model.
• ARIMA(0,1,1) = difference once, then MA(1) — equivalent to exponential smoothing, a classic forecasting method.
• ARIMA(1,1,1) = difference once, then a blend of one AR and one MA term on the returns.

For a stock price, you’d typically have $d = 1$ (difference the log price into log returns), then small $p$ and $q$ — because, as we know, returns are close to white noise, often leaving little for the AR/MA terms to do.

Fitting and choosing: how many terms?

Once you’ve settled $d$ (difference until stationary — check with ADF and a plot), you choose $p$ and $q$. Two complementary approaches:

1. Read the plots. Use the cheat sheet from last lesson: PACF cutoff suggests $p$, ACF cutoff suggests $q$. Good for a first guess.

2. Information criteria. Fit several candidate models and compare their AIC (Akaike) or BIC (Bayesian) information criterion. Both have the form $\text{IC} = -2\ln(\text{likelihood}) + (\text{penalty}) \times k,$ where $k$ is the number of parameters. The first term rewards fit; the penalty punishes complexity. Lower is better. AIC’s penalty is $2k$; BIC’s is $\ln(n)\,k$, which is harsher for large samples, so BIC tends to pick simpler models. Pick the specification with the lowest criterion, then validate: run Ljung–Box on the residuals — they should be white noise (a large p-value). If structure remains, add a term; if you’re over-parameterized, drop one.

Worked example — picking by BIC. Three candidates on the same returns:

AR(4) fits marginally best in raw likelihood, but its parameter penalty sinks it. ARMA(1,1) has the lowest BIC (1200.4), so you’d choose it — the small extra fit over AR(1) justifies one extra parameter, while AR(4)‘s extra terms don’t earn their keep.

Putting it together

The AR(p) model writes today as a weighted sum of its own past values plus a shock — inertia whose persistence lives in $\phi$ (stationary only when $|\phi| < 1$); its signature is a PACF cutoff at lag $p$. The MA(q) model writes today as the current shock plus echoes of the past $q$ shocks — a strictly finite memory; its signature is an ACF cutoff at lag $q$. ARMA(p,q) blends both (both plots decay) and needs a stationary input. ARIMA(p,d,q) adds the integration step: difference $d$ times to reach stationarity, then fit ARMA — so $(0,1,0)$ is a random walk, $(0,1,1)$ is exponential smoothing, and a stock is typically $(0,1,q)$ with small $q$. You choose $d$ by testing for stationarity, then $p$ and $q$ by reading the ACF/PACF and minimizing AIC/BIC (lower is better; BIC favors simpler models), and you validate by confirming the residuals pass Ljung–Box. The one thing ARIMA does not touch is the variance — it models the mean and assumes constant volatility, which is exactly the gap the next lesson fills.

Next up — volatility clustering & GARCH — we pivot from modeling the mean of returns to modeling their variance. Real markets have calm stretches and stormy stretches that bunch together, and the ARCH/GARCH family is how you turn that clustering into a forecast of tomorrow’s risk.