Markov Chains

A random walk (lesson 2) builds its whole future out of its whole past — every step is summed onto every step before it. But a huge swath of finance gets away with a far cheaper assumption: that the future depends on only where you are right now, not the winding road that got you there. A market is “in a bull regime” or “in a bear regime”; a bond is rated “AAA” or “BBB”; a customer is “active” or “churned.” Knowing the current label is enough to forecast the next one — the history is forgettable. That forgetfulness is the Markov property, and a process that obeys it, hopping between a finite set of states, is a Markov chain. This lesson is the machinery: states, transition matrices, matrix powers for multi-step forecasts, and the long-run stationary distribution that markets drift toward.

The Markov property: the future forgets the past

Analogy. Picture a token on a Snakes & Ladders board. Your next move depends only on the square you’re standing on and your dice roll — never on the convoluted path of snakes and ladders that delivered you there. Two players sitting on square 37 face identical futures, even if one climbed a ladder and the other slid down a snake to arrive. The board is memoryless.

Formally, a process $X_0, X_1, X_2, \dots$ taking values in a finite set of states has the Markov property when

$P(X_{n+1} = j \mid X_n = i,\, X_{n-1},\, \dots,\, X_0) = P(X_{n+1} = j \mid X_n = i).$

Read it slowly: the probability of landing in state $j$ next, given the entire history on the left, collapses to a probability that depends on $i$ (today’s state) alone. Everything before today is irrelevant once you know today. That collapse is the whole game — it’s what makes the math tractable.

Contrast that with genuinely path-dependent quantities. The running maximum of a price (“the highest it’s ever been”) is not Markov in the price: to know it tomorrow you need the whole past, not just today’s price. An Asian option’s payoff depends on the average price over its life — again, history matters. Markov is a real restriction, not a free lunch: it only fits processes where today is a sufficient summary.

Transition matrices

Once the chain is memoryless, all its behaviour is captured by one table: for each pair of states, the probability of hopping from one to the other in a single step. Stack those into a square matrix $P$ where

$P_{ij} = P(X_{n+1} = j \mid X_n = i).$

Row $i$ holds every probability of leaving state $i$. Because from state $i$ you must go somewhere next step, each row sums to 1. A square matrix with non-negative entries whose rows each sum to 1 is called a stochastic matrix — that’s the formal name for a transition matrix.

Worked example. Model a market in one of two daily moods: Calm or Volatile. Suppose a Calm day is followed by another Calm day 90% of the time, and tips into Volatile 10% of the time. A Volatile day stays Volatile 60% of the time and settles back to Calm 40% of the time. Write that as a 2×2 stochastic matrix, rows = today, columns = tomorrow:

To read a row: stand on today’s state, run along its row, and the entries are next-step probabilities. Sitting in Volatile today? Row 2 says 0.40 chance of Calm tomorrow, 0.60 chance of staying Volatile — and $0.40 + 0.60 = 1$, as every row must. The columns, by contrast, do not have to sum to anything in particular (here column “Calm” sums to 1.30); only rows are constrained.

n-step transitions = matrix powers

A single step lives in $P$. What about two steps, or ten? The beautiful fact: the probability of going from $i$ to $j$ in exactly $n$ steps is the $(i,j)$ entry of the matrix power $P^n$.

$P(X_{n} = j \mid X_0 = i) = (P^n)_{ij}.$

Why powers? To go $i \to j$ in two steps you must pass through some intermediate state $k$: $i \to k$ then $k \to j$. Sum over every possible middleman $k$, multiplying the two leg-probabilities, and that sum-of-products is exactly the definition of matrix multiplication. So $P^2 = P\cdot P$ gives two-step probabilities, $P^3$ three-step, and so on. This decomposition — “to get there in $m+n$ steps, go somewhere in $m$ steps then finish in $n$” — is the Chapman–Kolmogorov equation in words: $P^{m+n} = P^m P^n$.

Worked example. Take our Calm/Volatile matrix and compute $P^2$ by hand. The two-step “Calm today → Calm in two days” entry is the top-left of $P^2$ = (row 1 of $P$) · (column 1 of $P$):

$(P^2)_{\text{Calm,Calm}} = (0.90)(0.90) + (0.10)(0.40) = 0.81 + 0.04 = 0.85.$

Walk that arithmetic: $0.90\times0.90$ is the path Calm→Calm→Calm; $0.10\times0.40$ is Calm→Volatile→Calm; add the two routes and you get an 85% chance of being Calm two days out, having started Calm. Filling in the rest the same way:

$P^2 = \begin{pmatrix} 0.85 & 0.15 \\ 0.60 & 0.40 \end{pmatrix}.$

Notice the one-step “stay Calm” probability was 0.90, but the two-step “be Calm” probability dropped to 0.85 — the chain is already starting to forget its lucky start and slide toward its long-run mix.

The stationary distribution

Run the chain long enough and a remarkable thing happens: the probability of being in each state settles down to a fixed mix that no longer depends on where you started. That long-run mix is the stationary distribution $\pi$, a row vector that satisfies

$\pi P = \pi, \qquad \sum_i \pi_i = 1.$

Read $\pi P = \pi$ as: feed the long-run mix through one more step and it reproduces itself — it’s the distribution that’s invariant under the dynamics. Equivalently, $\pi_i$ is the long-run fraction of time the chain spends in state $i$. For an irreducible (you can eventually reach any state from any state) and aperiodic (it doesn’t cycle in lockstep) chain, this $\pi$ exists, is unique, and is the limit the chain converges to from any starting point.

Worked example. Solve $\pi P = \pi$ for our Calm/Volatile chain. Write $\pi = (c, v)$ with $c + v = 1$. The equation $\pi P = \pi$ gives, reading the first column:

$0.90\,c + 0.40\,v = c \;\Longrightarrow\; 0.40\,v = 0.10\,c \;\Longrightarrow\; v = \tfrac{1}{4}c.$

Substitute into $c + v = 1$: $c + \tfrac{1}{4}c = 1 \Rightarrow \tfrac{5}{4}c = 1 \Rightarrow c = 0.8$, so $v = 0.2$. The stationary distribution is $\pi = (0.8,\, 0.2)$: in the long run the market is Calm 80% of days and Volatile 20% — regardless of whether it opened in a Calm or Volatile mood. That’s why the two-step “be Calm” probability (0.85) had already drifted from the one-step 0.90 down toward this 0.80 limit.

Now meet a three-state version. Below is a Bull / Bear / Flat market chain. Its transition rule is fixed: from Bull, the token stays Bull with probability 0.80, slips to Bear 0.05, and goes Flat 0.15. From Bear, it recovers to Bull 0.10, stays Bear 0.70, and goes Flat 0.20. From Flat, it breaks up to Bull 0.25, down to Bear 0.25, and stays Flat 0.50. Each of those three rows sums to 1, as required. Run the simulation, let the token hop hundreds of times, and watch the visit-frequency bars stop wobbling and lock onto a fixed height — that fixed height is $\pi$, the solution of $\pi P = \pi$, emerging empirically rather than from algebra.

Regimes, and where Markov breaks

Markov chains are everywhere in finance precisely because so many objects are naturally described by a current label plus transition odds:

• Regime-switching models — the markets-mood chain above: bull, bear, crisis, each with its own return and volatility profile, and a transition matrix governing how regimes flip. The famous Hamilton regime-switching model is a Markov chain at its core.
• Credit-rating migration matrices — agencies publish yearly tables of $P(\text{AAA} \to \text{AA})$, $P(\text{BBB} \to \text{default})$, and so on. The whole apparatus assumes a rating is Markov: this year’s rating, not the firm’s rating history, drives next year’s. Powers of that matrix give multi-year default probabilities.
• Weather-of-the-market models — “trending vs choppy,” “risk-on vs risk-off” — coarse state machines that traders reason about with exactly this language.

Pitfall 1: a price alone is usually NOT Markov in the way people hope. It is tempting to treat today’s price as the state and call it a day. But real returns show volatility clustering (big moves cluster — today’s turbulence predicts tomorrow’s) and momentum/mean-reversion (recent trend carries information). Both mean recent history matters beyond today’s price, so price-alone violates the Markov property. The fix is to enrich the state: add a volatility level, a trend indicator, or a regime label, so the augmented state is again a sufficient summary and the Markov property is restored. The art is choosing a state rich enough to be Markov but small enough to estimate.

Pitfall 2: estimated transition matrices are noisy and assume stationarity. You estimate $P$ by counting historical transitions, but rare transitions (AAA → default in one year) have tiny sample counts and huge error bars. Worse, the whole framework assumes the transition probabilities are constant over time — yet crises are exactly when migration rates spike and the calm-period matrix you fitted stops applying. A Markov model calibrated in placid markets can badly understate tail risk in a meltdown.

Putting it together

A Markov chain is a process that hops between a finite set of states with the memoryless property: the next state depends only on the current one, $P(X_{n+1}=j \mid \text{history}) = P(X_{n+1}=j \mid X_n=i)$. All its single-step behaviour packs into a transition (stochastic) matrix $P$, where $P_{ij}$ is the $i\to j$ probability and every row sums to 1. Multi-step forecasts are matrix powers: $(P^n)_{ij}$ is the chance of $i\to j$ in $n$ steps (Chapman–Kolmogorov). Run long enough and an irreducible, aperiodic chain forgets its start and settles into the stationary distribution $\pi$ solving $\pi P = \pi$ — the long-run time share in each state. Finance leans on this for regime switching and credit-rating migration, but beware: a bare price is rarely Markov (clustering and momentum demand an enriched state), and estimated matrices assume a stationarity that crises shred.

Next up — Poisson Processes — we leave the fixed-step grid behind. Markov chains tick on a clock, one hop per period; but defaults, trades, and jumps arrive at random times, irregularly scattered along a continuous timeline. The Poisson process is the memoryless model of when the next event strikes — the continuous-time cousin of everything you just learned.