Recurrent Models: RNN, LSTM & GRU

Last lesson we watched a multilayer perceptron stumble in finance: with a tiny effective sample size (a few hundred non-overlapping regimes hiding behind thousands of autocorrelated rows), it either memorized noise or learned nothing. But the MLP had a deeper sin — it treated each day as an unordered bag of features. It didn’t know that yesterday came before today.

Markets are sequences. Returns arrive in order, volatility clusters (your GARCH lesson), and yesterday’s shock leaks into today. The natural fix is a model that reads the series one step at a time, carrying a running memory — a summary of everything it has seen so far. That is exactly what a recurrent neural network does. Whether it helps more than it hurts is the question we’ll keep poking at.

The recurrent cell and unrolling

A recurrent cell keeps a hidden state $h_t$ — its mental summary so far — and updates it as each new input $x_t$ arrives:

$h_t = \tanh\left(W x_t + U h_{t-1} + b\right)$

Read it left to right: the new input $x_t$ gets transformed by $W$, the old memory $h_{t-1}$ gets transformed by $U$, you add a bias $b$, and squash through $\tanh$ to keep values in $[-1, 1]$. An output (say, a return prediction) is read off $h_t$ via another linear layer. The matrices $W$, $U$, $b$ are the same at every step — they are the cell, and the cell is the model.

Analogy. You’re reading a sentence. You don’t re-read the whole paragraph at every word — you keep a running gist in your head and update it with each new word. $h_{t-1}$ is your gist before the current word, $x_t$ is the word, $h_t$ is your updated gist. “Unrolling” is just drawing that one reader as a chain of copies, one per word — but it’s still one reader, not a committee of $T$ readers.

Worked example

Take a scalar toy cell: $W = 0.5$, $U = 0.8$, $b = 0$, and a sequence of returns $x_1 = 1.0$, $x_2 = -0.5$. Start $h_0 = 0$.

• Step 1: $h_1 = \tanh(0.5 \cdot 1.0 + 0.8 \cdot 0) = \tanh(0.5) \approx 0.462$.
• Step 2: $h_2 = \tanh(0.5 \cdot (-0.5) + 0.8 \cdot 0.462) = \tanh(-0.25 + 0.370) = \tanh(0.120) \approx 0.119$.

Notice $h_2$ depends on $x_1$ only through $h_1$ — the memory is the channel through which the past reaches the present. Break that channel and history disappears.

When to use it

Reach for recurrence when order genuinely carries information and the same local update rule should apply at every step: tick-by-tick order-flow, intraday bars, a rolling window of returns feeding a next-step forecast. If your features are an unordered cross-section (today’s value, momentum, quality scores for 500 stocks), recurrence buys you nothing — that’s a job for the MLP or, more likely, gradient-boosted trees.

The vanishing (and exploding) gradient

To train, we backpropagate the loss through the unrolled chain — backpropagation through time (BPTT). The chain rule says the gradient of a late loss with respect to an early hidden state is a product of one Jacobian per step:

$\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} \frac{\partial h_k}{\partial h_{k-1}}$

If each factor has magnitude roughly $\rho$, the whole product scales like $\rho^{\,T-t}$. Products of many numbers are unforgiving: a little under 1 and they collapse toward zero (vanishing); a little over 1 and they detonate (exploding).

Analogy. A whispered rumor passed down a line of people. If each person preserves 60% of the message, after 20 people almost nothing of the original survives — that’s vanishing. If each person exaggerates it 1.2× , after 20 people you’ve got a screaming conspiracy theory — that’s exploding.

Worked example

Suppose the typical per-step factor is $\rho = 0.6$. Over $T = 20$ steps the gradient to the first input scales by

$0.6^{20} = e^{20 \ln 0.6} = e^{20 \cdot (-0.511)} = e^{-10.2} \approx 3.6 \times 10^{-5}.$

The distant past contributes essentially nothing to the weight update — the model is structurally blind to dependencies older than a handful of steps. Now flip it: $\rho = 1.2$ gives $1.2^{20} \approx 38.3$, and over 60 steps $1.2^{60} \approx 5.6 \times 10^{4}$ — a gradient that overflows and produces NaN losses.

The exploding case has a cheap patch — gradient clipping: if the gradient’s norm exceeds a threshold $\tau$, rescale it to $\tau$ before the update. That caps the blow-up. Vanishing has no such patch; you can’t rescale a near-zero gradient back into something informative, because the signal is gone, not just the magnitude. That asymmetry is why the architecture itself had to change.

Look back at the hero island: toggle Vanilla RNN and drag the length slider out. Watch the shading of the first input’s contribution fade to near-black within a dozen steps — that’s $\rho^{T}$ rendered as opacity.

When to use it

A vanilla RNN is fine when the genuine dependency is short (a few steps) and you want a cheap, low-parameter sequence model — sometimes a virtue on tiny finance samples. The moment you suspect dependence spanning dozens of steps (a slow regime shift, a multi-day vol build-up), the vanilla RNN is the wrong tool, and you graduate to a gated cell.

LSTM: the cell-state highway

The Long Short-Term Memory cell keeps two states: the usual hidden state $h_t$ and a separate cell state $c_t$ — a protected conveyor belt for long-term information. Three sigmoid gates (each outputs values in $[0,1]$, i.e. “how much to let through”) regulate it:

• Forget gate $f_t$ — how much of the old cell state to keep.
• Input gate $i_t$ — how much of the new candidate to write.
• Output gate $o_t$ — how much of the cell state to expose as $h_t$.

The candidate is $\tilde{c}_t = \tanh(W_c x_t + U_c h_{t-1} + b_c)$, and the cell state updates additively:

$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t, \qquad h_t = o_t \odot \tanh(c_t)$

($\odot$ is elementwise multiplication.) That addition is the whole trick. The path from $c_{t-1}$ to $c_t$ is roughly multiply by $f_t$, and if the cell learns $f_t \approx 1$, the gradient along the belt is multiplied by ~1 each step instead of ~$\rho < 1$. The information highway stays open.

Analogy. The vanilla RNN forces every memory through a lossy tanh mixer at every step — like photocopying a photocopy. The LSTM adds a sealed envelope (the cell state) you can choose to carry forward untouched, only opening it (output gate) or editing it (input gate) when you decide to.

Worked example

Compare the long-range survival of a vanilla factor versus an LSTM forget gate. Suppose the cell learns $f_t = 0.97$. Over 20 steps the cell-state contribution scales by

$0.97^{20} = e^{20 \ln 0.97} = e^{20 \cdot (-0.03046)} = e^{-0.609} \approx 0.54.$

Roughly half the signal survives 20 steps. The vanilla RNN’s $0.6^{20} \approx 3.6\times10^{-5}$ has already vanished by six orders of magnitude. Push to 60 steps: $0.97^{60} \approx 0.16$ still meaningful, versus $0.6^{60} \approx 4.7\times10^{-14}$ — dead.

The forget gate isn’t fixed at 0.97 — it’s learned per step, so the cell can hold a memory open across a regime and slam it shut when the regime breaks. That adaptivity is what raw $\rho^T$ can never do.

When to use it

Use an LSTM when you have genuine long-range dependence and enough data to feed the extra parameters — high-frequency or intraday series where samples are plentiful, or sequences where holding a state open for dozens of steps clearly matters. On a few hundred non-overlapping daily regimes, an LSTM is usually over-armed; regularize aggressively or reach for the leaner cousin below.

GRU: the streamlined cousin

The GRU is the LSTM after a diet. It keeps a single state vector (no separate cell state) and uses just two gates:

• Update gate $z_t$ — how much of the old state to keep versus overwrite (it does the forget and input job at once).
• Reset gate $r_t$ — how much past state to let into the new candidate.

$\tilde{h}_t = \tanh\!\big(W x_t + U (r_t \odot h_{t-1}) + b\big), \qquad h_t = (1 - z_t)\odot h_{t-1} + z_t \odot \tilde{h}_t$

That convex blend $h_t = (1-z_t)h_{t-1} + z_t\tilde{h}_t$ is the GRU’s highway: when $z_t \approx 0$ the state is carried almost untouched (long memory), when $z_t \approx 1$ it’s fully rewritten (react to a shock). Same additive-path trick as the LSTM, fewer moving parts.

Analogy. If the LSTM is a car with separate accelerator, brake, and a sealed glovebox, the GRU is an electric car with one regenerative pedal that does both accelerate and brake. Fewer controls, usually just as good a drive.

Worked example

Parameter count is the headline. For hidden size $H$ and input size $D$, the recurrent blocks need roughly:

With $D = 10$ inputs and $H = 32$ hidden units, one block is $32(10+32)+32 = 1376$. So GRU $\approx 4128$ and LSTM $\approx 5504$ recurrent parameters — the GRU saves ~25%. On a dataset with a few hundred effective observations, every parameter you don’t have is a parameter you can’t overfit. That’s why GRUs frequently match or beat LSTMs on small, noisy financial series despite being “less powerful.”

When to use it

Default to the GRU over the LSTM when data is scarce and you want gating without the parameter bloat — which describes most daily-frequency finance problems. Prefer the LSTM only when you have abundant data and empirically find its extra gate earns its keep (sometimes true on very long, information-rich sequences like full limit-order-book streams).

Do RNNs actually help in finance?

Here’s the sobering part. Everything above is elegant — and on most finance problems, an RNN still loses to a boosted tree. Why?

• Tiny effective sample (callback to lesson 1). Recall: thousands of autocorrelated daily rows collapse to a few hundred non-overlapping, independent regimes. Recurrent nets have more parameters than an MLP and the same starved data — overfitting risk goes up, not down.
• Sequential, un-parallel training. BPTT must process steps in order; you can’t parallelize across the time axis the way a Transformer or a tree-split scan can. Training is slow, which means fewer experiments, which means more overfitting via researcher degrees of freedom (your deflated-Sharpe instincts should be twitching).
• Trees and TCNs eat your lunch. Gradient-boosted trees devour engineered lag features and are wildly sample-efficient. Temporal Convolutional Networks (dilated causal convolutions) get long receptive fields with full parallelism and often better stability than RNNs.

Where do recurrent nets genuinely earn their keep? Large, structured, higher-frequency data with rich sequential micro-structure: limit-order-book (LOB) sequences, tick streams, intraday event sequences — places where samples are abundant and the order of micro-events truly carries signal. There, the effective sample problem eases and the sequence inductive bias pays off.

When to use it

Use a recurrent model in finance when (1) the data is high-frequency or otherwise large in effective sample, (2) sequential micro-structure plausibly carries signal, and (3) you’ve already confirmed a simpler baseline (regularized linear, GBM, TCN) doesn’t dominate. Otherwise, the honest default is: start with trees, and make the RNN prove it adds out-of-sample value. Regularize hard either way — dropout, weight decay, tiny hidden sizes, short windows.

Recap