Attention & Transformers for Markets

By now you’ve met two ways to make a network remember the past. The RNN/LSTM/GRU threads a single hidden state through time like a game of telephone — every step whispers to the next, and gradients fade out over the chain (lesson 2’s vanishing-gradient problem). The Temporal Convolutional Network instead stacks dilated, causal convolutions to widen a fixed receptive field (lesson 3) — fast and parallel, but the horizon is whatever you hard-wired into the dilation schedule.

Attention is the third answer, and it’s almost insultingly direct: don’t thread memory, and don’t widen a window — just let every time step look directly at every other time step and decide for itself who matters. No chain to decay through. No fixed window to outgrow. Step 200 can read step 1 in a single hop. That’s the whole trick, and the Transformer is what you get when you build an entire architecture out of nothing but that trick.

Let’s earn it.

Self-attention: who looks at whom

The analogy. Picture a trading desk meeting. You (the query) ask a question: “what’s relevant to where price goes next?” Every analyst in the room has a name tag (a key) describing what they know, and a briefing (a value) they’ll hand over. You glance at every name tag, score how well each matches your question, turn those scores into attention percentages, and walk out with a blend of briefings weighted by relevance. Self-attention is that meeting, run for every time step at once, with every step playing both analyst and asker.

Precise definition. Each input token (say, the feature vector at time $t$) is linearly projected into three vectors: a query $q_t$, a key $k_t$, and a value $v_t$, via learned matrices $W_Q, W_K, W_V$. Stack them into matrices $Q, K, V$. Scaled dot-product attention is:

$\text{Attention}(Q,K,V)=\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$

The matrix $QK^\top$ holds every query-key dot product (the raw relevance scores). Dividing by $\sqrt{d_k}$ — where $d_k$ is the key dimension — rescales those scores. The softmax turns each row into a probability distribution that sums to 1, and multiplying by $V$ produces the weighted average of value vectors.

Why $\sqrt{d_k}$? If $q$ and $k$ are roughly independent unit-variance entries, their dot product $q\cdot k=\sum_{i=1}^{d_k} q_i k_i$ has variance about $d_k$ — so it grows with dimension. Large scores push the softmax into saturation: one weight goes to ~1, the rest to ~0, and the gradient through softmax nearly vanishes. Dividing by $\sqrt{d_k}$ pulls the variance back to roughly 1, keeping the softmax in its responsive, well-conditioned regime.

Worked example — a tiny 3-key softmax. One query attends to three keys with $d_k = 4$, so $\sqrt{d_k}=2$. Suppose the raw dot products are $q\cdot k_1 = 8$, $q\cdot k_2 = 4$, $q\cdot k_3 = 2$.

Scale each by $\sqrt{d_k}=2$: scaled scores are $4,\ 2,\ 1$.

Exponentiate: $e^{4}=54.60$, $e^{2}=7.389$, $e^{1}=2.718$. Sum $=64.71$.

Normalize: $w_1 = 54.60/64.71 = 0.844$, $w_2 = 7.389/64.71 = 0.114$, $w_3 = 2.718/64.71 = 0.042$ (sums to 1.000).

Now blend the value vectors. Say $v_1=[1,0]$, $v_2=[0,1]$, $v_3=[1,1]$. The output is

$0.844\,[1,0] + 0.114\,[0,1] + 0.042\,[1,1] = [0.886,\ 0.156].$

Notice the output leans heavily on $v_1$ (the best-matching key) but isn’t a hard pick — it’s a soft, differentiable blend. Now compare what happens without scaling: the unscaled scores $8,4,2$ give $e^8=2981$, $e^4=54.6$, $e^2=7.39$, weights $0.980, 0.018, 0.002$ — the softmax has nearly collapsed onto key 1, throwing away most of the relevance structure and flattening the gradient. That’s exactly the saturation $\sqrt{d_k}$ is there to prevent.

For markets we add a causal mask: an output at time $t$ may only attend to keys at times $\le t$ (you can’t peek at the future). In the score matrix that zeroes out the upper triangle before the softmax, so future positions get weight 0.

When to use it

Reach for self-attention when relevance is content-dependent and irregular — when which past bar matters depends on what happened, not on a fixed lag. An earnings surprise, a regime break, or a specific level being retested are things attention can route to directly. If the dependency is a smooth, fixed-lag decay (yesterday matters most, then the day before, monotonically), a cheap exponential filter or a small AR term may match it with a fraction of the parameters — don’t summon a Transformer to relearn a moving average.

Why attention beats recurrence for long range

The analogy. The RNN is a message passed hand-to-hand down a line of 200 people — by the end it’s “purple monkey dishwasher,” and each handoff multiplies in another Jacobian factor, so the gradient either vanishes or explodes (exactly the failure from lesson 2). Attention is a group chat: position 200 @-mentions position 1 directly, and the message arrives intact in one hop.

Precise statement. Define maximum path length as the longest distance, in computational steps, that information must travel between any two input positions for them to interact. For a recurrent net, two positions $i$ and $j$ interact only through the chain of hidden states between them, so the path length is $O(|i-j|)$, up to $O(T)$. For self-attention, every position attends to every other in a single layer, so the path length is $O(1)$ — independent of $T$. Short paths mean gradients don’t have to survive a long product of Jacobians, so long-range learning doesn’t decay.

The bill arrives as compute and memory. The score matrix $QK^\top$ is $T\times T$, so self-attention costs $O(T^2 \cdot d)$ time and $O(T^2)$ memory. Double the sequence length and the attention cost quadruples.

Worked example — counting the cost. Take $T = 500$ bars and model width $d = 64$.

• RNN sequential path between the first and last bar: $\approx 500$ hops; the operation count to process the sequence is $O(T \cdot d^2) = 500 \times 64^2 = 2{,}048{,}000$, but it’s sequential — step $t+1$ waits for step $t$.
• Self-attention path between any two bars: $1$ hop; the dominant cost is $O(T^2 \cdot d) = 500^2 \times 64 = 16{,}000{,}000$ — about 8× the RNN’s arithmetic here, but fully parallel across positions (one big matmul, no waiting).

So attention trades more (parallelizable) arithmetic for a vastly shorter gradient path and no sequential bottleneck. On modern hardware the parallelism usually wins on wall-clock, right up until $T$ gets large enough that the $T^2$ memory term bites.

When to use it

Prefer attention over recurrence when the dependency is genuinely long-range and content-routed, and your sequences are short enough that $T^2$ is affordable. If $T$ is small (a handful of lags) and the signal is local, an LSTM or a TCN is cheaper and harder to overfit. If $T$ is enormous and the dependency is mostly local with rare long jumps, a TCN’s logarithmic-depth receptive field is often the sweet spot.

Multi-head attention & positional encoding

The analogy for heads. One analyst can only hold one thesis at a time. A desk hires several — a momentum specialist, an events specialist, a liquidity specialist — and each reads the same tape through a different lens, then you pool their notes. Multi-head attention runs $h$ attention computations in parallel, each with its own lower-dimensional $W_Q, W_K, W_V$ projections, and concatenates the results through an output projection. One head can learn to track recent drift, another to spike on a specific past event, another to watch a level.

Precise definition. With model dimension $d_{\text{model}}$ split across $h$ heads, each head works in dimension $d_k = d_{\text{model}}/h$:

$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V), \qquad \text{MultiHead} = \text{Concat}(\text{head}_1,\dots,\text{head}_h)\,W^O.$

Splitting the budget keeps total cost roughly the same as single-head while letting heads specialize.

Positional encoding. Because attention is permutation-invariant, we add a position-dependent vector $p_t$ to each token’s embedding $x_t$ so identical features at different times are distinguishable. Sinusoidal encodings use

$PE_{(t,2i)} = \sin\!\left(\frac{t}{10000^{2i/d}}\right), \qquad PE_{(t,2i+1)} = \cos\!\left(\frac{t}{10000^{2i/d}}\right),$

giving each position a unique multi-frequency fingerprint; learned positional embeddings are a trained lookup table instead. Either way, the model now knows when each token sat.

Worked mini-example — why order matters for returns. Consider two day-return sequences built from the same multiset of moves $\{+5\%, -5\%\}$:

• Sequence A: $+5\%$ then $-5\%$. Cumulative wealth: $1.05 \times 0.95 = 0.9975$ (down 0.25%).
• Sequence B: $-5\%$ then $+5\%$. Cumulative wealth: $0.95 \times 1.05 = 0.9975$ (also down 0.25%).

Compounding happens to commute here, so the terminal wealth is identical — but the path is not, and path is what risk lives on. Sequence A drew down to 1.05 then fell; Sequence B drew down to 0.95 first. A momentum signal, a trailing stop, a margin call, or a volatility estimate all read these two as different worlds. A model that can’t tell A from B (no positional info) literally cannot represent “the drop came first.” Order is not decoration; it’s signal.

When to use it

Use multiple heads whenever you suspect several distinct relational patterns coexist (trend, mean-reversion, event-response) — it’s nearly free given a fixed $d_{\text{model}}$. Always include positional encoding for time series; the only time you’d drop it is for genuinely orderless set inputs (e.g. attending across the cross-section of assets at a single timestamp, where there is no intrinsic order to encode).

Transformers on the limit order book

The analogy. Forecasting next year’s return from a few yearly observations is like predicting a coin’s lifetime average from three flips — mostly noise. Nowcasting the next mid-price tick from the live order book is more like reading a crowded auction room in real time: the bids and offers are right there, stacked and quantified, and the next move is mechanically related to that pressure. That’s the high-SNR, data-rich regime lesson 1 said DL was built for.

The limit order book (LOB) as input. At each instant the book lists the best $N$ price levels on each side with their resting sizes. A standard DeepLOB-style input is the top $N=10$ levels: for each level, the bid price, bid size, ask price, ask size — $10 \times 4 = 40$ features per snapshot — stacked over a window of, say, the last 100 snapshots. This is a structured, normalized, high-frequency tensor, not a handful of noisy daily bars.

The labels. Define the mid-price $m_t = (p^{\text{ask}}_t + p^{\text{bid}}_t)/2$. To denoise the tick-level wiggle, compare smoothed means before and after: let $m_-$ be the average mid over the previous $k$ ticks and $m_+$ the average over the next $k$. The label is the sign of the smoothed change against a threshold $\alpha$:

$\ell_t = \begin{cases} \text{up} & (m_+ - m_-)/m_- > +\alpha \\ \text{flat} & |m_+ - m_-|/m_- \le \alpha \\ \text{down} & (m_+ - m_-)/m_- < -\alpha \end{cases}$

A three-class up/flat/down classification over a horizon of a few ticks.

Worked example — labeling one event. Best bid $= 100.00$, best ask $= 100.02$, so $m_t = 100.01$. Over the previous $k=5$ ticks the mid averaged $m_- = 100.00$; over the next 5 it averages $m_+ = 100.05$. The smoothed return is $(100.05 - 100.00)/100.00 = 0.0005 = 5$ basis points. With a threshold $\alpha = 2$ bp, $5 > 2$, so this snapshot is labeled up. Had $m_+ = 100.01$ (a $1$ bp move), $1 < 2$, it would be flat — the threshold absorbs micro-noise so the model isn’t trained to chase rounding.

The DeepLOB lineage. The influential recipe (Zhang, Zohren & Roberts, 2019) is a stack: convolutions over the price/volume levels first compress the 40-feature snapshot into local microstructure features (treating adjacent levels like adjacent pixels), an inception module captures multi-scale patterns, and then a temporal model — originally an LSTM, in later variants an attention/Transformer block — integrates over the time window before a softmax over up/flat/down. Attention’s role: route the prediction to the moments in the recent book that actually mattered (a big sweep, a sudden imbalance) rather than smearing across the window.

When to use it

Deploy a Transformer-on-LOB when you have genuine high-frequency book data, a short horizon, and millions of labeled snapshots — and when your backtest models latency, fees, and fill realism honestly. Do not transplant the architecture to low-frequency return forecasting and expect the magic to follow; there the data is too scarce and too noisy for the model’s capacity, which is exactly the next section’s problem.

The data-hunger problem

The analogy. A convolution arrives with an opinion (“nearby things relate”), and recurrence arrives with another (“time is a chain”) — these inductive biases are guardrails that constrain what the model can fit. A Transformer arrives with almost no opinion: any position can relate to any position, learned freely. Maximum flexibility is a superpower when data is abundant and a curse when it’s scarce. On a few hundred noisy yearly returns, a bias-free, parameter-heavy model is a sponge that soaks up noise and calls it signal.

Precise statement. Generalization error decomposes loosely as bias plus variance. Strong inductive bias lowers variance at the cost of some bias; the Transformer’s weak bias and large parameter count $P$ maximize variance, and variance is governed by the ratio of $P$ to the effective number of independent samples $N$. Financial return series have a tiny effective $N$ (overlapping, autocorrelated, regime-shifting), so $P/N$ is enormous and the model overfits before it generalizes.

Worked example — the parameter-to-sample ratio. Suppose a modest Transformer block has $P \approx 200{,}000$ parameters and you train on 20 years of daily equity returns: $20 \times 252 \approx 5{,}040$ observations — and because returns are serially correlated and regimes overlap, the effective independent sample count might be closer to a few hundred. Take $N_{\text{eff}} \approx 250$. Then $P/N_{\text{eff}} \approx 200{,}000/250 = 800$ parameters per effective sample. A healthy supervised setup wants that ratio well below 1. At 800-to-1 the model can fit the training set perfectly and learn essentially nothing that survives out of sample.

Remedies.

When to use it

Use a Transformer for return prediction only when you can honestly raise the effective sample count — pooling a broad cross-section, leaning on relevant pretraining, or working in a high-frequency regime with millions of snapshots. Otherwise, the smaller-bias-stronger model (or even a regularized linear model) usually wins out of sample, and you’ve spent far fewer trials getting there. The sober bottom line — which lesson 6 will hammer — is that architecture rarely rescues a low-SNR, data-starved problem; matching model capacity to genuine information content is the whole game.

Recap