Embeddings for Categorical & Alternative Data

So far this course has fed your networks numbers — returns, volumes, volatilities, the nicely-behaved continuous stuff a matrix multiply is happy to chew on. But real market data is riddled with labels: the ticker JPM, the sector “Financials”, the country “US”, the exchange “NYSE”, the day-of-week “Monday”, the regime tag “risk-off”. And then there’s the truly unstructured swamp — news headlines, 10-K filings, earnings-call transcripts.

A neural net does not know what “Financials” means. It multiplies and adds. It cannot multiply the string "JPM". So before any of your hard-won RNN / TCN / transformer machinery can touch a category, you have to turn that label into a vector. The lazy way is one-hot encoding. The good way — the way that lets a bank learn from what other banks did — is an embedding. This lesson is about doing it well, and about the leakage traps that embeddings make extraordinarily easy to fall into.

One-hot encoding is a waste

Analogy. One-hot encoding is like giving every employee in a 3,000-person company a badge that is a single light in a 3,000-light panel — yours is light #1,742, lit; everyone else’s is dark. The badge tells you who someone is and absolutely nothing about what they’re like. Two analysts on the same desk are as “far apart” as the CEO and the night janitor.

Definition. One-hot encoding maps a category $c$ from a vocabulary of cardinality $K$ to the indicator vector $e_c \in \{0,1\}^K$ — a $K$-long vector that is $1$ in position $c$ and $0$ everywhere else. All distinct one-hot vectors are mutually orthogonal: $e_i \cdot e_j = 0$ for $i \neq j$, and $\lVert e_i - e_j \rVert = \sqrt{2}$ for every pair. Constant distance, zero similarity structure.

Worked example. Take a toy universe of four tickers and encode them:

Distance between the two banks, $\lVert e_{\text{JPM}} - e_{\text{BAC}} \rVert = \sqrt{(1-0)^2 + (0-1)^2} = \sqrt{2} \approx 1.41$. Distance between a bank and the oil major, $\lVert e_{\text{JPM}} - e_{\text{XOM}} \rVert = \sqrt{2} \approx 1.41$. Identical. The encoding insists JPM and BAC are no more similar than JPM and XOM.

Now scale up. With $K = 3{,}000$ tickers, each input is a 3,000-long sparse vector. A first dense layer with $h = 64$ hidden units needs $K \times h = 3{,}000 \times 64 = 192{,}000$ weights just to read the ticker — and most of them see a single non-zero example at a time, so they update glacially.

When to use it

One-hot is fine — even preferable for its simplicity — when cardinality is low (say $K \le 10$: day-of-week, a handful of regimes) and you have plenty of data per category. At that scale the parameter cost is trivial and the lack of learned similarity barely matters. It’s only at high cardinality that it becomes a liability.

Entity embeddings

Analogy. This is word2vec for stocks. Word2vec learned that “king − man + woman ≈ queen” not because anyone told it, but because words that appear in similar contexts ended up with similar vectors. An entity embedding does the same with tickers: two banks that respond alike to rate moves get nudged toward each other in the embedding space, because sharing a representation lowers the loss. You never hand-label “these are both banks” — the geometry falls out of the training signal.

Definition. An embedding maps each category id $c \in \{1, \dots, K\}$ to a dense vector $\mathbf{v}_c \in \mathbb{R}^m$ via a lookup into an embedding matrix $E \in \mathbb{R}^{K \times m}$, where row $c$ is $\mathbf{v}_c$. Equivalently it’s a dense layer with no bias and no activation applied to the one-hot vector: $\mathbf{v}_c = E^{\top} e_c$. The matrix $E$ is trained jointly with the task. The embedding dimension $m$ (with $m \ll K$) is a hyperparameter you choose.

Dimension rule of thumb. Two popular heuristics:

$m \approx \min\!\left(50, \left\lceil \tfrac{K}{2} \right\rceil\right) \qquad \text{or} \qquad m \approx K^{0.25}.$

The first (the fast.ai rule) caps the size for huge vocabularies; the second grows very slowly. Both encode the same instinct: more categories deserve a richer space, but with sharply diminishing returns.

Worked example. With $K = 3{,}000$ tickers, the min-rule gives $m \approx \min(50, \lceil 3000/2 \rceil) = \min(50, 1500) = 50$. The fourth-root rule gives $m \approx 3000^{0.25} \approx 7.4 \to 8$. People usually land somewhere in between; say $m = 32$. Parameter count of the embedding: $K \times m = 3{,}000 \times 32 = 96{,}000$ — versus the $192{,}000$ the one-hot dense layer needed above for $h=64$, and crucially every row now accumulates shared structure rather than memorising one ticker in isolation.

When to use it

Reach for an entity embedding when a categorical is high-cardinality and you have a large pooled cross-section so many categories can pool statistical strength (more on this below). It shines exactly where one-hot drowns. But it is not free: you’ve added $K \times m$ parameters that can memorise, and a learned space that — like every learned thing here — can leak the future if you’re careless.

Embedding alternative & text data

Analogy. A pretrained text embedding is a translator you hired who already speaks the language fluently. You don’t teach them English from scratch on your 4,000 trading days of headlines — you hand them the sentence and they hand you back a compact “meaning vector”. Your downstream model only has to learn what that meaning implies for returns, not what English is.

Definition & taxonomy. “Embedding alternative data” really means producing a dense vector representation of unstructured or semi-structured inputs:

The point is uniform: by the time it reaches the network, everything — ticker, sector, news, sentiment — is a dense vector, and the downstream layers don’t care which column came from a lookup table and which came from a language model.

Worked example (out-of-vocabulary). Your training universe has $K = 3{,}000$ tickers. In live trading a freshly-IPO’d name appears that the embedding table has never seen. With a raw lookup, there is no row for it — the model crashes or silently maps to garbage. The fix is to reserve an extra row at index $0$ for an <UNK> (“unknown”) token: build $E \in \mathbb{R}^{(K+1) \times m}$, route every unseen id to row $0$, and train that row by randomly mapping some rare tickers to <UNK> during training so it learns a sensible “generic new name” vector. Same trick handles out-of-vocabulary words in text encoders.

When to use it

Pretrained text embeddings earn their keep when the signal lives in the language (tone of a filing, novelty of a headline) and your own dataset is far too small to learn English from scratch — which is essentially always in finance. Skip the heavy encoder when a one-line sentiment score already captures everything you’ll act on; a 1,536-dim vector feeding a model with 3,000 training days is a fast track to overfitting.

Leakage traps embeddings make easy

This is the section that matters. Embeddings are a learned representation, and anything learned can memorise the future. Every leakage lesson from Machine Learning for Alpha — preprocessing-on-full-data, survivorship, look-ahead — has an embedding-shaped version that is easier to commit and harder to spot.

The five traps.

Worked example (regime look-ahead). Suppose you tag each day with a regime label and embed it. You define “risk-off” by checking whether the next 20 days had a drawdown exceeding 10%:

$\text{regime}_t = \begin{cases} \text{risk-off} & \text{if } \min_{1 \le k \le 20}\, r_{t+k}^{\text{cum}} < -0.10 \\ \text{risk-on} & \text{otherwise.} \end{cases}$

That label at time $t$ is a function of returns from $t{+}1$ to $t{+}20$. Embed it, feed it to a model predicting $r_{t+1}$, and your “feature” already contains the answer — the embedding for “risk-off” will look spectacularly predictive, and on live data, where you can’t see the next 20 days, it evaporates. The fix: define the regime using only information available at or before $t$ (a trailing volatility threshold, a moving-average crossover), and assign it causally.

When the leak is subtle

The nastiest case is (e), memorisation on tiny data. With $K = 3{,}000$ tickers, $m = 32$, and only ~1,000 days of pooled history, the embedding can quietly learn an id→average-return lookup — a fancy target-encoding fit on the training labels. It validates poorly out-of-sample and you’ll blame “regime change”. The tell: in-sample fit improves sharply with $m$ while out-of-sample stalls or degrades. Shrink $m$, add regularisation/dropout on the embedding, or question whether you have the data to embed this category at all.

When embeddings earn their keep

Analogy. Embeddings are a team sport. Their whole advantage is letting categories pool strength — one bank learning from a thousand bank-days. If you’ve got two banks and a long weekend of data, there’s no team to pool; you’re just memorising. Embeddings reward a crowded stadium, not an empty one.

The decision, made precise. Favour an embedding when both hold: (1) cardinality is high ($K$ in the hundreds-to-thousands), so one-hot is wasteful and similarity structure is worth learning; and (2) you have a large pooled cross-section — many entities $\times$ many periods — so the embedding has enough examples per latent factor to learn structure rather than ids. Avoid it when data is tiny (it memorises), when cardinality is low (an ordinal or target encoding, or just a one-hot, is simpler), or when a tree will do.

When to use it — the trade-off

The embedding trade is expressive power and shared structure bought with parameters, opacity, and a fresh surface for leakage. You take the trade when high cardinality and a deep pooled cross-section make the shared structure real and learnable. You decline it when the data can’t fund the parameters, or when a tree gives you 95% of the benefit with 5% of the foot-guns.

Recap