Distribution Shift & Concept Drift

The last three lessons starred a villain: an adversary deliberately feeding your model poisoned data and crafted inputs. Frightening, sure — but it requires someone to actually be out to get you. This lesson is the part that gets you even when nobody is attacking at all.

Here is the uncomfortable truth: the world your model was trained on simply stops existing. The data-generating process drifts. Regimes change. Other quants find your signal and trade it flat. Your own trading nudges the very prices you’re predicting. No malice required — the market is a moving target, and your model is a photograph of where it used to be.

That’s the difference worth holding onto: an adversarial attack is targeted and intentional (Lessons 1–3); distribution shift is diffuse and natural — entropy, not enemies. Both leave the same crater in your PnL. This lesson is the statistics of the natural half: how to name the kind of shift you’re suffering, why a fresh strategy decays on contact with reality, and how to catch the rot before your equity curve announces it for you.

The taxonomy of shift — what, exactly, is changing?

Analogy. Imagine a chef who trained for years cooking at sea level. Three things can ruin the next dish. The ingredients could change — fattier butter, saltier stock (the inputs drift). The menu mix could change — suddenly 80% of orders are dessert instead of 20% (the base rates shift). Or the physics of the kitchen could change — they’re now cooking at 4,000 metres, where water boils at 87°C and every old recipe is wrong even with identical ingredients (the rule connecting ingredients to outcome flipped). All three are “things went wrong,” but the diagnosis — and the fix — differ completely.

Definition. Any supervised model lives on a joint distribution of inputs $x$ and outcomes $y$, which factorises two ways:

$P(x, y) = P(y \mid x)\, P(x) = P(x \mid y)\, P(y).$

Distribution shift means the test-time joint $P_{\text{test}}(x, y)$ differs from the training joint $P_{\text{train}}(x, y)$. The three canonical kinds isolate which factor moved:

• Covariate shift — $P(x)$ changes, $P(y \mid x)$ stays fixed. Your inputs drift into regions of feature space you never (or rarely) trained on. The rule is still valid; you just keep getting asked questions far from where you learned. Detectable by watching the inputs alone, because that’s the thing that moved.
• Label shift (a.k.a. prior shift) — $P(y)$ changes, while the class-conditional $P(x \mid y)$ stays fixed. The base rates move: e.g. the fraction of up-days, or the unconditional default rate, shifts. A classifier calibrated on a 55% up-day base rate is miscalibrated when up-days become 48%, even if every “given an up-day, what do features look like” stays identical.
• Concept drift — $P(y \mid x)$ changes. The relationship itself moves: the same features now predict a different (sometimes opposite) outcome. This is the deadliest for trading, because the inputs can look perfectly familiar while the model is now systematically wrong. No input-only monitor will ever see it coming.

Worked example — the momentum feature that betrays you. Let $x$ be a normalised 12-month momentum score and $y$ next month’s return. In a trending regime (2017, say), high momentum predicts more gains — the data says $E[y \mid x] = +0.6\,x$, a positive slope: winners keep winning. The model learns “buy high momentum.”

Now the regime turns choppy/mean-reverting (a whipsawing 2018-style tape). The distribution of momentum scores is unchanged — you still see scores from roughly the same range, so $P(x)$ is stable. But the relationship flips to $E[y \mid x] = -0.4\,x$: high momentum now predicts a pullback. A stock with momentum $x = +2.0$ that your model expected to earn $0.6 \times 2.0 = +1.2$ instead earns $-0.4 \times 2.0 = -0.8$. Same input, flipped $y \mid x$ — a $2.0$ swing in the wrong direction, and your feature monitor flashes all-green the entire time. That’s textbook concept drift, and it is why momentum strategies suffer brutal “crashes” precisely when the world looks normal.

When to use it

Use this taxonomy as a triage the moment a live model underperforms. First ask: did my inputs move into unfamiliar territory (covariate)? Then: did the base rate of the thing I predict change (label)? Only if both look stable but you’re still bleeding do you indict concept drift — and reach for outcome-based detection (Section 4) rather than input monitors that will keep lying to you.

Why a strategy decays the moment it goes live

Analogy. A strategy’s backtest is a dating profile: flattering, carefully selected photos, no mention of the snoring. Live trading is moving in together. Three disappointments follow. Some of the glow was just a good-angle selfie (selection/luck). Once you’re seen in public, everyone else copies the look until it’s no longer special (crowding). And your presence changes the room — you can’t observe the party as it was before you walked in (reflexivity).

Definition — the three decay forces.

1. Overfitting & selection. Your backtest is the maximum over many trials — parameters, features, universes, start dates. The maximum of noisy estimates is biased upward (the deflated-Sharpe lesson from Machine Learning for Alpha). Part of that 2.0 was never edge; it was the luckiest draw among the configurations you tried, and it evaporates out-of-sample because there was nothing there to begin with.
2. Alpha crowding. Markets are competitive. The instant a signal works and word gets out — a paper, a conference talk, a defecting colleague — other quants trade the same thing, push prices toward fair value faster, and arbitrage the edge away. This is exactly the crowding and capacity problem from Systematic & Statistical Arbitrage: a spread that mean-reverts in 10 days when three funds trade it reverts in 2 days when thirty do, and the per-trade alpha shrinks toward your costs.
3. Reflexivity. Acting on a signal changes the market it predicts. Your buy orders push the price up, eating the very mispricing you spotted (market impact), and at scale you become part of the data-generating process. This is the central headache of Deep RL for Execution: you are not a passive observer of a fixed environment — you are an agent in an environment that reacts to you, so the world your model learned from no longer exists once the model starts acting in it.

Worked example — the publication haircut. McLean and Pontiff’s “Does Academic Research Destroy Stock Return Predictability?” tracked dozens of published anomalies and found returns decay roughly 58% out-of-sample, with a large chunk attributed to post-publication decline (investors learning and trading the signal) on top of the statistical bias. Take the round number the literature popularised — about a 50% haircut. Start with an in-sample Sharpe of $2.0$. Knock off the publication/crowding half: $2.0 \times (1 - 0.5) = 1.0$. Then layer reflexivity and your own costs on top — say a further 20% bite from impact at the size you actually trade: $1.0 \times (1 - 0.2) = 0.8$. The honest expectation for live, at-size Sharpe is well under half the headline. Anyone quoting the backtest number as their forward expectation is reading the dating profile, not living with the snoring.

When to use it

Apply the haircut before you size the position, not after the drawdown. When deciding capital allocation, plug in a deflated, crowding-adjusted, impact-adjusted Sharpe — not the backtest headline. And weight the three forces by your situation: a published factor is mostly crowding risk; a large book is mostly reflexivity risk; an over-searched backtest is mostly selection risk.

Detecting covariate shift — watch the inputs

Analogy. Covariate-shift detection is a bouncer with a photo of the regulars. Each night they compare who walks in against the reference crowd. If tonight’s clientele looks statistically like the usual regulars, all good. If suddenly everyone’s a stranger from a different neighbourhood, the bouncer raises a flag — before anything has gone wrong inside, just on the grounds that the inputs no longer match the place the rules were written for.

Definition — Population Stability Index (PSI). Bin a feature into $k$ buckets. Let $a_i$ be the fraction of training (reference) observations in bin $i$ and $b_i$ the fraction of live (current) observations in the same bin. Then

$\text{PSI} = \sum_{i=1}^{k} (b_i - a_i)\, \ln\!\left(\frac{b_i}{a_i}\right).$

(It is the symmetrised relative-entropy cousin of KL divergence — every term is non-negative, so PSI only grows as the two distributions pull apart.) Industry rules of thumb: PSI < 0.1 = stable (no meaningful shift); 0.1–0.25 = moderate shift (investigate); > 0.25 = significant shift (the population has moved — retrain or recalibrate). A complementary tool is the two-sample Kolmogorov–Smirnov (KS) test, whose statistic is the maximum gap between the two cumulative distribution functions; a small p-value says “these samples didn’t come from the same distribution.”

Worked example — PSI on a 2-bin feature. Suppose a binary regime feature was a 50/50 split in training, so $a_1 = 0.5,\ a_2 = 0.5$. Live, it has drifted to 30/70, so $b_1 = 0.3,\ b_2 = 0.7$. Term by term:

• Bin 1: $(b_1 - a_1)\ln(b_1/a_1) = (0.3 - 0.5)\ln(0.3/0.5) = (-0.2)\ln(0.6) = (-0.2)(-0.5108) \approx 0.1022$.
• Bin 2: $(b_2 - a_2)\ln(b_2/a_2) = (0.7 - 0.5)\ln(0.7/0.5) = (0.2)\ln(1.4) = (0.2)(0.3365) \approx 0.0673$.

Sum: $\text{PSI} \approx 0.1022 + 0.0673 = 0.169$. That lands squarely in the 0.1–0.25 “moderate shift, investigate” band. Notice the structure: both terms are positive (the negative difference in bin 1 multiplies a negative log, giving a positive product), which is why PSI can’t cancel itself out — any movement, in either direction, adds to the score.

When to use it

Run PSI/KS monitors on every model input on a rolling schedule (daily or per-rebalance), and alarm on the important features first. They are your cheapest, earliest warning — but they only catch covariate shift. Treat them as necessary, never sufficient: passing every input monitor is fully compatible with the model being dangerously wrong because the relationship — not the inputs — has drifted.

Detecting concept drift — watch the outcome

Analogy. Input monitors check that the ingredients match the recipe. Concept-drift monitors taste the dish. You can have flawless, on-spec ingredients and still serve something inedible if the oven’s thermostat secretly broke — and the only way to know is to taste the output, not re-inspect the flour. So you watch the result: realised hit rate, realised PnL, prediction residuals.

Definition — outcome-based drift detection. Because concept drift may leave $P(x)$ untouched, you monitor the model’s realised performance over time and fire when it degrades faster than noise:

• Rolling Sharpe / rolling hit-rate — a moving-window estimate of the live edge; a sustained slide toward (or through) zero is the headline symptom.
• CUSUM (cumulative sum) — accumulate the running sum of deviations of the metric from its expected (good) level; the cumulative sum stays flat under no drift and ramps once a small, persistent deterioration sets in, so it catches gradual drift a single-period test would miss.
• Page–Hinkley test — a CUSUM variant tuned for change-point detection on a stream: track the cumulative deviation and the minimum it has reached, and alarm when the gap between current cumulative and that running minimum exceeds a threshold $\lambda$. Classic for detecting a mean change in a residual or error stream online.

The teaching point, made concrete by the timeline beside this: a good detector fires before the PnL fully collapses, while there’s still edge left to protect — but detector sensitivity is a genuine tradeoff. A low threshold fires early but trips on noise (false alarms that pull you out of a still-working strategy); a high threshold fires late, after most of the edge is already gone. There is no free lunch, only a dial between “early but jumpy” and “late but sure.”

Worked example — Page–Hinkley on a hit-rate. Say your signal’s hit rate should be 54% when healthy, so each correct call scores $+1$ and each miss $-1$, with an expected per-trade mean of $0.54 - 0.46 = +0.08$. After concept drift quietly flips part of the edge, the true hit rate drops to 48%, an expected mean of $0.48 - 0.52 = -0.04$. The Page–Hinkley running statistic accumulates each trade’s shortfall below the expected $+0.08$ and tracks its running minimum; once the gap exceeds your $\lambda$, it alarms. The point: a single bad week (a few misses) won’t trip it — that’s just noise around $+0.08$ — but a persistent drop from $+0.08$ to $-0.04$ makes the cumulative shortfall grow without bound, so the alarm fires after a characteristic number of trades, not on any single unlucky one. Lower $\lambda$ shortens that number (earlier, jumpier); higher $\lambda$ lengthens it (later, surer).

When to use it

Run outcome monitors continuously, alongside (never instead of) input monitors. Pick the tool to the drift’s shape: a single-window rolling Sharpe for fast, obvious breaks; CUSUM / Page–Hinkley for slow, insidious degradation that a window would smear into noise. Set the threshold by your asymmetry: if a false alarm (needlessly pulling a working strategy) is cheaper than a missed drift (riding a dead one into the ground), bias the threshold low — and vice versa.

Responding to drift — the menu of moves

Analogy. Responding to drift is like adjusting to a city you’ve moved to. You can keep a rolling memory — remember only the last few months and forget the old city (rolling-window retrain). You can update constantly — learn from every new street (online learning), at the risk of being fooled by one misleading detour (poisoning). You can keep multiple mental maps and switch by neighbourhood (regime-switching/ensembles). Or — radically — you can stop driving until you’ve got the new map straight (turn it off). Wisdom is matching the response to how fast the city is changing.

Definition — the response menu.

• Retrain / rolling windows. Refit on a trailing window so the model tracks the current regime. The window length is a bias–variance dial: a short window adapts fast to new regimes but is estimated on few, noisy points (high variance); a long window is stable and well-estimated but stale — slow to notice a regime has turned (high bias toward the past).
• Online learning. Update the model incrementally as data arrives — maximally adaptive, but it re-opens the data-poisoning attack surface from Lesson 3: a model that trusts every new point can be steered by an adversary feeding it crafted updates. Adaptivity and attackability are the same coin.
• Regime-switching models. Explicitly model a hidden regime state (e.g. a Markov-switching model) and use the parameters for the regime you’re currently in, so a flip in $P(y \mid x)$ is expected rather than catastrophic.
• Ensembling across regimes. Keep a stable of models — one per regime or training era — and blend or select among them by current conditions. Robust to any single model going stale; this is the natural bridge to Lesson 5’s robustness-through-diversity theme.
• Turn it off. The honest, underrated move: when drift is confirmed and you don’t yet understand it, stop trading the strategy. A flat PnL beats riding a dead edge into a drawdown while you “wait for it to come back.”

Worked example — 60-day vs 250-day retrain window. Suppose a coefficient you re-estimate each rebalance has a per-observation estimation noise such that the standard error of the mean scales as $\sigma/\sqrt{n}$ with $n$ the window length. A 250-day window gives $\sigma/\sqrt{250} = \sigma/15.8$ — a tight, stable estimate. A 60-day window gives $\sigma/\sqrt{60} = \sigma/7.7$, roughly $15.8/7.7 \approx 2.0\times$ noisier per estimate. So the short window pays double the estimation noise. What does it buy? Speed: if a regime turns at day 0, the 60-day window is “mostly new regime” within ~30 trading days, while the 250-day window still has ~190 stale days dragging the estimate toward the dead regime for months. So you’re trading a clean ~2× noise penalty for a roughly 4× faster adaptation — worth it precisely when the alpha decays fast, a bad deal when it’s slow and stable.

When to use it

Choose the response by speed of decay and cost of being wrong. Fast decay → short window or online learning (and accept the poisoning exposure with input validation from Lesson 3). Unknown regime → regime-switching or ensembles. Confirmed drift you don’t understand → turn it off and investigate; never let “it might come back” keep a diagnosed-dead strategy live. The goal isn’t to never drift — everything drifts — it’s to detect it before the PnL does and respond proportionally.

Recap

The adversary in the first three lessons was a person. The adversary here is entropy: the data-generating process drifts whether or not anyone means you harm. You learned to name the rot — covariate shift ($P(x)$ moves), label shift ($P(y)$ moves), and the deadliest, concept drift ($P(y \mid x)$ flips while the inputs look innocent). You learned why a fresh strategy decays on contact with reality — selection, crowding (Systematic & Statistical Arbitrage), and reflexivity (Deep RL for Execution) — and that roughly half the backtest edge is the honest forward expectation. You learned to catch it: PSI / KS on the inputs for covariate shift, and rolling Sharpe / CUSUM / Page–Hinkley on the outcomes for the concept drift the input monitors can’t see — the timeline made the early-vs-late detection tradeoff vivid. And you learned to respond: rolling windows (with their bias–variance dial), online learning (and its poisoning exposure), regime-switching, ensembles (next lesson), and the honest option of turning the strategy off.