Leakage & Purged Cross-Validation

There is a specific, intoxicating moment in every quant’s life: the backtest comes back with a Sharpe of 3.2, an equity curve so smooth it looks drawn with a ruler, and a drawdown you could step over. You feel like a genius. You are not a genius. You are, almost certainly, the victim of data leakage — and the cruel joke is that the prettier the backtest, the more likely it’s lying to you.

This lesson is about the single most expensive mistake in financial machine learning: letting your model peek at information it would never have had in real life, scoring that cheating run, and mistaking the cheat for skill. We’ll name every way the future sneaks into the past, prove why the cross-validation everyone learned in school is invalid on financial time series, and then build the fix the field actually uses — purging, the embargo, walk-forward, and combinatorial purged cross-validation. By the end you’ll trust your validation scores for the right reasons, and distrust the gorgeous ones for the right reasons too.

What leakage actually is

Analogy. Imagine grading a student by giving them the exam, then letting them study the answer key the night before, then marveling at their perfect score. The score is real in the trivial sense that they wrote the right answers — but it predicts nothing about how they’ll do on a test they haven’t seen. Data leakage is handing your model the answer key. The model “learns,” scores brilliantly on validation, and then face-plants the instant it faces genuinely unseen data.

Definition. Leakage is any flow of information from the test set (or the future, or the target itself) into the training process that would not be available at decision time in production. The result is an upward-biased performance estimate: the backtest glows, live trading does not. The defining symptom is a gap — large in-sample (backtest) performance, collapsing out-of-sample (live) performance.

The four horsemen of leakage, each in one line:

• (1) Overlapping labels. A training sample’s label is computed over a forward window that spills into the test period, so train and test share the same future outcome. One-liner: you label Monday’s sample by the return over Mon–Fri, then “test” on Wednesday — the label already contains Wednesday.
• (2) Look-ahead features. A feature uses data not yet known at the sample’s timestamp. One-liner: using a company’s restated Q1 earnings (published in May) as a feature dated to March, when only the originally reported, point-in-time number existed.
• (3) Preprocessing on the full dataset. Scaling, imputation, or feature selection fit on train and test together leaks test-set statistics into training. One-liner: computing the mean/standard deviation for normalization over the whole history — including next year’s data — before splitting.
• (4) Survivorship bias. Training only on assets that survived to today silently excludes the ones that went to zero. One-liner: backtesting a stock strategy on the current S&P 500 members, so every delisted bankruptcy has been quietly deleted from history.

The first horseman — overlapping labels — is the subtle one, the one that survives even careful researchers, and the one this lesson spends the most ammunition on. It connects straight back to label uniqueness from Lesson 2: when labels are built over forward windows, adjacent samples are not independent, and that non-independence is exactly what poisons cross-validation.

Why ordinary k-fold cross-validation fails

Quick refresher. Cross-validation (CV) is how you estimate how well a model generalizes: split the data into k folds, train on k−1 of them, test on the one held out, rotate so every fold gets a turn as the test set, and average the scores. Standard k-fold also shuffles the data first, because it assumes every sample is i.i.d. — independent and identically distributed, like marbles drawn from a bag. For photos of cats and dogs, that’s fine. For financial time series, it’s a catastrophe, for two independent reasons.

Reason 1 — shuffling destroys the arrow of time. Markets are ordered: the only honest question is “given the past, can you predict the future?” Shuffling scatters future samples into the training set, so the model gets to learn from Friday to predict Monday. In production you can’t do that — Friday hasn’t happened yet. A shuffled CV score answers a question you will never get to ask live.

Reason 2 — overlapping labels leak across the train/test boundary, even with NO shuffling. This is the killer, and it’s why even a careful, time-ordered split is still invalid. Suppose your label at time $t$ is the sign of the return over the next 5 days, $[t, t+5]$. Now put the train/test boundary at day 50. The training sample at day 48 has a label computed over days $[48, 53]$ — which reaches into the test period (days 50+). Train and test now share the same future outcomes. The model isn’t predicting the test set; it’s remembering it.

Worked example of the boundary leak. Label horizon $h = 5$ days. Test fold begins at day 50. Every training sample from day 45 through day 49 has a label window that touches day 50 or beyond:

Five training samples are silently entangled with the test fold. Their labels are the test fold’s outcomes. The CV score that results is, in the technical sense, garbage — a beautiful number measuring nothing.

Purging — sever the shared-outcome leak

Analogy. Back to the exam. If a few questions on the practice test are literally on the final, you don’t get to study them — you tear those pages out of the practice packet. Purging tears out exactly the training samples whose answers overlap the test.

Definition. Purging removes from the training set any sample whose label window overlaps in time with the test fold’s span. Concretely: if a training sample at time $t$ has a label evaluated over $[t, t+h]$, and that interval intersects the test fold’s time range on either side, you drop that sample from training. What survives is training data whose outcomes are genuinely disjoint from the test outcomes — no shared future, no leak.

This is the direct operational consequence of Lesson 2’s overlapping labels / uniqueness point: because forward-looking labels make neighbors non-independent, the neighbors of the test fold are contaminated and must go. Purging is uniqueness logic applied at the validation boundary.

Worked example. Test fold spans days 50–60. A training sample sits at day 48 with a 5-day label window, covering $[48, 53]$. Does it overlap days 50–60? Yes — day 53 lands inside the test span. So day 48 is purged from training. Walk it through the neighborhood:

Purging happens on both sides of the test block: samples just before it (whose forward windows reach in) and, symmetrically, samples whose windows reach back into it. The chart below lets you feel this directly — drag the label horizon slider up and watch the purge band (dashed) grow on both sides of the orange test fold, eating into the training cells.

The embargo — kill the leak that survives purging

Analogy. You tore out the practice questions that were literally on the final. But suppose the practice questions right after them gave away the topic, the phrasing, the trick. Even without sharing an answer, they leak. So you also skip a few questions immediately following — a quarantine band. That quarantine is the embargo.

Definition. Even after purging exact label overlaps, features often carry serial correlation — today’s feature value is correlated with tomorrow’s (volatility clusters, momentum persists, microstructure echoes). That correlation lets information bleed from the test fold into training samples just after it, beyond the purge band. The embargo drops a small additional band of training observations immediately after the test fold — typically around 1% of total observations — to absorb that bleed.

Why only after and not before? Because the leak the embargo targets flows forward in time: test-period information serially correlates into the samples that follow it. The purge already handled the backward-reaching label overlap; the embargo handles the forward-reaching serial-correlation tail.

Worked example — sizing an embargo. You have $N = 10{,}000$ observations and choose an embargo fraction of 1%. Embargo size $= 0.01 \times 10{,}000 = 100$ observations. So after the test fold ends — and after any purge band on that side — you delete the next 100 training samples. If your features have a longer memory (say a 20-day exponentially-weighted volatility feature whose autocorrelation is still meaningful at 20 days), and a sample represents one day, you’d push the embargo to at least those 20 cells regardless of the 1% rule of thumb — match the embargo to how long your features actually “remember.” In the chart above, the darker dashed cells right after the test fold are the embargo; drag its slider to widen or shrink that quarantine band.

Walk-forward and combinatorial purged CV

Purging and the embargo make a single split honest. The remaining question is how to arrange the splits to get a trustworthy — and ideally robust — estimate of performance. Two schemes dominate.

Walk-forward

Walk-forward is the most intuitive: always train on the past, test on the immediate future, then roll the window forward. Train on Jan–Jun, test Jul; train Jan–Jul, test Aug; and so on. It mirrors live trading exactly — at every test point, the model only ever saw the genuine past.

That realism is its great virtue and its great limitation. Because it tests on one chronological sweep, walk-forward produces exactly one backtest path — a single historical trajectory. It’s also data-hungry (early periods have little training data) and high-variance: your verdict can hinge on whether the one path you happened to walk through included the 2020 crash or skipped it. One path, one story, and markets are generous with misleading stories.

Combinatorial purged cross-validation (CPCV)

CPCV (López de Prado) extends purging to a combinatorial design that manufactures many paths. Split the timeline into $N$ contiguous groups. Instead of holding out one group at a time, hold out every combination of $k$ groups as the test set, train (with purging + embargo) on the rest, and reassemble the held-out groups into backtest paths. The number of distinct train/test splits is the binomial coefficient

$\binom{N}{k} = \frac{N!}{k!\,(N-k)!}$

and these recombine into a whole family of backtest paths rather than a single one.

Worked example. With $N = 6$ groups and $k = 2$ held out per split, you get $\binom{6}{2} = 15$ distinct splits. Those recombine into 5 full backtest paths (the general formula gives $\binom{N}{k} \cdot k / N = 15 \cdot 2/6 = 5$ paths). Five trajectories instead of one. Now you don’t get a single Sharpe — you get a distribution of Sharpes, with a mean and a spread. That spread is the gold: it tells you how much your result depends on which slice of history you happened to test on.

That distribution is exactly the raw material for estimating the probability of backtest overfitting (PBO) — roughly, how often the strategy that looked best in-sample fails to be best out-of-sample. Hold that thought; it’s the spine of the next lesson.

Putting it together

Leakage is the reason backtests lie, and it’s nearly always one of four things: overlapping labels reaching across the train/test boundary, look-ahead features using data that didn’t exist yet, preprocessing fit on the full dataset, or a survivorship-filtered universe. Ordinary k-fold is doubly disqualified on financial time series — shuffling lets the model train on the future, and overlapping labels leak across the boundary even when you don’t shuffle. The fixes are concrete: purge training samples whose label windows overlap the test fold (both sides), embargo a small band right after it to kill serial-correlation bleed, and arrange your splits as walk-forward (one realistic path) or CPCV (many paths, a full distribution). Trust the score that survived all of that — and stay suspicious of the gorgeous one that didn’t.

You now know how to make a validation score honest — but honesty isn’t the same as robustness. Even with flawless purging, embargoing, and CPCV, there’s a deeper trap: try enough strategies on the same history and one will look spectacular by pure chance, the way the millionth monkey eventually types a coherent Sharpe ratio. That’s selection bias under multiple testing, and it’s what makes the best-looking backtest the most likely to disappoint. The distribution of CPCV paths is precisely the tool that lets us measure it. Next up, Lesson 4 — Backtest Overfitting & the Deflated Sharpe — where we put a number on how much of your gorgeous backtest is skill and how much is the monkey.