The Leakage Minefield

Lessons 1 through 4 sold you on the upside: pretrain on a giant corpus, then adapt (lesson 1), forecast time series zero-shot (lesson 2), ride the scaling laws (lesson 3), and even mine an LLM for research and signals (lesson 4). Every one of those superpowers shares a dirty secret, and this lesson is where we drag it into the light.

The secret: a foundation model has already read an enormous slice of the past — possibly including the very future your backtest is trying to “predict.” Classical models only know what you feed them. A pretrained model walks in already knowing things, and you usually have no idea what. That turns a normal modeling chore into a minefield, where one wrong step blows up your entire research conclusion without making a sound.

Let’s learn where the mines are buried.

Why foundation models make leakage worse

The analogy. Imagine hiring an analyst to predict today’s market, then discovering they spent last night reading tomorrow’s newspaper. Ask them to “forecast” and they’ll dazzle you — but they’re reciting, not reasoning. A foundation model pretrained on data through 2024 and pointed at a 2020 backtest is exactly that analyst. The brilliance you measure is memory wearing a forecasting costume.

Three precise terms. Leakage in foundation models comes in three named flavors, and you need all three in your vocabulary:

• Lookahead bias (look-ahead bias). Using, at decision time $t$, any information that was not actually available until after $t$. The classic version is a coding bug (you indexed tomorrow’s price). The foundation-model version is baked into the weights: the model saw the future during pretraining, so lookahead arrives for free, invisibly.
• Pretraining-data contamination. Your evaluation data — or its outcomes — sat inside the corpus used to pretrain the model. The model isn’t generalizing to your test set; it’s partly recalling it.
• Memorization. Large models provably store verbatim chunks of their training data. So contamination isn’t just “similar examples were seen” — specific values, sequences, or events can be reproduced from memory.

Why it is worse than ever. With a from-scratch model you control the inputs exactly: leakage is a bug you can audit line by line. With a foundation model, the leak is upstream, opaque, and enormous. You typically cannot inspect the corpus, cannot enumerate what it memorized, and cannot un-see what it saw. The leak is in the box before you open it.

When to use it

Treat every pretrained model as guilty-until-proven-innocent of contamination whenever its claimed skill is being measured on the past. If you are only ever running it forward on genuinely unseen, post-deployment data, the worry shrinks. The moment you backtest — the moment you score it on history — assume the corpus may have touched that history and design around it. The rest of this lesson is the “designing around it” part.

Leak #1 — temporal contamination (the knowledge cutoff)

The analogy. Knowledge cutoff is the date past which the model has read nothing. Everything before the cutoff is open-book; everything after is closed-book. Backtesting before the cutoff is handing the model an exam on chapters it already memorized.

Precise definition. A model’s knowledge cutoff is the latest date represented in its pretraining corpus. Temporal contamination occurs when your evaluation window falls (wholly or partly) before the cutoff, so the outcomes you score were available to the model during pretraining. A “forecast” of a pre-cutoff period is therefore a memory test, not a forecast — the model is, at best, interpolating data it has already seen and, at worst, regurgitating it.

Worked example — a backtest that is secretly a memory test. Suppose a vendor ships a foundation model with a stated knowledge cutoff of 31 December 2024. You evaluate it on a backtest spanning 2015–2023 and report a beautiful out-of-sample Sharpe.

• Every single trading day in 2015–2023 is before the 2024 cutoff.
• So 100% of your “out-of-sample” window is inside the model’s pretraining horizon.
• The model could have memorized the 2015–2023 price path verbatim; nothing structurally prevents it.
• Your Sharpe measures recall fidelity, not predictive skill. It will not survive contact with truly future data.

Now contrast a clean design. Same model, same 2024 cutoff, but you evaluate only on 2025 onward — dates the model provably could not have seen. That window can have leaked nothing through the cutoff (it postdates the corpus entirely), so a good Sharpe there is at least eligible to be real.

The fix. Backtest only on data strictly after the knowledge cutoff. That single rule kills temporal contamination outright. It is brutal — it can leave you with a painfully short evaluation window — but a short honest window beats a long contaminated one every time.

When to use it

Apply the strictly-post-cutoff rule any time the model’s training data is dated and your evaluation touches history. If the vendor won’t disclose a cutoff, you cannot establish a clean window at all — treat the whole backtest as suspect. The only fully safe evaluation of a dated model is forward, on data created after you froze the model. Everything else is a negotiation with contamination.

Leak #2 — benchmark / test-set contamination

The analogy. It’s an open-secret cheating ring: the exam questions were published online for years, and the model studied from a corpus that scraped the whole internet. Of course it aces the exam — it has seen this exact exam. A leaderboard win under those conditions tells you the model memorized the test, not that it’s smart.

Precise definition. Benchmark contamination (test-set contamination) is the special case of pretraining contamination where the leaked material is a public evaluation dataset. The classic time-series benchmarks — the M-competition series (M3, M4, M5), and the common open forecasting repositories everyone trains and tests on — are widely mirrored, blogged, and packaged. That makes them prime candidates to be swept into a web-scale pretraining corpus. A model that has effectively memorized M4 will look superhuman on M4 and merely human everywhere else.

Worked example — the leaderboard mirage. Two models report results on a public benchmark:

• Model A: reported error (sMAPE) of 9.8 on the public benchmark; on a fresh, never-published dataset of the same kind, 14.5.
• Model B: reported error of 11.0 on the public benchmark; on the same fresh dataset, 11.3.

Read naively, Model A “wins” the leaderboard. But Model A’s error jumps 48% the instant you move to data that couldn’t be in the corpus, while Model B barely budges. Model A’s leaderboard edge is contamination; Model B is the one that actually generalizes. The leaderboard ranked them backwards.

How to probe for it. You rarely get to read the corpus, so you test behaviorally:

1. Held-out-by-construction data. Evaluate on a private dataset you generated or licensed after the model’s cutoff. A big error gap (public vs. private) is a contamination fingerprint.
2. Perturbation tests. Slightly alter the public series (shift, rescale, mask a chunk) and re-evaluate. A model that generalizes degrades gracefully; a model that memorized collapses when the surface form changes.
3. Completion probing. Give the model the front of a known public series and ask it to continue. If it reproduces the held-out tail with eerie precision, it has memorized that series.

When to use it

Run benchmark-contamination probes whenever a foundation-model claim leans on public-dataset performance — which is almost always, since that’s how these models are marketed. Skip it only when your entire evaluation already runs on private, post-cutoff data (in which case you’ve sidestepped the problem). When in doubt, the completion-probe and the public-vs-private gap are the two cheapest tells.

Leak #3 — point-in-time, revision & survivorship leaks

The analogy. Using the revised number is like grading yesterday’s weather forecast with today’s corrected satellite data and then claiming the forecaster was clairvoyant. The forecaster only had the messy, soon-to-be-revised data of the moment — and so did the trader.

Precise definitions. Two clocks govern every data point:

• Event time — when the thing happened (the quarter the earnings cover, the month the GDP measures).
• Availability time (knowable time) — when the value first became known to the public (the release date, often weeks after the event; revisions arrive later still).

A point-in-time (PIT) dataset stamps each value with its availability time and preserves the vintage — the exact value as first published, before any restatement. Using the latest revised value at a past decision date is a revision leak: you injected information that didn’t exist yet. Two cousins complete the family:

• Survivorship bias — building a universe from today’s surviving names (e.g. current index members), silently deleting the firms that went bankrupt or were delisted. Your backtest only ever “buys” companies you already know survived.
• Index-membership / constituent leak — using today’s index composition for a past date. A stock added to the index in 2022 was not a member in 2018; pretending it was leaks the future.

Worked example — the revision that flatters a strategy. A “buy on strong earnings” rule keys off EPS:

Train on the revised $1.45 and your rule “knew” the firm would beat before the beat was announced. The backtest lights up with trades that were impossible to place. Swap to the vintage $1.20 and many of those trades vanish — because in the real Q1, the signal was lukewarm. The revised-data backtest is fiction; the vintage backtest is what you could have actually traded.

Where foundation models add a new wrinkle. A pretrained model doesn’t just risk your PIT bug — it may have absorbed the revised values during pretraining and have no concept of vintage at all. So even a perfectly PIT-clean feature pipeline can be undermined by a model that internally “knows” the restated figure. You now have to respect three boundaries at once: purge label-overlap, embargo after the test block, and restrict evaluation to after the model’s knowledge cutoff. The island below makes the first two visible; mentally add the cutoff as a third wall on the right.

When to use it

Insist on point-in-time, vintage data whenever your features touch anything that gets revised (macro releases, earnings, fundamentals) or anything defined by membership (index constituents, sector classifications, credit ratings). For pure price-only, never-revised series the revision leak is smaller — but survivorship and delisting still bite. With a foundation model in the loop, assume it absorbed revised, survivor-only data and lean even harder on a strictly-post-cutoff window where its internal “knowledge” can’t reach.

How to test for contamination

The analogy. Before you audit a witness’s testimony line by line, you check one thing: could they have been at the scene? The knowledge cutoff is the alibi check. If the model “was there” for your test period, nothing else it says about that period can be trusted.

A contamination test battery. Stack these from cheapest to most thorough:

1. Cutoff-vs-window check (free). Put the stated knowledge cutoff next to your evaluation dates. Any overlap = contaminated, full stop. Refuse to run the backtest until the window is strictly post-cutoff.
2. Future-event probing. Ask the model about events after its supposed cutoff. If it “knows” something it shouldn’t — a result, a price level, a headline from after the claimed cutoff — the real cutoff is later than advertised, and your “clean” window may not be clean.
3. Public-vs-private gap. Score on a public benchmark and on a private, post-cutoff dataset. A large gap is the contamination fingerprint from Leak #2.
4. Point-in-time replay. Re-run with strictly vintage features and reconstructed as-of-date universes. If the edge evaporates, it lived in revision/survivorship leaks, not skill.
5. Trial deflation. Whatever survives, deflate the reported Sharpe for the number of model variants, prompts, and windows you tried — the same multiple-testing discipline from the alpha-research lessons.

When to use it

Run the full battery before you stake any decision — capital, a research claim, a publication — on a foundation-model result. For quick internal exploration the free cutoff-vs-window check is the non-negotiable minimum; the expensive point-in-time replay can wait until a result looks promising enough to deserve it. But never present a number externally without at least the cutoff check and a trial-deflation caveat.

When you can trust a zero-shot foundation-model backtest

So when can you believe the number? Only when a backtest clears every wall at once. Here is the concrete checklist — treat it as a gate, where failing any one item sends the whole result back to “contaminated until fixed.”

The trust checklist.

1. Strictly post-cutoff window. Every evaluation date is after the model’s knowledge cutoff. No straddling. (Necessary; everything else is moot without it.)
2. Point-in-time, vintage features. Each feature carries an availability timestamp; you use the as-first-published value, and the universe is reconstructed as of each date (no survivorship, no future index membership).
3. No public-benchmark overlap. Evaluation runs on private or freshly generated data that demonstrably could not be in the pretraining corpus; the public-vs-private gap has been checked.
4. Deflated for trials. The reported Sharpe is adjusted for the number of variants, prompts, and windows you searched — no cherry-picked best run.
5. Costs and capacity modeled. Transaction costs, slippage, latency, and strategy capacity are subtracted; the “edge” survives realistic frictions and isn’t a paper artifact.

Worked example — running the gate. A team reports a zero-shot strategy with Sharpe 1.8.

• Window: 2025–2026, model cutoff 2024 → passes item 1 (strictly post-cutoff).
• Features: macro releases used at revised values → fails item 2. Stop. Re-run with vintage data; Sharpe falls to 1.1.
• Re-checked: private data, no benchmark overlap → passes item 3.
• Trials: 30 prompt/variant combinations searched, deflate → effective Sharpe ~0.7.
• Costs: after fees and slippage → ~0.5.

The headline 1.8 was real arithmetic on a fake setup. Walk it through the gate and the honest, eligible-to-be-true number is ~0.5 — still possibly worth trading, but a different decision entirely. The checklist didn’t destroy value; it revealed how much of the headline was leakage and luck.

When to use it

Run the full gate before any go/no-go decision on real capital or any external claim. For early-stage triage you can fail-fast on the cheap gates first (cutoff, then trial count) and only pay for vintage-data replay and cost modeling once a candidate survives. But the moment money or reputation is on the line, all five walls stand — no exceptions, no averaging.

Recap