The Honest Evaluation: Does Any of It Survive?

Five lessons in, you’ve taken the pretrain-and-adapt bet (lesson 1), met time-series foundation models and their zero-shot, few-shot, and scaling behavior (lessons 2–3), wielded LLMs as research tools and signal sources (lesson 4), and just spent a whole lesson learning how leakage can quietly fake every result you produce (lesson 5). Now comes the reckoning.

This is the lesson where we stop being impressed by leaderboards and ask the only question that pays rent: does any of it survive contact with a real, adversarial, non-stationary market — net of costs, deflated for luck, and free of leakage? Foundation models are genuinely useful in finance. They are also catnip for the over-eager. The difference between those two sentences is an honest evaluation, and that’s what we’re building here, brick by brick, as the synthesis of the whole course.

Let’s settle the account.

Re-arming the discipline

The analogy. Imagine a casino where 500 gamblers each flip a fair coin 100 times, and you crown a “champion” as whoever got the most heads. The champion will look gifted — maybe 65 heads out of 100 — but they have no skill; they’re just the luckiest of 500. If you only ever interview the winner, you’ll conclude coin-flipping is a learnable art. Backtesting a foundation model across hundreds of architectures, context lengths, and fine-tuning recipes is exactly that casino, and the deflated Sharpe ratio is the bouncer who asks the champion to prove they’re better than the luckiest-of-500 baseline.

Tool 1 — the deflated Sharpe ratio. Carried from Machine Learning for Alpha. The core fact: if you try $N$ independent strategies that all have zero true edge, the expected maximum of their in-sample Sharpe ratios is not zero — it climbs with $N$. A useful back-of-envelope for the hurdle (in annualized-Sharpe-ish units, when trial dispersion is near 1) is

$\text{hurdle}(N) \approx \sigma_{SR}\,\sqrt{2 \ln N},$

where $\sigma_{SR}$ is the dispersion of Sharpe estimates across trials. Your deflated edge is the observed best Sharpe minus this hurdle. If it’s $\le 0$, your “winner” is statistically indistinguishable from the luckiest noise-trader in the room.

Worked example — deflating a 500-trial foundation-model search. You fine-tuned and back-tested $N = 500$ configurations (architectures × context windows × adapter ranks) on the same price history, with trial dispersion $\sigma_{SR} \approx 1$. Compute the hurdle:

$\ln 500 = 6.215, \qquad 2 \ln 500 = 12.43, \qquad \sqrt{12.43} = 3.52.$

So the luck hurdle is about 3.5. Your champion’s in-sample Sharpe was $2.8$. Deflated edge $= 2.8 - 3.5 = -0.7$. Negative. That dazzling 2.8 is below what 500 worthless strategies would produce by chance — it’s not alpha, it’s the casino. To survive, your best configuration would have needed an in-sample Sharpe comfortably above 3.5, or you’d have needed to run far fewer trials.

Tool 2 — purged and embargoed cross-validation. Carried from the same toolkit and reinforced by lesson 5. Standard k-fold cross-validation shuffles rows, which is a catastrophe for time series: a training row from tomorrow can sit next to a test row from today, leaking the future. The fix has two parts:

• Purging — drop training observations whose label horizon overlaps the test set. If a label is built from a forward window (e.g. the next 5 days’ return), any training sample whose forward window reaches into the test period is contaminated and must be removed.
• Embargo — after each test fold, skip a buffer of observations before resuming training, so serial correlation and slow-moving features don’t bleed test information into the next training block.

For foundation models the stakes are higher, because the model’s pretraining corpus may already contain your test window — a leak no amount of clever CV can fix if you don’t control the pretrain cutoff (lesson 5’s central warning). Purged-embargoed CV protects the fine-tuning and evaluation split; a clean pretrain cutoff protects everything underneath it.

When to use it

Apply both tools to every foundation-model evaluation, always — they’re not optional extras for the paranoid. Reach for the deflated Sharpe the moment you’ve compared more than one configuration (which is always), and for purged-embargoed CV whenever labels have a forward horizon or features are serially correlated (which is always, in markets). The only time you can relax the deflation is a single, pre-registered model with no search — and even then you should report how many ideas you discarded before committing.

Near-stationary domains vs adversarial, non-stationary markets

The analogy. A foundation model that masters weather and electricity demand is like a chess engine that’s superhuman against a fixed opponent who plays the same opening every time. Markets are a chess opponent who watches you, learns your strategy, and changes their play the instant you start winning. Worse, the board itself is being rebuilt mid-game — pieces change how they move (regime shifts). Stationary benchmarks reward memorizing a stable pattern; markets punish exactly that, because any stable pattern gets found and erased.

Three reasons benchmark wins don’t translate.

1. Distribution shift (non-stationarity). The data-generating process moves. Volatility regimes, interest-rate regimes, liquidity regimes, and correlation structures all shift, sometimes abruptly. A model pretrained on one regime can be confidently wrong in the next. Weather has seasons, but it doesn’t suddenly decide to obey different physics in 2009.

2. Reflexivity / alpha crowding. A genuine edge is self-destroying. The moment a profitable pattern is discovered and traded at scale, the trades move prices toward fair value and the edge shrinks toward zero. Your forecast changes the thing you’re forecasting. Tomorrow’s temperature does not care that you predicted it; tomorrow’s price does, and so does everyone else’s prediction.

3. Signal-to-noise (SNR). Daily return predictability is tiny — an out-of-sample $R^2$ of even a few percent is exceptional, and most of the variance is irreducible noise. Electricity demand at 6pm is overwhelmingly driven by deterministic structure (time of day, temperature, day of week); the noise floor is low. With faint signal and a moving target, the very capacity that helps a foundation model nail demand becomes a liability that fits noise on returns.

Worked example — a benchmark that lies. Suppose a foundation model posts a MASE (mean absolute scaled error) of $0.62$ on a weather benchmark, beating the seasonal-naive baseline by 38%. Translated naively, you might expect a comparable “38% improvement” on returns. But on daily equity returns the seasonal-naive baseline is essentially zero (returns have ~no exploitable autocorrelation), so beating it by 38% of nearly-nothing is still nearly-nothing. A model can have a beautiful MASE on a stationary series and an out-of-sample return $R^2$ of, say, $0.3\%$ — and after costs, that $0.3\%$ is often negative PnL. The benchmark and the bank account are measuring different universes.

When to use it

Trust benchmark dominance as evidence of model quality in domains that resemble the benchmark — near-stationary, high-SNR, non-adversarial series. Treat it as nearly irrelevant the moment your target is liquid, heavily-researched, low-SNR return forecasting. The closer your problem sits to the stationary, non-adversarial end of the table above, the more a foundation model’s pretraining is worth; the closer to the adversarial, non-stationary end, the more you must prove edge from scratch on your own clean, deflated, cost-aware test.

Where pretraining genuinely helps

The analogy. A pretrained foundation model is a seasoned analyst who has read thousands of company histories. Drop them onto a brand-new IPO with three weeks of trading and they’re invaluable — they pattern-match from everything they’ve seen. Drop them onto a 30-year mega-cap that an army of quants has already dissected to the molecule, and their general wisdom adds almost nothing the local specialists don’t already know. Pretraining is a prior, and a prior matters most exactly when the data is too thin to overwhelm it.

Four places pretraining earns its keep.

1. Cross-asset / cross-market transfer. Pretraining on thousands of series — equities, FX, commodities, rates, across geographies — lets the model learn shared structure: volatility clustering, mean-reversion at certain horizons, common reactions to shocks. That borrowed structure can sharpen forecasts on a target series that, alone, is too short to reveal it.

2. Cold-start. A new listing, a freshly launched ETF, or a just-IPO’d stock has almost no history. A locally trained model has nothing to chew on; a pretrained model arrives with a sensible default and adapts from a few observations (the few-shot regime from lesson 3).

3. Data-poor / illiquid regimes. New products (a novel derivative), small or frontier markets, or thinly-traded names simply don’t generate enough clean observations to fit a hungry local model. The pretrained prior is the difference between a usable forecast and noise.

4. A strong zero-shot baseline / prior. Even when you will build a custom model, the foundation model’s zero-shot forecast is a high-quality baseline to beat and a sensible prior to shrink toward — better than a naive seasonal or random-walk default (lesson 3).

Worked example — data-rich target vs data-poor target. Two forecasting jobs, same foundation model:

On the data-rich large-cap, the tuned local model already extracts 1.8%; the foundation model fine-tuned reaches 1.9% — a rounding-error gain that almost certainly evaporates after deflation and costs. On the 3-week IPO, the local model cannot even be fit (15 observations), while the foundation model’s borrowed prior delivers a real, if modest, 1.1%. Same model, opposite verdicts — and the deciding variable is local data abundance, not the model’s cleverness.

When to use it

Lean on pretraining when local data is the binding constraint: cold-start assets, illiquid or new products, frontier markets, or any forecast where you have dozens of observations rather than thousands. Treat it as a strong default prior and zero-shot baseline even in data-rich settings — but expect the marginal gain over a well-tuned local model to shrink toward zero as your local sample grows. The rule of thumb: the value of a borrowed prior is inversely proportional to how much of your own clean data you already have.

Where it’s pure hype

The analogy. Chasing leaderboard MASE on liquid return forecasting is like polishing your time in a footrace where the finish line is teleporting away from you faster than you can run. The metric improves; you get no closer to money. A lower MASE is not a higher PnL. MASE rewards accurate point forecasts of a series; trading profit depends on the sign and timing of moves net of costs, on a target whose edge has already been arbitraged by thousands of well-funded competitors.

The hype zone, precisely. Foundation models are most over-sold when all of these hold:

• Liquid and heavily-researched — the asset is traded by everyone, so any persistent pattern has been found and erased (maximal reflexivity/crowding).
• Low signal-to-noise — daily/weekly return predictability is faint, so the model’s spare capacity fits noise.
• Metric ≠ money — the reported win is a forecasting-error metric (MASE, MAE, RMSE) presented as if it were strategy performance, with no cost, capacity, or deflation accounting.

In that zone, a foundation model’s enormous pretrained capacity is not an advantage — it’s an over-powered noise-fitter, and the leaderboard number is a vanity metric. The honest expectation is no tradable edge, and the burden of proof is entirely on anyone claiming otherwise.

When to use it

Honestly? In the hype zone, don’t — at least not as your alpha engine. If you’re forecasting returns on a liquid, heavily-researched, low-SNR target, assume the foundation model has no tradable edge until a full end-to-end test proves otherwise, and never report a MASE/MAE improvement as if it were a strategy result. The legitimate role for a foundation model in that regime is supporting infrastructure (a baseline, a feature, a prior), not the headline signal.

The end-to-end test

The analogy. A benchmark win is a great dress rehearsal; live trading is opening night with hecklers, a moving stage, and the lighting crew working against you. Plenty of strategies that nail the rehearsal collapse the moment real costs, real size, and real time take the stage. The end-to-end test is the full dress-and-tech run that catches the collapse before you wire up real capital.

Precise statement — the gauntlet. A theoretical or benchmark win is not money until it survives, in order, every one of these:

Miss any one and the “edge” is an artifact. Notice the order isn’t arbitrary: deflation and leakage checks are cheap and should gate the expensive live work — don’t spend three months on a production pipeline for a strategy that never cleared the luck hurdle.

Worked example — a strategy dies one stage at a time. A foundation-model signal looks spectacular:

• Gross in-sample Sharpe: 2.8. Impressive.
• Costs: turnover is high; realistic spread + fees + slippage knock roughly $1.0$ off the Sharpe → 1.8 net. Still good.
• Deflation: it was the best of $N = 500$ configs, hurdle $\approx 3.5$. Deflated edge $= 1.8 - 3.5 = -1.7$. Dead here — but suppose for argument it had cleared, say a pre-registered single model with net Sharpe 1.8 and hurdle 0.5.
• Post-cutoff OOS: re-run only on bars after the pretraining cutoff. The Sharpe drops to 0.6 — much of the apparent edge was pre-cutoff contamination (lesson 5).
• Capacity: at target size, market impact shaves it to 0.3.
• Live-pipeline parity: a 1-bar timing mismatch between research and production erases the rest → ≈ 0.

A 2.8 that became a 0. Every stage was honest; the strategy simply never had real, tradable, deflated, leakage-free edge. This is the normal outcome, and an evaluation that doesn’t run the full gauntlet would have shipped a money-loser with great-looking slides.

The decision: pretraining likely helps vs likely hype

The synthesis — and the bridge to your final exam. Foundation models are neither a miracle nor a fraud; they are a prior, and a prior is exactly as valuable as the gap it fills. Where your own clean data is thin — cold-start, illiquid, frontier, new products — that borrowed structure is genuine leverage. Where your data is abundant and your target is liquid, low-SNR, and picked over by an army of competitors, the prior adds almost nothing and the leaderboard is a vanity metric. The instrument that tells the two apart is not enthusiasm — it’s the honest evaluation: deflate the best-of-N Sharpe, purge and embargo your folds, subtract real costs and capacity, test only on post-cutoff out-of-sample bars, and demand that your live pipeline reproduces the research result exactly. Carry that discipline into the final exam, where you’ll have to apply every piece of it — the bet, the models, the scaling, the LLM tooling, the leakage traps, and this gauntlet — to decide what survives and what’s a mirage. You now know how to use a foundation model in markets without letting it use you. Go prove it.

Recap