The Sim-to-Real Gap & Honest Evaluation

You spent the last five lessons building deep agents that learn to execute parent orders and quote two-sided markets. Some of them, on the backtest, are spectacular. Sharpe of 4. Smooth equity curve. Inventory neatly mean-reverting around zero. You are, frankly, a genius, and you are about to be very, very poor.

Here’s the brutal joke at the heart of this entire field: your agent did not learn to trade. It learned to trade your simulator. Every flaw in the environment you built — every place the sim under-charges for aggression, ignores your footprint, fills you at impossible prices, or replays a market that can’t see you coming — is a loophole, and a deep RL agent is a tireless, creative, utterly amoral loophole-finder. This is the capstone lesson, and its job is to humble everything that came before it. Not because the methods are wrong, but because the gap between a green backtest and a green P&L is where careers, funds, and a great deal of money go to die. Let’s learn to cross it honestly.

The sim-to-real gap, defined

Picture a student who gets the exam answer key a week early. They ace the test, top of the class, and conclude they are a brilliant physicist. Then you hand them a real, unseen problem and they freeze. They never learned physics — they learned this exam. That is the sim-to-real gap: the chasm between performance in the training environment and performance in the real world, opened by the agent learning the environment’s quirks instead of the underlying task.

Precisely: an RL agent maximizes expected return under the dynamics of the environment it trains in. If your simulator’s transition function $P(s' \mid s, a)$ and reward $r(s,a)$ differ from the live market’s true dynamics $P^\star, r^\star$ — and they always do, because no simulator is the market — then the policy $\pi^\star$ that is optimal for your sim is not the policy that’s optimal live. The agent has solved the wrong MDP. Worse, it has solved it exactly, squeezing out every drop of advantage your imperfect physics allowed. The more capable your function approximator (and a deep net is very capable), the more thoroughly it exploits the gap.

Why is this worse for RL than for ordinary supervised machine learning? Two reasons, both structural:

• The agent chooses its own training distribution. A supervised model is fed a fixed dataset; it overfits, sure, but to a distribution you handed it. An RL agent acts, and its actions determine which states it visits next. It actively explores the states it likes — the ones where your sim is generous — and gravitates toward the corners of the environment where the physics is flakiest. It is not a passive victim of distribution shift; it is an active prospector for it.
• The search space of policies is enormous. Supervised learning searches over functions from a fixed input to a label. RL searches over policies — entire strategies of behavior over time, in a space that grows combinatorially with the horizon and action set. More degrees of freedom means vastly more ways to find a spurious “solution” that fits the simulator’s accidents rather than the market’s structure.

When to use it

The sim-to-real framing is the lens for every evaluation decision in this course. Before you trust any backtested number from any of lessons 1–5, ask the sim-to-real question: which difference between my environment and the live market could be generating this result? If you cannot name the mechanism by which a number would survive contact with reality, treat it as fiction until proven otherwise.

The market impact model is the crux

Of all the ways a simulator can lie, one dominates: the market impact model. Here’s the inescapable problem. A backtest replays historical data — but history happened without your order in it. The tape you’re replaying never felt your million-share buy. So the simulator must invent what would have happened: how much your order moved the price (impact), whether and when you got filled (fill probability and queue position), and how the book reacted. That invented physics is the impact model, and it is the single most consequential, least observable assumption in the whole pipeline.

Get it wrong in the friendly direction and the consequences are catastrophic in exactly the way an RL agent loves:

• Under-charge temporary impact and a deep execution agent learns to slam the book — trade enormous size in single slices — because the sim barely punishes aggression. Live, that aggression pays real, convex impact and the strategy hemorrhages.
• Ignore the agent’s footprint and a market-making agent assumes its own quotes never move the market and always get filled at the posted price with no adverse selection. Live, the moment it posts a tight quote, informed traders pick it off (they trade against it precisely when it’s wrong), and its “guaranteed spread capture” becomes a guaranteed loss.
• Skip queue position and the sim fills your passive limit order the instant the price touches it. Reality: you’re 4,000 shares deep in the queue at that price, the level trades 1,000 shares, the price moves, and you got nothing — except when the market is about to run you over, in which case you get filled in full right before the loss. Adverse selection is the queue’s cruel asymmetry.
• Skip partial fills and latency and the agent assumes its orders arrive instantly and complete atomically. Live, there’s a delay between decision and execution, orders fill in pieces, and the price you saw is not the price you get.

A worked example — friendly vs. realistic fills

Take a market-making agent quoting a stock, posting a passive bid to capture the spread. Over a backtest day it posts 10,000 bids that the price touches. Suppose the quoted half-spread it tries to capture is $0.02 per share, 100 shares a clip.

Friendly sim (the lie): every touch fills, in full, at the posted price, with no adverse selection. The agent “captures” the half-spread on all 10,000 touches:

• Gross spread capture: 10,000 × 100 × $0.02 = $20,000.
• Modeled adverse selection: $0. Modeled fees: ignored.
• Reported P&L: +$20,000. Beautiful. Ship it.

Realistic sim (the physics): queue position means only 60% of touches actually fill (the rest, the queue ahead of you eats the prints) → 6,000 fills. And fills are adversely selected: you preferentially get filled when the price is about to move against you. Say the realized adverse move averages $0.035 per share on the filled trades — larger than the $0.02 you hoped to earn. Add an exchange fee of $0.001 per share.

• Gross spread capture on real fills: 6,000 × 100 × $0.02 = $12,000.
• Adverse selection cost: 6,000 × 100 × $0.035 = −$21,000.
• Fees: 6,000 × 100 × $0.001 = −$600.
• Realized P&L: 12,000 − 21,000 − 600 = −$9,600. A loss.

Same agent, same actions, same day. The only thing that changed was the honesty of the fill model — and it flipped a $20k winner into a $9.6k loser. The agent didn’t get worse; the simulator stopped lying. This is why impact and fill modeling is the crux, not a footnote.

Non-stationarity & regime shift

In a video game, the rules never change. The physics of Pac-Man in 2024 are the physics of Pac-Man in 1980. This is why RL conquered Atari, Go, and StarCraft: the environment is stationary — its dynamics don’t drift — so a policy learned on a billion frames is still optimal on the billion-and-first. The market grants you no such mercy.

Markets are non-stationary: the data-generating distribution drifts over time. Microstructure changes (tick sizes get reformed, new venues open, latency arms races escalate). Volatility regimes flip — the placid grind of 2017 and the convulsions of March 2020 are not draws from the same distribution. Liquidity migrates. A policy fit to 2015–2019 data is solving a subtly different MDP than the one live in 2020–2024; its $P^\star$ and $r^\star$ have moved underneath it. The closed-form world where a single optimal policy exists forever simply isn’t where you live.

And then there’s the part that has no analogue in any board game: the market is adversarial and adaptive. The other participants are not fixed environment dynamics — they are other learning agents, and they adapt to you. If your execution agent is predictable (always front-loads at the open, always pulls quotes at the same inventory threshold), other players will detect the pattern and trade against it — front-running your buys, fading your quotes. In game theory terms, you are not playing against nature; you are in a non-stationary game where your counterparties’ strategies co-evolve with yours. A policy that was profitable becomes a signature, and a signature becomes a liability the moment someone reverse-engineers it.

Honest evaluation — the discipline

So your backtest is suspect, your impact model is a guess, and the world won’t hold still. How do you ever know if you have something real? You adopt a discipline — a checklist of evaluations designed not to flatter your agent but to try to kill it. The mindset is adversarial: assume your edge is fake and demand it survive a gauntlet. Six practices, in order of how often they’re skipped (most-skipped first):

(1) Always compare against the analytic baseline — never against “doing nothing.” This is the cardinal sin and the easiest to commit. Your deep MM agent makes money — compared to what? Compared to not trading? Of course it beats not trading; that’s a meaningless bar. The honest comparison is against the strong, simple, closed-form baseline that already exists for your problem: Almgren–Chriss or TWAP/VWAP for execution, Avellaneda–Stoikov for market making. If your deep RL agent can’t beat a one-line analytic formula out-of-sample, you don’t have a result — you have a very expensive way to lose to a formula. (And recall from the execution lesson: in the formula’s own assumed world, a correct RL agent should merely re-derive the baseline, not beat it.)

(2) Walk-forward / purged & embargoed cross-validation — to stop leakage. Naive k-fold cross-validation leaks the future into the past on time-series data, because a sample’s label is realized over a forward window. If a training sample’s label window overlaps the test fold, the model is literally trained on the answer it’s being tested on. The fix (López de Prado): purge the training samples whose label horizons overlap the test fold, and embargo an extra band just after it to kill serial-correlation bleed. Walk-forward evaluation — train on the past, test on the strictly-later future, roll forward — is the time-respecting backbone underneath it.

(3) Deflated Sharpe ratio — pay the multiple-testing tax. Here is the quiet killer. You didn’t try one reward function — you tried 500. Different risk-aversion λ, different state features, different network sizes, different terminal penalties. You picked the best. But if you try 500 configurations on the same history, the best one looks great by pure luck, even when none has any real edge: under the null hypothesis (true Sharpe = 0), the expected maximum Sharpe across N trials climbs steadily with N. The deflated Sharpe ratio haircuts your observed best Sharpe by this rising “luck hurdle.” If your winner doesn’t clear the hurdle for the number of configurations you actually tried, it is statistically indistinguishable from luck — overfitting, not alpha.

(4) Out-of-sample on a held-out regime. Beyond rolling windows, deliberately reserve an entire regime you never touched during development — a crisis, a low-vol grind, a different year — and test there. Strong in-sample, strong same-regime out-of-sample, then dies on a held-out regime is the classic signature of a policy that memorized one market mood. (5) Paper-trade / shadow-trade. Run the agent live on real-time data with no real money (or microscopic size), generating the orders it would send and scoring them against actual fills. This is the first time your agent meets the real impact and latency it could never simulate. (6) Ablations vs. the baseline. Strip the agent’s components one at a time — remove the alpha signal, freeze the inventory skew, replace the net with a linear policy — and re-measure against the analytic baseline. If removing your fanciest component doesn’t hurt, that component was decoration, and the baseline was carrying you.

When to use it

This discipline is not optional polish for the final write-up — it is the research process. Run the gauntlet as you develop, not at the end: benchmark against the baseline from day one, use purged CV for every model-selection decision, track how many configurations you’ve tried so you can deflate honestly, and hold out a regime you refuse to peek at. The teams that skip this don’t find out their edge was fake in the lab; they find out in production, with real money.

Deployment discipline

Suppose your agent survived the gauntlet: it beats Avellaneda–Stoikov out-of-sample, clears the deflated-Sharpe hurdle, holds up on a held-out regime, and shadow-trades cleanly. Congratulations — now you get to be paranoid in production, because the live market is still the one place your simulator never visited. Deployment is its own discipline:

• Start small. Deploy at a tiny fraction of target size. The first job of live trading is not to make money; it’s to measure the sim-to-real gap with real fills and small losses, before scaling.
• Position and loss limits. Hard caps on inventory, per-trade size, daily loss. The agent does not get to bet the firm because its value function is confident.
• A kill-switch. A human-triggerable (and auto-triggerable) hard stop that flattens positions and halts the agent instantly. Non-negotiable.
• Live-vs-sim monitoring. Continuously compare realized fills, slippage, and P&L against what the simulator predicted for the same actions. If live diverges from sim beyond a threshold, stop — the gap you feared has opened, and every additional trade is now uncharted.
• Continual re-training cadence. Because the market is non-stationary, a frozen policy decays. Re-train and re-validate on a schedule (and after regime breaks), re-running the whole honest-evaluation gauntlet each time — never hot-swap a model that hasn’t cleared it.
• Humility about capacity. An edge that works on $1M of size may vanish at $100M, because you become the impact. Estimate capacity honestly and don’t scale past where your own footprint eats the alpha.

When to use deep RL at all

Time for the most honest sentence in the course: most of the time, the one-line analytic baseline wins, and that is the correct, publishable result. Reach for a deep RL agent only when you have specific, defensible reasons the closed form leaves money on the table — a genuine short-horizon signal to condition on, nonlinear or regime-dependent impact the analytic model can’t express, hard non-smooth constraints, or a high-dimensional correlated basket. Even then, the deep agent has to beat the baseline out-of-sample, after deflation, after shadow-trading — or you ship the baseline. “We tried deep RL and Avellaneda–Stoikov / Almgren–Chriss was just as good net of complexity and risk” is not a failure; it’s a rigorous, valuable finding that saves your firm from deploying a fragile black box for no edge. The mark of an expert is not loving the fanciest method — it’s knowing exactly when the simple one is already optimal and having the discipline to say so.

Where this goes next

You’ve now completed the arc: from tabular control to deep function approximation, through DQN’s instabilities on financial state, policy gradients and actor–critic for continuous control, deep RL for execution, deep RL for market making — and finally this, the lesson that tells you to distrust all of it until it survives contact with reality. That’s not cynicism; it’s the working posture of every serious practitioner. The sim-to-real gap, the impact-model crux, non-stationarity, the honest-evaluation gauntlet, and deployment paranoia are the difference between a beautiful backtest and a surviving strategy.

This was the last teaching lesson of the course. What remains is the Final Exam — a graded, one-shot run across the whole course: function approximation and DQN instability, policy gradients and actor–critic, execution and market-making agents, and everything in this capstone about why the green backtest is the beginning of the work, not the end of it. Bring the skepticism you just learned. It’s the most valuable thing in your toolkit.