This is the capstone. Six lessons took you from the moment a lookup table dies on a continuous order book to the moment a beautiful backtest dies in production. You replaced the table with a deep function approximator and learned that the substitution — generalize, don’t memorize — is deep RL. You met the Deep Q-Network and the deadly triad (function approximation + bootstrapping + off-policy learning) that makes the naive swap diverge, plus the experience-replay and target-network fixes, and the four ways financial state re-arms what those fixes were built to defuse. You threw out the value-table mindset for policy gradients and actor–critic — REINFORCE’s catastrophic variance, the advantage as the cure, PPO’s clipped trust region, and the brutal truth that on-policy exploration in markets is paid for in real spread, which is why offline RL exists. You pointed all of it at optimal execution, where implementation shortfall is the scoreboard and reward shaping silently rewrites the policy into a do-nothing machine or a directional gambler if you let it. You turned the chair around for market making, a two-sided continuous-control minefield where Avellaneda–Stoikov is the baseline to beat, skew steers inventory, the easy fill is the toxic one, and every naive reward breeds a different broken agent. And you ended on the lesson that humbles all the rest: the sim-to-real gap, where the agent learns the bugs in your physics, the impact model is the crux you can never replay, and only an adversarial honest-evaluation gauntlet — beat the baseline, purge-and-embargo, deflate the Sharpe, hold out a regime, shadow-trade, watch live-vs-sim — tells you whether you have alpha or expensive luck. No formula sheet, no hints, no take-backs.
Big picture
Deep RL for Execution & Market Making — the whole ladder
- Deep RL for Execution & Market Making
- Control → deep approximation
- Tables die on continuous order-book state (curse of dimensionality)
- Network generalizes across nearby states = deep RL
- Baselines to beat: Almgren–Chriss, Avellaneda–Stoikov
- DQN & instability
- Bellman → bootstrapped TD loss on a Q-network
- Deadly triad: approximation + bootstrapping + off-policy
- Fixes: experience replay + target network (Double/Dueling)
- Markets: non-stationary, noisy, partial, ugly rewards
- Policy gradients & actor–critic
- Output the action directly — no argmax over a continuum
- REINFORCE variance → baseline → advantage A = Q − V
- Actor–critic; PPO clips into a trust region
- Exploration costs real spread → offline/conservative RL
- Deep execution
- State = LOB + inventory + time + signal; continuous action
- Reward = −shortfall − λ·risk − η·q² terminal
- Beat signal-blind baselines; re-derive A-C with no signal
- Shaping rewrites the policy; mask hard constraints
- Deep market making
- Two-sided continuous (skew, half-spread) → PPO/SAC
- Spread is the wage, inventory is the risk; skew to flat
- Adverse selection: the easy fill is the toxic one
- Reward minefield: never-fill / speculator / bleeds
- Sim-to-real & honest evaluation
- Agent learns the sim, not the market
- Impact model is the crux — you cannot replay your own order
- Deflated Sharpe, purged CV, held-out regime, shadow-trade
- Live-vs-sim divergence → stop; often the baseline wins
- Control → deep approximation
One run, one shot
This exam is graded and irreversible. Each question locks the moment you submit it — there is no Back button, no retry, and no Restart. A wrong answer simply fails that question and the exam moves on. Your pass/fail score appears only at the very end, and the pass mark is 70%. Read every option before you commit.
You discretize 6 order-book features into 100 bins each instead of 10. Compared to the 10-bin grid, what happens to the number of table cells, and what does it prove about tabular RL on an order book?
Select an answer to continue.
Course Recap
The whole-course map is at the top of this page — give it one last look before you submit, then trust what you learned.
Key Takeaways
What you now own
You can take a learner from a dead lookup table to a live, monitored deep-RL trading agent — and, more importantly, you know when not to. Replace the table with a network only because it generalizes; keep that network from diverging by respecting the deadly triad and its replay/target-network fixes; learn the policy directly when the action is a continuous spread or size, taming REINFORCE’s variance with the advantage and fencing updates with PPO’s clip; design execution rewards that finish the order and can’t be hacked into a do-nothing machine or a directional gambler; make markets by trading spread against inventory, skewing to flat and fearing the toxic fill; and — above all — assume your simulator is lying until an adversarial gauntlet (beat the baseline, purge-and-embargo, deflate the Sharpe, hold out a regime, shadow-trade, watch live-vs-sim) proves otherwise. The deepest lesson echoes every course before it: the algorithm is never your biggest risk; the human who trusted a flawless backtest always is.