Deep RL for Execution & Market Making

Two of the previous courses taught you halves of the same agent. Reinforcement learning for trading gave you the decision-making half: framing the market as a Markov decision process, designing rewards, and choosing actions in a loop where your move changes the world. Deep learning for market data gave you the perception half: neural networks that read a noisy, high-dimensional limit order book and turn it into features. This course welds the two together — deep reinforcement learning — and points the result at the two control problems where it has the strongest claim to belong: optimal execution (slice a large order to beat the cost benchmark) and market making (quote both sides of the book and get paid the spread for holding the risk).

Deep RL is genuinely powerful and genuinely overhyped, and finance is where the gap between the two is widest. The promise is real: a deep policy network can read raw order-book state, condition on a short-term signal, and learn a quoting or execution rule no closed form can express. But every danger from the two parent courses compounds here. The neural network brings its hunger for data into a domain with a tiny effective sample size and a non-stationary, adversarial target. Reinforcement learning brings the deadly triad — function approximation, bootstrapping, and off-policy learning, the combination that makes deep value methods diverge — and an agent that gets to choose its own training distribution, which is overfitting with extra steps. The thread through every lesson is the same one as the courses before it: a backtest that looks too good usually is, and the only cure is a brutally honest baseline.

Here is the arc, from the machinery to the verdict:

• From tabular control to deep function approximation — why you cannot keep a lookup table for an order book, what a neural network replaces it with, and the two canonical problems (execution, market making) with their analytic baselines (Almgren–Chriss, Avellaneda–Stoikov) that every learned agent must beat to justify itself.
• Deep Q-networks and their instability — the value-based family with a neural network: experience replay, target networks, and the deadly triad that makes DQN diverge on financial state more readily than on a video game.
• Policy gradients and actor–critic — learning the policy directly: REINFORCE and its punishing variance, baselines and advantage, A2C and PPO, and why continuous quoting actions push trading toward policy methods.
• Deep RL for optimal execution — the state/action/reward design for slicing an order, how the learned schedule beats TWAP/VWAP and Almgren–Chriss when a signal exists, and how reward shaping silently rewrites the policy you thought you specified.
• Deep RL for market making — quoting as continuous control, the Avellaneda–Stoikov skeleton as the baseline, inventory-penalised spread capture, adverse selection, and the minefield of perverse incentives a market-making reward hides.
• The sim-to-real gap and honest evaluation — why a learned agent that prints money in your simulator loses it live: imperfect impact models, non-stationarity, overfitting the simulator, and the deflated-Sharpe / purged-cross-validation discipline that keeps the verdict honest.

By the end you will be able to build a deep execution or market-making agent and, more importantly, to distrust it correctly — to know which of its backtested gains are real edge and which are the simulator’s bugs wearing a neural network. A graded final exam runs the whole discipline back at you at the end, one locked question at a time.