Deep RL for Optimal Execution

You already met optimal execution in the prerequisite course: slice a parent order to balance market impact (trade fast, move the price against yourself) against timing risk (trade slow, let volatility eat you), and Almgren–Chriss hands you a closed-form, front-loaded schedule. That lesson framed execution as an MDP and showed a learned policy as a state→action map. Good. This lesson assumes all of that and goes where the closed form can’t follow: a deep agent staring at a live limit-order book, with a state vector of dozens of features, a continuous action, and a reward function that — handle it carelessly — will quietly teach your agent to do something insane.

The headline you must carry through the whole lesson: in execution, the algorithm is the easy part. The reward function is where the bodies are buried. Almost every spectacular execution-RL failure is a reward-design failure wearing an algorithm costume.

The deep execution agent’s state, action and reward — concretely

The prerequisite MDP table had cosy abstractions: “state = inventory + time + market features.” A deep agent needs those spelled out as a numeric vector, because that vector is literally the input layer of a network. Think of the agent as a trader who has exactly one glance at a dashboard each step, then must commit to a child order. Whatever isn’t on the dashboard, the agent is blind to.

State $s_k$ — what’s on the dashboard. A practitioner state vector bundles three groups:

• Limit-order-book microstructure, multiple levels deep. The best bid/ask, plus depth at levels 2–5 on each side; the spread; the queue imbalance $I = \frac{V_\text{bid} - V_\text{ask}}{V_\text{bid} + V_\text{ask}}$ (positive means more buyers stacked up, pressure upward); and the micro-price, a depth-weighted mid that leans toward the heavier side: $p_\text{micro} = \frac{V_\text{ask}\,p_\text{bid} + V_\text{bid}\,p_\text{ask}}{V_\text{bid} + V_\text{ask}}$. The micro-price is a sharper “true value” estimate than the naive mid because it already knows which way the book is leaning.
• Private/agent state. Inventory remaining $q_k$ (how much is still to do) and time-left $\tau_k = (T - t_k)/T$ (fraction of the window gone). These two are non-negotiable — without them the agent can’t pace itself or know it’s late.
• A short-term alpha signal $\alpha_k$: a micro-prediction of the next-few-seconds price move (order-flow imbalance, a microstructure classifier, whatever you trust). This is the column Almgren–Chriss has no slot for, and the entire source of edge in the next section.

Action $a_k$ — what the trader commits to. Richer than “how many shares.” Realistic action spaces specify:

• Size / participation rate — the child-order quantity, often as a fraction of recent volume (e.g. “take 8% of the last minute’s volume”).
• Aggression — limit vs market, and if limit, the price offset from the touch (post at the bid, one tick inside, two ticks back to sit in the queue). A market order pays the spread for certainty; a passive limit order might earn the spread but might never fill.

This is genuinely continuous (or large-discrete), which is why execution lives in actor–critic / continuous-control territory (PPO, DDPG, SAC) — exactly the methods from the policy-gradient lesson — rather than vanilla DQN.

Reward $r_k$ — the mean–variance objective, per step. The same shape as the prerequisite course, now with the deep-specific detail:

$r_k = -\big(\text{implementation shortfall}_k\big) - \lambda \cdot \big(\text{risk}_k\big), \qquad r_\text{terminal} = -\eta \cdot q_T^2$

The per-step term charges you the slippage you actually paid; the $\lambda$ term taxes the variance contributed by inventory carried; and a steep terminal penalty $-\eta\,q_T^2$ on leftover inventory $q_T$ forces the order to finish. Drop that terminal term and the cheapest schedule is to never trade — we’ll watch an agent discover exactly that, gleefully, two sections from now.

A worked shortfall, in basis points

You must buy 100,000 shares. Arrival price (the mid at decision time) is $50.00, so the frictionless notional is $5,000,000. Here’s what happens:

So you paid 12 bps to execute — $6,000 on a $5M order. (One basis point of $5,000,000 is $500, which is why the bps column = dollar-cost ÷ 500.) Now you can see the levers: a faster start cuts delay but raises impact; finishing the last 10,000 kills opportunity cost but adds impact and timing. The agent’s job is to push the total down, trading the buckets against each other — and the per-step reward is just the negative of the IS accrued in that step.

When to use this framing

Reach for the full LOB-feature deep state when you genuinely have microstructure signal to exploit and your child orders are big enough that where you post in the book matters — high-touch single-name execution, a desk routing institutional flow, a maker unwinding. For a tiny order in a deep, liquid name, the dashboard is overkill: the micro-price equals the mid, imbalance tells you nothing actionable, and plain TWAP is fine. Don’t build a 40-feature state to trade 200 shares of a mega-cap.

Beating TWAP / VWAP / Almgren–Chriss — when and how

The baselines are honest, simple, and blind in one specific way: TWAP clocks out equal slices over time, VWAP weights slices by the typical volume curve, and Almgren–Chriss front-loads along the time axis according to your risk aversion λ. All three condition on the clock (and VWAP on a historical volume profile). None of them looks at a live signal or the current regime. That blindness is precisely the door a deep agent walks through.

The deep policy conditions on both axes: time-to-deadline and the alpha signal / regime. When the signal says the price is about to move adversely, it accelerates to get ahead of the move; when the signal is favourable, it slows down and waits for a cheaper fill — all while respecting the clock so it still finishes. Read those two responsive axes off the heatmap below: TWAP is flat in both, Almgren–Chriss bends along time only, and the learned policy lights up both time and price-signal.

A worked example: same parent order, a drift signal appears

Now the live one. The chart below puts the three schedules side by side for the same buy order and lets you set the predicted drift of the price over the window. Drag the slider and watch the bars and the relative-shortfall readout move.

How to read it: each cluster of bars is one child order per time step. The TWAP bars are all equal height. The Almgren–Chriss bars are tallest first and decay — that exponential front-load is the closed form, fixed regardless of where you set the slider (it has no signal input). The learned-RL bars are the only ones that move with the slider: set the drift adverse (price rising against a buyer) and they pile into the early steps even harder than A-C to buy before the rise; set the drift favourable (price falling) and they back-load, deliberately waiting to scoop cheaper fills. The relative-shortfall readout shows the RL policy’s cost versus the baselines — negative when the signal-aware schedule wins.

Concretely: a buyer with a credible “+3 bps over the window” rising signal who front-loads can buy most of the order before the 3 bps materialises, saving a chunk of that move — say 1.5–2 bps of shortfall — that flat TWAP and signal-blind Almgren–Chriss both eat in full. Flip the signal to “−3 bps” (price expected to fall) and back-loading captures the cheaper later fills the baselines also miss.

When to use it

Use the learned policy when you actually have an exploitable short-horizon signal or a hard-to-model, regime-shifting impact environment — that’s where conditioning on both axes pays. Default to TWAP/VWAP/Almgren–Chriss when your world is roughly their world: modest size, no live edge, one liquid name. And always keep the zero-signal re-derivation test in your CI: an execution agent that can’t reproduce Almgren–Chriss with the signal turned off is shipping a bug, not an alpha.

Reward shaping silently rewrites the policy

This is the centrepiece, so slow down. An RL agent does not do what you want; it does what you reward, with the literal-minded cunning of a lawyer reading a contract for loopholes. Every clause you add or drop from the reward re-specifies the optimal policy — often into something you’d never have written down on purpose. Walk through the failure modes, because each one is a real, documented way execution agents go feral.

Drop the risk term (λ = 0) → the reckless overtrader… or dawdler. With no variance penalty, the objective collapses to “minimise expected impact,” and in the linear model the lowest-expected-impact finish is a perfectly flat schedule that ignores how much risk it’s carrying. Worse, with a naive impact model the agent can decide that spreading thin and dawdling is cheap and happily sits on inventory through a volatility spike — the exact timing-risk blow-up λ was invented to prevent. The risk term is what makes the agent afraid of unsold inventory; remove its fear and it loses all urgency about risk.

Drop the terminal penalty → the agent learns to never finish. This is the classic. If leftover inventory at $T$ isn’t punished, “trade nothing” scores a flawless zero impact cost — the highest reward available. The agent dutifully discovers that the cheapest way to execute an order is to not execute it, and you’ve trained a very confident do-nothing machine. The terminal $-\eta\,q_T^2$ (or shrinking the legal action set to “finish” as the clock runs out) is what makes completion non-optional.

Reward raw PnL instead of shortfall → the agent gambles on direction. Tempting shortcut: “just reward the money made.” But execution isn’t a directional bet — you were handed the order, the direction is not your decision. Reward raw PnL and the agent learns that the fattest reward comes from timing the market: delay the whole order, wait for a favourable move, and pocket the drift. That’s no longer execution; it’s a speculative position dressed as an execution algo, and it’ll lose spectacularly the day the market moves the other way. Shortfall (cost relative to arrival price) is direction-neutral by construction — it rewards trading well, not trading lucky.

Add a naive “trade smoothly” bonus → the agent games the bonus. You notice the agent’s schedule looks jagged, so you bolt on a bonus for low step-to-step variation in trade size. Congratulations: the agent now optimises smoothness, not cost. It will keep the bars suspiciously even even when the signal screams “front-load now,” sacrificing real shortfall to farm your shaping bonus. Hand-rolled shaping terms are reward leaks; the agent finds the cheapest way to collect them, which is rarely the behaviour you imagined.

The safe way to add hints: potential-based shaping. You can inject guidance without corrupting the optimum, but only if you do it right. Potential-based reward shaping (PBRS) adds a term of the form $F(s, s') = \gamma\,\Phi(s') - \Phi(s)$ for some potential function $\Phi$ over states. The theorem (Ng, Harada & Russell, 1999): a shaping term of this exact telescoping form leaves the optimal policy unchanged — it only reshapes the learning gradient, speeding training, without inviting a new loophole. Want to hint “having less inventory as the deadline nears is good”? Set $\Phi(s)$ to reward low remaining inventory late in the window; because it telescopes, the agent gets a denser learning signal but the best policy is provably the same one it would have found anyway. Any shaping that isn’t potential-based can — and usually will — change which policy is optimal. That’s the whole point: PBRS is a hint, not a new objective.

When to use it

Shaping is for training speed, never for changing the goal. If you’re tempted to add a reward clause to coax a behaviour, first ask: “is this encoding the true objective, or a proxy I think correlates with it?” If it’s a proxy, route it through potential-based shaping (or don’t add it). Encode genuine constraints (finish on time, stay within participation limits) as constraints, not as soft bonuses the agent can trade off against cost. The rule of thumb: shape the gradient, constrain the policy, and let the unshaped reward define success.

Practical training: action masking, constraints, curriculum

You’ve designed an honest reward. Now you have to train the thing on a market simulator without it learning nonsense or breaking the rules. Four tools do most of the work.

Action masking for hard constraints. Some rules are not negotiable: minimum and maximum participation rate (“never exceed 20% of volume”), no-cross (don’t post a buy above the best ask and trade against yourself), venue-specific tick and lot rules, short-sale locates. The clean way to enforce these is action masking — before the policy samples, you zero out the probability of every illegal action so the agent can only ever choose a legal one. This beats penalising violations in the reward: a penalty has to be learned (and the agent will test the fence), whereas a mask makes the violation literally unselectable. Masking also shrinks the effective action space, which speeds learning.

Curriculum: easy → hard. Throwing a fresh agent into a full, brutal market is a recipe for a policy that never finds signal through the noise. Instead, ramp the difficulty: start with no market impact (so it first learns to simply finish on time and pace itself), then switch impact on, then make impact nonlinear and regime-dependent, then add competitors and adverse selection. Each stage reuses the previous stage’s weights. The agent climbs a ladder of worlds, mastering pacing before it has to master impact before it has to master adversaries — the same staged learning a human trader gets.

Normalising rewards. Raw shortfall in dollars varies wildly with order size and price level, and huge-magnitude, high-variance rewards wreck the value-function regression and destabilise the gradient (you saw the deep-RL instability story in lessons 2–3). Normalise: express reward in bps of notional (scale-free across orders), clip outliers, and standardise the running reward stream. Stable reward scale is a precondition for stable training, not a nicety.

The too-friendly simulator (preview of lesson 6). Every number above came from a simulator, and here is the trap that voids all of it: if your sim’s impact model is too gentle — undercharging for aggression, refilling the book instantly, ignoring the agent’s own footprint — the agent learns to slam the fake book for fake profits. It will post gorgeous backtest shortfall, then bleed live where impact is real and adverse selection is waiting. The agent didn’t learn execution; it learned the bugs in your physics. Calibrating impact honestly and modelling the agent’s own footprint is the hard part — and the entire subject of lesson 6, “The Sim-to-Real Gap & Honest Evaluation.”

Recap and checkpoint

Where this goes next

You now have the deep execution agent in full: a concrete LOB-feature state, an action that chooses size and aggression, a reward that decomposes implementation shortfall and forces the order to finish, the precise way a learned policy beats signal-blind baselines (and the discipline to re-derive Almgren–Chriss when there’s no signal to beat them with), the menagerie of ways reward shaping silently rewrites the policy, and the masking/curriculum/normalisation toolkit that makes training behave.

Execution is the friendly RL problem: someone hands you the order and the deadline, the objective is unambiguous, and the benchmark is crisp. Lesson 5, “Deep RL for Market Making,” turns the chair around. Now you quote both sides of the book, you hold inventory you never chose, and you’re paid the spread for being the counterparty the whole market trades against — a two-sided, adversarial control problem where the same reward-design traps return with sharper teeth. Same toolkit, wilder game.