Optimal Execution as an RL Problem

Your fund just told you to buy a million shares before the close. You cannot simply slam one giant market order into the book — there isn’t a million shares sitting at the best offer, and even if there were, gobbling all of it would yank the price up against you so violently that the last shares cost far more than the first. So you slice the order into pieces and feed them in over time. But the moment you decide to go slow, a new enemy appears: while you dawdle, the price can drift away on its own. Buy slowly and the stock might rally before you finish; sell slowly and it might crater. You are squeezed between two costs that pull in opposite directions.

That squeeze is the problem. Trade fast and you pay market impact — you move the price by demanding liquidity faster than the market can supply it. Trade slow and you pay timing risk — more time in the market means more exposure to plain old volatility. Execution is not a forecasting problem (“will the stock go up?”); it’s a control problem (“given that I must trade this size, what schedule minimizes my total cost?”). And a control problem with a state, actions, and a reward is exactly the shape of an MDP — which is why optimal execution is the canonical, undisputed win for reinforcement learning in trading.

The execution problem

Strip it to essentials. You hold $Q$ shares to liquidate over a horizon $T$, split into $N$ equal time steps. At each step you choose how many shares $n_k$ to trade, with $\sum_k n_k = Q$. The sequence $\{n_1, n_2, \ldots, n_N\}$ is your execution schedule, and the running total still to be done is your inventory. Your goal: finish the whole quantity while minimizing total cost — where “cost” has two flavours.

Temporary impact is the price concession you pay to consume liquidity right now. Demand $n_k$ shares this instant and you sweep up the book past the best quote; the deeper you reach, the worse your average fill. Crucially, it’s temporary — once you stop pushing, the book refills and the price recovers. Temporary impact penalizes the rate of trading: trade twice as fast in a slice and you pay disproportionately more per share.

Permanent impact is different: it’s the lasting shift in the equilibrium price caused by the information your trade reveals. The market sees persistent buying, infers something, and reprices for good. It doesn’t recover when you stop, and it depends on how much you trade in total, not how fast. In the classic linear model, permanent impact is a fixed tax on your whole order — it shifts the level but, being schedule-independent in the simplest case, doesn’t change the optimal shape of the schedule.

Timing (volatility) risk is the third character. The longer your remaining inventory sits unsold, the more it’s exposed to the stock’s own random walk. Slow schedules have lower expected impact cost but higher variance of cost, because price could wander far while you’re still holding. This is the tension the whole field is built around.

When to use it

This impact-versus-risk framing applies whenever you move size that’s large relative to available liquidity: a pension fund rebalancing, an index reconstitution, a market maker unwinding inventory, or a quant desk routing a parent order to an execution algo. For a 100-share retail order it’s irrelevant — you’re a rounding error in the book, impact is nil, and you should just cross the spread and move on.

Almgren–Chriss, the closed form

In 2000, Almgren and Chriss gave the problem a clean answer. Their move: don’t minimize expected cost alone (that just says “go slow to dodge impact” and ignores risk), and don’t minimize variance alone (that says “go instantly to dodge volatility” and ignores impact). Minimize a mean–variance blend:

$\min_{\{n_k\}} \; \mathbb{E}[\text{cost}] + \lambda \, \mathrm{Var}[\text{cost}]$

Here $\lambda \ge 0$ is your risk aversion — the exchange rate between expected cost and its uncertainty. Sweep $\lambda$ and the optimal schedule morphs:

• $\lambda = 0$ (risk-neutral). You don’t care about variance at all, so you only fight impact. The answer is to trade as evenly as possible — a flat, uniform schedule, one slice per step. This is the spirit of TWAP (time-weighted average price); weight the slices by typical volume instead and you get VWAP.
• $\lambda > 0$ (risk-averse). Now unsold inventory frightens you, so you want it gone early. The optimal schedule front-loads: trade more in the first steps, less later, following an exponentially decaying trajectory of remaining inventory. The more risk-averse you are, the steeper the decay — in the limit, you fire almost everything immediately and eat the impact to kill the risk.

Plot the achievable (expected cost, variance) pairs as you vary $\lambda$ and you trace the efficient frontier of execution: each point is the lowest possible variance for a given expected cost (and vice versa). Schedules below the frontier are impossible; schedules above it are strictly dumb — you could cut variance for free. Your $\lambda$ just picks where on the frontier you want to live.

A worked example

Liquidate $Q = 100$ shares over $N = 4$ steps. Suppose, for round numbers, the risk-neutral plan trades 25 each step, and a chosen risk-averse $\lambda$ yields the front-loaded plan 40 / 30 / 20 / 10.

Average inventory carried (the thing volatility taxes): TWAP holds $(75+50+25+0)/4 = 37.5$ shares on average; the front-loaded plan holds $(60+30+10+0)/4 = 25$ shares. The front-loaded schedule carries a third less inventory through time, so it eats less timing risk — at the price of bunching trades early and paying more impact up front. That exact trade, less variance for more expected impact, is one step along the efficient frontier.

Framing execution as an MDP

Lessons 1–3 trained you to see any sequential decision under uncertainty as an MDP. Execution snaps right into the mould:

The reward line is a direct callback to lesson 2’s reward design: you are encoding the exact same mean–variance objective Almgren–Chriss minimized, but now as a per-step signal an agent can learn from. Maximizing the discounted sum of $-\text{impact} - \lambda\cdot\text{risk}$ is identical to minimizing $\mathbb{E}[\text{cost}] + \lambda\,\mathrm{Var}[\text{cost}]$ — the negation flips “minimize cost” into “maximize reward,” and the per-step risk penalty accumulates into the variance term. The terminal penalty is the part you must not forget: an agent rewarded only for cheap trading learns the cheapest schedule of all, which is to never finish. The deadline constraint, “inventory must hit zero by $T$,” is what makes the problem honest.

Reading a learned policy

Here’s the payoff of the MDP view: a solved policy is just a map from state to action — “given my remaining inventory, time left, and the current price signal, trade this rate.” You can visualize it as a heatmap. Read the grid below as a lookup table the agent consults each step.

Now read the three behaviours off the picture:

• TWAP is a flat field — the same trade rate in every cell. It is blind to the price signal (every row looks identical) and blind to urgency (every column looks identical). Dead simple, impossible to game, and a perfectly respectable benchmark.
• Almgren–Chriss varies down the time axis: heavy early, fading late (front-loading), but it too ignores the price-signal rows — the classic model has no alpha to condition on, so every row is the same.
• A learned RL policy lights up both axes. It ramps into the deadline like Almgren and conditions on the price signal: when the buy looks cheap it leans in and trades extra, when the price runs away from it the policy pulls back and waits, all while respecting the clock. That two-dimensional responsiveness is the signature of a policy that has learned something the closed form never modelled.

What RL adds (and what it just re-derives)

Time for the honest accounting, because the hype here is thick. Take the exact world Almgren–Chriss assumed: linear permanent and temporary impact, constant volatility, a single asset, no predictive signal, a mean–variance objective. Point an RL agent at that world and let it train. What does it learn?

It learns Almgren–Chriss. Essentially the same front-loaded, exponentially-decaying schedule the closed form hands you in one line of algebra. This is not a disappointment — it’s a sanity check. If your execution agent fails to recover Almgren–Chriss in the setting where Almgren–Chriss is provably optimal, your agent is broken, and you’ve just found the bug cheaply. Re-deriving the known answer is the unit test of execution RL.

RL earns its keep precisely where those tidy assumptions crack — which is to say, in reality:

• Nonlinear or uncertain impact. Real impact isn’t a neat linear function; it bends, it depends on regime, and you don’t know its parameters exactly. RL learns the schedule from data without needing you to write the impact model down correctly.
• A short-term alpha signal. If you have a genuine micro-prediction (“price likely ticks up in the next minute”), the optimal schedule should speed up to buy before the rise and slow down into adverse moves. Almgren–Chriss has no slot for this; RL conditions on it natively.
• Regime-dependent liquidity. Spreads widen, depth evaporates, volatility clusters. A policy that reads current market conditions and adapts its aggression beats any fixed schedule computed at the open.
• Hard constraints. Minimum and maximum participation rates, dark-pool routing, “don’t be more than X% of volume.” RL handles messy, non-smooth constraints that closed-form mean–variance can’t.
• Multi-asset / portfolio execution. Liquidating a basket with correlated prices and cross-impact is a high-dimensional control problem with no clean closed form. This is RL’s home turf.

When to use it

Default to Almgren–Chriss (or plain TWAP/VWAP) when your world is roughly its world: modest size, no exploitable signal, a single liquid name, well-understood impact. Reach for RL when you have a real edge to exploit (a signal), a hard-to-model impact regime, non-smooth constraints, or a correlated basket — and only after you trust your simulator’s impact model enough that you’re not just learning its bugs.

Where this goes next

Execution is the friendliest RL problem in trading because the objective is unambiguous (finish the order cheaply), the benchmark is clear (beat TWAP/Almgren–Chriss), and reality breaks the closed form in exactly the ways RL is built to absorb. You’ve now seen the canonical win end to end: the impact-versus-risk trade-off, Almgren–Chriss and its efficient frontier, the MDP framing, a learned policy read off a heatmap, and the honest line between what RL re-derives and what it genuinely adds.

But execution still assumes someone hands you the order and a deadline. Lesson 5, “Market-Making Agents,” flips the chair around: instead of consuming liquidity on a schedule, you provide it — quoting both sides of the book, managing inventory you never chose to hold, and getting paid the spread for the privilege of being the counterparty everyone else trades against. Same MDP toolkit, a wilder, two-sided reward.