Deep RL for Market Making

Execution gave you a clean mandate: someone hands you an order and a deadline, you slice it cheaply, you go home. Market making rips that mandate up. Nobody tells you what to buy or sell. You stand in the middle of the book quoting a price you’ll buy at (your bid) and a price you’ll sell at (your ask), and you get paid the gap between them — the spread — every time both sides trade against you. The catch: you don’t choose your trades. The market chooses them, by hitting whichever quote it likes, and it tends to hit the one that’s about to hurt you. You hold inventory you never wanted, you carry directional risk you never asked for, and your whole job is to keep collecting pennies of spread without getting flattened by the dollars of risk those pennies sit on top of.

You already met the analytic version of this fight in the high-frequency market-making course: Avellaneda–Stoikov, reservation prices, inventory skew, adverse selection. This lesson does the same job with a deep RL agent — and the punchline up front is that market making is a far nastier RL problem than execution, because the action is two-sided and continuous, the reward is a minefield of perverse incentives, and the benchmark you have to beat (AS) is genuinely good.

Market making as continuous control

Picture a vending machine that resets its prices every second. It posts what it’ll pay for a can (bid) and what it’ll charge for one (ask). Customers wander up and either sell it a can or buy one — the machine doesn’t get to pick. Its only lever is where it sets the two prices each second. Tilt them, widen them, narrow them — that’s the entire job. A market maker is that machine, except the cans are shares and the customers occasionally know something the machine doesn’t.

Formally, every step the agent emits two numbers: a bid offset and an ask offset relative to the mid (equivalently, a reservation-price skew plus a half-spread). That action lives in a continuous 2D space — you can quote 1.3 ticks below mid or 2.7 ticks above, not just a fixed menu. This is the single most important structural fact about market-making RL, and it’s why the algorithm choice is forced.

• DQN struggles. A Deep Q-Network outputs one Q-value per discrete action. Two continuous knobs means you’d have to chop each into a grid and take the Cartesian product — say 21 bid levels × 21 ask levels = 441 discrete actions, and you’ve still lost the fine resolution that matters when an edge is a fraction of a tick. Worse, neighbouring quotes have nearly identical value, so the argmax flickers noisily between them.
• Policy-gradient / actor–critic fit. PPO and SAC parameterize a policy that outputs a continuous action directly — typically a Gaussian over (skew, half-spread). They optimize the quotes as real numbers, with built-in exploration (the policy’s own variance) and stable updates (PPO’s clipped objective, SAC’s entropy bonus). Continuous action → actor–critic. That’s the rule.

The state the agent conditions on is rich: inventory $q$, the mid price, the current spread, recent volatility, order-book imbalance (more bids than asks resting, or vice versa), recent fills (did you just get hit?), and time (toward a session close or an inventory-reset point). The reward over a step is the whole P&L story compressed into one number:

$r_t = \underbrace{\text{spread captured on fills}}_{\text{good}} \; - \; \underbrace{\lambda_{\text{inv}}\, q_t^2}_{\text{inventory risk}} \; - \; \underbrace{\text{adverse-selection loss}}_{\text{getting picked off}} \; - \; \underbrace{\text{fees}}_{} \; + \; \underbrace{\text{rebates}}_{}$

Every term in that line is a place the agent can be tricked. Hold that thought — the last section is devoted to it.

When to use it

Reach for actor–critic market-making RL when your action is genuinely continuous and your state is high-dimensional — multiple book levels, microstructure features, a vol estimate. If you’ve deliberately collapsed the problem to “wide / medium / tight” three-way quoting on a slow, liquid name, a discrete method (even tabular) can be fine and far easier to debug. The continuous-control machinery earns its complexity only when fractional-tick resolution and many state features actually move your P&L.

Avellaneda–Stoikov, the baseline to beat

Before you let a neural network anywhere near the book, you write down the answer you’re trying to beat. Avellaneda–Stoikov (2008) is that answer. It is the analytic optimum for a market maker who faces a mid that random-walks with constant volatility $σ$, an exponential fill intensity (the further your quote sits from the mid, the exponentially less likely it gets hit), and terminal aversion to leftover inventory. In one block:

$r = s - q\,\gamma\,\sigma^2\,(T - t) \qquad \delta = \gamma\,\sigma^2\,(T - t) + \frac{2}{\gamma}\ln\!\left(1 + \frac{\gamma}{k}\right)$

The first equation is the reservation price $r$: your personal risk-adjusted fair value, shoved away from the mid $s$ in proportion to your inventory $q$, your risk aversion $γ$, variance $σ^2$, and time remaining $(T-t)$. Get long ($q>0$) and $r$ drops below the mid — you’ve quietly decided you’d rather sell. The second is the optimal half-spread $δ$: half from inventory risk, half from the log term that trades off fill probability against per-fill profit via the arrival intensity $k$. Your quotes are then $\text{ask} = r + δ/2$ and $\text{bid} = r - δ/2$. Drag the sliders below and watch both quotes slide as a unit while the spread sets their width.

Here is the course creed, and it is not optional: a deep market-making agent that can’t beat Avellaneda–Stoikov out-of-sample has earned nothing. AS is one block of algebra. If your PPO agent, with its GPU-hours and its 400-dimensional state, merely ties AS on held-out data, you have built an extremely expensive way to reproduce a 2008 formula. And the sharper version: in AS’s own idealized world — constant vol, exponential intensity, no signal — a correct agent should re-derive AS, the same way an execution agent re-derives Almgren–Chriss. Failing that recovery isn’t a stylistic difference; it’s a bug.

When to use it

Use bare AS (no RL) when reality genuinely resembles its assumptions: a single liquid name, roughly constant volatility over your horizon, no exploitable short-term signal, symmetric uninformed flow. It’s transparent, instant to compute, and impossible to overfit. Graduate to a learned agent only when those assumptions crack — toxic flow you can detect, regime-dependent volatility, multi-level book dynamics, or asymmetric flow — and even then, keep AS running as the benchmark you report against on every out-of-sample slice.

Inventory-penalised spread capture — the core trade-off

This is the beating heart of market making, so slow down here. You have one dial that controls everything: how wide you quote.

Quote tight (bid and ask hugging the mid) and you win the race for fills — flow hits you constantly, spread piles up. But every fill dumps inventory on you, and inventory is directional risk: a pile of long inventory is just a leveraged bet that the price goes up, made by a desk whose entire thesis is “I have no opinion on direction.” Quote wide and inventory risk plummets — but so do your fills, and a market maker with no fills earns no spread. Tight = more spread, more risk. Wide = less risk, less spread. There is no setting that maximizes both.

The escape hatch is skew, the AS reservation-price move made concrete. When you’re long, lower both quotes: your now-cheap ask gets lifted (someone buys your excess off you) while your now-stingy bid rarely gets hit (you stop accumulating). When you’re short, raise both. Skew is a steering wheel that pushes inventory back toward flat without your having to cross the spread and pay up yourself. The chart below shows the whole linear relationship — sweep inventory and watch both quotes slide together while the spread (the gap) holds roughly constant.

A worked example

Start flat ($q=0$) on a stock with mid $100.00, quoting a symmetric bid $99.95 / ask $100.05 (a $0.10 spread, so $0.05 captured each time you’re on the right side). Walk through four fills of 100 shares each, and crucially let the mid drift while you carry inventory.

Spread captured: steps 1+2 are a clean round trip — you sold at $100.05 and bought back at $99.95, banking 100 × $0.10 = $10. Steps 3+4 look like another $10 round trip on paper. But between steps 3 and 4 you were long 100 shares while the mid fell $0.10, a paper loss of 100 × $0.10 = $10 on that inventory. Net P&L across the four fills: $10 (clean round trip) + $10 (gross spread on the second pair) − $10 (inventory mark-to-market) = $10, not the $20 the spread capture alone suggested. The inventory you carried in step 3 quietly ate one whole round trip’s worth of profit. That single line — spread captured minus what your carried inventory did against you — is the entire game.

Adverse selection — why getting filled is bad news

Here’s the cruelty at the centre of market making: the fill you wanted is the fill that hurts. You posted a bid hoping a random seller would hand you cheap shares. Sometimes that’s who shows up. But sometimes it’s a trader who knows the price is about to drop — they sell to you at your bid a heartbeat before the mid falls, and now you’re long at a price that’s already stale. You got your fill. You also got robbed. That’s adverse selection: the systematic tendency for informed flow to pick off the quote that’s about to be wrong.

The analogy is a used-car lot that buys any car at a posted price. Most days you buy ordinary cars and resell at a markup. But the people most eager to sell at your posted price are disproportionately the ones whose cars are secretly lemons — they know something you don’t. Quote a fixed buy price and you’ll find yourself owning a yard full of lemons. The fix isn’t to stop buying; it’s to read the signals of who’s selling and adjust.

For a market maker, the loudest such signal is order-flow imbalance: when the resting and incoming volume is lopsided toward one side, that side is more likely informed. A wave of aggressive sells eating the bid is exactly when your bid is about to be the wrong price. A deep agent — unlike vanilla AS, which assumes symmetric uninformed flow — can condition on imbalance and recent-fill features to widen and skew defensively when toxicity rises: pull the bid back, fatten the spread, refuse to keep catching the falling knife. The same imbalance feature that warns you also tells you which way to skew. AS’s blind spot is precisely this: it has no slot for “this flow looks informed,” and that’s one of the clearest places a learned agent earns its keep.

The reward minefield

Now the centerpiece, and the reason market-making RL is so much harder than execution. In execution the objective is almost unambiguous: finish the order cheaply, with a terminal penalty forcing completion. Market making has no such anchor. You’re trying to encode “earn spread, but don’t take too much inventory risk, and don’t get picked off, net of fees” into a single scalar reward — and almost every honest-looking way to write it down hides a perverse incentive the agent will find and exploit with malicious glee. Reward design is the job here.

Tour the minefield, because each mine has blown up a real desk’s prototype:

• Reward only realized spread. “Pay the agent for spread it actually captures on round trips.” Sounds clean. The agent discovers the safest way to never realize a loss is to never fill — it quotes absurdly wide, sits at zero inventory, takes zero risk, collects zero reward, and a flat-line zero looks safe next to any strategy that occasionally dips negative. You’ve trained a very expensive agent to refuse to make markets.
• Reward mark-to-market P&L with no inventory penalty. “Just reward total P&L including unrealized.” Now the agent notices that the fastest way to make P&L go up is to be right about direction — so it stops balancing its quotes and starts loading up inventory on whichever way it thinks the mid is headed. Congratulations, you’ve trained a directional speculator wearing a market-maker costume. It’ll look brilliant in a trending backtest and detonate in a reversal.
• Penalize inventory too hard. Overcorrect from the last bug by slapping a huge $λ_{\text{inv}}\, q^2$ penalty on any nonzero position. The agent concludes that the only truly safe inventory is none, and the only way to guarantee none is to not quote — or to quote so wide it never fills. Same dead market as the first mine, reached from the opposite direction.
• Ignore adverse selection. Reward gross spread and inventory control but say nothing about who you traded with. The agent quotes tight to rack up fills, gets systematically picked off by informed flow, and bleeds — each fill scores positive spread the instant it happens, and the loss only shows up later as the mid moves against the freshly-acquired inventory, by which time the credit assignment is muddy. The agent never connects “tight quote into toxic flow” with “loss,” so it never stops.

The thread through all four: a market-making reward must simultaneously reward fills (or the agent won’t quote), penalize inventory (or it speculates) — but not so hard it goes catatonic — and account for adverse selection (or it bleeds on toxic flow). The $q^2$ inventory penalty has to be tuned, not cranked: enough to discourage piling on, not so much that flat-and-silent dominates.

The disciplined tool is potential-based reward shaping. Add shaping terms of the form $F(s,s') = \gamma\,\Phi(s') - \Phi(s)$ for some potential $Φ$ (e.g. a smooth function of inventory) — this provably leaves the optimal policy unchanged while making the gradient point the right way during learning. It nudges the agent toward flat inventory and away from toxic fills without secretly rewriting the objective into “never trade.” Crude penalties change what’s optimal; potential-based shaping only changes how fast you get there. When you must add an inventory term, prefer the shaped, potential-based version over a raw fixed penalty.

The price-ladder below is the picture to keep in your head while reasoning about reward. It’s the order book with the maker’s own bid and ask sitting in it; push the inventory slider and watch both quotes skew (the reward’s inventory term made visible), push the half-spread slider and watch the gap fatten (the spread-vs-fill-probability term). Reading it: the column of resting orders is the book, your two highlighted levels are where you are exposed, and the distance from the mid is exactly the knob the reward is fighting over — too tight and you fill into everything (including toxic flow), too wide and you never earn.

When to use it

Spend your effort on reward design before your effort on architecture — a perfectly-tuned PPO on a broken reward learns the wrong thing flawlessly. Start from the AS objective (it already encodes spread minus inventory risk sensibly), add an explicit adverse-selection / toxicity term only once you have a usable imbalance feature, and prefer potential-based shaping over raw penalties whenever you need to steer behaviour. And always sanity-check: in AS’s idealized world the shaped reward should reproduce AS, and out-of-sample the agent must beat AS net of fees, or the whole apparatus has earned nothing.

Where this goes next

You’ve now seen market making as a two-sided continuous-control problem: the action is (skew, half-spread), which forces actor–critic methods like PPO and SAC; the benchmark is Avellaneda–Stoikov, which a serious agent must re-derive in its world and beat out-of-sample; the core trade-off is spread capture against inventory risk, steered by skew; the cruelty is adverse selection, where the easy fill is the toxic one; and the whole problem is really a reward-design problem, a minefield where every naive objective breeds a different broken agent.

But every “beats AS out-of-sample” claim in this lesson rests on a simulator — a model of fills, of impact, of who’s on the other side. And market-making sims lie even more seductively than execution sims, because they get to invent the counterparty. Lesson 6, “The Sim-to-Real Gap & Honest Evaluation,” is where we stop trusting the backtest: how a too-friendly fill model or absent adverse selection manufactures fake profits, and how to evaluate a quoting agent honestly enough that “it beat AS” means something live instead of meaning “it learned the bugs in my physics.”