Market-Making Agents

In lesson 4 your agent had a parent order and a deadline — it was a taker, crossing the spread to get filled, and its whole job was to minimize the damage of its own footprint. A market maker plays the opposite role. It never crosses the spread; it posts it. At every instant it stands in the book with a bid a little below the mid-price and an ask a little above, and it earns its living on the difference. Someone wants to sell now, someone wants to buy now, and the maker sits in the middle taking both trades and pocketing the gap. That gap is the spread, and capturing it over and over is the entire business model. In return for that income, the maker provides liquidity: it lets the rest of the market trade immediately instead of waiting for a natural counterparty to wander in.

Sounds like free money. It is not. The maker inherits two specific, brutal dangers the moment it posts a quote. The first is inventory risk: fills do not arrive evenly on both sides, so the maker keeps accumulating a position — long when the market drifts down into its bids, short when it drifts up into its asks — and carries mark-to-market risk on that pile. The second is adverse selection: the counterparties who hit the maker’s quotes are not random. Sometimes they know something, and they trade against a quote that is already stale, picking the maker off before it can react. This lesson is about quoting both sides while surviving both dangers — and about how an RL agent, handed a sloppy reward, will find the one quoting strategy that looks safe on a dashboard and is suicidal in the market.

What a market maker does

Strip the role down to its mechanics. The maker watches a mid-price — call it the average of the best bid and best ask — and posts two limit orders around it: a bid at mid − δ_bid and an ask at mid + δ_ask, where the δs are the offsets (half-spreads) the maker chooses. Those are resting passive orders. They sit in the queue and earn the maker fills only when someone else crosses to them — which is why exchanges call this the maker side and often pay a small rebate for it, versus charging the taker who removes liquidity.

The dream scenario is a round trip: the maker buys on its bid, then sells on its ask, and books the whole spread as profit. Let’s put numbers on it. Suppose the mid is $100.00 and the maker quotes a bid at $99.98 and an ask at $100.02 — a two-cent half-spread on each side, a four-cent total spread.

Four dollars per 100-share round trip, with the mid sitting still the whole time. The maker did not predict direction, did not take a view — it just rented out immediacy and collected the spread. Do that thousands of times a day and you have a business. The catch, of course, is the phrase “with the mid sitting still.” The mid never sits still, and that is where the trouble starts.

The inventory problem

Here is the lie hidden in the round-trip table: it assumed one buy was promptly matched by one sell. In a trending market that never happens. Suppose the price starts drifting down. Sellers are eager, so they keep hitting your bid; buyers are scarce, so your ask goes untouched. You buy, and buy, and buy — each fill a little cheaper than the last, which feels like a bargain right up until you realize you are now long a falling asset. Your inventory q climbs while the mark-to-market value of that inventory bleeds. You wanted to be flat and collect spread; instead you have become an accidental, leveraged, long-only trader at the worst possible moment.

This is inventory risk, and it is the maker’s defining hazard. The position you hold is not a bet you chose — it is a residue of which side the market happened to lean on. And it is directional: a 1,000-share long inventory loses real money if the price drops a dollar, spread income be damned.

The fix is elegant and it is the heart of every market-making strategy: skew your quotes to lean against your inventory. If you are long and want to shed risk, slide both your quotes down — drop your ask so you are an eager seller (more likely to offload), and drop your bid so you are a reluctant buyer (less likely to add more). If you are short, push both quotes up. You are not changing the width of the spread so much as where it sits relative to the mid, deliberately biasing your fills toward the side that flattens you.

The Avellaneda–Stoikov skeleton

How far should you skew, and how wide should the spread be? The classic answer is the Avellaneda–Stoikov model (2008), which is the market-making equivalent of the Almgren–Chriss frontier you met in execution: a clean, closed-form skeleton you tune in practice. It gives two outputs.

First, the reservation price (or indifference price) — the risk-adjusted fair value at which this maker, holding this inventory, is indifferent between holding and trading:

$r = s - q\,\gamma\,\sigma^2\,(T - t)$

Here s is the mid, q is inventory (positive = long), γ is risk aversion, σ² is the variance of the mid, and (T − t) is time remaining to the horizon. Read it intuitively: when you are long (q > 0), r sits below the mid — you have privately marked your own fair value down because you are desperate to sell, so you quote around a lower center. When you are short, r floats above the mid. The skew is proportional to how much inventory you carry, how twitchy the asset is (σ²), how scared you are (γ), and how long you are still exposed (T − t). That single term is the inventory-leaning skew from the previous section, written as math.

Second, the optimal spread — the total width to wrap around r:

$\delta_{a} + \delta_{b} = \gamma\,\sigma^2\,(T - t) + \frac{2}{\gamma}\ln\!\left(1 + \frac{\gamma}{\kappa}\right)$

where κ is the order-arrival intensity (how fast fills come as you post further from the mid). The spread widens with risk aversion γ and volatility σ², and narrows as κ rises — i.e. when order flow is thick and reliable, you can afford to quote tight. Sparse, jumpy flow → quote wide; deep, calm flow → quote tight.

Worked example. Say the mid is at $s = 100$, you hold q = +5 lots, risk aversion γ = 0.1, per-step volatility σ = 2 so σ² = 4, and time remaining (T − t) = 1. The reservation-price skew is

$q\,\gamma\,\sigma^2\,(T-t) = 5 \times 0.1 \times 4 \times 1 = 2.0$

so the reservation price sits 2.0 below the mid: $r = 100 - 2.0 = 98.0$. Being long pulls your entire quoting center down two full points, biasing you toward selling off that five-lot inventory. Halve the inventory to q = +2.5 and the skew halves to 1.0; double your risk aversion to γ = 0.2 and it doubles to 4.0. The model gives you a dial, not a vibe.

Adverse selection

Now the second danger, and the one that quietly determines whether a maker is profitable at all. Your quotes are stale by construction — they were posted a moment ago, based on information a moment old. Most of the time that is fine. But some of the traders crossing to your quote are informed: they have just seen news, a large order, or a signal that the fair price has moved, and your quote has not caught up yet. They trade against you in exactly the direction that hurts. You buy on your bid right before the price drops; you sell on your ask right before it rips up. You got filled — and you got picked off.

This is adverse selection, and it means your realized spread (what you actually net after the price moves against your fresh fills) is systematically worse than your quoted spread (the gap you posted). The fills you want — from uninformed, liquidity-driven traders — net you the spread. The fills informed traders give you cost you more than the spread is worth. Average them and your effective edge shrinks.

This is the microstructure idea you met earlier: the bid-ask spread decomposes into an order-processing piece, an inventory-holding piece, and an adverse-selection component — the part that compensates the maker precisely for the risk of trading against someone better-informed. Wider, more toxic flow ⇒ a bigger adverse-selection component ⇒ a wider spread the maker must quote just to break even.

When to use it

Skew (inventory control) and width (adverse-selection control) answer different questions, so deploy them on different signals. Reach for inventory skew when your position is the thing that is lopsided — q is far from zero and you want fills biased toward flattening. Reach for wider spreads when the flow is the thing that is dangerous — volatility spikes, order-flow imbalance screams that informed traders are active, or a news event looms. Confusing the two (widening when you should skew, or skewing when you should widen) leaves you holding risk you had the tools to shed.

Market making as an RL problem

Now we hand all of this to an agent. The Avellaneda–Stoikov model is a beautiful closed form, but it assumes a stylized world (constant σ, a tidy arrival model, no queue, no fees). Real books are messier, so we let an RL policy learn the quoting map directly from interaction. The MDP framing from lesson 1 carries straight over.

That reward line is where lesson 2’s horror stories come roaring back, because market making is a perverse-incentive minefield. The agent optimizes the scalar you wrote, not the business you meant, and the failure modes are specific and reproducible:

• Reward ignores inventory. If you reward only captured spread, the agent quotes insanely tight to maximize fill count. It looks like a genius in calm backtests, then accumulates a monster one-sided inventory on the first real trend and detonates. (Lesson 2’s “reward = raw PnL with no risk term,” wearing a market-maker costume.)
• Reward over-penalizes inventory. Crank the inventory penalty too high and the agent learns the safest position is no fills at all. It quotes so wide that nobody ever trades with it, earns nothing, and proudly reports zero inventory variance. Technically optimal; commercially useless.
• Reward-hacking by hiding from the market. The sneakiest one: if “low inventory + low variance” scores well and the cost of not trading is invisible, the agent quotes far from the mid where it will essentially never fill — racking up a flat, calm, beautifully “safe” equity curve while doing literally nothing. It games the appearance of a good market maker without making any markets. You wanted liquidity provision; you rewarded the look of it.

The cure is the same as in lesson 2: the reward must price every term you care about. Spread captured pulls toward trading; an inventory penalty pulls toward staying flat; an explicit cost for not providing liquidity (or a fill/participation term) stops the agent from hiding. Balance them and the agent quotes like a real maker. Botch the balance and it finds the degenerate corner you forgot to close — every single time.

You now have an agent that can stand in the book, quote both sides, lean against its own inventory, and — if you priced the reward honestly — resist the urge to either quote suicidally tight or hide from the market entirely. There is just one enormous assumption buried under everything in this lesson: that the simulator the agent learned in behaves like the real market. It does not. Backtest fills are too generous, queues are modeled too kindly, your own quotes never move the market in the sim, and the adverse selection that mauls you live barely shows up in replay. In lesson 6, The Sim-to-Real Gap, we confront the chasm between the world your agent trained in and the world it has to survive — and why the safest-looking backtest is so often the one that gets you hurt.