From Tabular Control to Deep Function Approximation

You spent the prerequisite courses learning two halves of one machine. One course taught you to see a market — turn raw order-book data into features with a neural network. Another taught you to decide under uncertainty — frame a sequential problem as a Markov decision process and solve it with reinforcement learning. Both halves worked beautifully on toy problems. Then you point them at a live order book and the decision half quietly explodes.

The reason is embarrassingly simple. The RL you learned stored its knowledge in a table — one cell per (state, action) pair. That works when the world has a few dozen states. An order book has effectively infinite states: continuous prices, continuous spreads, depth at ten levels, your inventory, the seconds left on your clock, three alpha signals. The table you’d need has more cells than there are atoms you’ll ever get data points for, and almost every cell is visited exactly zero times. The table doesn’t learn — it just sits there, mostly empty.

This course is about the fix: replace the table with a function approximator — usually a deep network — that generalizes across states instead of memorizing them. That single substitution is the whole definition of deep RL, and it reanimates execution and market making as learnable control problems. But it also drags in a swarm of new failure modes the table never had. This lesson sets up the problem, the tools, and the two analytic baselines that will keep us honest for the rest of the course.

Why a lookup table dies on an order book

Picture tabular RL as a giant filing cabinet. Every possible situation gets its own drawer, and inside each drawer is a slip of paper telling you the value of each action you could take there. To learn, you visit a drawer, try an action, see the reward, and update the slip. Brilliant — if you have few enough drawers that you can visit each one many times.

Now build the cabinet for an order book. A realistic execution state might track, at minimum, six numbers: the mid-price (continuous), the bid–ask spread (continuous), short-term volatility (continuous), order-book depth (continuous), your remaining inventory (continuous), and time-left on the clock (continuous). Add an alpha signal or two and you’re past six. None of these are naturally discrete, so to even build a table you must chop each continuous feature into bins.

Here’s the arithmetic that ends the conversation. Discretize 6 features into 10 bins each. The number of states is

$10 \times 10 \times 10 \times 10 \times 10 \times 10 = 10^{6}.$

A million states. Now give the agent, say, 20 discrete actions (trade-rate buckets). The table has $10^6 \times 20 = 2 \times 10^7$ cells. To learn a reliable value for a cell you need to visit it many times — call it 30 visits. That’s $6 \times 10^8$ informative experiences required. A few years of minute-bar data gives you maybe $10^6$ steps. You’re short by three orders of magnitude — and that’s with a laughably coarse 10-bin grid that throws away most of the price resolution that actually matters. Refine to 100 bins per feature and the state count jumps to $10^{12}$; the data deficit becomes hopeless.

This is the curse of dimensionality: the number of table cells grows exponentially in the number of features, while your data grows only linearly in time. The cells you visit are a measure-zero sprinkle in an enormous empty space. And the table cannot help you in a cell you’ve never seen — it has no notion that a state with spread 0.011 should behave like the neighbor at spread 0.012. Every drawer is an island.

When to use it

Tabular RL is not useless — it’s the right tool when the state really is small and discrete: a 3×3 grid-world, a coarse inventory-only market-making toy with five inventory levels and three actions, a teaching example where you want to inspect every cell. The moment a single continuous feature enters and you can’t bin it without either blurring the signal or exploding the count, you’ve left tabular territory. On a production order book you left it before you started.

Function approximation: the network as the table’s replacement

The fix is to stop storing a separate number for every state and instead store a function that computes the number on demand. Feed in the state $s$; the function returns the value $V(s)$ — or the action-value $Q(s,a)$, or the action itself $\pi(s)$. The function has a modest number of tunable parameters $\theta$, far fewer than the table had cells, and you fit $\theta$ to the data you actually have. The payoff is generalization: because the function is smooth, a state you’ve never visited gets a sensible answer interpolated from the nearby states you have visited. The drawers stop being islands.

Formally, the table $Q(s,a)$ becomes a parameterized $Q_\theta(s,a)$. Two flavours matter:

• Linear approximation. $Q_\theta(s,a) = \theta^\top \phi(s,a)$, a weighted sum of hand-designed features $\phi$. Cheap, stable, easy to reason about — but it can only represent relationships you already baked into $\phi$. If impact is nonlinear in size and you didn’t hand it a size-squared feature, linear can’t find it.
• Deep approximation. $Q_\theta$ (or $\pi_\theta$) is a neural network — stacked nonlinear layers that learn their own features from raw-ish state. It can represent the convex impact curve, regime-dependent liquidity, and signal interactions without you specifying the functional form. The price is fragility (more on that all course long).

This is the weld. The perception half — turning order-book state into useful features — is exactly what the deep-learning-for-market-data course taught. The decision half — choosing actions to maximize long-run reward in an MDP — is exactly what the RL course taught. Bolt the network in as the policy/value function inside the MDP and you have deep reinforcement learning: reinforcement learning where the policy and/or value function is represented by a deep neural network function approximator rather than a lookup table. That one sentence is the entire premise of this course.

The two canonical control problems

This course pursues two problems, and it’s worth being precise about why both are control rather than prediction. A prediction problem asks “what will happen?” and your answer doesn’t change the world — guess tomorrow’s return and the market doesn’t care. A control problem asks “what should I do?”, your action changes the state (you move the price, you accumulate inventory), and that change feeds back into your next decision. Self-affecting, sequential, and therefore an MDP. Both of ours qualify.

Optimal execution. You’re handed a parent order — “buy a million shares by the close” — and must slice it into child orders over time. Trade fast and you pay market impact (you shove the price against yourself); trade slow and you pay timing risk (the price wanders while you dawdle). The objective is to minimize implementation shortfall: the gap between the price when the decision was made and the average price you actually achieved. You’re a liquidity taker working off a known quantity against a deadline.

Market making. You quote both sides of the book — a bid and an ask — and earn the spread when someone trades against each. But every fill leaves you holding inventory you never chose: buy filled and you’re long, sell filled and you’re short, and that inventory is exposed to price moves. The objective is to capture spread while keeping inventory near zero. You’re a liquidity provider with no deadline but an ever-present inventory-risk leash.

The analytic baselines every learned agent must beat

Before you let a deep network anywhere near these problems, you must know the answer the math already gives you — because there’s a beautiful closed form for each, derived decades before anyone trained a network on a GPU. These are not strawmen. They are tight, near-optimal solutions under their assumptions, and they cost one line of algebra to evaluate.

Almgren–Chriss (execution, 2000). Minimize a mean–variance blend of cost:

$\min_{\{n_k\}} \; \mathbb{E}[\text{cost}] + \lambda \, \mathrm{Var}[\text{cost}],$

where $\lambda \ge 0$ is your risk aversion. At $\lambda = 0$ you ignore variance and trade a flat, uniform schedule — the spirit of TWAP. As $\lambda$ rises, unsold inventory frightens you, so the optimal schedule front-loads: the remaining-inventory trajectory becomes an exponential decay, steeper for larger $\lambda$. Sweep $\lambda$ and you trace the efficient frontier of execution — the lowest achievable variance for each expected cost. The closed form hands you the whole schedule.

Avellaneda–Stoikov (market making, 2008). Quote around a reservation price that shifts away from the mid as your inventory grows, then post a symmetric optimal half-spread around it. The reservation price is

$r(s, q, t) = s - q \, \gamma \, \sigma^2 (T - t),$

where $s$ is the mid, $q$ your inventory, $\gamma$ risk aversion, $\sigma$ volatility, and $T-t$ time remaining. Long inventory ($q>0$) pushes your reservation price below the mid, so you quote lower on both sides to encourage selling and discourage buying — that’s inventory skew. The optimal total spread is

$\delta^a + \delta^b = \gamma \sigma^2 (T-t) + \frac{2}{\gamma} \ln\!\left(1 + \frac{\gamma}{\kappa}\right),$

with $\kappa$ the order-flow liquidity parameter. One formula gives you both quotes at any inventory.

These two formulas are the spine of the course’s discipline. Here is the creed, and we will repeat it until it’s reflex:

A deep agent that cannot beat — or at least match — these one-line baselines has earned nothing. Compare against the baseline, not against doing nothing.

It is trivially easy to build a deep RL agent that “makes money” or “finishes the order.” That proves nothing, because TWAP also finishes the order and a fixed spread also makes money. The only question that matters is whether your expensive, fragile, hard-to-debug network beats the free, robust, one-line formula. If it doesn’t, you’ve spent a month re-deriving Almgren–Chriss the hard way.

When to use it

Reach for the closed form by default. If your world genuinely matches its assumptions — modest size, linear impact, a single liquid name, no exploitable signal for execution; constant volatility and Poisson order flow for market making — the baseline is not just a benchmark, it’s the production answer. You only graduate to a learned agent once you can point to a specific assumption that reality breaks and show the agent exploits that break to beat the baseline. Until then, the formula wins on cost, robustness, and your sanity.

What deep RL genuinely adds vs the hype

Time for the honest accounting, because the marketing is loud. Point a deep RL agent at the exact world a closed form assumes — linear impact, constant volatility, one asset, no signal — and a correctly built agent will essentially re-derive the closed form. That’s not a triumph; it’s a unit test. If your execution agent fails to recover Almgren–Chriss where Almgren–Chriss is provably optimal, your agent is broken and you found the bug cheaply. Reproducing the known answer is how you earn the right to trust the agent elsewhere.

Deep RL earns its keep precisely where the tidy assumptions crack — which is to say, in reality:

• Nonlinear or uncertain impact. Real impact bends (think square-root law), shifts with regime, and you don’t know its parameters exactly. A network learns the schedule from data without you having to write the impact model down correctly.
• Exploitable signals. A genuine short-horizon prediction (“price likely ticks up next minute”) should speed up a buy and slow a sell. The closed forms have no slot for alpha; deep RL conditions on it natively.
• Regime-dependent liquidity. Spreads widen, depth evaporates, volatility clusters. A policy that reads current conditions and adapts aggression beats any fixed schedule computed at the open.
• Messy, non-smooth constraints. Min/max participation rates, “don’t exceed X% of volume,” dark-pool routing, discrete lot sizes — constraints closed-form mean–variance can’t swallow.
• High-dimensional baskets. Liquidating a correlated basket with cross-impact, or making markets in many names at once, is high-dimensional control with no clean closed form. Home turf.

But every one of those wins comes bundled with a danger, and the rest of the course is a tour of them. The deadly triad — function approximation + bootstrapping + off-policy learning, together — can make value estimates diverge (lesson 2). The sim-to-real gap means an agent that crushes your simulator may bleed live because the sim mismodeled impact (lesson 6). And the agent can overfit its own generated data, learning the quirks of your environment rather than the market. The expressive power that lets deep RL beat the baseline is the same power that lets it fool you spectacularly.

Where this goes next

You now have the frame the whole course hangs on. Tabular RL dies on a continuous order book because the curse of dimensionality leaves its table astronomically large and almost entirely empty. A deep network replaces the table by generalizing across states — that substitution, and only that, is deep RL. Our two targets are control problems, not prediction: optimal execution (take liquidity, beat implementation shortfall) and market making (provide liquidity, capture the spread, tame inventory). Each has a one-line analytic baseline — Almgren–Chriss and Avellaneda–Stoikov — and the creed that governs everything ahead: beat the baseline, not nothing. Deep RL’s genuine value lives where those baselines’ assumptions crack; everywhere else it should merely re-derive them.

The catch is that the moment you swap the table for a network, the math stops being polite. Lesson 2, “Deep Q-Networks & Their Instability on Financial State,” takes the first deep-RL algorithm — DQN — and shows why naively combining a neural Q-function with bootstrapping and off-policy replay (the deadly triad) makes value estimates diverge, and what experience replay and target networks do to drag them back from the brink. The table is gone; now we learn to keep its replacement from blowing up.