Deep Q-Networks & Their Instability on Financial State

In 2015, a single algorithm learned to play forty-nine Atari games from raw pixels, beating humans on most of them, using one set of hyperparameters and no game-specific tricks. That algorithm was the Deep Q-Network (DQN), and it kicked off the whole deep-RL boom. The recipe looks almost insultingly simple: take ordinary Q-learning, which you already met as a table of values, and replace the table with a neural network. Done. Publish. Collect citations.

Except it is not done, and the reason is the whole point of this lesson. The naive version — “just swap in a neural net” — diverges. Value estimates spiral off to infinity and the agent learns nothing. DQN works only because of two specific engineering fixes that defuse a structural landmine in the math. That landmine has a name, the deadly triad, and on financial state it is armed and twitchy in ways Space Invaders never was. So this lesson is really two stories: how value-based deep RL is supposed to work, and why it keeps blowing up in your face the moment you point it at an order book.

From Q-table to Q-network (DQN)

Start with what you already know. Q-learning learns an action-value function $Q(s,a)$: the expected discounted return from taking action $a$ in state $s$ and then acting optimally forever after. The whole edifice rests on the Bellman optimality equation, which says the value of a state-action pair equals the immediate reward plus the discounted value of the best thing you can do next:

$Q^*(s,a) = \mathbb{E}\big[\, r + \gamma \max_{a'} Q^*(s', a') \,\big]$

Tabular Q-learning turns that fixed-point equation into an update rule. After observing a transition $(s, a, r, s')$, you nudge your current estimate toward the bootstrapped target:

$Q(s,a) \leftarrow Q(s,a) + \alpha \big[\, \underbrace{r + \gamma \max_{a'} Q(s',a')}_{\text{TD target}} - Q(s,a) \,\big]$

The bracket is the temporal-difference (TD) error: the gap between what you now think the pair is worth and what your bootstrapped estimate says it should be worth. The learning rate $\alpha$ controls how big a step you take to close it. For a small, discrete world — a few dozen states — a lookup table holds one number per cell and this converges to $Q^*$.

Markets do not give you a few dozen states. The state of an execution or market-making agent is a vector of real numbers — inventory, time remaining, spread, queue position, volatility, a fistful of order-book and signal features — living in a continuous space you could never enumerate. A table is hopeless: you would need infinitely many cells, and you’d never visit the same one twice to learn anything. So you replace the table with a function approximator: a neural network $Q(s,a;\theta)$ with weights $\theta$ that generalizes across nearby states. Feed it a state, it outputs one Q-value per discrete action. States it has never seen get a sensible value because the network interpolates from similar ones it has.

Training swaps the tabular nudge for gradient descent on a loss. The loss is the squared TD error — your prediction versus the bootstrapped target:

$L(\theta) = \big(\, r + \gamma \max_{a'} Q(s',a';\theta) - Q(s,a;\theta) \,\big)^2$

You differentiate $L$ with respect to $\theta$, take a gradient step, repeat. Same TD logic as the table; the table’s single-cell update has just become a global parameter update that shifts the value of every state at once. That generalization is the superpower — and, as you’ll see in two sections, the loaded gun.

A worked TD update

Make it concrete with one step. Your execution agent is in state $s$ (say, 60% of the slice left, halfway to the deadline) and takes action $a$ = “trade 50% of the remaining slice.” Suppose:

• The network currently predicts $Q(s,a;\theta) = 4.0$ (in reward units — negated cost, so higher is better).
• You execute the action and observe immediate reward $r = -1.0$ (you paid some impact this step).
• You land in $s'$, where the network’s outputs for the available next actions are $\{2.0,\ 3.0,\ 2.5\}$, so $\max_{a'} Q(s',a';\theta) = 3.0$.
• Discount $\gamma = 0.9$.

The TD target is $r + \gamma \max_{a'} Q(s',a') = -1.0 + 0.9 \times 3.0 = 1.7$. The TD error is $1.7 - 4.0 = -2.3$ — the network was too optimistic about this pair by 2.3. The squared loss is $(-2.3)^2 = 5.29$. A gradient step on $\theta$ pushes $Q(s,a;\theta)$ down from 4.0 toward 1.7, and — because it’s a shared network — drags the values of similar states down a little too. Do this across millions of transitions and the network converges toward satisfying Bellman’s equation everywhere. That’s the dream, anyway.

Why naive DQN diverges: the deadly triad

Here is the lesson’s beating heart. The naive swap blows up because of an interaction of three ingredients, christened the deadly triad by Sutton and Barto. Each is harmless, even desirable, on its own. Pair any two and you’re usually fine. Combine all three and your value estimates can diverge to infinity — not because of a bug, but because the math genuinely loses its guarantee.

The three ingredients:

1. Function approximation. You represent $Q$ with a parametric model (the neural net) instead of an exact table. Updating one state’s value spills over onto others, because they share weights. Generalization, the feature, is also leakage.
2. Bootstrapping. Your update target is built from your own current estimate of the next state’s value — that $\max_{a'} Q(s',a')$ term — rather than from a real, fully-observed return. You learn a guess from a guess.
3. Off-policy learning. You learn about the optimal (greedy) policy while the data you train on was generated by a different, exploratory behaviour policy. The distribution of states you update on is not the distribution the target policy would actually visit.

The failure mechanism, in one breath: bootstrapping makes the target depend on $\theta$. Function approximation means a step taken to fix one state’s value perturbs the next state’s value too — which is part of that very target. Off-policy data means nothing reliably corrects the states that are getting over- or under-valued, because you’re not visiting them in proportion to how much they matter to the target policy. The result is a positive feedback loop: an over-estimate inflates the bootstrap target, which inflates the estimate, which inflates the target. Round and round, up and to the right, until the numbers are nonsense. This is not the same as ordinary supervised-learning overfitting — it’s a dynamics problem in the training process itself, where the labels you regress toward are generated by the model you’re training.

When it bites

The triad is most dangerous when bootstrapped targets are noisy or when the behaviour and target policies diverge sharply — both of which describe trading to a tee. Long horizons (many bootstrap steps compounding), high discount factors $\gamma$ near 1 (the future weighs heavily, so target errors propagate far), and aggressive off-policy reuse of stale data all crank up the risk. Keep this list in mind: every item on it is worse in markets than in a video game.

The two fixes that made DQN work

DQN didn’t repeal the deadly triad — you can’t, not while keeping all three ingredients, and DQN wants all three (it’s a neural net, it bootstraps, it’s off-policy). Instead it engineers around the instability with two now-standard tricks. Each one neutralizes a different part of the feedback loop.

Fix 1 — Experience replay. Don’t train on transitions in the order they happen. Store every transition $(s, a, r, s')$ in a large buffer (the replay buffer, often a million transitions), and each training step sample a random minibatch from it. Two benefits. First, it breaks temporal correlation: consecutive transitions in any episode are highly correlated, and training a network on a stream of correlated samples is like fitting a regression to data you fed in sorted order — the gradients are biased and the network forgets the past as it tracks the present. Random sampling de-correlates the minibatch back toward i.i.d., which is what gradient descent assumes. Second, it gives data efficiency: each expensive transition gets reused many times across many updates instead of being seen once and discarded.

Fix 2 — Target network. This one strikes directly at bootstrapping. Keep two copies of the network: the online network $Q(s,a;\theta)$ you’re actively training, and a target network $Q(s,a;\theta^-)$ that is a periodically-frozen snapshot of it. Build the bootstrap target with the frozen weights:

$L(\theta) = \big(\, r + \gamma \max_{a'} Q(s',a';\theta^-) - Q(s,a;\theta) \,\big)^2$

Every few thousand steps you copy $\theta$ into $\theta^-$ (or blend slowly, $\theta^- \leftarrow \tau\theta + (1-\tau)\theta^-$ for small $\tau$ — a “soft” update). Between copies, $\theta^-$ is constant, so the target you regress toward holds still. You are no longer chasing a target that jumps every time you update — you’re fitting toward a stable label for a while, then refreshing it. That cuts the positive-feedback loop where an estimate inflates its own target inflates the estimate, because for thousands of steps the target simply does not move with $\theta$.

Two refinements you should know by name, because they patch real biases in vanilla DQN:

• Double DQN. The $\max$ operator in the target is a systematic over-estimator: with noisy Q-values, $\max_{a'} Q(s',a')$ tends to pick whichever action got lucky with positive noise, so the target is biased high. Double DQN decouples the two roles of that max — it selects the next action with the online network ($\arg\max_{a'} Q(s',a';\theta)$) but evaluates that action with the target network ($Q(s',\arg\max;\theta^-)$). Same action wouldn’t be argmax under both unless it’s genuinely good, so the optimistic bias shrinks. In noisy domains (read: markets) this matters a lot.
• Dueling DQN. Split the network’s head into two streams — a scalar state value $V(s)$ and a per-action advantage $A(s,a)$ — then recombine them into $Q(s,a) = V(s) + (A(s,a) - \text{mean}_a A)$. The intuition: in many states which action you pick barely matters (the value is dominated by the situation, not the choice), and learning $V(s)$ once is far more efficient than learning a near-identical $Q$ for every action. Helpful when most actions in a state are roughly equivalent — common in execution, where most micro-decisions hardly move the outcome.

Why financial state breaks DQN harder than a video game

Everything above was domain-agnostic. Now the bad news, specific to trading. DQN’s two fixes lean on assumptions that Atari satisfies beautifully and markets violate flagrantly. Each violation re-arms a part of the triad the fixes were supposed to defuse.

Non-stationarity. Atari is a fixed program: the rules of Breakout in 2015 are the rules of Breakout today, so a transition you stored an hour ago is still a perfectly valid sample of the same environment. Markets are not fixed. The data-generating process drifts — volatility regimes shift, liquidity dries up, a competitor’s algo changes, a regulation lands. This guts experience replay at its root: replay assumes the buffer is a representative sample of a roughly stationary distribution. When the MDP itself moves, your million stored transitions are partly a museum of a market that no longer exists. You train the agent to be excellent at yesterday’s regime, and the staleness that was harmless in Atari becomes actively misleading.

Low signal-to-noise. In Breakout the reward is clean and causal — break a brick, get points, and the value of a state is genuinely learnable because the future is largely determined by your actions. In markets the next-state value is dominated by noise: price moves are mostly unpredictable, so the bootstrapped target $r + \gamma\max_{a'}Q(s',a')$ is mostly noise with a faint signal buried inside. Bootstrapping off a noisy target is the triad’s bootstrapping leg on hard mode — you’re learning a guess from a guess where the guess is 95% static. The max operator’s overestimation bias (which Double DQN fights) is also worse the noisier your Q-values are, because there’s more positive noise for the max to cherry-pick.

Partial observability. DQN’s whole framework assumes the state $s$ is Markov — that what you observe contains everything you need to predict the future. The true state of a market (everyone’s hidden intentions, the full latent order flow, pending news) is not observable from the order book. What you feed the network is a noisy, incomplete projection. That breaks the Markov assumption underneath the Bellman equation, so even a perfectly trained $Q$ is fitting an approximation to a problem that isn’t quite the one you’re solving. (This is why real trading agents bolt on recurrence or stacked-history features to reconstruct a pseudo-state — a partial patch, not a cure.)

Reward sparsity and scale. Atari rewards are frequent and live on a sane integer scale. Trading rewards can be sparse (P&L realized only at the end of an episode), wildly skewed (long stretches of small gains punctuated by rare large losses), and badly scaled (tiny per-step impact costs versus occasional huge terminal penalties). Gradient descent hates that: the loss is dominated by a few outlier transitions, the network’s value scale drifts, and the careful balance the target network was protecting gets harder to hold.

Discrete actions: DQN’s natural fit and its ceiling

One last structural fact about DQN, and it’s the hinge to the next lesson. DQN’s output layer is one Q-value per action, and the target uses $\max_{a'}$ over those actions. That math only works for a discrete, smallish action set — you need to enumerate every action to take the max. That makes DQN a natural fit for coarse, chunky decisions, and an awkward fit for fine continuous ones.

Where DQN fits cleanly: discretize the decision into a handful of buckets. For execution, “trade 0% / 25% / 50% / 75% / 100% of this slice” is five clean actions — DQN eats that for breakfast. For market making, a small grid of quote offsets (“post my bid 1, 2, or 3 ticks behind mid; same for the ask”) is a modest discrete set DQN can handle. Coarse control, discrete buckets, small max — perfect.

Where it strains: genuinely continuous quoting. A market maker’s real action is a real number — a half-spread of, say, 1.7 ticks, or a precise inventory-dependent skew. To force that into DQN you must discretize, and discretization costs you on both ends. Too few buckets and you can’t express the fine quote the optimal policy wants; too many and the action set explodes, the max becomes expensive, and each action is visited too rarely to learn well. The curse compounds in multiple dimensions: independently quoting bid and ask offsets across a grid multiplies into a combinatorial action space.

A worked discretization

Say the optimal half-spread for the current state is 1.7 ticks. You discretize the half-spread into actions $\{0, 1, 2, 3, 4\}$ ticks — five buckets, one tick apart. The closest DQN can quote is 2 ticks, so you’re 0.3 ticks too wide on every fill — a permanent, structural haircut on your captured spread that no amount of training removes, because the right answer literally isn’t in the action set. Tighten the grid to half-tick resolution $\{0, 0.5, 1.0, \ldots, 4.0\}$ and you can quote 1.5 or 2.0 (still missing 1.7, now by 0.2) — but you’ve doubled the actions. Now do it for the ask side too and the joint action space squares. Add inventory-dependent skew as a third dimension and DQN’s tidy “take the max over actions” turns into a search over a combinatorial grid, with most cells starved of data.

The lesson writes itself: discrete actions are DQN’s home and its ceiling. Fine continuous control wants a method that outputs the real number directly instead of choosing from a menu — which is exactly what policy-gradient and actor–critic methods do, and exactly where the next lesson goes.

When to use it

Reach for DQN when your action space is naturally discrete and small and you want the simplicity and sample-efficiency of a value-based method: coarse execution (a few participation-rate buckets), routing decisions (which venue), or discrete regime switches. Avoid it — or at least don’t force it — when the action is genuinely continuous and resolution matters (precise quoting, fine hedging), or when the action space is high-dimensional. And whatever you choose, remember the triad and the four market-specific stressors above: even on a discrete action set, a noisy non-stationary MDP will test every gram of stability your replay buffer and target network can provide.

Where this goes next

You now have the full value-based picture: the Bellman optimality equation turned into a bootstrapped TD update, the Q-table swapped for a generalizing $Q(s,a;\theta)$, the deadly triad that makes the naive swap diverge, the experience-replay and target-network fixes that tame it (plus Double and Dueling DQN), and the four ways financial state — non-stationarity, noise, partial observability, ugly rewards — re-arms the very instabilities those fixes were built to defuse. The recurring punchline: the tricks that conquered Atari assume a stationary, fully-observed, clean-reward world, and markets are none of those things.

The hard ceiling we ended on is the action space. DQN thinks in a small discrete menu and takes a max over it — wonderful for coarse buckets, hopeless for the real-valued quotes and skews a market maker actually needs. Lesson 3, “Policy Gradients & Actor–Critic for Continuous Control,” throws out the value-table-shaped thinking entirely: instead of scoring every action and taking the max, you parameterize the policy itself and nudge its parameters in the direction of higher return — letting the agent output a continuous half-spread of 1.7 ticks directly, no rounding, no exploding grid. Same MDP, a fundamentally different lever.