Policy Gradients & Actor–Critic for Continuous Control

Lesson 2 left you with a DQN — a value network that scores every action and trades the one with the highest score. That works beautifully when the actions are a tidy little menu: buy a lot, buy a little, hold. But a real quoting or sizing decision isn’t a menu. Your half-spread is a continuous number. Your child-order size is a continuous number. Your bid offset from the mid is a continuous number. The moment the action becomes a knob you can turn anywhere on a dial, “take the max over all actions” stops being a quick lookup and becomes its own optimization problem you’d have to solve inside every single decision. That’s a non-starter on a trading clock.

So we change the question. Instead of learning how good is each action and then maximizing, we learn the policy directly — a network that takes the state and just outputs the action (or a distribution over actions). No max, no menu, no inner optimization. This family — policy-gradient and actor–critic methods — is what runs essentially every modern continuous-control RL system, and it’s the toolkit lessons 4 and 5 will point at execution and market making. This lesson builds it from the ground up: the raw gradient, why it’s catastrophically noisy in markets, and the three tricks (baselines, critics, trust regions) that beat the noise into submission.

Why learn the policy directly

Picture two ways to decide where to eat. The value way: rate every restaurant in town from 1 to 10, then go to the highest. The policy way: skip the ratings and just learn the habit “when it’s late and I’m tired, go to the noodle place.” The value way needs you to enumerate and score every option. The policy way maps the situation straight to an action. When options are few, scoring is fine. When options are infinite — every possible spread, every possible size — you want the habit.

Formally, a policy is $\pi(a \mid s; \theta)$: a function, with learnable parameters $\theta$ (the weights of a neural net), that gives the probability of taking action $a$ in state $s$. For a discrete action set it’s a softmax over the choices. For a continuous action it’s typically a Gaussian — the network outputs a mean $\mu(s)$ and a standard deviation $\sigma(s)$, and you sample your spread or size from $\mathcal{N}(\mu, \sigma^2)$. The mean is “what I think I should do”; the spread $\sigma$ is “how much I’m still exploring.” Training nudges $\theta$ so good actions get more probability mass.

This is exactly the shape a quoting or sizing agent wants. Ask it for a half-spread and it hands you a number — and, for free, a distribution over numbers, which is your built-in exploration and your built-in measure of confidence. No argmax, no discretizing your spread into clumsy buckets, no inner optimization on the critical path.

When to use it

Reach for a policy method whenever the action is continuous (a price offset, a participation rate, a hedge ratio, a dollar size) or when you genuinely want a stochastic policy — a randomized quoting strategy that’s harder for adversaries to predict and front-run. Stick with a value method (DQN and friends) when the action set is small and discrete and you mostly want the cleanest possible “best action” lookup. Most execution and market-making agents in production are policy-based for exactly the continuous-action reason.

REINFORCE and the policy-gradient theorem

Here’s the beautiful idea at the core of all of this. You want to maximize $J(\theta)$, your expected return — the average total reward your policy collects. You’d love to do gradient ascent: nudge $\theta$ in the direction that raises $J$. But the return depends on $\theta$ through the actions you sampled, and through the environment’s response — how do you differentiate through a coin flip and a market?

The policy-gradient theorem is the trick that makes it work. It says the gradient of expected return has a clean, sampleable form:

$\nabla_\theta J(\theta) = \mathbb{E}\big[\, \nabla_\theta \log \pi(a \mid s; \theta) \cdot G \,\big]$

where $G$ is the return — the total (discounted) reward from this action onward, estimated by just rolling out the episode (Monte Carlo). Stare at it for a second, because the intuition is everything. The term $\nabla_\theta \log \pi(a\mid s)$ points in the direction of $\theta$-space that makes action $a$ more probable. Multiply it by $G$. If the return was big and positive, you take a big step toward making that action more likely. If the return was negative, $G$ flips the sign and you step away, making it less likely. Push up the probability of actions that led to good outcomes; push down the ones that led to bad outcomes. That’s REINFORCE, the original (1992, Williams) policy-gradient algorithm, in one sentence.

A worked gradient step

Toy market. State: the asset looks cheap. Your policy is a softmax over two actions, buy and hold, and right now it assigns $\pi(\text{buy}) = 0.6$, $\pi(\text{hold}) = 0.4$. You sample — you happen to buy. The trade works out: the episode return is $G = +5$.

REINFORCE updates $\theta$ by $\alpha \cdot \nabla_\theta \log \pi(\text{buy}\mid s) \cdot G$ with, say, learning rate $\alpha = 0.1$. The $\nabla_\theta \log \pi(\text{buy})$ term is positive in the direction that raises $\pi(\text{buy})$, and $G = +5$ is a fat positive multiplier, so the step firmly increases the probability of buying when the asset looks cheap. After the update you might see $\pi(\text{buy})$ tick up from 0.60 to, say, 0.63. Sample hold instead and earn $G = -3$, and the same machinery would have lowered $\pi(\text{hold})$. Repeat over millions of episodes and the probability mass flows toward whatever actually pays.

Variance reduction: baselines and advantage

The fix is almost suspiciously simple. The problem with REINFORCE is that it judges an action by its raw return $G$ — but $G = +5$ tells you nothing unless you know what was normal for that state. If every action from this state earns about +5, then +5 is unremarkable and you shouldn’t credit this action for it. So subtract off a baseline $b(s)$ — a reference level for the state — and update on the difference:

$\nabla_\theta J(\theta) = \mathbb{E}\big[\, \nabla_\theta \log \pi(a \mid s;\theta) \cdot \big(G - b(s)\big) \,\big]$

The magic: because $b(s)$ depends only on the state, not on the action you chose, subtracting it does not bias the gradient at all — it’s provably the same expected gradient. But it slashes the variance, because now you’re updating on “how much better than the state’s baseline was this,” a far more stable signal than the raw return swinging with the whole market.

What’s the best baseline? The value function $V(s)$ — the expected return from this state under the current policy, i.e. exactly “what’s normal here.” Plug $V(s)$ in as the baseline and the multiplier $G - V(s)$ becomes the celebrated advantage:

$A(s,a) = Q(s,a) - V(s)$

Read it in plain English: $Q(s,a)$ is the value of taking action $a$ here; $V(s)$ is the average value of the state over all actions. Their difference is “how much better than average was this particular action?” Positive advantage → above-average action → raise its probability. Negative → below-average → lower it. The advantage is the single most important quantity in modern policy-gradient RL, and almost every algorithm you’ll meet is some clever way of estimating it.

In practice you don’t have clean $Q$ and $V$; you estimate the advantage from data. GAE (Generalized Advantage Estimation) is the standard recipe: in one line, it’s an exponentially-weighted average of multi-step TD errors $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$, with a knob $\lambda \in [0,1]$ that smoothly trades bias for variance — small $\lambda$ leans on the bootstrap (low variance, some bias), large $\lambda$ leans on the long Monte-Carlo return (low bias, more variance).

A worked advantage

State: cheap asset. The critic estimates the state value at $V(s) = +2$ — on average, decent things happen from here. You sample buy and the realized return is $G = +5$. Then the advantage is $A = G - V(s) = 5 - 2 = +3$: buying was 3 better than average, so REINFORCE-with-baseline pushes $\pi(\text{buy})$ up — but only by the surprise portion, +3, not the full +5. Now suppose instead the lucky rally gave $G = +5$ for an action that’s normally mediocre in a state where $V(s) = +6$. The advantage is $5 - 6 = -1$: even though the raw return looks great, the action underperformed the state, so its probability is correctly nudged down. That’s the baseline saving you from crediting an action for luck that belonged to the state.

Actor–critic: the marriage

Now combine the two networks. The actor is your policy $\pi(a\mid s;\theta)$ — it proposes actions. The critic is a value network $V(s; w)$ — it scores how good states are, and so supplies the baseline/advantage the actor needs. They train together: the actor acts, the critic watches and learns to predict returns, and the critic’s estimate becomes the low-variance teaching signal for the actor.

The decisive upgrade over plain REINFORCE is timing. REINFORCE has to wait until the episode ends to compute the Monte-Carlo return $G$ — only then does it know how to update. Actor–critic doesn’t wait. After every single step, the critic produces a bootstrapped advantage from the TD error $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$, and the actor updates immediately. You get a credit signal each step instead of once per episode, dramatically lower variance (the bootstrap replaces the noisy full return), and you can learn in long-running, never-quite-ending environments like a market that’s open all day.

Think of it as an apprentice and a coach. The actor (apprentice) tries a quote. The critic (coach) instantly says “that was a bit above average for this situation” — without waiting to see how the whole day ends. The apprentice adjusts on the spot. A2C (Advantage Actor–Critic) is the clean synchronous version of this loop; A3C is the older asynchronous variant that runs many actors in parallel to decorrelate their experience. The agent–environment loop below is this loop — read the agent box as the actor, and imagine a critic sitting beside it scoring every reward as it comes back.

PPO — the workhorse

Actor–critic still has a landmine: step size. Policy gradients are on-policy — the data you collect was generated by the current policy, and it’s only valid for a small region around it. Take one over-eager gradient step, and the new policy can lurch far from the old one into a region where all your collected data is misleading. The policy collapses, starts generating garbage experience, and trains on its own garbage. One bad large update can wreck the whole run. Early fixes (TRPO, Trust Region Policy Optimization) enforced a hard constraint on how far the policy could move per update — correct, but heavy, with a constrained-optimization step that’s a pain to implement.

PPO (Proximal Policy Optimization) is the beloved, dead-simple answer that took over the field. It keeps each update inside a trust region not by a hard constraint but by clipping. Define the probability ratio $r(\theta) = \pi_\theta(a\mid s) / \pi_{\text{old}}(a\mid s)$ — how much more (or less) likely the new policy makes the action you actually took, versus the old policy that collected the data. PPO maximizes a clipped surrogate objective:

$L(\theta) = \mathbb{E}\Big[\, \min\big(r(\theta)\,A,\; \mathrm{clip}(r(\theta),\, 1-\epsilon,\, 1+\epsilon)\,A\big) \,\Big]$

The clip (with $\epsilon$ around 0.1–0.2) is the whole idea. When the advantage $A$ is positive, you want to raise that action’s probability — but the $\min$ with the clipped term caps how much you’re rewarded for pushing the ratio past $1+\epsilon$, so there’s no incentive to leap. When $A$ is negative you want to lower it, and the clip stops you from over-correcting below $1-\epsilon$. Either way, the objective flattens out once the policy has moved “enough,” gently fencing each update inside a trust region. The result: stable on-policy learning that’s simple to implement and reasonably sample-efficient — which is why PPO is the default first thing you reach for in trading control, and the algorithm lessons 4 and 5 assume under the hood.

Beyond PPO sits the off-policy continuous-control family, which buys sample efficiency by reusing old data from a replay buffer — at the cost of more delicate stability. DDPG (Deterministic Policy Gradient) learns a deterministic actor plus a Q-critic; TD3 patches DDPG’s overestimation bugs with twin critics and delayed updates (echoing the Double-DQN fix from lesson 2); SAC (Soft Actor–Critic) adds an entropy bonus that explicitly rewards keeping the policy stochastic, which makes it both sample-efficient and unusually robust. The headline trade-off: on-policy PPO is stable and forgiving but data-hungry; off-policy DDPG/TD3/SAC are sample-efficient but fussier to stabilize.

Exploration that costs real money + offline RL

Here’s the deepest difference between trading RL and the benchmarks these algorithms were born on. In a video game, exploration is free: send Mario off a cliff a thousand times to learn the level, and you’ve lost nothing but compute. In markets, every exploratory action pays real money — every off-distribution quote you post, every “let’s see what happens” trade, crosses a real spread, eats real impact, and can move the price against you (flip on the impact toggle in the loop above to watch a single trade tax its own reward). On-policy exploration, where you must act in the live market to gather the data you learn from, is therefore genuinely, brutally expensive. You’re paying tuition in slippage.

That cost reframes the whole problem and motivates offline (batch) RL: learn a policy from a fixed log of historical data — your firm’s millions of past trades and quotes — without ever interacting with the live market during training. No exploratory bleeding; you mine the experience you already paid for. The catch is distributional shift: the logged data only covers states-and-actions the old behavior policy actually visited, so if your learned policy wants to do something the logs never tried, the value estimates for that action are pure, confident hallucination. Off-policy value methods are notorious for over-estimating exactly these unseen actions and then chasing them off a cliff.

The fix is a family of conservative / behavior-regularized offline algorithms — CQL (Conservative Q-Learning), BCQ, IQL and kin — that all share one instinct: stay close to what the data supports. They penalize value estimates for out-of-distribution actions or constrain the learned policy to resemble the logging policy, deliberately trading away some upside for the safety of not extrapolating into the unknown. In trading, where extrapolating confidently into an unseen action can mean a very expensive surprise, that conservatism isn’t timidity — it’s survival.

When to use it: value vs policy vs actor–critic

The decision guide that ties this course’s three algorithm lessons together:

The reusable comparison to keep in your head, across the families:

Recap and checkpoint

You’ve climbed the policy-based ladder: from why trading control wants to learn the policy directly (continuous spreads and sizes break the argmax), through REINFORCE and its variance problem, to the trio of fixes — baselines/advantage, the actor–critic loop, and PPO’s clipped trust region — and finally the real-money cost of exploration that pushes you toward offline RL. Build the map, then run the mixed checkpoint.

Where this goes next

You now hold the full policy-based toolkit: learn $\pi(a\mid s;\theta)$ directly so continuous spreads and sizes come straight out of the network; tame REINFORCE’s variance with a baseline and the advantage; let a critic feed the actor a low-variance, per-step signal; fence each update with PPO’s clip; and respect that in markets, exploration is paid for in real slippage — which is what makes offline RL and conservative methods matter. Crucially, you also have the decision guide for when each family is the right call.

Lesson 4, “Deep RL for Optimal Execution,” points this machinery at the first real problem: slicing a large parent order to minimize implementation shortfall. You’ll see why a PPO or actor–critic agent with a continuous participation-rate action is the natural fit, how the advantage signal maps onto the impact-versus-timing-risk trade-off you met in the prerequisite course, and how the exploration-cost worries from this lesson force an honest, simulator-and-offline-data-aware training setup. Same algorithms — now earning their keep on a live execution clock.