Reinforcement Learning for Trading

Supervised machine learning answers one question: given this, predict that. It is a brilliant fortune-teller and a useless decision-maker. It will forecast tomorrow’s return and never tell you how much to buy, when to stop, or what to do after the trade you just placed nudged the price. Trading is not a prediction problem — it is a sequential decision problem, played against an adversary that reacts to your every move. The discipline built for exactly that shape of problem is reinforcement learning (RL): an agent takes actions, the world responds with rewards and a new situation, and the agent learns a policy — a rule mapping every state to an action — that maximizes reward over the long run, not just the next tick.

This course takes the RL toolkit and points it at markets, honestly. RL has genuine claims here that supervised learning cannot make: it optimizes the thing you actually care about (cumulative risk-adjusted profit, including the cost of trading), it handles the feedback loop where your action moves the environment, and it naturally expresses problems — optimal execution, market making, dynamic sizing — that are control problems, not forecasting ones. But RL also amplifies every danger from the machine-learning course before it. The data is non-stationary, the simulator is always wrong in ways that matter, the agent will gleefully hack any reward you misspecify, and an RL backtest is even easier to overfit than a supervised one because the agent gets to choose its own training distribution. The thread running through every lesson is the same enemy as before — overfitting — wearing a new and craftier disguise.

Here is the arc, from the framing to the frontier:

• Markets as a Markov decision process — the formal language of RL (state, action, reward, transition, policy, return, discounting), why trading fits it and forecasting does not, and the single most important difference from a video game: in markets, the action moves the environment.
• Reward design & its pitfalls — the hardest and most dangerous part of applied RL: turning “make money carefully” into a scalar. Risk-adjusted rewards, the cost of trading, reward shaping, myopia versus the discount factor, and the reward-hacking horror stories that follow from getting it wrong.
• Value-based vs policy-gradient methods — the two great families: learning what each situation is worth (Q-learning, DQN) versus learning the policy directly (policy gradients, PPO, actor–critic). When each fits, why continuous actions push you toward policy methods, and the exploration-versus-exploitation problem under transaction costs.
• Optimal execution as an RL problem — the canonical win: slicing a large order to balance market impact against timing risk. How this relates to the closed-form Almgren–Chriss solution, what RL adds (and what it merely re-derives), and how to read a learned execution policy.
• Market-making agents — quoting both sides of the book while managing inventory: the inventory-versus-spread trade-off, the Avellaneda–Stoikov skeleton, adverse selection, and why a market-making reward is a minefield of perverse incentives.
• The sim-to-real gap — the lesson that humbles everything above: why an RL agent that prints money in your backtest loses it live. Non-stationarity, an imperfect simulator, the agent overfitting its own environment, and the deployment discipline that keeps you honest.

By the end you will see trading the way a control engineer sees a thermostat or a self-driving car — as a loop of observe, act, and adapt — while keeping the quant’s hard-won paranoia about a backtest that looks too good. A graded final exam runs the whole discipline back at you at the end, one locked question at a time.