Markets as a Markov Decision Process

Supervised machine learning is a fortune-teller. You hand it a feature vector, it hands back a number — tomorrow’s return, the probability of a fill, next hour’s volatility — and then it shrugs and goes home. It never tells you how much to buy, when to stop, or what to do once the order you just placed has moved the very price you were trying to predict. Forecasting is a one-shot answer to a question nobody actually asked. Trading is a sequence of decisions unfolding in time, each one changing the board you play the next one on.

Reinforcement learning (RL) is the discipline built for exactly that shape: an agent takes actions, the world answers with a reward and a new situation, and the agent learns a rule — a policy — that maximizes reward over the long run, not just the next tick. That long-run framing is also where our recurring villain returns. Across the machine-learning course the enemy was overfitting: a model that memorized noise and called it signal. It is still the enemy here — only now it wears a craftier disguise, because an RL agent gets to choose its own training experience, and a backtest that looks like free money is usually an agent that has overfit a simulator nobody should have trusted.

The vocabulary of sequential decisions

RL has a small, precise vocabulary, and almost every confusion in the field comes from blurring two of these terms together. Define them once, carefully, and the rest of the course is just engineering.

• State $s_t$ — everything the agent observes at time $t$ that is relevant to its decision. In trading: current price, your inventory (position), recent returns, order-book imbalance, time-of-day, remaining order to execute.
• Action $a_t$ — what the agent does. Buy/hold/sell, a target position, a quoted bid–ask pair, or how many child orders to release this minute.
• Reward $r_t$ — the scalar feedback after acting. Often a risk-adjusted, cost-inclusive P&L increment — and designing it correctly is so dangerous it gets its own lesson next.
• Transition / dynamics $P(s_{t+1} \mid s_t, a_t)$ — the rule of the world: the probability of landing in the next state given the current state and action. In markets this is the price process plus the impact of your own trade. You almost never know it; you sample from it.
• Policy $\pi(a \mid s)$ — the agent’s strategy: a (possibly randomized) mapping from states to actions. This is the object you are actually learning.
• Trajectory / episode — one full play-through: $s_0, a_0, r_0, s_1, a_1, r_1, \dots$ A trading episode might be one day, or the lifetime of unwinding one parent order.
• Return $G_t$ — the cumulative (discounted) reward from time $t$ onward. The thing the agent maximizes. Not a price forecast — an ugly coincidence of vocabulary we will be very careful about.

A market is a Markov decision process

Stack those pieces together and you have a Markov decision process (MDP) — the formal object RL studies. Mechanically it’s a loop: the agent observes the state, its policy picks an action, the environment returns a reward and the next state, and around it goes, forever.

The “Markov” in MDP is a promise about the state. A process has the Markov property when the next state depends only on the current state and action — not on the entire history of how you got here:

$P(s_{t+1} \mid s_t, a_t, s_{t-1}, a_{t-1}, \dots) = P(s_{t+1} \mid s_t, a_t).$

This is not a law of nature; it is a design constraint on what you put in your state. If “last traded price” doesn’t summarize everything relevant — and in markets it never does — then the price alone is not Markov, and an agent that conditions only on it is flying blind. The fix is not to bolt on memory of the past; it is to enrich the state until it is sufficient: add your inventory (so the agent knows its own exposure), recent returns (momentum/mean-reversion context), and order-book features (imbalance, depth, spread). A well-engineered state makes the Markov property true enough to be useful.

The chain above shows the same idea on regimes: the token decides its next hop using only where it is now, yet the long-run share of time in each regime settles to a fixed mix. Memoryless transitions, stable statistics — that is the Markov property earning its keep.

Worked example — a tiny 2-state trading MDP

Picture a toy market with two regimes as the relevant state: Calm and Volatile. Each step the agent picks a position size, but first the world transitions. From Calm, the dynamics are:

$P(\text{Calm}\mid \text{Calm}) = 0.8,\qquad P(\text{Volatile}\mid \text{Calm}) = 0.2.$

Suppose your policy in Calm goes long, and the reward (P&L increment) is $r=+2$ if you stay in Calm and $r=-5$ if you tip into Volatile while still long. The expected one-step reward of being long in Calm is

$0.8 \times (+2) + 0.2 \times (-5) = 1.6 - 1.0 = +0.6.$

Notice what made that number: the transition probabilities, not a forecast of price. The agent isn’t predicting the next return — it’s weighing actions against the dynamics of the world, which is exactly what an MDP forces you to do.

The return and the discount factor γ

The agent doesn’t maximize the next reward — it maximizes the return, the cumulative discounted reward from now on:

$G_t = r_t + \gamma\, r_{t+1} + \gamma^2 r_{t+2} + \dots = \sum_{k=0}^{\infty} \gamma^k\, r_{t+k}.$

The discount factor $\gamma \in [0,1)$ decides how far the agent looks ahead. At $\gamma = 0$ the agent is purely myopic — it cares only about $r_t$ and would happily blow up next step for a penny now. As $\gamma \to 1$ it becomes far-sighted, treating a reward ten steps out almost like a reward today. Discounting also keeps the infinite sum finite and encodes a preference for sooner-rather-than-later profit.

Worked example — discounting a reward stream

Take a short episode with rewards $r_0 = 10,\; r_1 = 0,\; r_2 = -4,\; r_3 = 6$ and a discount of $\gamma = 0.9$. The return from $t=0$ is:

$G_0 = 10 + 0.9(0) + 0.9^2(-4) + 0.9^3(6).$

Compute the weights: $0.9^2 = 0.81$ and $0.9^3 = 0.729$. So

$G_0 = 10 + 0 + 0.81(-4) + 0.729(6) = 10 - 3.24 + 4.374 = 11.134.$

Now redo it myopically with $\gamma = 0$: $G_0 = 10$ flat — the agent never even sees the $-4$ shock or the $+6$ recovery. Same reward stream, completely different sense of “good,” set entirely by one knob.

Prediction vs control — the deep difference

Here is the chasm between the previous course and this one. Supervised learning is prediction; RL is control.

In prediction, your data is (roughly) i.i.d., you estimate a fixed mapping to a label that exists independently of you, and — crucially — your prediction does not change the world. Guessing tomorrow’s rain doesn’t alter the weather. The label was going to be whatever it was going to be.

In control, you choose actions, and those actions change which states you visit next — they change the future data distribution your agent will learn from. In markets this is not a metaphor: a large buy moves the price (market impact), consumes liquidity, and shifts the order book the next decision is conditioned on. The environment is not a passive label generator; it’s a counterparty that reacts to you.

Flip on “Market impact” in the loop island above and you can watch this directly: the agent’s own trades push the price against it, shrinking every subsequent reward. That feedback — your action moves the environment — is the one feature a video game and a market share and a forecasting model can never represent.

The policy and the value function — a preview

The policy $\pi(a \mid s)$ is the whole game: a (possibly stochastic) rule that, given a state, hands you an action — or a probability distribution over actions, so the agent can explore rather than always exploiting its current best guess. Everything in this course is, ultimately, a method for finding a good $\pi$.

But how do you know a policy is good without playing infinitely many episodes? You estimate value. The state-value $V^\pi(s)$ is the expected return from state $s$ if you follow policy $\pi$ forever after; the action-value $Q^\pi(s,a)$ is the expected return from taking action $a$ in state $s$ and following $\pi$ thereafter. These two functions are the load-bearing beams of nearly every RL algorithm — they let you compare actions without enumerating the future — and lesson 3 builds them properly.

Recap

You now speak MDP: you can frame a trading problem as states, actions, rewards, transitions and a policy, you know why a price-only state is rarely Markov, and you can see the one feature — your action moves the environment — that makes this control rather than forecasting. But notice how much we leaned on the reward simply existing. Next, in Reward Design & Its Pitfalls, we put the whole weight of the agent on that single scalar — and watch what happens when “make money carefully” is even slightly mistranslated into math. It’s where most RL trading projects quietly die, and where our old villain, overfitting, finds its craftiest disguise yet: an agent that hacks the reward you wrote instead of the goal you meant.