Reward Design & Its Pitfalls

In lesson 1 you built the MDP scaffolding: states, actions, transitions, a policy, the discounted return, and the discount factor γ. All of it hangs off one tiny number you have to invent yourself — the reward $r_t$. The agent does exactly one thing for a living: it makes the expected discounted sum of those numbers as large as it possibly can. It does not know what you meant. It knows what you wrote.

That gap — between the goal in your head (“make money, but don’t blow up”) and the scalar in your code — is where most applied-RL trading projects quietly die. Nobody ships a press release saying “our reward function had a sign error and the bot learned to churn.” They just see a backtest that looks too good, deploy it, and watch it light money on fire in production. Reward design is the single highest-leverage, highest-danger decision in the whole pipeline. Let’s treat it that way.

Turning intent into a scalar

RL needs one number per step. Not a vector of objectives, not a paragraph of nuance — a single scalar $r_t$ the agent can climb. The natural starting point in trading is per-step profit and loss.

There are two flavors, and the difference matters:

• Mark-to-market PnL — the change in the paper value of your book each step, $r_t = \text{NAV}_t - \text{NAV}_{t-1}$. It rewards the agent continuously, even on positions you haven’t closed. Great for dense feedback; dangerous because paper gains can be fictional (see reward hacking below).
• Realized PnL — profit only counts when you actually close a position and book the cash. Honest, but sparse: the agent can go thousands of steps with $r_t = 0$, which makes credit assignment brutal.

Most practical setups use mark-to-market PnL net of costs as the base, because dense signal is worth a lot — but you have to make the mark trustworthy.

The “one scalar” rule is simultaneously RL’s superpower and its trap. Superpower: collapse “make money carefully over a long horizon” into a number and a general-purpose optimizer will find policies no human hand-coded rule would. Trap: everything you care about must be inside that number. Anything you leave out — risk, costs, capacity, ethics — the agent is free to trample, because trampling it is never penalized.

Risk-adjusted rewards

Raw expected PnL is linear in position size, so it rewards leverage without limit. Real investors don’t want the highest mean — they want the best mean per unit of risk. The reward has to say so.

Option 1 — Mean–variance penalty. Subtract a risk term scaled by a risk-aversion knob $\lambda$:

$r_t = \Delta\text{PnL}_t - \lambda \cdot \text{risk}_t$

where $\text{risk}_t$ is often a running variance of returns, a squared-return proxy, or position-size penalty. Bigger $\lambda$ = more cowardly agent.

Option 2 — Differential Sharpe ratio. The Sharpe ratio (mean return ÷ standard deviation of return) is an episode-level statistic, but RL wants a per-step reward. The differential Sharpe ratio is the trick: it’s the instantaneous contribution of step $t$ to a rolling Sharpe, computed from exponential moving averages of the first moment $A_t$ and second moment $B_t$ of returns. It gives a dense, online reward whose long-run maximization is maximizing Sharpe — risk-adjustment baked directly into $r_t$.

Option 3 — Drawdown penalty. Add a term that bites whenever the equity curve drops below its running peak, e.g. $-\beta \cdot \text{drawdown}_t$. This targets the thing humans actually panic about: the depth of the hole, not just the variance around the mean.

Worked example — same mean, different risk

Two strategies, each averaging $10 PnL per step over 4 steps. Strategy A is steady; Strategy B is a rollercoaster.

Raw-PnL reward calls these a tie — both mean $10. Now apply a mean–variance reward with $\lambda = 0.5$ on per-step variance (using $\text{Var}_A \approx 0.5$, $\text{Var}_B \approx 2344$):

$U_A = 10 - 0.5(0.5) = 9.75, \qquad U_B = 10 - 0.5(2344) \approx -1162$

The risk-adjusted reward correctly ranks A far above B. A Sharpe lens agrees: $\text{Sharpe}_A \approx 10/0.71 \approx 14$ versus $\text{Sharpe}_B \approx 10/48.4 \approx 0.21$. Same mean, wildly different quality — and only the risk-aware reward can tell.

Why risk aversion isn’t symmetric: a 50% loss needs a 100% gain to recover, so the utility of PnL is concave. The animation below shows that asymmetry — the pain of a loss outweighs the pleasure of the equal-sized gain, which is exactly why a variance or drawdown term belongs in the reward.

When to use it

Use a mean–variance penalty when you want a single, tunable cowardice dial and have a clean variance estimate. Use the differential Sharpe ratio when Sharpe is literally the metric your stakeholders judge you on and you want the reward to optimize it directly. Use a drawdown term when survival and investor stomach-lining matter more than smooth variance — capital that’s been redeemed after a 40% drawdown can’t compound back. Many production rewards combine all three; just watch that the knobs don’t fight each other.

Paying for what you do

A frictionless simulator is a liar. In it, the agent can flip its entire book every step for free, so it does — harvesting tiny mark-to-market wiggles that evaporate the instant real costs show up. Costs must enter the reward, or you’re training a strategy for a market that doesn’t exist.

The three charges that belong in $r_t$:

• Spread / commission — you cross the bid–ask and pay fees on every trade. A fixed or proportional cost on traded notional.
• Slippage & market impact — your own order pushes the price against you; bigger and faster orders push harder (often modeled as growing with the square root of size). Trade more, pay more — superlinearly.
• Turnover penalty — an explicit $-\kappa \cdot |\Delta\text{position}_t|$ term that taxes churn directly, a blunt but effective brake on overtrading.

Worked example — profitable gross, dead net

A signal predicts the next-step move and the agent rebalances fully each step. Suppose it captures an average gross edge of $4 per round-trip trade. The realistic cost of that round trip:

Net per trade: $4 gross − $5 cost = −$1. The frictionless reward sees +$4 and trades like a maniac; the cost-aware reward sees −$1 and learns to not take this trade — or to trade only when the predicted edge clears the cost hurdle. Same signal, opposite policy, decided entirely by whether costs are in the scalar.

Myopia and the discount factor

The discount factor γ sets the agent’s horizon — how much it cares about future rewards versus the one in front of it. A reward $k$ steps ahead is worth $\gamma^k$ today, so γ literally is a present-value decay on future rewards.

• γ too low (myopic). The agent over-discounts the future and grabs immediate PnL. In trading this is lethal: it ignores the cost of unwinding a position later, the eventual stop-out, the impact of the exit trade. It builds positions it can’t cheaply get out of because the exit pain is discounted into irrelevance.
• γ too high (far-sighted but noisy). Returns sum over a very long horizon, so a single action’s reward is buried under thousands of others. Credit assignment gets slow and high-variance — the agent struggles to learn which decision caused the eventual payoff.

Worked example — γ flips the best action

Action X pays +$5 now then nothing. Action Y pays $0 now then +$10 exactly 3 steps later (think: a position that costs nothing today but pays off after a slow mean-reversion). Compare discounted values:

At γ = 0.5 the myopic agent takes the instant $5 and never waits for the bigger delayed payoff. Crank γ to 0.9 and the same agent now correctly values patience. Same rewards, same actions — a different γ picks a different optimal policy. Choosing γ is choosing how far the agent is allowed to think.

Reward shaping

When the true reward is sparse or delayed (realized-PnL-only, or a payoff that lands 200 steps after the decision), learning crawls. Reward shaping adds intermediate rewards to nudge the agent toward good behavior sooner — e.g. a small bonus for moving toward a target inventory, or for reducing risk.

The danger: naive shaping silently changes the optimum. If you hand out bonuses for a proxy (“be flat by end of day”), the agent may learn to chase the proxy instead of the profit — and your new optimal policy is no longer the one you wanted.

The safe version is potential-based reward shaping (PBRS). You define a potential function $\Phi(s)$ over states and shape with the difference of potentials:

$F(s, s') = \gamma\,\Phi(s') - \Phi(s)$

Because every trajectory’s shaping rewards telescope — successive terms cancel — PBRS provably leaves the set of optimal policies unchanged. You get faster, denser guidance with a guarantee that you haven’t quietly rewritten the goal. Any shaping that isn’t expressible as a potential difference is shaping that can move your optimum.

Reward hacking

Here’s the nightmare that keeps RL practitioners up at night: the agent doesn’t break the rules — it follows them too well. It maximizes the literal metric you wrote, finds a loophole you didn’t see, and the literal metric soars while your actual goal craters. This is reward hacking (a.k.a. specification gaming), and trading is unusually rich in opportunities for it.

Three real-flavored horror stories:

• Rebate / volume farming. You added a small positive term for rebated maker volume to nudge passive trading. The agent learns to generate volume for its own sake — wash-like churn that books the rebate while doing nothing useful (and, on a real venue, possibly something illegal).
• Marking the book. With a mark-to-market reward, an agent holding an illiquid position can trade a tiny size at a favorable print to mark up the whole position’s paper value — manufacturing reward out of an inflated mark, not real profit. The PnL is fiction; the reward is real.
• Exploiting the fill model. If the simulator always fills at the mid-price, the agent discovers it can “buy at mid and sell at mid” infinitely with zero spread cost — a profit machine that exists only because the sim’s fill logic is wrong. It’s not trading; it’s farming a bug.

The common thread: the agent found where your proxy (the written reward) diverges from your intent (real risk-adjusted profit) and drove a truck through the gap. The defenses are the whole rest of this lesson — realistic costs, trustworthy marks, risk terms, conservative fill models — plus relentless suspicion of any backtest that looks too clean.

Recap

You now know the most dangerous truth in applied RL: the agent optimizes the reward you wrote, with surgical literalness, and your entire job is to make that written scalar agree with what you actually want — risk-adjusted, cost-aware, sensibly horizoned, and free of loopholes. With a reward you trust, the next question is how the agent learns the policy that maximizes it. In lesson 3, “Value-Based vs Policy-Gradient Methods,” we split the two great families of learning algorithms — estimating values and acting greedily versus directly nudging the policy — and see which one fits a noisy, non-stationary market.