Calibration, Bias & Arbitrage

So far you’ve taken the market’s word for it: the price of a YES share is the crowd’s probability, the mechanics move USDC around, and UMA settles who was right. All true. But a probability you can’t score is just a vibe with a dollar sign on it. This lesson does two ruthless things. First, it grades the crowd — a rigorous test of whether prices priced at 30¢ actually come true 30% of the time, and where they reliably don’t. Second, it shows you how to turn those grading errors, plus a quirk of binary contracts, into edge — and the cleanest edge of all, arbitrage, where the profit is locked in before the event even resolves.

You bring two superpowers from earlier courses: Bayesian thinking (priors, base rates, updating) and Kelly (sizing a known edge). Here you’ll learn where the edge comes from in the first place. The punchline: markets are mostly very good, systematically wrong in one famous direction, and patrolled by arbitrageurs so relentless that the one truly free lunch — YES plus NO costing something other than a dollar — is usually gone before you can click.

Calibration — is the crowd’s 30% really a 30%?

Analogy. A weather forecaster says “30% chance of rain” on a hundred different days. You don’t grade any single day — rain or no rain proves nothing about a 30% call. You grade the batch: gather every “30%” day and ask whether it rained on roughly 30 of them. A forecaster who passes this test for every probability level is well-calibrated — their numbers mean what they say. A prediction market is just a forecaster whose forecast is its price, so we grade it the same way.

Definition. A market is well-calibrated if, among all the contracts it priced at $x$, the event later resolved YES about $x$ of the time. The standard tool is the calibration plot (a.k.a. reliability diagram): bucket every historical contract by its price, put the predicted probability (the price) on the x-axis and the realized YES frequency on the y-axis, and plot a point per bucket. Perfect calibration is the 45° diagonal $y = x$ — priced at 0.3, comes true 0.3 of the time. Points below the diagonal mean the market charged more than the event was worth (overpriced); points above mean it charged too little (underpriced).

Worked example. Take every Polymarket contract over a season that traded near 70¢ at some snapshot — say 1,000 of them. Count how many resolved YES. If 712 did, the realized frequency is $712 / 1000 = 0.712$, essentially bang on the 0.70 price: that bucket sits on the diagonal, well-calibrated. Now the 5¢ bucket: of 1,000 contracts priced at 0.05, perfect calibration predicts about 50 YES resolutions. If only 31 actually hit, realized frequency is $31/1000 = 0.031$ — the point lands at $(0.05,\,0.031)$, below the diagonal. The crowd paid 5¢ for something worth 3.1¢. Multiply that gap across thousands of cheap contracts and it’s a real, systematic leak.

Drag the sliders below. The bias slider bends the realized-frequency curve away from the diagonal; the bucket slider inspects one price and reads off what the market charges versus what actually happens, labelling the bucket as overpriced, underpriced, or calibrated.

Calibration vs resolution vs sharpness — being right-on-average isn’t enough

Analogy. A clock stuck at the average temperature is “calibrated” in a useless way: a forecaster who answers “the base rate, 50%” to every question will, across a balanced set of questions, be perfectly calibrated — half its 50% calls come true — while telling you absolutely nothing about any individual question. Calibrated, and worthless.

Definition. Calibration asks whether your stated probabilities match reality on average. Resolution (closely tied to sharpness) asks whether you actually discriminate — whether you push confidently toward 0 and 1 when you know something, instead of huddling near the base rate. The gold standard is a forecaster that is both calibrated and sharp: it makes bold, near-0/near-1 calls and those calls come true at the rate claimed. A market can be beautifully calibrated and still unsharp if it rarely strays from 50/50 — accurate but uninformative.

For our purposes the takeaway is light but important: when you test a market, calibration is necessary but not sufficient. The markets worth trading are the ones that are sharp — confidently far from the base rate — because that’s where a calibration error is large enough in dollar terms to be worth exploiting.

The favorite–longshot bias — the crowd’s most reliable mistake

Analogy. The lottery and the blue-chip stock, side by side. People cheerfully overpay for a $2 ticket with a one-in-300-million jackpot because the dream is worth a premium, and they slightly underpay for the boring near-certainty because boring is, well, boring. Prediction markets inherit the same human tilt: bettors overpay for the thrilling longshot and underpay for the dull favorite.

Definition. The favorite–longshot bias is the best-documented inefficiency in betting and prediction markets: longshots are overpriced — contracts priced cheap (say 5¢) resolve YES less often than 5% — and favorites are slightly underpriced — heavy favorites (say 90¢) win more often than 90%. On the reliability diagram this is the curve sagging below the diagonal at the low end and rising above it at the high end (exactly what the bias slider above produces). Cause: a taste for lottery-like payoffs (people pay a premium for a small chance of a big win), plus the cost and hassle of betting NO on a near-zero longshot, which lets the overpricing persist.

Worked example — the longshot leak. A basket of 5¢ longshots that truly resolve YES only ~3% of the time. Two ways to read the edge:

Buying the cheap YES bleeds 2¢ per contract on average; the classic edge is the other side — systematically buy NO on overpriced longshots (or, equivalently, back the underpriced favorites), harvesting the 2¢ gap the lottery-lovers leave on the table. Tiny per contract, but it’s a repeatable, positive-expectation bet — precisely the kind Kelly was built to size.

Binary arbitrage — the one genuinely free lunch

Analogy. A vending machine that always dispenses a dollar coin when you insert one red token plus one blue token together — guaranteed, every time, no matter what. If the shop next door is selling a red token for 60¢ and a blue token for 38¢, you buy one of each for 98¢, feed the machine, get $1 back, and you’ve made 2¢ with zero uncertainty. You don’t care which color “wins”; the pair is worth a dollar by construction.

Definition. A binary market has YES and NO shares where 1 YES + 1 NO merge to exactly $1 at resolution (one pays $1, the other $0). So the no-arbitrage price is YES + NO = $1. Whenever the market strays:

• If YES + NO < $1, the pair is cheap: buy both, hold to resolution, redeem the pair for $1. Locked profit per set = $1 − (YES + NO).
• If YES + NO > $1, the pair is expensive: mint a fresh set for $1 (deposit a dollar, the protocol hands you 1 YES + 1 NO) and sell both into the market. Locked profit per set = (YES + NO) − $1.

Either way the outcome is irrelevant — you’ve hedged both sides — so the profit is risk-free (before fees). This is true arbitrage, not the favorite–longshot edge: there’s no probability estimate, no being-right-on-average, just a price identity violated.

Worked example. YES is 60¢, NO is 38¢, so YES + NO = 98¢, which is below $1. Buy one of each for 98¢. At resolution exactly one share pays $1 and the other pays $0, so the pair redeems for $1 regardless of who wins. Profit = $1 − 0.98 = 2¢ per set, risk-free, before fees. Buy 10,000 sets for $9,800 and you lock $200 — no opinion about the event required. Now flip it: if YES is 60¢ and NO is 45¢, the sum is $1.05, which is above $1. Mint a set for $1, sell the YES for 60¢ and the NO for 45¢ to collect $1.05, and keep the 5¢ — again risk-free.

Use the bars below. Set the YES and NO prices, watch the sum cross the $1.00 redemption line, and see the strategy flip between buy the pair and mint and sell — with a fee slider that quietly eats the thin ones.

Why arbitrage doesn’t make you rich — and why that’s the point

Analogy. A $20 bill spotted on a busy sidewalk. In theory, free money. In practice, on a crowded street it’s gone in seconds — and if it’s still lying there, it’s probably glued down (a fee, a catch, a reason). Visible, persistent arbitrage in a liquid market is the glued-down twenty: by the time you see it, either it’s already taken or there’s a cost that makes it not worth the bend.

Definition. Real arbitrage profit is eaten by frictions: trading fees, gas (each on-chain action costs money), the bid–ask spread (you don’t trade at the mid; you cross the spread to buy YES and NO), execution risk (prices move while your two legs fill — you might get one side and not the other), and limited depth (only a few hundred dollars sit at the good price; size up and you walk the book into worse fills). A 2¢ gross gap can easily net to zero or less once all of that is paid — which the fee slider in the bars above makes visceral.

And here’s the beautiful part. Arbitrageurs competing to grab these gaps are exactly the force that keeps YES + NO ≈ $1. Every time the sum drifts off a dollar, someone pounces and shoves it back. So the no-arbitrage identity isn’t a lucky coincidence — it’s an enforced equilibrium. And since YES + NO = $1 means the price of YES is one minus the price of NO, that enforcement is precisely why “price = probability” holds in the first place: if a YES priced at $p$ and a NO priced at $1-p$ together always pay exactly $1, then$p$ behaves like a probability — bounded in $[0,1]$, complementary to its NO, and arbitraged flat. The thing you couldn’t earn (the free lunch) is the very thing that makes the market’s prices trustworthy.

Putting it together

You can now grade a market and find an edge. Calibration is the report card: bucket contracts by price, plot realized YES frequency against price, and read off where the market lands on (or off) the 45° diagonal — but remember calibration alone isn’t enough; you want markets that are sharp too. The most reliable off-diagonal pattern is the favorite–longshot bias: cheap longshots are overpriced (back NO) and heavy favorites are underpriced (back YES), a small-but-real, Kelly-sizable edge that varies by venue and gets nibbled by costs. The cleanest edge of all is binary arbitrage: because 1 YES + 1 NO always redeem for $1, a sum below a dollar means buy the pair, a sum above means mint and sell, and the profit is the distance from $1 — risk-free before fees. And the reason that free lunch is usually gone (fees, gas, spread, execution risk, thin depth) is the same reason price = probability: arbitrageurs enforcing YES + NO = $1 are what make the price a trustworthy probability in the first place.

Next up — sizing the bet with Kelly on a binary, and the real risks. You’ve found where edges live; now you’ll size them. We’ll convert your estimated probability versus the market price into a binary-bet edge, feed it through the Kelly criterion (and a sober fractional Kelly, because your edge estimate is noisy and these markets are unforgiving), and then stare hard at the risks that no formula prices: resolution and oracle disputes, illiquidity and slippage when you try to exit, smart-contract and platform risk, and the simple, humbling possibility that the crowd was right and you were wrong.