Bayes' Rule & Base Rates

Last lesson you met the cast — prior, likelihood, posterior — and watched them combine to update a belief. This lesson promotes that intuition into an actual machine: Bayes’ theorem, written out in full, with every term named and the arithmetic shown to the last digit. Then we use it to expose the single most expensive mistake in quantitative trading: confusing “this signal fires when there’s a real edge” with “when this signal fires, there’s a real edge.” Those two sentences sound like twins. They are not even cousins, and the gap between them is where strategies — and careers — quietly go to die.

Here’s the trap in one breath: a detector that fires on 90% of genuine trading edges can still be wrong two times out of three about whether any given firing is real. Sounds impossible. It’s arithmetic, and by the end of this page you’ll compute it yourself.

Bayes’ theorem, written out in full

The analogy. Think of Bayes’ rule as a currency converter for evidence. You walk in holding a belief in one currency — your prior, what you thought before seeing the data. The data arrives at an exchange rate — the likelihood, how strongly this data favors your hypothesis. The converter hands you back your belief in a new currency — the posterior, what you should think now. The whole formula is just the exchange desk’s receipt.

The formula. For a hypothesis $H$ (say, “today has a real edge”) and observed data $D$ (say, “the signal fired”):

$P(H\mid D)=\frac{P(D\mid H)\,P(H)}{P(D)},\quad P(D)=P(D\mid H)P(H)+P(D\mid \neg H)P(\neg H).$

Name every piece, because confusing them is exactly how the disasters happen:

• $P(H\mid D)$ — the posterior: probability the hypothesis is true given you saw the data. This is what you actually want to know.
• $P(D\mid H)$ — the likelihood: probability of seeing this data if the hypothesis were true. (Your signal’s sensitivity, “fires on 90% of real edges.”)
• $P(H)$ — the prior: probability of the hypothesis before any data. (The base rate of edges.)
• $P(D)$ — the marginal (or evidence): the total probability of seeing this data at all, across every way it could happen. It’s the normalizer that makes the posterior a proper probability.
• $\neg H$ — “not $H$”: every world where the hypothesis is false. $P(D\mid \neg H)$ is your false-positive rate.

The bottom line $P(D) = P(D\mid H)P(H) + P(D\mid \neg H)P(\neg H)$ is doing the heavy lifting. It says: the data can appear two ways — when $H$ is true and when $H$ is false — and you must count both sources. Skip the second term and you’ve already committed the fallacy below.

The inversion fallacy: P(D|H) is not P(H|D)

This is the load-bearing idea of the entire lesson, so we’ll hammer it. The likelihood $P(D\mid H)$ and the posterior $P(H\mid D)$ are different numbers, and swapping them is the inversion fallacy — known in courtrooms as the prosecutor’s fallacy.

The courtroom version. A prosecutor announces: “There’s only a 1-in-a-million chance this DNA matches an innocent person — so the defendant is a million-to-one guilty!” That’s $P(\text{match}\mid \text{innocent})$ being passed off as $P(\text{innocent}\mid \text{match})$. But in a city of ten million, roughly ten innocent people also match. The probability the defendant is the culprit given the match isn’t a million-to-one; with one true culprit and ten innocent matches, it’s closer to 1 in 11. Same evidence, wildly different conclusion — because the prosecutor ignored how many innocent people there were to begin with (the base rate).

The trading version. “My signal fires on 90% of real edges” is $P(\text{fire}\mid \text{edge})$. It tells you how good the detector is. It does not tell you that a firing is 90% likely to be real — that would be $P(\text{edge}\mid \text{fire})$, which also depends on how rare edges are and how often the signal cries wolf on edge-less days. A backtest that brags about hit rate while hiding the base rate is selling you the wrong conditional.

Base rates and natural-frequency reasoning

Why does a rare prior wreck a strong test? Because the test runs against a population, and when real edges are rare, the huge pile of edge-less days generates a flood of false alarms that can drown out the genuine hits — even if each individual day is only rarely a false alarm.

Raw probabilities make this hard to feel. The fix, due to the psychologist Gerd Gigerenzer, is natural-frequency reasoning: stop juggling percentages and instead imagine a concrete crowd — “of 1000 signals…” — and count heads. The same Bayes’ rule falls out, but now it’s just sorting people into boxes, and the base rate is impossible to forget because it’s literally the size of a box.

How natural frequencies rebuild Bayes

Walk a population of 1000 through the test:

1. Split by the prior. A base rate of 5% means 50 of the 1000 are real edges; 950 are not. (There’s the base rate, front and center.)
2. Apply the likelihood to the real ones. Sensitivity 90% means 45 of the 50 real edges fire.
3. Apply the false-positive rate to the rest. A 10% false-positive rate means 95 of the 950 edge-less days also fire.
4. Read the posterior off the flagged box. Total flagged = 45 + 95 = 140. Of those, only 45 are real. So P(edge given fire) = 45 of 140.

No formula anxiety, no flipped conditional — just four counts. That’s the whole method.

Fully worked example: the “90% accurate” signal

Let’s nail the numbers from the natural-frequency walk above, with the arithmetic laid bare.

The setup.

• Base rate of a real edge: $P(\text{edge}) = 5\% = 0.05$.
• Sensitivity: $P(\text{fire}\mid \text{edge}) = 90\% = 0.90$.
• False-positive rate: $P(\text{fire}\mid \neg\text{edge}) = 10\% = 0.10$.

Count out of 1000 signals.

The posterior.

$P(\text{edge}\mid \text{fire}) = \frac{45}{140} \approx 0.321 = 32\%.$

Read that and let it sting: a signal everyone would call “90% accurate” is, when it fires, wrong about the edge roughly two times out of three. The 90% sensitivity was real; it just answered a question — “how many real edges do I catch?” — that nobody’s P&L was asking. The question your money asks — “given a firing, is it real?” — comes back 32%, because the 950 edge-less days threw off 95 false alarms, and 95 is more than double the 45 true ones.

Plugging into the formula confirms the count-based answer exactly:

$P(\text{edge}\mid \text{fire}) = \frac{P(\text{fire}\mid \text{edge})P(\text{edge})}{P(\text{fire}\mid \text{edge})P(\text{edge}) + P(\text{fire}\mid \neg\text{edge})P(\neg\text{edge})} = \frac{0.90 \times 0.05}{0.90 \times 0.05 + 0.10 \times 0.95} = \frac{0.045}{0.045 + 0.095} = \frac{0.045}{0.140} \approx 0.321.$

Watch the posterior collapse

The grid below is this example, made of 1000 little squares. Brand-blue squares are real edges the signal caught (true positives); the villainous accent squares are false alarms (false positives). The badge up top is P(real given flagged) — the posterior — computed live as TP / (TP + FP).

Now do the experiment that teaches the whole lesson: drag the base-rate slider DOWN toward 1% and watch the posterior collapse even though you never touched the sensitivity or the false-positive rate. The detector didn’t get worse — the world just got rarer, and a rarer prior means almost every firing is a false alarm. Then drag the base rate up and watch the posterior climb. Same test, opposite verdicts, decided entirely by the base rate.

Same test, different prior: the posterior swings

The posterior is not a property of the test — it’s a property of the test times the world. Change only the base rate and the conclusion can flip from “ignore it” to “act on it.” Hold the signal fixed (sensitivity 90%, false-positive rate 10%) and compare two worlds: edges rare (5%) versus edges common (30%).

Same detector, untouched. In the rare world a firing is 32% likely to be real — mostly noise, ignore it. In the common world the identical firing is 79% likely to be real — worth acting on. The conclusion more than doubled, and nothing about the test changed. This is why “is the signal good?” is an incomplete question: a signal is only as trustworthy as the base rate it’s hunting in. Hunt for rare edges and even a sharp signal mostly lies; hunt where edges are plentiful and the same signal becomes reliable.

P(D|H) or P(H|D)? Sort the statements

The fastest way to inoculate yourself against the prosecutor’s fallacy is to drill which conditional a sentence is actually stating. Each statement below is either a likelihood P(D given H) — a property of the test, “given the truth, how does the data behave?” — or a posterior P(H given D) — what you actually want, “given the data, what’s the truth?”

Misconceptions that cost real money

Misconception 1 — ignoring the base rate entirely. The headline error: quoting sensitivity (“90% accurate!”) and treating it as the chance a firing is real. It forgets the 950 edge-less days that throw off false alarms. Always demand the prior.

Misconception 2 — treating “test accuracy” as the answer. “Accuracy” is a slippery word. A signal that never fires is “95% accurate” when edges are 5% rare — it’s right on every quiet day. Accuracy bundles together two conditionals you need separated (sensitivity and specificity) and still tells you nothing about P(edge given fire) without the base rate. Ask for sensitivity, false-positive rate, and the prior — never a single “accuracy” number.

Misconception 3 — double-counting correlated signals. Bayes’ rule multiplies in independent evidence. Stack three signals that all secretly read the same momentum factor and you’ll multiply their likelihoods as if you had three independent witnesses — when really you have one witness shouting three times. The posterior balloons toward false confidence. Genuinely independent confirmations are gold; correlated ones are the same evidence wearing disguises.

Misconception 4 — “the p-value is the probability the strategy is junk.” A backtest p-value is $P(\text{data this good}\mid \text{no real edge})$ — a likelihood, conditioned on the null. It is not $P(\text{no real edge}\mid \text{data this good})$, the posterior you actually want. Flipping them is the inversion fallacy in a lab coat. To get the posterior you’d need the prior odds that a randomly tried strategy has an edge — usually grim — which is exactly why a “significant” backtest so often dies live.

How to not get fooled

A three-part field kit for the next time someone waves a flashy signal at you:

1. Always ask for the base rate. “How often is this true before the signal?” If they can’t tell you the prior, they can’t tell you the posterior — full stop. A hit rate without a base rate is half a fraction.
2. Reason in counts, not percentages. Translate every claim into “of 1000…” and physically sort the crowd into true positives, false positives, and the rest. The base rate becomes a box you can’t misplace, and the prosecutor’s fallacy becomes visibly absurd.
3. Beware multiple testing and data mining. Test 1000 random signals at a 5%-significance threshold and roughly 50 will look “significant” by pure luck — a pile of false positives manufactured by the search itself. The more signals you try, the lower the prior that any survivor is real, and the harder the posterior collapses. The cure is honest accounting: track how many hypotheses you tried, demand out-of-sample confirmation, and treat a single dredged-up backtest with the suspicion it deserves.

Putting it together

Bayes’ theorem, $P(H\mid D)=P(D\mid H)P(H)/P(D)$, converts the conditional you can measure — the likelihood $P(D\mid H)$, your signal’s sensitivity — into the one you actually need — the posterior $P(H\mid D)$, the chance a firing is real. The bridge between them is the base rate $P(H)$, and forgetting it is the inversion fallacy (a.k.a. the prosecutor’s fallacy): “90% of edges fire” is not “a firing is 90% an edge.” When edges are rare, the marginal $P(D)$ fills up with false alarms, and even a 90%-sensitive signal can be right only 32% of the time — wrong about the edge two tries in three. The antidote is natural-frequency reasoning: sort 1000 events into true and false positives, where the base rate is a box you can’t lose. Always ask for the prior, always reason in counts, and always treat a flashy backtest — born from a search over many signals — as the false positive it most likely is.

Next up, we leave detectors behind and turn Bayes into a continuous dial. When the evidence arrives bit by bit and the hypotheses live on a spectrum rather than a true/false switch, we’ll meet conjugate priors — the elegant trick that lets you update a belief with a pencil instead of an integral — and watch a prior distribution sharpen into a posterior as the data rolls in.