Survivorship and Selection Bias

Picture a wall of glossy fund brochures, each one bragging about its returns. They all look pretty good. Here’s the unsettling part: that’s not because the funds are good — it’s because the bad ones aren’t on the wall. They closed, merged, got quietly buried, and stopped sending brochures. What you’re staring at is a graveyard with the headstones removed and only the trophies left standing. This lesson is about the most dangerous bias for anyone evaluating a track record, a fund, a backtest, or a “guru”: you judge the world by the survivors you can see, and forget the corpses you can’t. Learn to count the corpses and half the investment industry’s marketing stops working on you.

Guess before reading

You met availability and base rates back in Availability, Representativeness and Base Rates — the habit of judging by the vivid examples in front of you instead of the full population. Survivorship and selection bias are that same disease, weaponised by an industry that controls which examples reach you. The cure is the same: always ask, “what’s the full sample, including the parts I can’t see?”

Survivorship bias — armour the holes you can’t see

The cleanest story about this bias comes from a war, not a market.

In World War II, the U.S. military studied the bombers that came back from missions to decide where to add armour. The planes returned peppered with bullet holes — concentrated on the wings, the fuselage, the tail. The obvious move: reinforce the spots with the most holes. The statistician Abraham Wald said the exact opposite. Armour the places with no holes — the engines, the cockpit. Why? Because the data only included planes that survived. A plane shot in the engine didn’t limp home to be measured; it crashed and never entered the dataset. The undamaged-looking areas weren’t safe — they were the fatal spots, and the evidence of their danger had been removed from the sample by the simple fact of dying.

That is survivorship bias: drawing conclusions from a sample that only contains the things that “made it,” while the failures have silently dropped out. The survivors aren’t representative of the original group — they’re a filtered, flattering slice of it. And the failures don’t just go missing at random; they go missing precisely because they failed, which is exactly the information you needed.

In investing, the dead planes are dead funds. Mutual funds and ETFs that perform badly get liquidated (shut down and the money returned) or merged into a better-performing sibling fund. When that happens, their lousy track record usually disappears from the database — or gets absorbed into the survivor’s. So when you look at “the funds available today and how they did over the last ten years,” you are looking at survivors. The duds have been buried.

Worked example — the brochure average vs the true average

Imagine a fund family launched 100 funds ten years ago. Over the decade:

• 64 funds survived. They averaged 9.0% a year. These are the ones still being advertised.
• 36 funds were closed or merged. They were the laggards — say they limped along at roughly 1.0% a year before being put down.

The brochure proudly quotes the survivor average: 9.0%. But the average a real investor faced, picking blind from all 100 funds back then, weights every fund that existed:

$\text{true average} = \frac{(64 \times 9.0\%) + (36 \times 1.0\%)}{100} = \frac{576\% + 36\%}{100} = 6.1\%$

The honest, whole-cohort number is 6.1%, not 9.0% — the survivors’ average overstates reality by nearly 3 percentage points in this stylised case. Real-world studies (Brown, Goetzmann, Ibbotson and Ross in 1992; Malkiel in 1995) put the typical survivorship inflation at a more modest but still serious 1 to 1.5 percentage points per year. Compounded over decades, even 1.5pp a year is the difference between a comfortable retirement and an anxious one.

When it matters

Survivorship bias bites hardest whenever you compare yourself to, or buy into, a group that has already been filtered by success: choosing an actively managed fund from today’s menu, reading “average hedge-fund returns,” copying the habits of rich investors, or trusting any “here’s what the winners did” study. The longer the time window, the more failures have been buried, and the more flattering — and misleading — the survivor average becomes.

Selection and backfill bias — the database is rigged before you arrive

Survivorship is about who exits the sample. Selection bias is the broader sibling: it’s about who enters the sample in the first place — and, just as importantly, on whose terms. If the things that get into your dataset were chosen in a way that’s correlated with success, the dataset is flattering before you’ve done a single calculation.

Three of these games are everywhere in track records.

Backfill bias. Many hedge-fund and managed-strategy databases are voluntary — managers choose whether to report. When a fund finally decides to join a database, it often gets to backfill its past returns into the record. Guess which funds choose to join and backfill? The ones with a good history to show off. A fund that started rough and stayed rough never joins, or never backfills. So the database’s early years are stuffed with retroactively added winners, and the historical “average hedge fund” looks better than the average fund actually performed. Studies of these databases have found backfilled returns running several percentage points a year higher than the same funds’ later, live, reported returns.

Incubation bias. A fund company quietly launches, say, ten small “incubator” funds with little or no outside money. It waits a year or two, then kills the losers and takes the winners public — opening them to investors with a shiny one- or two-year track record already attached. The investing public only ever sees the survivors of an in-house tournament, presented as if that one fund had simply been brilliant from birth. The eight dead incubators are never mentioned. It’s survivorship bias, but manufactured on purpose as a marketing pipeline.

Cherry-picked start dates. The same fund can be a genius or a dunce depending on where you start the clock. “Up 300% since March 2009!” sounds incredible — until you notice March 2009 was the exact bottom of the financial crisis, so everything tripled from there. Pick the start date that flatters the story, and any record can be made to gleam. This is selection bias applied to time rather than to funds.

Worked example — backfill inflation

Suppose a hedge-fund database reports its “average fund returned 12% a year.” You dig in and find that for each fund, the first three years in the record were backfilled when the fund chose to join, and the later years were live. Split them:

The backfilled early years are running 7 percentage points hotter than the live years — not because hedge funds get worse over time, but because only funds with great early years chose to join and backfill them. The “12% average” is a blend of a real number (the live ~9%) and a selection artefact (the backfilled ~16%). The number you should believe is the live, out-of-sample one: roughly 9%, before you even subtract the fund’s notoriously high fees.

Applying it: funds, backtests and gurus

Now put the toolkit to work on the three places these biases do the most damage to real money.

Funds and the menu you’re shown

When you scan “the best-performing funds of the last ten years,” remember you’re reading a survivor list. The funds that existed ten years ago but performed badly enough to be closed aren’t on it. The right comparison isn’t “this fund beat its surviving peers” — it’s “this fund beat the full original universe, including the ones that died, and beat a cheap index fund after fees.” Most active funds, measured honestly against the whole starting cohort net of costs, trail a simple index over 10–15 years — exactly the base rate you should anchor on (the Availability and Base Rates lesson again).

Backtests — the lab results that fall apart in the wild

A backtest is when someone runs a trading strategy on historical data to show how it “would have” performed. Backtests are the single most abused number in finance, because they’re easy to rig — usually without the rigger even realising it. Two killers:

• No out-of-sample test. If you try thousands of strategy variations on the same history and keep the one that scored best, you haven’t discovered a winning strategy — you’ve discovered the one that best fit the noise of that particular past. This is overfitting: the financial twin of cherry-picking. The only honest proof is out-of-sample performance: does it still work on data the strategy was never tuned on, ideally a future period after it was published? Most backtested miracles quietly die out of sample.
• No transaction costs. A backtest that ignores trading fees, the bid-ask spread, taxes, and slippage (the price moving against you as you buy) can turn a real-world loser into a paper winner. A strategy that trades constantly looks brilliant on costless paper and bleeds out once the real bill arrives.

Gurus — the one who was right once

Every crash and every boom mints a guru: the pundit who “called it.” Before you hand them your attention or money, run the base-rate math you learned earlier. If ten thousand commentators each make a bold market call, pure chance guarantees that hundreds of them will be spectacularly “right” — and the media, by availability bias, will interview exactly those few and ignore the thousands who were wrong. Being right once is what randomness produces, not evidence of skill. The guru is just the surviving bomber of a forecasting war you never saw the casualties of.

Sort these track-record claims into the ones that should make you suspicious and the ones that are genuinely credible.

The whole picture

A mixed recap pulling from the whole lesson:

Key Takeaways