Natural Experiments & Difference-in-Differences

You have learned that a backtest measures association, that allocation is a causal bet, and that confounders fake edges that no train/test split can catch. So the obvious next question is: where do you ever get to run a real experiment in markets? You can’t randomise which stocks join the S&P 500. You can’t coin-flip a short-sale ban onto half the tape. Randomised controlled trials — the gold standard of causal inference — seem impossibly out of reach.

Except the world runs them for you, constantly, and forgets to tell you. An index committee shoves a stock across a mechanical cutoff. A regulator bans short-selling on one list of names and not another. An exchange halves the tick size on a pilot set of stocks. Each of these is an assignment to treatment that you did not engineer and the firm did not choose — and that is exactly what makes it a near-experiment. This lesson teaches you to recognise these gifts, to measure their effect cleanly with event studies and difference-in-differences, and to interrogate the one assumption — parallel trends — that the whole method silently rests on.

Natural experiments: when the world randomises for you

Analogy. Imagine a school that admits every child taller than exactly 120 cm into the morning class and everyone shorter into the afternoon class. A child measuring 120.1 cm and one measuring 119.9 cm are, for all practical purposes, the same child — a millimetre is noise. Yet a hard rule throws them into different rooms. If the morning class later scores higher on a test, you can credibly blame the class, not the children, because near the cutoff the two groups were interchangeable. The world’s arbitrary line did the randomising you couldn’t.

Definition. A natural (or quasi-) experiment is a situation where assignment to treatment is determined by some external process — a rule, a shock, an accident of timing — that is plausibly independent of the outcome’s other determinants, as if a researcher had randomised it. The closer assignment is to genuinely random (conditional on what you can observe), the closer the quasi-experiment approximates a true RCT, and the more credibly the post-event difference between treated and untreated is a causal effect rather than a confounded one.

The canonical finance specimens:

Worked example. The Russell indices are reconstituted once a year purely on end-of-May market-cap rank. Suppose two near-identical mid-cap firms sit at ranks 998 and 1002. Firm A (rank 998) stays in the Russell 1000; Firm B (rank 1002) drops into the Russell 2000, where it now commands a far larger index weight because the 2000 holds smaller companies. Passive Russell 2000 funds must buy Firm B; passive Russell 1000 funds must sell it. Nothing about Firm B’s business changed between rank 998 and 1002 — a sliver of market cap, well inside measurement noise, flipped its membership. So any divergence in their prices over the following weeks is attributable to the demand shock of reclassification, not to fundamentals. That is a causal estimate handed to you for free.

When to use it

Reach for a natural experiment whenever you suspect a confounder you cannot measure or control away — the situations the previous lessons warned you about. Instead of trying to adjust for every back-door path, you find a slice of reality where treatment was assigned for reasons unrelated to those paths, and let the world’s randomisation do the work a regression control never could. The price you pay is generality: a clean reconstitution estimate is about reconstitutions, not about every demand shock everywhere.

Event studies done right: abnormal returns and CAR

Analogy. A doctor doesn’t judge a fever by reading the thermometer once; she compares it to your baseline temperature. 38.5°C means nothing until you know you normally run 36.6°C — the abnormal part is the 1.9°C above what’s expected for you. An event study does the same to a stock: it builds the stock’s “normal temperature” from a quiet pre-event period, then measures how far the event-window return runs above or below that baseline. The deviation is the diagnosis.

Definition. Pick an estimation window (e.g. 120 trading days, well before the event) and fit a market model: $R_{i,t} = \alpha_i + \beta_i R_{m,t} + \varepsilon_{i,t}$, where $R_{i,t}$ is the stock’s return, $R_{m,t}$ the market’s, and $\hat\alpha_i, \hat\beta_i$ the fitted intercept and beta. Then for each day $t$ in the event window, the abnormal return is

$AR_{i,t} = R_{i,t} - \left(\hat\alpha_i + \hat\beta_i\, R_{m,t}\right).$

The cumulative abnormal return sums these over the window: $CAR_i = \sum_{t} AR_{i,t}$. If the event had no effect, the $AR$s scatter around zero and the $CAR$ stays flat; a genuine effect shows up as a $CAR$ that steps off zero and stays there.

Worked example. A stock is added to the S&P 500. From its estimation window we fit $\hat\alpha = 0.01\%$ per day and $\hat\beta = 1.2$. Over the three-day event window we observe its returns and the market’s:

The stock rose 4.0% raw over three days — but 1.1% of that was just the market dragging it up (its beta of 1.2 on positive market days). Strip out the expected piece and the inclusion effect is a $CAR$ of about +2.89%: the abnormal jump concentrates on the announcement day, then fades to near zero by t = +2, exactly the demand-shock signature index-inclusion studies famously report.

When to use it

Use an event study when you have a dated, identifiable shock to a specific set of names and a sane benchmark for “normal.” It is the workhorse for inclusion/deletion effects, M&A announcements, regulatory dates, and earnings surprises. It is not the tool when the event hits everything at once (no clean benchmark return) or when the date is fuzzy and anticipation smears the window — there you need a control group that absorbs the common shock, which is the whole point of the next section.

Difference-in-differences: subtract the common trend

Analogy. Two neighbouring farms have different soil, so their yields are never equal — that is a fixed, unfair head start you cannot wish away. This year one farm installs irrigation; the other doesn’t. A drought hits both. If you only looked at the irrigated farm’s before-vs-after, you’d blame the drought on the irrigation. Instead you measure how much each farm’s yield changed, and subtract: the irrigated farm fell less than the dry one, and that gap — the difference of the two differences — is the irrigation effect, cleaned of both the permanent soil gap and the shared drought.

Definition. With a treated group and a control group, each observed before and after treatment, the DiD estimator is

$\widehat{DiD} = \big(\bar Y_{\text{treated, after}} - \bar Y_{\text{treated, before}}\big) - \big(\bar Y_{\text{control, after}} - \bar Y_{\text{control, before}}\big).$

The first difference (treated, after minus before) contains the treatment effect plus any common time trend plus the treated group’s fixed level. The second difference (control, after minus before) contains the common time trend plus the control group’s fixed level. Subtracting cancels the common time trend and the time-invariant group differences in one move, leaving the treatment effect — if the two groups would have trended together absent treatment.

Worked example. A short-sale ban is imposed on a list of bank stocks (treated). We compare their average bid-ask spread (a cost, so lower is better) to a matched set of unaffected financials (control), before and after the ban:

Read it two equivalent ways. Rows: treated spreads rose +6.0 bps, control spreads rose +2.0 bps, so $DiD = 6.0 - 2.0 = +4.0$ bps. Columns: the treated-minus-control gap was +2.0 before and +6.0 after, so $DiD = 6.0 - 2.0 = +4.0$ bps. Either way, the ban widened spreads by an estimated 4.0 bps beyond whatever was happening to all financials. Notice what got subtracted away: the treated group always had wider spreads (the +2.0 fixed gap, a time-invariant difference), and both groups widened by 2.0 bps from the market-wide stress (the common trend). DiD removed both and kept only the ban’s marginal damage.

When to use it

Use DiD when a shock hits one identifiable group and not another, you can observe both before and after, and you have a defensible control that shares the treated group’s environment. It shines exactly where a plain event study struggles: a market-wide shock that contaminates the event window gets differenced out by the control. The catch — the entire validity of the method — lives in one assumption you must defend, not assume.

The parallel-trends assumption: the load-bearing wall

Analogy. Two cars are driving down a highway, one always 10 metres ahead of the other but both holding the same speed — a constant gap. You hit the brakes on the rear car. Did the gap change because of your brakes, or because the front car was already accelerating away? DiD assumes the front car (control) would have kept the same speed, so any change in the gap is your braking. If the front car was secretly speeding up the whole time — non-parallel trends — you’ll blame your brakes for a gap that was widening anyway. The constant 10-metre offset never mattered; the speeds matching is everything.

Definition. Parallel trends states that in the absence of treatment, the average outcome for the treated and control groups would have followed the same trajectory — equal changes, not equal levels. The control group’s observed before-to-after change is then a valid counterfactual for what the treated group would have done untreated. You can never observe this counterfactual directly (the treated group was treated), so you stress the assumption indirectly: pre-trend tests (did the two groups move in parallel in the periods before treatment?) and placebo periods (run DiD on a fake “treatment” date when nothing happened — it should return ≈ zero).

Worked example — a violated trend. Reconsider the short-sale ban, but now suppose the banned banks were already deteriorating faster than the controls before the ban — a divergent pre-trend. The honest data:

The naive DiD uses only Pre-1 and After: $DiD = (14.0 - 8.0) - (8.0 - 6.0) = 6.0 - 2.0 = +4.0$ bps — the same +4.0 we got before. But look at the pre-trend: between Pre-2 and Pre-1 the gap already widened from 0.0 to +2.0 bps with no ban in sight. The treated group was on a divergent path. If that pre-existing drift of +2.0 bps per period simply continued, the gap would have reached +4.0 after the ban anyway, with no ban effect at all. So the true causal effect is closer to $6.0 - 4.0 = +2.0$ bps, not +4.0. The naive DiD double-counts the pre-trend and overstates the ban’s damage by 100%. Same arithmetic, wrong answer — because the load-bearing wall (parallel trends) had a crack the 2×2 table couldn’t see.

When to use it

Always test parallel trends before you trust a DiD — never report a DiD estimate without showing the pre-trends and, where possible, a placebo. If the groups visibly diverged before treatment, either fix it (better-matched controls, group-specific trends, a synthetic control built to match the pre-trend) or downgrade your claim from “this is the effect” to “this is an upper/lower bound.” The discipline is identical to the rest of the course: state the counterfactual, then attack it before your P&L does.

Putting it to work in trading

Analogy. A natural experiment in trading is a free sample at a supermarket: it tells you, honestly, what that product tastes like — but it doesn’t tell you whether you’ll like the whole cuisine. You can use a reconstitution to learn precisely how much a forced-buy demand shock pushes price, or use Reg SHO to learn precisely how short-sale constraints move volatility. What you can’t do is assume the next, differently-shaped shock behaves the same. The sample is clean; the extrapolation is on you.

Worked example — turning the design into a trade. Combine the two tools: use an event study to size the reconstitution demand shock, and a DiD to clean it of the market move. Suppose your event-study CAR on additions is reliably +2.9% concentrated on the announcement-to-effective window, while a matched control of near-miss stocks (ranked just outside the cutoff) shows +0.4% over the same days — a common small-cap rally. The DiD-cleaned, attributable price-pressure effect is $2.9\% - 0.4\% = +2.5\%$. If it reliably reverses afterward (price pressure unwinds once index funds finish buying), the trade is to provide liquidity into the forced flow and fade the overshoot. That is a causal hypothesis — “the move is mechanical pressure, not information, so it mean-reverts” — earned from a quasi-experiment, not data-mined from a backtest.

The honest limits. Quasi-experimental edges come with a bill:

When to use it

Lead with natural experiments when the causal stakes are high and confounding is the real risk — sizing a new strategy, validating a microstructure assumption, or settling a “does demand really move price?” debate that a backtest can only beg. Treat each one as a precise but local measurement: bank the internal validity, discount heavily for external validity and small N, and never let a single clean experiment talk you into a position bigger than its handful of events can support.

Recap

Markets don’t run randomised trials, but they hand you quasi-experiments by the fistful: index reconstitutions whose mechanical cutoffs make membership as-if random, regulatory shocks that treat one list of names and spare another, microstructure pilots that randomise the rules themselves. To measure the effect cleanly you have two tools. An event study subtracts the expected return (from a pre-event market model) from the actual one, summing abnormal returns into a CAR that shows whether — and when — the event moved the price. A difference-in-differences goes further when a market-wide shock would contaminate the window: it subtracts the control group’s change from the treated group’s change, cancelling both the permanent level gap and the common time trend in one stroke. But DiD stands or falls on parallel trends — that absent treatment the two groups would have moved together — an assumption you stress with pre-trend tests and placebos but can never fully prove. Used well, these designs defend against the unmeasured confounder that no regression can reach; used naively, they double-count a pre-trend or misattribute a market-wide move. The edge they buy is sharp, credible, and local — bank the internal validity, stay humble about external validity, and never lever a one-off event past what its small N can carry.