Instrumental Variables & Regression Discontinuity

The previous lessons handed you a deal: identify the confounders, measure them, condition on them, and the back door slams shut. But what happens when the confounder is something you cannot measure — managerial skill, private information, the unobserved fundamental that drives both your signal and the return? You cannot control for a variable you never see. The back door stays wide open, and no amount of careful regression will close it.

This lesson is about the escape hatch. Instead of blocking the contaminated path, you go around it: you find a lever that nudges the treatment for reasons that have nothing to do with the outcome, and you read the effect off how the outcome responds to the nudge. Two designs share this logic — instrumental variables (IV) and regression discontinuity (RDD) — and both trade the impossible demand “measure every confounder” for a different, often more defensible demand: “find a source of as-good-as-random variation in the treatment.”

The intuition for an instrument

Analogy. Picture a stuck, rusted bolt (the causal effect you want to read) buried under a heap of mud (the confounder). You can’t wipe the mud away — it’s bottomless. So instead you find a long wrench whose handle sticks out of the mud cleanly. You push the handle a known amount and watch how far the bolt turns. As long as the wrench touches only the bolt — and not, say, a hidden second bolt — the bolt’s rotation per unit of push tells you exactly how the bolt responds. The wrench is your instrument: a clean handle on a dirty mechanism. The classic textbook version is using rainfall as an instrument for crop supply when estimating how supply affects price — rain shoves the harvest around for purely meteorological reasons, so it cannot be in cahoots with whatever else moves prices. In experiments, an encouragement or a lottery plays the same role: randomly nudging some people toward a treatment they’re then free to take or refuse.

Definition. An instrument $Z$ for the effect of treatment $X$ on outcome $Y$ is a variable that (i) causes variation in $X$ and (ii) influences $Y$ only through $X$ — never directly and never through any confounder. The instrument injects exogenous variation into the treatment: a slice of $X$ that wiggles for reasons unconnected to the unmeasured common cause. Because that slice is clean, regressing $Y$ on it recovers the causal effect even with the back door open.

Why it isolates exogenous variation. Decompose the treatment into two parts: the part driven by the confounder (dirty) and the part driven by the instrument (clean). Ordinary regression uses all the variation in $X$, so it mixes the dirty part back in and inherits the bias. IV throws away every wiggle of $X$ except the wiggles the instrument is responsible for — and those, by assumption, carry no confounding. You pay for this purity: you use far less of the data’s variation, so IV estimates are noisier. But noisy-and-unbiased beats precise-and-wrong when you’re about to bet capital on the answer.

When to use it

Reach for the instrument idea the moment you suspect an unobservable common cause — the variable you’d love to control for but can never measure (skill, private information, latent demand). If your confounder is measurable, just measure and condition on it; instruments are the tool for when conditioning is off the table. They are also the natural frame for any setting with a quasi-random nudge already baked in: a policy that assigns eligibility by a rule, a lottery, an index rebalance that forces flows.

The two IV conditions

Analogy. Two boxes must both be ticked for the wrench to work. Relevance: the wrench must actually grip the bolt — push the handle and the bolt had better turn, or you’re measuring nothing. You can check this by pushing and watching. Exclusion: the wrench must touch only the bolt, nothing else under the mud. You cannot check this by looking, because the mud hides everything — you can only argue, from how the tool was built, that it physically cannot reach a second bolt.

Definition — the two conditions.

(A third, often-listed condition — independence/exogeneity, that $Z$ is itself as-good-as-randomly assigned — usually travels with exclusion and is defended the same way.)

Worked example — a finance instrument. Suppose you want the causal effect of passive ownership $X$ (the fraction of a stock held by index funds) on the stock’s return volatility $Y$. The trouble: a latent “fundamental quality” confounder plausibly drives both how much passive money holds a name and how volatile it is. You cannot measure quality. So you propose an instrument $Z$ = whether the stock sits just inside the Russell 1000 vs. the Russell 2000 at the annual reconstitution.

• Relevance: crossing the index boundary mechanically forces index funds to buy or dump the name to track their benchmark, so $Z$ powerfully moves passive ownership $X$. Testable, and here the first stage is strong.
• Exclusion: you must argue that landing on one side of the rank cutoff affects volatility only through the resulting change in passive ownership — not through, say, analyst coverage or lending supply that also jump at the boundary. This is a claim about market plumbing, defended with institutional knowledge, and it is exactly where this instrument gets contested.

When to use it

Spell out both conditions, in writing, before you estimate anything. Relevance is a quick empirical check — if the first stage is weak, stop now (see the next section on why). Exclusion is the hard intellectual work: list every alternative path from the instrument to the outcome and argue each one shut. If you cannot articulate why $Z$ has no second route to $Y$, you do not have an instrument; you have a correlate wearing a disguise.

Two-stage least squares (2SLS)

Analogy. Back to the wrench. You push the handle by 2 inches (the instrument moves) and the bolt turns 6 degrees (the outcome moves). But you don’t want degrees-per-inch-of-handle; you want degrees-per-turn-of-the-bolt. So you also measure: that same 2-inch handle push turned the bolt’s shaft by 3 degrees (the first stage). The answer you actually want is $6 \div 3 = 2$ degrees of outcome per degree of treatment. You divided the handle’s effect on the outcome by the handle’s effect on the treatment. That division is the entire trick of IV.

Definition — 2SLS in two stages.

• Stage 1: regress the treatment on the instrument, $X = \pi_0 + \pi_1 Z + \text{noise}$, and keep the fitted values $\hat{X}$. These fitted values are the clean, instrument-driven part of the treatment — the variation that, by exclusion, carries no confounding.
• Stage 2: regress the outcome on those fitted values, $Y = \beta_0 + \beta_1 \hat{X} + \text{noise}$. The slope $\beta_1$ is your IV estimate of the causal effect.

For a single instrument this is algebraically identical to the IV / Wald estimator:

$\hat{\beta}_{\text{IV}} = \frac{\text{Cov}(Z, Y)}{\text{Cov}(Z, X)}$

and, when $Z$ is binary, it simplifies to the reduced form over the first stage — the difference in mean $Y$ between $Z{=}1$ and $Z{=}0$, divided by the difference in mean $X$ between $Z{=}1$ and $Z{=}0$.

Worked numeric example — a Wald estimate from group means. A regulator randomly encourages some retail brokerages to default their clients into a low-cost index portfolio ($Z{=}1$); the rest get no nudge ($Z{=}0$). The treatment $X$ is whether a client actually ends up index-invested; the outcome $Y$ is the client’s annual net return. Encouragement is as-good-as-random (it’s a lottery) and plausibly affects returns only by changing whether clients go passive — so it’s a candidate instrument.

The Wald estimate divides the reduced form by the first stage:

$\hat{\beta}_{\text{IV}} = \frac{6.2\% - 5.0\%}{0.70 - 0.40} = \frac{1.2\%}{0.30} = 4.0\%.$

So going passive causes roughly a 4-percentage-point lift in annual net return — for the clients whose behavior the nudge actually changed. Notice the logic: the nudge moved returns by 1.2%, but it only moved 30% of clients into the treatment, so each mover’s effect must be larger — $1.2\% / 0.30 = 4.0\%$. Dividing by an incomplete first stage scales the diluted reduced-form effect back up to the per-treated effect.

The weak-instrument problem. Look at that denominator. If the encouragement had barely moved anyone — say the first stage were $+0.03$ instead of $+0.30$ — you’d divide by a tiny, noisy number. A first stage near zero makes the IV estimate explode and become severely biased (toward the very OLS bias you were trying to escape), with confidence intervals so wide they’re useless. This is why relevance isn’t a formality: a weak instrument ($F$ well below 10) doesn’t just add noise, it produces confidently wrong answers. Dividing by almost-nothing is how small errors in the numerator become enormous errors in the estimate.

LATE — whose effect did you get? The 4.0% is not the average effect for everyone. It is the Local Average Treatment Effect (LATE): the effect on compliers — the clients who went passive because they were encouraged and would not have otherwise. It says nothing about always-takers (who’d go passive regardless) or never-takers (who refuse no matter what). Different instruments move different compliers, so two valid IVs for the “same” treatment can legitimately return different numbers. IV answers “what is the effect on the people this particular lever moves?” — not “what is the effect on everyone.”

When to use it

Use 2SLS when you have a credible instrument and want a single causal number with honest standard errors; use the bare Wald ratio when the instrument is binary and you just want the intuition or a quick estimate from group means. Always lead with the first-stage F-statistic — it is the price of admission. And state your estimate as a LATE: name the compliers (“the effect among clients whose allocation the nudge actually changed”), because pretending it’s the average-for-everyone is how IV results get over-generalized into trades that don’t exist for most of the universe.

Finance instruments are scarce and contested

Analogy. Finding a clean instrument in markets is like finding a wire in a junction box that connects to exactly one thing. In a tidy circuit that’s easy. In a financial market — a junction box where every wire is soldered to a dozen others — you tug what looks like an isolated wire and three lights you didn’t expect flicker on. That flicker is exclusion failing: your instrument just revealed a second path to the outcome.

The honest catalogue. A few instruments recur in financial research. None is uncontested; for each, the exclusion worry is the live debate.

Worked example — exclusion sneaking a second path. Take index reweighting as an instrument for the effect of passive ownership $X$ on return $Y$. Relevance is airtight: cross the boundary and index funds must trade you. But suppose that, the moment a stock enters the index, sell-side analysts also start covering it (more eyeballs, lower information asymmetry) and securities-lending desks make it easier to short. Now the instrument $Z$ has three paths to returns: the intended one through passive ownership, plus coverage, plus lending. Your IV estimate lumps all three together and attributes them to passive ownership alone. The number is biased not because the math failed but because exclusion failed — the wrench was touching two extra bolts under the mud.

When to use it

In finance, treat instruments as a high-burden-of-proof tool, not a default. Use one only when (a) you can point to a genuinely mechanical or quasi-random source of treatment variation, and (b) you can enumerate the alternative channels and argue each closed — ideally backed by a placebo or falsification test (does the instrument “predict” an outcome it shouldn’t, if exclusion held?). When you can’t clear that bar, an RDD around a sharp rule, or honest sensitivity bounds, is often more defensible than a contested IV.

Regression discontinuity design (RDD)

Analogy. Imagine a school that admits every applicant scoring 60 or above and rejects everyone at 59 or below. A student who scored 60 and one who scored 59 are, for all practical purposes, the same student — same ability, same background, same luck — separated by a single point that’s mostly noise. Yet one gets the treatment (admission) and the other doesn’t. Compare those two groups — the just-admitted vs. the just-rejected — and any gap in their later outcomes is caused by admission, because the threshold sorted near-identical people into different treatments. The cutoff manufactured a tiny randomized experiment for free.

Definition. A regression discontinuity design exploits a treatment rule of the form “treated if a continuous running variable $R$ crosses a known cutoff $c$.” Right at $c$, treatment status jumps discontinuously while everything else — fundamentals, characteristics, the confounders — varies smoothly. The causal estimate is the size of the jump in the outcome at the cutoff: the limit of $Y$ as $R$ approaches $c$ from above, minus the limit from below.

$\tau_{\text{RDD}} = \lim_{r \downarrow c}\mathbb{E}[Y \mid R = r] - \lim_{r \uparrow c}\mathbb{E}[Y \mid R = r]$

Any smooth confounding influence is the same on both sides of an infinitesimally thin line, so it cancels — leaving only the treatment’s effect.

Finance examples of sharp cutoffs.

• Index-membership rank: the Russell 1000 / 2000 reconstitution assigns membership by a rank cutoff — firms just above vs. just below the 1000th rank are near-twins, but only one set joins the small-cap index and absorbs passive flows.
• Credit-rating thresholds: the investment-grade vs. high-yield boundary (e.g., BBB- vs. BB+). Mandates force many institutions to dump a bond that crosses from IG to junk, so the rating line is a sharp treatment cutoff for forced selling and spread changes.
• Margin / eligibility cutoffs: a price or market-cap threshold that flips a security from marginable to not, or eligible for a mandate to not.

Worked numeric example — the jump at the cutoff. Study the effect of crossing into high-yield on a bond’s credit spread. The running variable is the rating, with the cutoff between investment-grade (just above) and high-yield (just below). Take bonds in a narrow band on each side of the line and average their spreads:

If spreads varied smoothly with credit quality, the BBB- and BB+ buckets — adjacent notches — should differ only modestly, in line with the roughly 30–40 bp gaps you see elsewhere on the ladder. Instead the jump from just-above (210) to just-below (340) is:

$\tau_{\text{RDD}} = 340 - 210 = 130\ \text{bps}.$

The smooth deterioration in fundamentals explains perhaps 35 bps of that (the typical notch-to-notch step); the remaining $\approx 95$ bps is the discontinuous effect of crossing the line — the forced selling and lost-mandate demand that hit a bond the instant it’s labeled junk. That excess jump, not the level, is the local causal effect of the downgrade-across-the-line.

Sharp vs. fuzzy RDD. In a sharp design, crossing the cutoff changes treatment with certainty (rank below 1000 ⇒ definitely in the small-cap index). In a fuzzy design, crossing the cutoff only changes the probability of treatment — it bumps the odds but doesn’t guarantee it (e.g., a downgrade makes forced selling likely but not universal, since some holders aren’t mandate-bound). A fuzzy RDD is estimated like IV: divide the jump in the outcome by the jump in the treatment probability at the cutoff — the same reduced-form-over-first-stage move as the Wald estimator. Sharp RDD is just the special case where that treatment jump equals 1.

When to use it

Use RDD wherever a hard, mechanical rule with a known threshold assigns the treatment — index reconstitutions, rating boundaries, eligibility and margin cutoffs, regulatory size thresholds. It is often the most defensible design in finance precisely because the rule is exogenous to any single firm and the comparison is local, so you lean less on the untestable exclusion story that haunts IV. Before trusting it, always (1) plot the running variable’s density and check for bunching, (2) show the estimate is stable across sensible bandwidths, and (3) state plainly that the answer is local to the cutoff — don’t extrapolate the threshold effect to firms nowhere near it.

Recap

When the confounder is unmeasurable, conditioning fails and you must go around the back door instead of blocking it. Instrumental variables find a lever $Z$ that shoves the treatment for reasons unrelated to the outcome — valid only if it clears two hurdles: relevance (it really moves the treatment; testable via the first-stage F) and exclusion (it reaches the outcome only through the treatment; untestable, argued from the mechanism). 2SLS isolates the clean, instrument-driven part of the treatment and reads the effect off it; for a binary instrument that’s the Wald ratio — reduced form over first stage — and a weak first stage makes the whole thing explode toward the bias you fled, while the estimate you do get is only a LATE on compliers. In finance, valid instruments are scarce and contested because dense market plumbing keeps breaking exclusion: index flows, supply shocks, and assignment rules all sneak second paths to returns. Regression discontinuity is the most credible cousin — a sharp rule (index rank, the IG/HY rating line, eligibility cutoffs) sorts near-identical units into different treatments, so the jump in the outcome at the cutoff is the local causal effect. Guard it against manipulation (bunching), pick the bandwidth honestly, and remember the answer lives only at the line.