Backtesting VaR and Its Limits

A VaR number is a confident little sentence: “we are 99% sure we won’t lose more than $4.2 million tomorrow.” Lovely. But confidence is cheap — anyone can promise a number. The whole point of a 99% VaR is that it makes a falsifiable claim: across many days, losses should blow past the line on about 1% of them, no more, no less. That’s not a vibe; it’s a count you can check.

Backtesting is the audit. You line up every day’s predicted VaR against the loss that actually happened, and you tally the days reality punched through. Too many breaches and your model is dangerously optimistic. Too few and it’s a coward, locking up capital you didn’t need to. A model that’s never wrong is a model that’s lying about how often it’s right. Let’s learn to count — and then learn exactly where VaR keeps lying anyway.

Did reality agree? Count the breaches

Analogy. A weather app that says “10% chance of rain” isn’t wrong when it rains — it’s wrong if, over a hundred such days, it rains on forty of them. You judge a probabilistic forecast not by any single day but by its hit rate over many days. VaR is exactly that kind of forecast, and backtesting is exactly that kind of audit.

Definition. An exception (or breach) is a day on which the realised loss exceeded the VaR predicted the day before. Backtesting compares the time series of predicted VaR to the time series of realised P&L and counts the exceptions. The count is the raw evidence; everything that follows — Kupiec, Basel — is just a rule for deciding whether that count is acceptable.

Drag the confidence slider below. A stricter 99% VaR sets a deeper loss limit, so fewer days punch through it; loosen to 95% and the line rises, catching more breaches. The exceptions light up in red, and the summary tallies them against the count you’d statistically expect.

The expected exception rate

Before you can say a model is wrong, you need to know what right looks like. For a confidence level $c$ over $N$ days, the expected number of exceptions is simply:

$\text{Expected exceptions} = (1 - c)\,N$

Worked example. A 99% VaR over one trading year ($N = 250$): expected exceptions $= (1 - 0.99) \times 250 = 0.01 \times 250 = 2.5$. So a healthy model should breach roughly 2 or 3 times a year. A 95% VaR over the same year expects $0.05 \times 250 = 12.5$ breaches — twenty times as many, because 95% is a far less demanding promise.

Now the part people forget: both directions are failures.

• Too many breaches (say 8 when you expected 2.5) → your VaR is too low. You’re systematically underestimating risk, holding too little capital, and one bad week could be ruinous. This is the dangerous failure.
• Too few breaches (say 0 when you expected 2.5) → your VaR is too high. The model is over-cautious, freezing capital you could have deployed and lying about how safe you really are. A risk number that’s never tested isn’t conservative; it’s uncalibrated.

A good model lands near the expected rate from both sides. “Never breached” earns no gold star.

The Kupiec POF test

Counting is easy; deciding is the hard part. If you expected 2.5 breaches and got 4, is the model broken — or is that just luck? Random sampling means even a perfect model won’t hit exactly 2.5. We need a statistical test.

Analogy. It’s the same logic as testing whether a coin is fair. Flip it 250 times, expect 125 heads, get 138 — is it biased, or just noise? You compute how surprising 138 is under the “fair coin” assumption, and reject fairness only if it’s surprising enough.

Definition. The Kupiec proportion-of-failures (POF) test asks: is the observed breach rate $\hat{p} = x/N$ (with $x$ exceptions in $N$ days) statistically consistent with the promised rate $p = 1 - c$? It’s a likelihood-ratio test — it compares how likely the data is under the model’s promised rate versus under the rate you actually observed:

$LR_{POF} = -2\ln\!\left[\frac{p^{x}(1-p)^{N-x}}{\hat{p}^{x}(1-\hat{p})^{N-x}}\right]$

When the observed rate matches the promise, the two probabilities are nearly equal, the ratio is near 1, its log is near 0, and $LR_{POF} \approx 0$ — no evidence against the model. As the observed rate drifts from the promise, $LR_{POF}$ grows. You compare it to a chi-squared critical value with 1 degree of freedom (about 3.84 at 95% significance); exceed it and you reject the model.

Worked intuition. 5 breaches when 2.5 were expected ($N = 250$, $p = 0.01$) gives an $LR_{POF}$ around 2 — below the 3.84 threshold, so it’s borderline but not rejected. Twice the expected rate sounds alarming, but with only a handful of events the noise is large, and the test (rightly) refuses to condemn on thin evidence. Push to 9 or 10 breaches and $LR_{POF}$ sails past 3.84 — now you reject.

The Basel traffic light

Regulators can’t have every bank inventing its own pass/fail rule, so the Basel Committee wrote a dead-simple one: run a 99% 1-day VaR over the last 250 trading days, count the exceptions, and read off a colour.

Analogy. It’s a literal traffic light. Green: drive on, your model’s fine. Yellow: slow down — we’re not sure, so we’ll make you hold more capital as a penalty until you prove yourself. Red: stop — your model is rejected, and supervisors step in.

The bite is in the capital multiplier. Required market-risk capital is roughly your VaR times a multiplier $k$ (floor of 3). In the yellow zone, $k$ is bumped up in steps the more breaches you have; in red, the model is thrown out entirely. More breaches → higher multiplier → more capital frozen. Backtesting isn’t an academic exercise — it directly sets how much money the regulator makes you park.

Note the asymmetry baked in: green tolerates 0–4, comfortably bracketing the 2.5 you statistically expect, because regulators would rather not punish honest noise. But cross into yellow and the cost climbs fast — the system is tuned to make underestimating risk expensive.

Where VaR lies to you

Backtesting can tell you a model is mis-calibrated. It cannot save you from the things VaR was never designed to see. Even a VaR that sails through every backtest carries deep, structural blind spots — and ignoring them is how firms blow up.

Stress testing and scenario analysis

VaR is a fair-weather statistic — it describes the distribution it was fed, and that distribution is mostly placid days. It is structurally bad at the rare, violent event that actually matters. The fix isn’t a better VaR; it’s a different tool that doesn’t lean on the recent distribution at all.

Stress testing asks a blunt “what if?” — what does this portfolio lose under a deliberately brutal, named scenario? You don’t estimate the scenario’s probability from recent data; you simply impose it and measure the damage:

• Historical replays: revalue today’s book through the 2008 crash, the COVID-19 February 2020 plunge, the 1987 Black Monday, or a 1994-style rate shock.
• Hypothetical shocks: rates +300 bp overnight, equities −25%, credit spreads doubling, a key correlation snapping to 1.

Where VaR says “1% of normal days look this bad,” stress testing says “if this specific catastrophe hits, here is the bill” — no probability, no thin-tail assumption, no recent-calm bias. Regulators now require it alongside VaR precisely because the two cover each other’s blind spots: VaR for the routine, stress tests for the tail VaR can’t see.

Putting it together — the whole topic

Step back and look at the arc you’ve climbed. VaR compresses an entire portfolio’s risk into one falsifiable sentence — a horizon, a confidence level, a dollar loss. You learned to build that number three ways: historical simulation (replay the past), the parametric / variance–covariance method (assume a bell curve and use algebra), and Monte Carlo (simulate thousands of synthetic tomorrows) — each with its own assumptions and failure modes. You scaled it across horizons with the square-root-of-time rule, then met Expected Shortfall, which answers the question VaR refuses — how bad is it once you breach? — and patches VaR’s coherence flaws. Here you backtested the whole edifice: count exceptions against $(1-c)N$, run Kupiec and Christoffersen, read the Basel traffic light. And finally you stared at where it all lies — fat tails, tail-blindness, procyclicality, gaming — and added stress testing to cover the catastrophe VaR can’t model. You can now compute a VaR three ways, average its tail, scale it, validate it against reality, and explain with numbers exactly when to distrust it. That is what an expert does: trusts the number, and knows its lies.

That’s the topic. You’ve built VaR three ways, sharpened it with Expected Shortfall, scaled it across time, validated it against reality, and learned exactly where it breaks. One thing remains: prove it. The final exam is a single graded run across the whole topic — one question at a time, each answer locked the moment you submit, score revealed only at the end. No notes, no second chances. Go earn the number.