Hypothesis Testing & Significance

A backtest spits out a number: your strategy made 0.4% a month on average. Glorious — except a coin-flipping monkey would also post some average, and sometimes a flattering one, by sheer luck. The whole job of hypothesis testing is to stand between you and that monkey: it forces you to answer one ruthless question before you celebrate. If my edge were actually zero, how often would random noise alone hand me a result this good? If the answer is “all the time,” you’ve discovered nothing. If it’s “almost never,” you might — might — have something real.

This is the capstone of the statistics course. Everything before it — distributions, the normal, expectation and variance, the standard error and the Central Limit Theorem — was building the machinery you’ll now point at a single, career-defining question: is this edge real, or is it luck?

The logic of testing: assume nothing, then look for surprise

Analogy. A hypothesis test is a courtroom. The defendant — your strategy’s edge — is presumed innocent of having any effect. “Innocent” here means the true edge is zero; that presumption is the null hypothesis $H_0$. The prosecution’s claim, the alternative hypothesis $H_1$, is that there is an effect. You don’t get to declare guilt because the strategy “looks profitable.” You convict only if the evidence is so improbable under innocence that “it was just luck” stops being credible.

Definitions.

• Null hypothesis $H_0$ — the boring, no-effect baseline. For a strategy: the true mean return is $0$. For a regression coefficient: the true slope is $0$. We assume it’s true and try to embarrass it.
• Alternative hypothesis $H_1$ — the interesting claim that there is an effect (mean $\neq 0$, slope $\neq 0$). A two-sided $H_1$ says “different from zero in either direction”; a one-sided $H_1$ says “specifically greater than zero” (or specifically less).

The asymmetry is the whole point. We never prove $H_1$; we only reject $H_0$ when the data is too surprising to square with it — or fail to reject when it isn’t. “Fail to reject” is an acquittal, not a declaration of innocence: absence of evidence isn’t evidence of absence.

The test statistic and the t-test: turning an edge into a z-distance

Analogy. Raw averages aren’t comparable — a 0.4% monthly edge is impressive for a sleepy bond fund and trivial for a wild crypto strategy, because the noise differs. A test statistic rescales the edge into a universal unit: how many standard errors away from zero is it? It’s the z-score idea from the normal-distribution lesson, applied to an estimate instead of a single observation.

Definition. For a sample of $n$ returns with mean $\bar{x}$ and standard deviation $s$, recall from the sampling lesson that the standard error of the mean is $\mathrm{SE} = s / \sqrt{n}$. The t-statistic for testing $H_0:\ \mu = 0$ is

In words: the edge, measured in standard errors. A big $|t|$ means the estimate sits far out in the tail of the “no-effect” distribution — the data is surprising if $H_0$ is true. By the Central Limit Theorem, for a decent sample size $\bar{x}$ is approximately normal, so $t$ behaves like a standard normal (the t-distribution it’s named after just adds slightly fatter tails for small $n$). A rough rule of thumb: $|t| > 2$ is the classic “significant at roughly 5%” threshold.

Worked example. A strategy has a monthly mean return of $\bar{x} = 0.8\%$ with a monthly standard deviation of $s = 4\%$, measured over $n = 36$ months.

A t-stat of $1.20$ is below $2$ — the edge is barely more than one standard error from zero. Verdict: not significant. With this much month-to-month noise and only three years of data, an average of 0.8% is the kind of thing luck produces routinely. Now watch what more data does: keep the same $\bar{x}$ and $s$ but stretch to $n = 144$ months (12 years). Then $\mathrm{SE} = 4\%/12 = 0.333\%$ and $t = 0.8\%/0.333\% = 2.40$ — now significant. The edge didn’t change; the standard error shrank because $n$ grew. That $\sqrt{n}$ in the denominator is doing all the work.

The p-value: the probability of data this extreme — under the null

Analogy. The p-value is your surprise meter. Crank the machine assuming the null is true, then read off: how often would pure noise produce a result at least as extreme as the one I actually got? A tiny p-value means “noise basically never does this” — strong evidence against the null. A big p-value means “noise does this all the time” — shrug.

Definition. The p-value is the probability, computed assuming $H_0$ is true, of observing a test statistic at least as extreme as the one you got. You compare it to a pre-chosen significance level $\alpha$ (the threshold you’ll accept for crying “real,” conventionally $0.05$):

• If $p \leq \alpha$: reject $H_0$ — call the result statistically significant.
• If $p > \alpha$: fail to reject $H_0$.

A t-stat of $2.0$ corresponds to a two-sided $p \approx 0.046$; a t-stat of $1.20$ (our first worked example) to $p \approx 0.23$ — i.e. noise would reproduce that “edge” about 23% of the time. Of course it wasn’t significant.

A second, subtler trap: statistical significance is a yes/no verdict at your chosen $\alpha$, not a measure of effect size. A microscopic, useless edge can be “highly significant” with enough data, and a huge, valuable edge can be “not significant” with too little. The p-value answers “is it distinguishable from zero?” — never “is it big?”

Type I and Type II errors, and the power to tell them apart

Analogy. Back in the courtroom: two ways to blow the verdict. Convict an innocent defendant (declare an edge that isn’t there) — a Type I error, a false positive. Or acquit a guilty one (miss an edge that’s real) — a Type II error, a false negative. You can’t drive both to zero at once: a jury that never convicts the innocent also lets more of the guilty walk.

Definitions.

• Type I error (false positive), rate $\alpha$ — rejecting $H_0$ when it’s actually true. You “found” an edge that’s pure noise. You set this rate: choosing $\alpha = 0.05$ means accepting a 5% false-positive rate per test.
• Type II error (false negative), rate $\beta$ — failing to reject $H_0$ when $H_1$ is actually true. A real edge slipped through undetected.
• Statistical power $= 1 - \beta$ — the probability of correctly detecting a real effect. High power = a sensitive test that rarely misses genuine edges.

The trade-off. Tighten $\alpha$ (demand more proof before convicting — say $\alpha = 0.01$) and you cut false positives, but you also make it harder to detect real effects, so $\beta$ rises and power falls. The cure for both is the same lever from the last two lessons: more data. A larger $n$ shrinks the standard error, which sharpens the whole machine — you can hold $\alpha$ low and gain power. There’s no free lunch except sample size.

The finance trap: data snooping, p-hacking, and backtest overfitting

Analogy. Buy one lottery ticket and winning is a miracle. Buy a million tickets and someone’s ticket wins — but that winner isn’t skilled, just selected. Test one strategy at $\alpha = 0.05$ and a false positive is a 1-in-20 fluke. Test 1,000 strategies on the same data and, by pure luck, about 50 of them will clear the 5% bar with zero real edge. Cherry-pick the best-looking one, publish its glorious backtest, and you’ve “discovered” noise. This is data snooping — and it is the single most lethal mistake in quantitative finance.

The mechanism. Each test has a 5% false-positive rate in isolation. But significance was never designed to survive being run a thousand times and reporting only the winner. The probability that at least one of $m$ independent true-null tests fires a false positive is $1 - (1 - \alpha)^m$. For $\alpha = 0.05$:

By 50 tries it’s a coin flip against you; by 1,000 a false discovery is essentially guaranteed. This is why a single dazzling backtest, divorced from the count of how many strategies were tried to find it, is nearly worthless evidence. The same disease wears other names: p-hacking (tweaking the rule, the window, the universe until $p$ slips under $0.05$) and backtest overfitting (adding parameters until the strategy memorizes the history’s noise).

The defenses. The fix is to make the bar harder the more you searched. The crude-but-honest Bonferroni correction divides your threshold by the number of tests: testing $m$ strategies, demand $p \leq \alpha/m$ for each (so $1{,}000$ strategies at an overall $5\%$ means each must clear $p \leq 0.00005$ — a t-stat near $4$, not $2$). Practitioners also use the deflated Sharpe ratio, which discounts a strategy’s Sharpe for the number of trials it took to find and for non-normal returns, and they insist on out-of-sample validation: hold back data the strategy never saw, because noise that was memorized in-sample won’t reappear out-of-sample.

Significance is not economic relevance

Analogy. A bathroom scale precise enough to detect a single grain of rice can prove you gained weight after a sneeze — a real, statistically significant change that no one would ever act on. In markets, a tiny edge can be unmistakably real and still utterly worthless, because the market charges admission: bid-ask spreads, commissions, slippage, and market impact all skim the return before it reaches your pocket.

The point. Statistical significance answers “is this edge distinguishable from zero?” Economic relevance answers a completely different question: “is what’s left after trading costs worth trading?” With enough data, a $0.01\%$ monthly edge can post a t-stat of $5$ — bulletproof significance — and yet vanish the instant you subtract a $0.05\%$ round-trip cost. Significant, and net-negative.

Worked example. A high-turnover signal earns a gross edge of $0.04\%$ per trade, established over a huge sample with a t-stat of $6$ — about as statistically real as evidence gets.

The edge is real, significant, and loses money on every trade once costs bite. A practitioner who stops at the p-value ships a strategy that’s statistically flawless and financially suicidal. Always run the second test: significant, sure — but does it survive costs, capacity, and turnover?

Putting it together

Hypothesis testing is the discipline that separates an edge from a fluke. Assume the null (no effect), rescale your estimate into a t-statistic ($t = \bar{x}/\mathrm{SE}$), and read off a p-value: how often noise alone would fake a result this extreme. Reject the null only when that p clears your $\alpha$ — never mistaking the p-value for the probability the null is true. Respect the $\alpha/\beta$ trade-off and chase power with more data. Then survive the two finance-specific gauntlets: data snooping — correct for every strategy you tried, because 1,000 tests breed ~50 lucky “discoveries” — and economic relevance — a significant edge still has to beat the costs. Significance is where the analysis starts, not where it ends.

That closes the statistics course: you can now take a number off a backtest and interrogate it like a quant — is this real, or is it luck, and even if it’s real, is it worth trading? Every later course that claims an “edge,” a “factor,” or a “signal” is implicitly making a hypothesis-testing claim. Now you know how to audit it.