Credible Intervals & MCMC

So far this topic has handed you whole posteriors — Beta distributions, Normal distributions, tidy conjugate shapes you can write down on a napkin. But a distribution is a mouthful. Your boss doesn’t want a probability density; they want a number with a margin: “the win rate is around 67%, give or take.” That summary is a credible interval, and it’s the single most useful thing you can say out loud about a posterior. There’s a catch, though — it has an evil twin from frequentist statistics, the confidence interval, that looks identical and means something subtly, infuriatingly different. Getting them mixed up is the most common error in all of applied statistics. Then we’ll face the harder problem: most real posteriors have no napkin formula at all, and to read them we have to sample — which is where Markov-chain Monte Carlo, the workhorse of modern Bayesian finance, comes in.

The credible interval — the summary you actually wanted

Analogy. A posterior distribution is a mountain range of plausibility — tall where the parameter is likely, low in the foothills where it isn’t. A credible interval is you drawing a fence around enough of that range to enclose, say, 95% of the total dirt. Inside the fence: where the truth most plausibly lives. The fence is the headline.

Definition. A credible interval at level 95% is a range that contains the parameter with 95% posterior probability. Formally, you integrate the posterior density until 95% of its area sits inside the interval. Because a Bayesian treats the unknown parameter $\theta$ as a random variable with a distribution, you are fully entitled to say “there is a 95% probability that $\theta$ is in here” — the probability is a genuine statement about the parameter, conditioned on your prior and data.

There are two common flavours, differing only in where you put the fence:

• Equal-tailed interval: chop off 2.5% of the posterior probability from each tail. Simple, symmetric in probability, and what most software reports by default. The downside: for a skewed posterior it can include low-density points near one tail while excluding higher-density points near the other.
• Highest-posterior-density (HPD) interval: the shortest interval containing 95% of the probability, built so every point inside is more probable than every point outside. For a symmetric posterior the two coincide; for a skewed one the HPD hugs the peak and is the more honest “most plausible values” region.

The confidence interval — the twin that means something else

Analogy. Imagine a carnival ring-toss machine that, by design, lands a ring on the peg 95% of the times you play. After one throw the ring is lying on the ground, already landed — it either ringed the peg or it didn’t; nothing is random anymore. “95% success rate” describes the machine, not this finished throw. A confidence interval is one finished throw of that machine. The 95% is a property of the method, certified before you looked, not a probability attached to the particular interval now sitting on your desk.

Definition. A frequentist treats the true parameter as a fixed, unknown constant — it has no probability distribution, so no interval can have “a 95% probability of containing it.” Instead, a 95% confidence interval is the output of a procedure with this guarantee: if you repeated the whole experiment many times, building one interval each time, about 95% of those intervals would contain the true value. The randomness lives in the data (and hence in the interval’s endpoints), not in the parameter. For this particular interval, the truth is either in it or not — you simply don’t know which. You may not legally say “95% chance the true value is in this interval”; you may only say “this interval came from a procedure that’s right 95% of the time.”

Here’s the contrast laid out — read each row across, because every cell flips between the two columns:

Worked example — a width that shrinks as data arrives

Let’s reconnect to the win-rate posterior from earlier in this topic. Suppose you start with a flat $\text{Beta}(1,1)$ prior on a strategy’s true win probability and you observe 7 wins and 3 losses. With a Beta–Binomial conjugate update, the posterior is

$\text{Beta}(1+7,\ 1+3) = \text{Beta}(8,4).$

Its posterior mean is $8/(8+4) = 0.667$, and the equal-tailed 95% credible interval works out to roughly 0.39 to 0.89. The correct sentence: “Given my prior and these 10 trades, there is a 95% probability the true win rate is between about 39% and 89%.” That’s an enormous range — and that’s the posterior being honest. Ten trades is almost nothing; the data barely budged the prior, so the interval is wide.

Now pour in more of the same win rate. Observe 70 wins and 30 losses — the same 70% rate, ten times the data:

$\text{Beta}(1+70,\ 1+30) = \text{Beta}(71,31),$

with mean $71/102 \approx 0.696$ and a 95% credible interval of roughly 0.60 to 0.78. Same central estimate, but the interval has collapsed from 50 percentage points wide to 18. Ten times more data, and the fence pulls in tight around the truth.

That shrinking width is the whole point of a credible interval: it reports not just your best guess but how much you should trust it. Little data, wide fence; lots of data, narrow fence. A point estimate alone would have hidden the difference between “70%, basically a guess” and “70%, pinned down.”

Why MCMC exists — the constant you can’t compute

Analogy. Bayes’ rule is a recipe with one monstrous ingredient. Posterior $\propto$ likelihood $\times$ prior is easy to write; the trouble is the proportionality. To turn it into a real probability distribution you must divide by a normalising constant that makes the whole thing integrate to 1 — and that constant is an integral over every possible value of every parameter at once. For the conjugate cases in lessons 3 and 4 the integral has a closed form, so the constant cancels out for free. For almost everything else, it’s a high-dimensional integral with no formula. Computing it directly is like weighing a cloud by measuring every water droplet.

Definition. Bayes’ rule says the posterior is

$P(\theta \mid D) = \frac{P(D \mid \theta)\,P(\theta)}{P(D)},$

where the denominator, the marginal likelihood $P(D) = \int P(D \mid \theta)\,P(\theta)\,d\theta$, is the nasty normalising integral over the whole parameter space. Conjugate priors are precisely the special cases where this integral resolves to a known constant. The moment your model is even slightly realistic — multiple correlated parameters, a non-conjugate likelihood, a hierarchy across assets — $P(D)$ becomes intractable, and you cannot write the posterior down.

The escape hatch is profound: you don’t need $P(D)$ to draw samples from the posterior. If you can produce a large pile of samples distributed according to $P(\theta \mid D)$, then every summary you want — the posterior mean, the variance, a 95% credible interval, the probability that $\theta$ exceeds some threshold — is just a quantity you read off that pile. Sampling replaces integration. That’s what Markov-chain Monte Carlo (MCMC) delivers.

Markov-chain Monte Carlo, at a glance

Unpack the name in two halves:

• Monte Carlo = estimate quantities by averaging over random samples instead of doing the integral analytically. Want the posterior mean? Average your samples. Want a credible interval? Take the 2.5th and 97.5th percentiles of your samples. The pile of draws stands in for the distribution.
• Markov chain = a sequence where each sample depends only on the previous one, not the whole history. MCMC generates its samples by walking a chain across parameter space, each step nudged from the last, engineered so that — in the long run — the chain visits each region in proportion to its posterior probability.

The Metropolis(-Hastings) algorithm

Here’s the trick that makes the impossible constant irrelevant. The Metropolis algorithm walks the parameter space like this:

1. Propose a nearby move: from your current position, jump to a candidate point a small random step away.
2. Compute the ratio of the unnormalised posterior densities, new over old: $r = \dfrac{P(D \mid \theta_{\text{new}})\,P(\theta_{\text{new}})}{P(D \mid \theta_{\text{old}})\,P(\theta_{\text{old}})}$.
3. Accept or reject. If the new point is more probable ($r \ge 1$), always move there. If it’s less probable, move there only with probability $r$ (flip a biased coin); otherwise stay put.
4. Record wherever you now stand — that’s one sample.
5. Repeat thousands of times.

The magic is in step 2. Both densities are missing the same normalising constant $P(D)$, so when you take their ratio, the constant cancels. You never compute the intractable integral — you only ever need the easy product likelihood $\times$ prior, evaluated at two points. Because the chain accepts upward moves always and downward moves probabilistically, it lingers in high-probability regions and only occasionally wanders into the foothills. Run it long enough and the histogram of visited points approximates the posterior — exactly the pile of samples you wanted.

Play with the sampler above to feel the algorithm. With the step size tiny, almost every proposed move is accepted (it’s barely different from where you are) — yet the chain inches along and takes forever to explore the whole posterior. With the step size huge, proposals routinely land in low-probability regions and get rejected, so the marker sits frozen, also wasting samples. A moderate step size — a sensible acceptance rate, often quoted near 25%–50% for these random-walk samplers — mixes fastest, painting the histogram into the target curve in the fewest steps.

Burn-in, mixing, and convergence

Two practical wrinkles keep MCMC honest:

• Burn-in. The chain starts wherever you dropped it, possibly in a silly low-probability spot, and needs time to wander into the meaty part of the posterior and forget its arbitrary starting point. The early samples are biased by that start, so you discard them — the burn-in period — and keep only what comes after. Toggle burn-in in the sampler and watch those first stray samples vanish from the histogram.
• Autocorrelation and mixing. Because each sample is a small nudge from the last, consecutive samples are correlated — they’re not independent draws. A step size that’s too small gives high autocorrelation and slow exploration (poor “mixing”); too big gives constant rejection and the chain stalls. Good mixing means the chain roams freely and consecutive samples decorrelate quickly. To check the chain actually converged, practitioners eyeball trace plots (the parameter value over iterations should look like a stationary fuzzy caterpillar, not a drifting trend) and compute the R-hat statistic, which compares several independent chains and should sit near 1.0.

Gibbs, Hamiltonian Monte Carlo, and NUTS

Metropolis is the conceptual ancestor; modern practice uses sharper tools. Gibbs sampling updates one parameter at a time by drawing from its conditional distribution given all the others, which is efficient when those conditionals have known forms. Hamiltonian Monte Carlo (HMC) and its self-tuning variant NUTS (the No-U-Turn Sampler) borrow physics — they treat the posterior as a landscape and “roll” a particle across it using gradient information, so they take long, well-aimed steps instead of a blind random walk, exploring high-dimensional posteriors far more efficiently. These are what production tools like Stan and PyMC run under the hood; you’ll see their names whenever someone fits a serious Bayesian model.

Finance uses — where closed forms run out

Conjugate posteriors are lovely but rare; real financial models almost never have them, which is exactly why MCMC is everywhere in quantitative finance:

• Stochastic-volatility models. Volatility that itself wanders randomly (think of fitting models where the variance is a hidden, time-varying state) gives a posterior with no closed form. Bayesian estimation via MCMC is a standard route to infer the volatility path and its parameters together with honest uncertainty.
• Parameter uncertainty in option models. Pricing models depend on parameters you must estimate, and plugging in point estimates hides how shaky they are. MCMC gives a full posterior over those parameters, which you can propagate into a distribution of prices rather than a single misleadingly precise number.
• Hierarchical models across assets. When you estimate, say, the alpha of many funds at once, a hierarchical (multilevel) model lets each fund borrow strength from the group — shrinking wild individual estimates toward the pooled mean. These models are almost never conjugate, so MCMC is the standard engine, returning credible intervals for every asset and for the shared group parameters.

The common thread: anywhere the posterior has no closed form, MCMC turns “I can’t write this distribution down” into “I can draw 10,000 samples from it and read off any summary I want.”

Pitfalls

Calling a confidence interval a credible interval (the #1 error). It is wrong, and ubiquitous, to attach a posterior-probability statement (“95% chance the truth is in here”) to a frequentist confidence interval. Only a credible interval — which requires a prior and treats the parameter as random — carries that meaning. If you didn’t put a prior on the parameter, you didn’t build a credible interval, and you can’t make the credible-interval statement.

Thinking more samples fix a biased or unconverged chain. MCMC’s guarantee is asymptotic only if the chain has actually converged to the posterior. If it’s stuck in one mode, hasn’t reached the high-probability region, or is mixing terribly, drawing ten times more samples just gives you ten times more samples from the wrong distribution. Convergence is not the same as sample count. Diagnose convergence (R-hat, multiple chains, trace plots) before you trust the answer; volume alone never rescues a broken chain.

Forgetting burn-in. Keep the early, start-contaminated samples and you bias every summary toward wherever you happened to initialise the chain. Always discard the burn-in.

Reading a single lucky trace as proof of convergence. One smooth-looking trace plot is necessary but not sufficient. A chain trapped in a single mode can produce a perfectly calm trace while completely missing other regions of the posterior. Run several independent chains from different starting points and confirm they agree (R-hat near 1.0) before declaring victory.

Credible vs confidence — say it right

Burn the distinction in, because it’s the line where most people slip:

• Credible interval (Bayesian): the parameter is random (it has a posterior); the 95% is the probability the truth is in this interval; needs a prior. Legal sentence: “There is a 95% probability the true value lies in this interval.”
• Confidence interval (frequentist): the parameter is a fixed constant; the 95% is the procedure’s long-run hit rate across hypothetical repeats; no prior. Legal sentence: “95% of intervals built this way contain the true value.” Illegal: “95% chance the truth is in this one.”

If you only remember one thing from this lesson: the everyday phrase “95% chance the true value is in this interval” is true for a credible interval and false for a confidence interval. They look the same on the page; they mean different things.

Putting it together

A credible interval is the headline summary of a posterior — a range carrying a stated posterior probability, so you can legitimately say “there’s a 95% probability the truth is in here,” the very thing people wrongly say about a frequentist confidence interval (whose 95% is really about the long-run procedure, not your one interval). Credible intervals come in equal-tailed and shortest-HPD flavours, and they widen with little data and narrow as evidence piles up — honesty about how much to trust your estimate. When the posterior has no closed form — which is almost always, the moment a model gets realistic — the normalising constant $P(D)$ becomes an intractable integral, and MCMC sidesteps it by sampling: Metropolis proposes nearby moves and accepts them via a density ratio in which that constant cancels, so the histogram of visited points converges to the posterior. Mind the craft — burn-in, mixing, and genuine convergence (R-hat, multiple chains) — and remember that more samples never fix an unconverged chain. From Gibbs to modern HMC/NUTS in Stan and PyMC, this machinery is what lets Bayesian finance fit stochastic-volatility, option-pricing, and hierarchical models that no formula could ever express.

You can now read a posterior out loud — a credible interval that says exactly what it means, never confused with its frequentist twin — and you can get a posterior even when no formula exists, by sampling it with MCMC and cancelling the constant nobody can compute. That completes the toolkit: from priors and likelihoods, through conjugate updates, to sampling the intractable. Next, we put the whole topic to the test.