Updating Return & Volatility Estimates

Last lesson you updated a win-rate — a probability between 0 and 1 — with the Beta-Binomial machinery, because counting wins and losses is naturally a coin-flip problem. But most of finance isn’t coin flips: it’s continuous numbers. An asset’s expected return $\mu$ could be 4%, or 7.3%, or −2%. Volatility is a positive real number. To do Bayesian inference on quantities like these, we need the continuous cousin of Beta-Binomial — and for a quantity that lives on the whole number line, the natural choice is the Normal distribution. This lesson is the Normal–Normal conjugate update: a prior bell curve on $\mu$, meeting a noisy bell curve of data, producing a sharper posterior bell curve. It is, hands down, the most-used Bayesian update in quantitative finance.

And it solves a genuinely painful problem. Estimating an asset’s true expected return from its past returns is shockingly hard — far harder than almost anyone’s intuition suggests.

The problem: a sample mean return is shockingly noisy

Analogy. Suppose you want to know someone’s true height, but your only ruler reports a different number every time you measure — sometimes off by a foot. One reading is nearly worthless; you’d want many, and you’d still hedge with prior knowledge (“adults are usually between 5 and 7 feet”). Estimating an asset’s expected return from past returns is exactly this, except the ruler is much worse than a foot off.

Why the sample mean is so bad. The standard error of a sample mean shrinks only as $1/\sqrt{n}$. For returns, the per-period volatility $\sigma$ is large relative to the mean $\mu$ you’re hunting for. Take a stock with mean return $\mu = 8\%$ and volatility $\sigma = 20\%$ a year. The standard error of the sample mean over $n$ years is $\sigma/\sqrt{n}$:

$\text{SE}(\bar{x}) = \frac{\sigma}{\sqrt{n}} = \frac{0.20}{\sqrt{n}}.$

With 10 years of data, $\text{SE} = 0.20/\sqrt{10} \approx 6.3\%$ — an error bar almost as big as the 8% you’re trying to measure. To get the standard error down to a still-mushy 2%, you’d need $n = (0.20/0.02)^2 = 100$ years of data. This is the famous, deflating fact: you need decades — often centuries — of data to pin down a mean return, while the asset’s character has long since changed underneath you. Pure data-driven estimates of $\mu$ are, bluntly, terrible.

Why this is good news for Bayesians. When the data speaks softly and uncertainly, a sensible prior carries real weight — and should. The whole point of the Normal–Normal update is to combine your weak, noisy sample with a structured prior so the estimate doesn’t lurch around with every new quarter.

When to use it

Reach for the Normal–Normal update whenever you’re estimating a continuous, roughly-symmetric quantity — an expected return, an alpha, a factor loading, a drift — from noisy observations, and you have a defensible prior (a market-implied return, a long-run average, an analyst view). It’s the engine under return-forecast blending, and it’s the mathematical heart of the shrinkage methods in the next lesson.

Precision: the right currency for Gaussian updating

Analogy. Variance is “how blurry is my belief”; precision is its opposite — “how sharp is my belief.” When you combine two photographs of the same scene, you don’t average their blurriness; you let the sharper one dominate. Bayesian updating with Gaussians works the same way, and it’s cleanest when we speak in sharpness, not blur.

Definition. Precision is simply the reciprocal of the variance:

$\tau = \frac{1}{\sigma^2}.$

A tight bell curve (small $\sigma^2$) has high precision; a wide, uncertain one has low precision. The reason precision is the natural currency: in the Gaussian update, precisions add, and the posterior mean is an average weighted by precision. Both rules are ugly in variance and beautiful in precision — so we switch coordinates.

Worked micro-example. A prior standard deviation of $\sigma_0 = 3\%$ corresponds to a variance of $0.03^2 = 0.0009$ and a precision of $\tau_0 = 1/0.0009 \approx 1111$. A data standard error of $4\%$ gives variance $0.0016$ and precision $625$. We’ll use these exact numbers below — the bigger the precision, the louder that source gets to speak.

The Normal–Normal conjugate update

Here is the load-bearing result of the lesson. Put a Normal prior on the unknown expected return $\mu$, and assume the data variance is known. Then observing a sample mean $\bar{x}$ updates the prior into another Normal — the families match, which is what conjugate means.

Setup.

• Prior: $\mu \sim N(\mu_0, \sigma_0^2)$, with prior precision $\tau_0 = 1/\sigma_0^2$.
• Data: a sample mean $\bar{x}$ from $n$ observations, with data precision $\tau_{\text{data}} = n/\sigma^2$ (equivalently $1/\text{SE}^2$, since the standard error is $\sigma/\sqrt{n}$).

The update.

$\tau_{\text{post}} = \tau_0 + \tau_{\text{data}}, \qquad \mu_{\text{post}} = \frac{\tau_0\,\mu_0 + \tau_{\text{data}}\,\bar{x}}{\tau_0 + \tau_{\text{data}}}.$

Read those two equations slowly, because together they are Bayesian inference for a mean:

1. Posterior precision is the SUM of the precisions. $\tau_{\text{post}} = \tau_0 + \tau_{\text{data}}$. You always end up more certain than either source alone — combining evidence can only sharpen your belief. The posterior standard deviation is $\sigma_{\text{post}} = 1/\sqrt{\tau_{\text{post}}}$, which is smaller than both $\sigma_0$ and the standard error.
2. Posterior mean is a PRECISION-WEIGHTED average of the prior mean $\mu_0$ and the sample mean $\bar{x}$. The weights are the precisions themselves. Whichever source is sharper (higher precision) pulls the estimate toward it.

The intuition in one line. More data (larger $n$) or lower noise (smaller $\sigma$) raises $\tau_{\text{data}}$ → the data wins. A strong, confident prior (high $\tau_0$) → the prior wins. The posterior is a tug-of-war, and precision is the strength of each team.

Worked example 1: confident prior meets noisy data

Let’s run the exact numbers. You believe a stock’s expected annual return is $\mu_0 = 6\%$ and you’re fairly confident, with a prior standard deviation $\sigma_0 = 3\%$. Then you collect a sample whose mean return is $\bar{x} = 12\%$, but it’s noisy — its standard error is $4\%$.

Step 1 — convert to precisions.

$\tau_0 = \frac{1}{0.03^2} = \frac{1}{0.0009} \approx 1111, \qquad \tau_{\text{data}} = \frac{1}{0.04^2} = \frac{1}{0.0016} = 625.$

Step 2 — posterior precision (the sum).

$\tau_{\text{post}} = \tau_0 + \tau_{\text{data}} = 1111 + 625 = 1736.$

Step 3 — posterior mean (precision-weighted average). Working in percent for the means:

$\mu_{\text{post}} = \frac{\tau_0\,\mu_0 + \tau_{\text{data}}\,\bar{x}}{\tau_0 + \tau_{\text{data}}} = \frac{1111 \times 6 + 625 \times 12}{1736} = \frac{6667 + 7500}{1736} = \frac{14167}{1736} \approx 8.16\%.$

(If you prefer the per-unit view: $1111 \times 6\% = 66.67$ and $625 \times 12\% = 75.0$, summing to $141.67$, over $1736$, giving $\approx 8.16\%$ — the same thing.)

Step 4 — posterior standard deviation.

$\sigma_{\text{post}} = \frac{1}{\sqrt{\tau_{\text{post}}}} = \frac{1}{\sqrt{1736}} \approx \frac{1}{41.7} \approx 0.024 = 2.4\%.$

Read the result. The noisy 12% sample pulled your estimate up — but only from 6% to about 8.16%, not all the way to 12%. Because the data was noisy (lower precision than the prior), it got the smaller vote. And notice the posterior standard deviation of 2.4% is tighter than both the prior’s 3% and the data’s 4% — the precisions added, so your uncertainty shrank. The Bayesian refused to be stampeded by one suggestive-but-noisy sample, exactly as it should.

Worked example 2: pile on the data and watch it overwhelm the prior

Keep the same prior — $\mu_0 = 6\%$, $\sigma_0 = 3\%$, so $\tau_0 \approx 1111$ — but now imagine you’ve gathered a much larger, cleaner sample. The sample mean is still $\bar{x} = 12\%$, but with so many observations the standard error drops to $0.8\%$ instead of 4%. The data precision explodes:

$\tau_{\text{data}} = \frac{1}{0.008^2} = \frac{1}{0.000064} \approx 15625.$

Now the update:

$\tau_{\text{post}} = 1111 + 15625 = 16736, \qquad \mu_{\text{post}} = \frac{1111 \times 6 + 15625 \times 12}{16736} = \frac{6667 + 187500}{16736} = \frac{194167}{16736} \approx 11.6\%.$

$\sigma_{\text{post}} = \frac{1}{\sqrt{16736}} \approx 0.0077 = 0.77\%.$

Read the contrast. Same prior, same observed sample mean of 12% — but precise, abundant data overwhelms the prior and drags the posterior almost all the way to 11.6%, with a razor-tight 0.77% standard deviation. This is the reassuring limit: as data precision $\to \infty$, the posterior mean $\to \bar{x}$ and the prior fades to irrelevance. The prior is training wheels for the data-starved regime — and the more (good) data you have, the less it matters, exactly as honest inference should behave.

Watch the precision-weighted average move

The chart below is the Normal–Normal update made physical. Three curves: your prior belief about $\mu$, the likelihood (what the data says), and the posterior that fuses them. Drag “Prior mean” and “Data says” to set the two centers; “Prior confidence” sharpens or blurs the prior; “Data strength” stands in for more observations / lower noise (higher data precision).

The thing to feel: start with weak data, then raise “Data strength.” Watch the posterior slide off the prior and onto the data — and, crucially, get narrower than both curves as you do it. That narrowing is the precision sum $\tau_{\text{post}} = \tau_0 + \tau_{\text{data}}$ in motion; the sliding is the precision-weighted average choosing its winner. Crank prior confidence way up instead, and the posterior barely budges from the prior no matter what the data says — the stubborn-prior regime.

Estimating volatility, briefly

So far the data variance was known. In reality you must estimate volatility too — and here the conjugate prior changes shape. Variance is a positive quantity with a skewed, asymmetric uncertainty (it can’t go below zero, but it can spike high), so a symmetric Normal prior is the wrong tool. The conjugate prior for a Normal variance is the Inverse-Gamma distribution (equivalently a scaled inverse chi-squared).

You don’t need the closed-form here — the idea is what matters and it rhymes with everything above:

• Start with a prior on the variance (e.g. “this asset’s variance is around its long-run level”).
• Observe a batch of returns and compute their squared deviations from the mean.
• The posterior is another Inverse-Gamma, combining your prior “pseudo-observations” with the actual observed sum of squared deviations.

The practical payoff: a Bayesian volatility estimate is a shrunken, more stable blend of your prior and the raw sample variance — instead of the raw sample variance alone. This matters because sample variances from short windows are jumpy: a single turbulent month can spike a one-year volatility estimate, and a calm stretch can lull it to sleep. Shrinking toward a sensible prior tames that jitter. (Reassuringly, estimating $\sigma$ is much easier than estimating $\mu$ — variance information accumulates faster — but a prior still buys you stability when windows are short.)

Shrinkage: the precision-weighted average is the bridge

Step back and look at what the posterior mean is:

$\mu_{\text{post}} = \frac{\tau_0\,\mu_0 + \tau_{\text{data}}\,\bar{x}}{\tau_0 + \tau_{\text{data}}}.$

This is a weighted average that pulls the noisy sample mean $\bar{x}$ toward the prior mean $\mu_0$ — and the noisier the data, the harder the pull. That operation has a name across statistics and finance: shrinkage. The Normal–Normal posterior mean is literally a shrinkage estimator, and the prior precision controls how much shrinkage you apply.

That’s the bridge to Lesson 5 (shrinkage estimators & Black-Litterman). The James–Stein estimator, shrinkage of covariance matrices, and the Black-Litterman model — which blends a market-implied “prior” return with investor “views” as data — are all the same precision-weighted average you just learned, scaled up to whole vectors and matrices of returns. Master this one update, and those machines stop looking magic.

Why this matters for portfolios

Here’s the stakes. Mean-variance optimizers are violently sensitive to their inputs, and the input they’re most sensitive to is the vector of expected returns — exactly the quantity whose raw sample estimate we just showed is garbage (recall: you’d need a century of data to pin it down). Feed a portfolio optimizer raw sample means and it will gleefully pour your money into whatever asset got lucky in the sample, producing extreme, unstable, untradeable weights. The fix is to shrink those mean estimates toward a sensible prior first — to hand the optimizer a precision-weighted posterior instead of the noisy sample. That is the entire reason shrinkage and Black-Litterman exist, and it’s why this unassuming weighted average is one of the most valuable ideas in quantitative portfolio construction.

Pitfalls

Pitfall 1 — trusting a short-window sample mean. A fund’s trailing 3- or 5-year average return has an error bar nearly as wide as the number itself. Treating it as the true expected return is the single most common, most expensive estimation mistake in investing. The standard error scales like one over the square root of the years, and you never have enough years.

Pitfall 2 — thinking the posterior is the simple average. The posterior mean is the precision-weighted average, not the equal-weighted midpoint. In Example 1, prior 6% and data 12% did not give 9%; they gave 8.16%, because the prior was more precise and earned a bigger vote. If you ever compute a plain midpoint, you’ve thrown away the most important information — how reliable each source is.

Pitfall 3 — forgetting that estimating $\mu$ is far harder than estimating $\sigma$. Volatility information accumulates fast; mean information crawls. People reflexively trust trailing returns and distrust risk models, when the truth is the reverse: your $\sigma$ estimate is far more trustworthy than your $\mu$ estimate. Plan your modeling effort — and your priors — accordingly.

Putting it together

To estimate a continuous quantity like an asset’s expected return, put a Normal prior on it and update with Normal data — the conjugate Normal–Normal case. Switch coordinates to precision ($\tau = 1/\sigma^2$), because then the two rules are clean: posterior precision is the sum ($\tau_{\text{post}} = \tau_0 + \tau_{\text{data}}$, so you always end up sharper), and the posterior mean is the precision-weighted average of the prior mean and the sample mean. More data or lower noise → the data wins; a confident prior → the prior wins. Because the sample mean of returns is brutally noisy — decades of data just to get a usable error bar — the prior does real, valuable work, and the resulting estimate is a shrunken one. Volatility gets the same Bayesian treatment via an Inverse-Gamma prior, yielding stabler vol estimates than a jumpy short-window sample variance. And that precision-weighted average is shrinkage — the exact idea the next lesson scales up into James–Stein and Black-Litterman to rescue portfolio optimizers from their own input sensitivity.

Next up — we scale this single precision-weighted average from one number to whole vectors of returns and matrices of risk. That’s shrinkage estimators and the Black-Litterman model: the industrial-strength version of “pull the noisy sample toward a sensible prior,” and the standard cure for the input-sensitivity that makes naive portfolio optimizers self-destruct.