Variational Autoencoders for Market Data

The GAN from last lesson was a brawler: sharp, fast, and prone to throwing the same punch over and over (mode collapse). The variational autoencoder (VAE) is the calmer sibling. Instead of a generator and a critic locked in combat, a VAE has two cooperating halves: an encoder that squeezes a price window down into a tiny latent code, and a decoder that tries to rebuild the window from that code. Train them together and the network learns a compact, reusable summary of “what a plausible price window looks like.”

The clever extra ingredient — the variational part — is that the VAE doesn’t just learn a code, it learns a smooth, samplable space of codes. It forces that latent space to look like a tidy bell-shaped cloud, so you can reach in, draw a brand-new code you’ve never seen, decode it, and get a novel but plausible path. That is what turns a humble compressor into a generative model: new codes in, new market data out.

The trade against GANs runs through this whole lesson. VAEs are stable (no adversarial tug-of-war to balance), they hand you an interpretable latent you can interpolate and steer, and they almost never collapse to a single mode. But they pay for it: their samples tend to come out soft and over-smoothed — and “smooth” is exactly the wrong word when the entire job is reproducing the fat tails that make markets dangerous. Keep that tension in mind; the last section is where it bites.

Autoencoders → variational autoencoders

A plain autoencoder is a compress-then-rebuild machine. The encoder maps an input $x$ (say, a 60-day window of log-returns) to a low-dimensional code $z$; the decoder maps $z$ back to a reconstruction $\hat{x}$. You train it to make $\hat{x} \approx x$. It learns a great compression — but it learns nothing about the shape of the latent space. Encoded points land wherever minimizing reconstruction error puts them, leaving vast empty regions in between. Sample a random code and you’ve almost certainly landed in a hole, and the decoder hands you nonsense.

Analogy. Think of the latent space as a filing cabinet. A plain autoencoder is a cabinet stuffed by someone in a hurry: every document is filed somewhere, and they can retrieve any one they’ve seen, but the drawers have huge empty gaps and no system. Ask for “the file halfway between these two” and you pull out an empty space. A VAE is the cabinet of an obsessive organizer: files are spread evenly and continuously, every spot you reach into holds something sensible, and neighboring spots hold similar things.

The VAE achieves this with two changes:

1. The encoder outputs not a single point but a distribution per latent dimension — a mean $\mu$ and a (log-)variance $\sigma^2$. So $x$ maps to $q(z \mid x) = \mathcal{N}(\mu, \sigma^2)$, a little fuzzy ball rather than a pinpoint.
2. A regularizer pulls every one of those balls toward a shared standard normal prior $\mathcal{N}(0, I)$. This packs the balls together to tile the space with no holes, so the whole region near the origin is now samplable.

At generation time you ignore the encoder entirely: draw $z \sim \mathcal{N}(0, I)$, push it through the decoder, and out comes a fresh, plausible price window.

When to use it

Reach for the autoencoder framing whenever you want both a compressor (a compact representation of market windows for downstream models) and a generator from the same network. If you only ever need to reconstruct or denoise — never sample new data — a plain autoencoder is lighter. The moment you need to draw novel paths, you need the variational machinery.

The ELBO: reconstruction vs KL

A VAE is trained by maximizing the evidence lower bound (ELBO), or equivalently minimizing its negative. For a single input $x$ the loss is:

Two forces, pulling opposite ways:

• Reconstruction wants $\hat{x}$ to match $x$ — it would happily encode every window to its own private, far-flung code if that gave a crisper rebuild.
• KL wants every posterior $q(z \mid x)$ to sit near the prior $p(z) = \mathcal{N}(0, I)$ — it would happily collapse every input to the same bland code at the origin if that minimized divergence.

The coefficient $\beta$ is the knob between them ($\beta = 1$ recovers the original VAE; the general case is the $\beta$-VAE). Watch the extremes:

Worked example. Suppose with $\beta = 1$ your VAE reconstructs a window with mean-squared error $0.018$ and a KL of $12.4$ nats, total loss $\approx 0.018 + 12.4$. Crank $\beta$ to $8$. The optimizer now finds it cheaper to shrink the KL: it pushes every posterior toward $\mathcal{N}(0, I)$, the latent stops carrying information about $x$, and the decoder — getting no useful signal — falls back on emitting the average window for every input. Reconstruction error balloons (say to $0.061$) but total loss drops because the KL term cratered. That degenerate state is posterior collapse, and its symptom on the page is blur: every sample looks like a washed-out, mean-reverting smudge.

When to use it

Treat $\beta$ as a dial you tune to the job. Pushing $\beta$ above 1 buys a more disentangled, controllable latent (good for the scenario knobs below) at the cost of fidelity. Pulling it toward 0 buys sharper reconstructions at the cost of samplability. For market data, where you already fight blur, you rarely want a large $\beta$ unless you specifically need clean, steerable latent axes.

The reparameterization trick

Here’s the snag. Training needs gradients of the loss with respect to the encoder’s outputs $\mu$ and $\sigma$. But between the encoder and the loss sits a random draw $z \sim \mathcal{N}(\mu, \sigma^2)$, and you can’t backpropagate through “roll a die.” The randomness blocks the gradient.

The trick is to move the randomness out of the way. Instead of sampling $z$ directly, sample a fixed, parameter-free noise variable $\epsilon \sim \mathcal{N}(0, I)$ and build $z$ deterministically from it:

where $\odot$ is elementwise multiplication. Now $z$ is a differentiable function of $\mu$ and $\sigma$ — the only random thing, $\epsilon$, is an external input that carries no gradient and no learnable parameters. The stochasticity has been shoved outside the computation graph.

Analogy. You can’t take the derivative of “spin a roulette wheel.” But you can take the derivative of “take a fixed random spin $\epsilon$, then stretch it by $\sigma$ and shift it by $\mu$.” The casino still injects randomness — you just moved your tunable knobs ($\mu$, $\sigma$) to where calculus can reach them.

Worked example. Take one latent dimension. The chain rule now flows cleanly, because $z = \mu + \sigma \epsilon$:

So if the upstream gradient of the loss at $z$ is $g = \partial \mathcal{L} / \partial z$, then $\partial \mathcal{L} / \partial \mu = g \cdot 1 = g$ and $\partial \mathcal{L} / \partial \sigma = g \cdot \epsilon$. Concretely, with a sampled $\epsilon = 0.7$ and an upstream $g = -0.4$, the encoder’s mean gets gradient $-0.4$ and its standard deviation gets $-0.4 \times 0.7 = -0.28$. Both are finite, both are usable — training proceeds.

When to use it

Always, for a Gaussian-latent VAE — it’s not optional. The same idea generalizes: any time a model must differentiate through a sampling step, look for a “sample fixed noise, then transform deterministically” rewrite (the same logic powers the noise schedule you’ll meet in the diffusion lesson).

A smooth latent buys scenario knobs

A holey latent is good for nothing but the points it memorized. A smooth latent is a playground. Because every nearby code decodes to something sensible and similar, the space picks up usable geometry — and that geometry is what turns a VAE into a scenario generator (recall the synthetic-data uses from lesson 1: stress testing, augmentation, privacy-safe sharing).

Two moves the smooth latent gives you for free:

• Interpolation. Encode a calm window to code $z_{\text{calm}}$ and a turbulent one to $z_{\text{turb}}$, then decode the straight-line blend $z(t) = (1 - t)\, z_{\text{calm}} + t\, z_{\text{turb}}$ for $t$ sweeping from 0 to 1. Each decoded $z(t)$ is a plausible window between the two regimes — markets that are calmer than the storm but choppier than the lull. A plain autoencoder’s interpolations pass through holes and decode to mush.
• Latent directions as dials. If a particular axis of the latent correlates with realized volatility, adding a multiple of that direction to a code dials the turbulence up or down while leaving other features roughly intact — a ready-made “crank up the vol” knob for stress scenarios.

Worked example (conceptual). You want a stress path that’s “this quarter, but 50% rougher.” Encode the quarter to $z_0$. Find the volatility direction $v$ (e.g. the latent axis along which decoded windows get choppier). Decode $z_0 + 0.5\,v$. Out comes the same broad regime — same drift, same rough shape — but with visibly fatter intraday swings, ready to feed a risk engine. You generated a counterfactual scenario by steering, not by re-sampling from scratch. That controllability is something a GAN’s tangled latent struggles to offer.

When to use it

Lean on latent interpolation and steering when you need controlled, interpretable scenarios — “show me the path between these two regimes,” “give me a high-vol variant of last week.” For raw realism with no steering requirement, the latent geometry is a nice-to-have, not the point. It’s most valuable for stress testing and what-if generation, where you care how a sample differs from a baseline.

Why VAEs blur the fat tails

Now the catch that makes VAEs a risky default for market data. Everything that makes a VAE stable and tidy also makes it regress toward the mean — and “the mean” is precisely where the danger isn’t.

Three forces conspire:

1. Gaussian likelihood. The standard decoder maximizes a Gaussian log-likelihood, which is equivalent to minimizing squared error. Squared error is minimized by predicting the conditional mean, and it punishes large deviations brutally — so the model learns to avoid extreme outputs. Real return distributions are leptokurtic (lesson 2’s fat tails); a Gaussian objective treats those tails as expensive mistakes to be smoothed away.
2. Averaging over the posterior. The decoded output effectively integrates over the fuzzy latent ball $q(z \mid x)$. Averaging is a low-pass filter: sharp spikes and abrupt regime switches get smeared toward something gentler.
3. KL pressure toward the prior. The far-flung latent codes that would decode to genuine crashes are exactly the ones the KL term discourages — they sit in the low-density skirts of $\mathcal{N}(0, I)$. Push everything toward the center and you push the extremes off the table.

Graded against the stylized-facts rubric from lesson 2, a vanilla VAE gets a mixed report card:

In one line: a VAE nails the middle of the distribution and fails the edges — the opposite of what a risk model needs, where the edges are the product.

The picture below shows the tell-tale signature: a VAE’s synthetic return distribution sits too narrow and too thin in the wings compared to the real, fat-tailed thing — the blur made visible.

When to use it

Use a VAE when stability, a controllable latent, and a faithful distribution bulk matter more than nailing the extremes — exploratory scenario sketches, data augmentation for the common case, representation learning. Avoid a vanilla VAE as your sole engine for tail-risk work (VaR, stress, extreme clustering) without explicit tail fixes. When sharp tails are non-negotiable, a GAN (sharper, lesson 4) or a diffusion model (lesson 6) is usually the better starting point.

Recap

A variational autoencoder pairs an encoder that compresses a price window into a distribution over a latent code with a decoder that rebuilds the window — and the variational twist forces that latent into a smooth, samplable $\mathcal{N}(0, I)$-shaped cloud, so new codes decode to novel paths. Training minimizes the ELBO: a reconstruction term versus a $\beta$-weighted KL term, with $\beta \to 0$ degenerating into a holey plain autoencoder and large $\beta$ triggering posterior collapse. The reparameterization trick ($z = \mu + \sigma \odot \epsilon$) shoves the randomness outside the gradient path so the encoder is trainable at all. The payoff is a smooth latent you can interpolate and steer for scenarios and stress tests — far more stable and interpretable than a GAN. The price is blur: a Gaussian likelihood, posterior averaging, and KL pressure all push the model toward the mean, so it under-produces the fat tails that risk work lives and dies by. Stable and faithful in the middle; soft and dangerous at the edges.