Why Generate Market Data?

Here is the uncomfortable secret of machine learning in finance: you are trying to learn from a river that flowed exactly once. A vision model gets to see ten million cat photos. A language model inhales the entire internet, twice. You, the quant, get one realized path of market history — one 2008, one 2020 crash, one COVID melt-up — and that’s it. The data-generating process ran the experiment a single time and handed you the result with no replays.

You already know from Deep Learning for Market Data why this is fatal: the effective sample size is tiny. Twenty years of daily prices looks like thousands of rows, but those rows are autocorrelated, regime-clustered, and overlapping, so the number of genuinely independent observations is closer to dozens than thousands. Deep nets are hungry beasts with millions of parameters; feed them dozens of effective examples and they don’t learn the market — they memorize it, then post a glorious backtest that evaporates the instant real money touches it.

So the move is audacious: if history only ran once, manufacture more history. Train or build a generator that emits new, never-realized return paths that still look like the market — same fat tails, same volatility clustering — and feed those to your hungry model. It’s a brilliant idea with a knife hidden in it, and that knife frames this entire course: a generator trained on your history has, in a sense, seen the future relative to any backtest you run on it. Get sloppy and it leaks your own data right back into your tests, dressed up as “synthetic.” Hold that thought — it’s the spine of lesson 4 onward and the warning we’ll keep returning to.

The one-history problem

Analogy. Imagine a chemist who can run their reaction exactly once, ever. No repeats, no error bars, no “let’s try it again at a different temperature.” Whatever happened, happened — and now they must publish universal laws from that single beaker. That chemist is you, and the single beaker is market history. You cannot rerun 2008 with a different Lehman decision; the tape is a one-shot experiment.

Definition. Market history is a single realized path drawn from an unknown, non-stationary data-generating process (DGP). You observe one trajectory; you never observe the distribution it came from, and you certainly can’t sample it again. The effective sample size $N_{\text{eff}}$ is the number of independent observations that path is worth — always far below the raw row count once you account for autocorrelation and regime persistence.

Worked example. Take 20 years of daily data: roughly

Sounds rich. But a crude effective-sample correction for autocorrelation shrinks it. If consecutive observations have lag-1 autocorrelation $\rho \approx 0.97$ at the volatility level (volatility clusters hard), one common approximation gives

Seventy-seven. And that’s the generous read — if instead you ask “how many genuinely distinct market regimes did I observe?” (a bull run, the 2008 crash, the slow QE grind, the COVID crash-and-rip, a rate-hike cycle), you’re counting on your fingers. One twenty-year history is, for the questions that matter, roughly one draw from the DGP.

Look at what the single path actually looks like — non-stationary price wandering on top, but stationary, clustered returns underneath:

When to use it

Invoke the one-history framing whenever someone waves a big row count at you as proof a deep model “has enough data.” It’s the diagnostic that tells you a model is starving even when the spreadsheet looks full — and it’s the entire motivation for manufacturing more paths.

The four uses of synthetic data

Analogy. A flight simulator. Real pilots can’t crash a real 747 to practice, can’t summon an engine fire on demand, can’t wait for the once-a-decade microburst to teach them. So they generate those scenarios. Synthetic market data is the flight simulator for strategies and risk systems — and like a flight sim, it’s only useful insofar as the physics it fakes are honest.

Definition. Synthetic market data is artificially produced data — return paths, order flows, prices — that mimics the statistical properties of real markets without being a literal copy of any observed period. It gets used for four distinct jobs, each buying you something different and each carrying its own risk.

Worked mini-example. Say you augment a training set from 5,000 real return rows to 50,000 paths — a 10x boost. What does it fix? It gives your optimizer more gradient steps over more varied trajectories, which can genuinely reduce variance in what the model learns and tame the worst overfitting. What does it not fix? If your generator was built from that same one history, those 50,000 paths still encode the same handful of regimes. You did not observe a new 2008; you observed echoes of the old one. The effective information content barely moved — you went from ~77 independent observations to maybe ~80, not 800. More paths ≠ more independent information.

When to use it

Match the use to the question. Thin model starving for gradient variety? Augmentation. Need to survive a shock history never dealt? Scenario generation. Legally can’t share the raw positions? Privacy generator. Want robustness across many futures, not one past? Synthetic backtesting — with the leakage guard from lesson 7 firmly in place.

Two philosophies: learn vs simulate

Analogy. Two ways to build a flight simulator. Mechanism (simulate): an engineer writes the equations of aerodynamics — lift, drag, thrust — from first principles. You can read every line and argue with it. Learned: you strap sensors to ten thousand real flights and train a net to predict “what the plane does next” — eerily realistic, but ask it why it did something and it shrugs. Both fly the sim; one you can audit, the other you mostly have to trust.

Definition. There are two philosophies for producing synthetic market data:

• Simulate a mechanism (bottom-up, assumption-driven): you specify the dynamics explicitly — geometric Brownian motion, a block bootstrap, a regime-switching model — and sample from your equations. Interpretable, data-light, but only as realistic as your assumptions. (Lesson 3.)
• Learn a generator (top-down, data-driven): you train a neural network — GAN, VAE, or diffusion model — to reproduce the data’s distribution. Flexible and high-fidelity, but opaque and data-hungry, with real leakage risk. (Lessons 4–6.)

Worked example. Suppose you want fat-tailed returns with volatility clustering. Mechanism path: plain GBM gives you neither — its returns are Gaussian and homoscedastic, so kurtosis $\approx 3$ (no fat tails) and no clustering. You’d need to upgrade to a regime-switching or stochastic-vol mechanism to get there, and you’d know exactly which assumption produced the fat tails. Learned path: a well-trained GAN might reproduce kurtosis of, say, $\approx 8$ — matching the real fat tails beautifully — but you couldn’t point to which parameter did it, and you’d have to run memorization tests to be sure it didn’t just photocopy a crash.

Here’s the part people skip: classical simulators are the baseline the learned models must beat. A GAN that doesn’t reproduce stylized facts better than a well-tuned regime-switching simulator — while costing 100x the compute and carrying leakage risk — has earned nothing. We start with the simple mechanisms in lesson 3 precisely so you have a yardstick.

When to use it

Reach for mechanisms when you need interpretability, have little data, or want an auditable stress scenario you can defend to a regulator. Reach for a learned generator when you have enough data, the stylized facts are too rich to hand-specify, and you’ve budgeted the evaluation effort (lesson 7) to prove it isn’t leaking. In practice: start mechanistic, graduate to learned only when it provably earns its keep.

The leakage trap that frames everything

Analogy. A student writes the exam, then is hired to write next year’s exam. If they reproduce questions from memory, every future student who “studies the practice exam” is just memorizing the real answers. The practice exam looks legitimate — different cover sheet, official font — but it’s the actual test in disguise. A generator that memorizes history is that student, and your backtest is the future class scoring suspiciously high.

Definition. Leakage is when information from the data your model is being tested on contaminates the data it was trained on (or vice versa), inflating measured performance. With learned generators it’s insidious: train a generator $G$ on your full history $\mathcal{H}$, then run train-on-synthetic, test-on-real (or backtest on $G$‘s output). Because $G$ saw all of $\mathcal{H}$ — including the period your test pretends is the unseen future — any memorization in $G$ silently feeds the answers forward. The “synthetic” data isn’t independent of the truth; it’s a lossy copy of it.

Worked example — the generator that cheats and gets away with it. Picture the worst case: a generator that perfectly copies its training set (it just samples real historical windows and relabels them “synthetic”).

Every distribution test gives it a clean bill of health, because matching the distribution is exactly what a perfect copy does. Yet the moment you use it for synthetic backtesting, the strategy’s Sharpe balloons — it’s being “validated” on the very prices it was fit to. A generator that memorizes is indistinguishable from a great generator on every distributional test, and catastrophically worse on the one thing you cared about. That’s why distribution-matching alone can never certify a generator.

The discipline (the thesis of this course). Evaluate the generator before you trust a single synthetic-data result. Specifically, run memorization / nearest-neighbour tests — for each synthetic sample, check it isn’t suspiciously close to a real training example — before any train-on-synthetic or synthetic-backtest claim. That evaluation toolkit is lesson 7, and it’s the gate every learned generator must pass.

When to use it

This isn’t an occasional check — it’s a permanent gate. Every time you train a generator and intend to use its output for anything you’ll trust (augmented training, a backtest, a shared dataset), the leakage question comes first. If you can’t show the generator didn’t memorize, you can’t believe any downstream number it produced. Carry this through lessons 4, 5, and 6; cash it out in lesson 7.

Recap

Machine learning starves in finance because you observe markets running once: thousands of autocorrelated rows collapse to a handful of independent regimes, so the effective sample size is tiny and hungry deep models memorize instead of learn. The fix is to manufacture more market history with a generator, and that synthetic data does four jobs — augmenting thin training sets, generating stress scenarios history never delivered, sharing statistics instead of private positions, and backtesting across many paths. You can build that generator two ways: simulate a mechanism (interpretable, assumption-driven, leak-resistant, the baseline) or learn a generator (flexible, data-hungry, opaque, leak-prone). And threading through all of it is the leakage trap: a generator trained on your history can memorize it and pass every distribution test while quietly feeding your backtest the answers — so you evaluate the generator for memorization before you trust any result it produces. That last sentence is the course in one line.