Evaluating Synthetic Data Without Fooling Yourself

We have spent six lessons learning to make fake market data — classical baselines, GANs, VAEs, diffusion. This lesson is the one the whole course has been quietly pointing at, and it is the hard one: a generator is only ever as trustworthy as your ability to evaluate it. You can build the most elegant diffusion sampler in the building, but if you cannot prove its output is both realistic and honest, it is a liability, not an asset. Making the data is the fun part. Judging it is the part that keeps you out of trouble.

Here is the punchline, stated bluntly so it sticks: the single easiest way to “win” every realism test is to memorize the training set. A generator that simply replays the paths it was trained on will have flawless tails, perfect autocorrelations, a discriminator that can’t tell real from fake — a gold star on every distributional report card. And it is also the most dangerous failure mode, because memorized data is the past photocopied back at you. Drop that into a backtest and you are no longer simulating the future; you are leaking it. Your Sharpe ratio will look spectacular, right up until real money meets real markets.

So evaluation is a gauntlet, not a number. No single test certifies a generator. You run a battery — marginals and dependence, a discriminator, a use-oriented predictive score, an explicit memorization probe, and a leakage audit — precisely because each test has a blind spot that the next one is designed to cover. Skip any rung of the ladder and the memorizer walks straight through the gap. Let’s build the ladder.

Distributional & dependence tests

Analogy. Imagine grading a forger. The first thing you check is whether each individual brushstroke looks like the master’s — the colours, the texture, the weight of the line. That’s the marginal distribution. But a master is also recognizable by how strokes relate: rhythm, recurring motifs, the way a dark passage follows a bright one. That’s the dependence structure. A forger who nails the colours but botches the rhythm is still a forger.

Definition. Distributional tests ask whether the synthetic data’s marginal statistics match the real ones; dependence tests ask whether its relationships across time match. Concretely, you re-run the stylized-facts rubric from lesson 2 on the synthetic series and compare, fact by fact:

• Fat tails — tail index and excess kurtosis (real returns are leptokurtic, kurtosis far above 3).
• No linear return autocorrelation — the ACF of raw returns should die almost immediately.
• Volatility clustering — the ACF of absolute (or squared) returns should decay slowly, over many lags.
• Leverage effect — negative correlation between today’s return and tomorrow’s volatility.

The instruments are standard: QQ-plots for visual tail comparison, the Kolmogorov–Smirnov and Anderson–Darling distances to quantify how far apart two distributions are (Anderson–Darling weights the tails more heavily, which is exactly where finance lives), and side-by-side ACF curves for the dependence facts.

Worked example. You sample 5,000 synthetic daily returns and tabulate against the real series:

Synthetic A tracks every fact — fat tails (kurtosis 5.8, tail index 3.3), no return autocorrelation, persistent volatility clustering (ACF of absolute returns still 0.16 at lag 10). Synthetic B is a near-Gaussian impostor: kurtosis 0.3, tail index 8.0 (far too thin), and absolute-return ACF that vanishes — no volatility clustering at all. The KS distance of 0.21 confirms B’s marginal is way off. B fails; A passes this rung.

When to use it

Always, and first. This is the cheap, fast smoke test: if a generator can’t even match the marginals and the volatility-clustering ACF, you stop here and fix it before spending effort on anything downstream. It rules out bad generators; it can never rule in a good one.

The discriminative score (can a classifier tell them apart?)

Analogy. Hire the sharpest art authenticator in the world and hand them a stack of paintings, half genuine, half forged, with the labels hidden. If the expert’s guesses are no better than a coin flip, the forgeries are indistinguishable. If the expert nails every fake, the forgeries have a tell. The discriminative score automates that expert.

Definition. Train a binary classifier (gradient-boosted trees, a small net — whatever is strong) to predict the label “real” vs. “synthetic” on a mixed, held-out set. Report its AUC (area under the ROC curve):

• AUC ≈ 0.5 → the classifier can’t beat chance. The synthetic and real distributions overlap. Good.
• AUC ≈ 1.0 → the classifier separates them perfectly. The fake has an obvious tell — wrong tails, a giveaway artifact, the wrong dependence. Bad.

This is strictly stronger than the marginal tests, because a powerful classifier can exploit joint and higher-order structure a handful of summary statistics miss.

Worked example. Three candidate generators, same real data:

The diffusion model is winning the realism contest. But note the trap waiting in the next callout.

When to use it

After the stylized facts pass, as your headline realism metric — it’s the single number that best summarizes “does the fake look real, accounting for joint structure.” Track it while tuning a generator. Just never let a 0.5 lull you into thinking you’re done; you’ve cleared realism, not memorization.

The predictive score (Train on Synthetic, Test on Real)

Analogy. A flight simulator is only worth the hangar space if pilots who train in it fly real planes well. Nobody cares how photoreal the simulator’s clouds are if graduates crash the actual 737. Train-on-Synthetic, Test-on-Real (TSTR) is the simulator-graduation test for data.

Definition. TSTR: train your downstream model (a return predictor, a classifier, a strategy) entirely on synthetic data, then evaluate it on a real held-out set you never generated from. Compare against TRTR — Train-on-Real, Test-on-Real, the same downstream model trained on real data. If the synthetic data carries the signal the model needs, then:

$\text{TSTR} \approx \text{TRTR}$

A small TSTR–TRTR gap means the synthetic data is useful — it can stand in for, or augment, real data. A large gap means the generator dropped the signal (often because it smoothed away the very structure the downstream task depends on).

Worked example. Same downstream classifier, three training regimes, evaluated on the same real test set:

Synthetic A nearly matches the real benchmark (accuracy 0.59 vs 0.61; Sharpe 0.98 vs 1.05) — it preserved the predictive signal and is genuinely useful for augmentation. Synthetic B collapses to near-chance accuracy and a feeble Sharpe — realistic-looking marginals, but the signal is gone. This is the test where a “pretty but useless” generator finally gets caught.

When to use it

Whenever the goal is augmentation, privacy-preserving sharing, or stress-testing — i.e. almost always. TSTR is the most use-oriented test in the gauntlet: it speaks directly to “will this data help my actual task?” rather than “does it look pretty?” Make it your acceptance criterion for shipping a generator into a training pipeline.

The memorization & nearest-neighbour test — the one that catches a copier

Analogy. A student who understands the material writes new sentences that are correct. A student who plagiarizes writes sentences that are word-for-word identical to the source — and a plagiarism checker doesn’t grade quality, it measures how close each sentence is to something in the source library. That’s the nearest-neighbour test, applied to data.

Definition. For each synthetic sample, find its nearest neighbour in the training set (smallest distance under a sensible metric on returns/paths) and record that distance. Then compare two distributions of distances:

• Synthetic → train nearest-neighbour distances, and
• a baseline: holdout → train distances (real samples the generator never saw, measured to the same training set).

If the synthetic-to-train distances are far smaller than the holdout-to-train distances, the generator is sitting on top of its training data — it’s copying, not generating. A healthy generator produces synthetic-to-train distances comparable to holdout-to-train distances: novel, but drawn from the same neighbourhood.

This is the rung that catches what every previous test misses. Drag the slider below and watch it happen.

Walk through it deliberately: at the Novel end the verdict is healthy — distinct paths, comfortable nearest-neighbour distance, safe to train on. Slide toward Copying and the synthetic curves snap onto real training paths while the distance readout plummets. That collapse-to-zero is the number you compute in production. Everything to the left of “Copying” is a generator you can trust; everything sliding toward zero distance is a backtest about to lie to you.

Worked example. You compute median nearest-neighbour distances (smaller = closer to training data):

Generator A’s synthetic samples are about as far from the training set as unseen real data is — exactly right. Generator B’s samples are practically glued to training rows (0.06 vs a 1.00 baseline). B is a photocopier wearing a generator costume, and only this test exposed it.

When to use it

On every generator, every time, before it is allowed near a backtest or a training set. It’s cheap (a nearest-neighbour search) and it is the only defence against the failure mode that all the realism metrics are blind to. Treat a passing memorization test as a precondition for trusting any of the other scores.

The leakage minefield

This is the warning the whole course has been circling, now made concrete. Recall the creed from Deep Learning for Market Data: leakage is the silent Sharpe-inflator, and a generator is a brand-new, sneaky way to commit it.

Analogy. Imagine a “history teacher” who has read tomorrow’s newspaper, then sets you a quiz about yesterday. Their questions will subtly steer you toward what actually happened next — and you’ll ace the quiz, learning nothing about whether you could have predicted it in real time. A generator fit on your full history is that teacher. It has seen data that is future relative to any earlier backtest window, and the synthetic paths it produces can quietly encode that future.

Definition. Generator leakage is the contamination that occurs when a generator is fit on data that overlaps or post-dates the evaluation window, so its synthetic output carries information the strategy could not have known in real time. Run train-on-synthetic / test-on-real on those paths and the leaked future inflates your Sharpe — exactly the way fitting features on the test set does, but harder to spot because the synthetic data looks clean and the memorization may be diffuse rather than literal.

The fixes — and they are non-negotiable:

1. Fit the generator ONLY on the training window. Never let it touch the test period. The generator is part of your model, so it must obey the same temporal wall every other component does.
2. Nest generation inside purged, embargoed cross-validation. Re-fit (or refuse to use) the generator within each fold, purging samples whose information overlaps the test fold and embargoing a buffer after it — the same purged/embargoed CV from Deep Learning for Market Data. A generator fit once on everything and reused across folds defeats the entire scheme.
3. Deflate for the generator search. Every generator architecture, seed, and hyper-parameter you tried is a trial. Searching across many generators and keeping the best inflates Sharpe by multiple testing — so apply the deflated-Sharpe correction for the number of generator trials, just as you would for strategy trials.

When to use it

This isn’t a test you run once; it’s a discipline you build into the pipeline. Audit for leakage every time a generator feeds a backtest or a model-selection loop — i.e. continuously. The memorization test catches literal copying; the leakage audit catches the subtler temporal contamination that survives even when nearest-neighbour distances look fine.

The evaluation checklist

No single test suffices — that has been the drumbeat of every section. Each rung covers a blind spot in the one before it, so you run them in order and a generator must clear all of them:

Notice the pattern: rungs 1–3 are all blind to memorization, which is why rung 4 exists; and rungs 1–4 can all be passed by a generator that nonetheless leaks, which is why rung 5 is a process, not a metric. Skip a rung and the failure it was meant to catch walks straight through.

Recap

Evaluating synthetic market data is harder than generating it, and the reason is a single adversary: a generator that memorizes the training set passes every realism test while leaking the future into your backtests. So you don’t trust a number — you run a gauntlet. Start with the stylized-facts comparison (marginals, tails, the ACFs of returns and absolute returns, leverage) to confirm the data looks real; add the discriminative score so a strong classifier can’t find a tell (AUC near 0.5 is the target); add the predictive TSTR-vs-TRTR test so you know the data is actually useful on real held-out data. Then — crucially — run the memorization / nearest-neighbour test, the only rung that distinguishes “learned the distribution” from “stored the data”: if synthetic-to-train distances collapse below the holdout-to-train baseline, reject the generator. Finally, treat leakage as a process discipline: fit generators on the training window only, nest them inside purged and embargoed CV, and deflate for the generator search. Each rung patches a blind spot in the last. Pass all five, in order, and you have a generator you can actually believe.