The Pretrain-and-Adapt Bet

For a decade, the machine-learning recipe in finance read like a craft project: pick one task, hand-label a dataset for exactly that task, train a model from scratch, ship it, and start over from zero for the next task. Want to forecast volatility? Build a vol model. Want to classify regimes? Build a regime model. Nothing carried over. Every problem was an island, and you paid full freight to settle each one.

Then language, vision, and weather quietly switched to a different recipe — pretrain a single big model on a mountain of broad data once, then adapt that one model cheaply to dozens of downstream tasks. It worked so spectacularly that “foundation model” became the default way to build. This course asks the only question that matters for us: does that bet pay off when the downstream task is a financial market? This first lesson sets up the wager — what a foundation model is, why the economics are seductive, and exactly which assumption markets are poised to violate. Lessons 2–6 then stress-test it with real models (Chronos, TimesFM, Moirai, Lag-Llama), scaling laws, leakage traps, and honest evaluation.

Let’s define the bet before we grade it.

What a foundation model actually is

The analogy. The old recipe is hiring a fresh specialist for every disease that walks through the door — a cardiologist for chest pain, a separate dermatologist for a rash, each trained from zero on just their slice. A foundation model is the seasoned generalist resident who has rotated through every ward and seen tens of thousands of cases. They don’t know your patient yet, but they walk in carrying a vast, transferable sense of how bodies work — and a short briefing (“here’s this patient’s chart”) is enough to put them to work. The expensive education happened once; each new case is a cheap adaptation.

Precise definition. The term comes from Bommasani et al. (Stanford, 2021): a foundation model is a model trained on broad data at scale (typically with self-supervision) that can be adapted — fine-tuned, prompted, or used zero-shot — to a wide range of downstream tasks. Two clauses do all the work: broad pretraining (one model, lots of generic data, no specific task in mind) and adaptation (cheaply specializing that one model to many concrete tasks).

The two recipes, side by side. The shift is from one labeled dataset → one model → one task to one giant pretraining run → one shared model → many tasks:

The phrase “foundation” is deliberate: the pretrained model is meant to be the base layer everything else is built on, not the finished building.

When to use it

The pretrain-and-adapt framing earns its place when you face many related tasks that plausibly share structure, and labeled data for any single task is scarce or expensive. If you only ever have one task and you have abundant clean labels for it, a focused from-scratch model is simpler and you skip the whole bet. The framing’s value scales with task count and shared structure — which is exactly the economics we make precise next.

The two-phase economics: amortization

The analogy. Amortization is the logic of a season pass. A single lift ticket is cheap; a season pass costs a fortune up front but almost nothing per day after. One weekend of skiing? Buy day tickets. Forty days? The pass is a steal. A foundation model is a season pass for ML tasks: a brutal fixed pretraining cost, then near-free entry to each new task. Whether the pass beats day tickets depends entirely on how many days you’ll ski — i.e., how many downstream tasks you have.

Precise model. Let $N$ be the number of downstream tasks. Split costs into two regimes:

• From scratch: a fixed per-task training cost $c$, paid every task, with no sharing. Total cost is purely linear: $C_{\text{scratch}}(N) = c\,N.$
• Foundation model: a one-time pretraining cost $P$, paid once regardless of $N$, plus a small marginal adaptation cost $a$ per task: $C_{\text{found}}(N) = P + a\,N.$

The foundation line starts high (you owe $P$ before task one) but climbs slowly (slope $a \ll c$). The from-scratch line starts at zero but climbs steeply (slope $c$). Two lines, different slopes and intercepts — they cross.

Solving for the crossover. The foundation model wins when $C_{\text{found}}(N) < C_{\text{scratch}}(N)$: $P + aN < cN \quad\Longrightarrow\quad N > \frac{P}{c - a}.$

Worked example — show the arithmetic. Use the illustrative units $c = 10$, $P = 40$, $a = 1$.

• From scratch over $N$ tasks: $C_{\text{scratch}}(N) = 10N$.
• Foundation over $N$ tasks: $C_{\text{found}}(N) = 40 + N$.

Set them equal to find break-even: $10N = 40 + N \Rightarrow 9N = 40 \Rightarrow N = 40/9 \approx 4.44$. Since you can’t have a fractional task, the foundation model becomes cheaper at $N = 5$ and beyond. Let’s tabulate to make the crossover concrete:

At $N = 20$ the foundation model covers all twenty tasks for 60 units while the from-scratch route burns 200 — more than 3× cheaper, and the gap widens forever after. Drag the slider below and watch the bars trade places exactly at the crossover.

When to use it

The amortization bet is worth making when $N$ is large and the per-task adaptation genuinely produces a competitive model — many related tasks, cheap transfer, shared structure. It’s a bad bet when $N$ is tiny (you never reach the crossover) or when transfer is so weak that the adapted models underperform bespoke ones (you reach the crossover on cost but lose on quality). Markets, we’ll see, threaten both conditions at once.

Why pretrain-and-adapt conquered language, vision, and weather

The analogy. Imagine learning a language by being handed every book ever written with the last word of every sentence covered up, and being asked to guess it — billions of times. You’d never pay a teacher; the text grades itself. That’s self-supervision: the data secretly contains its own answer key. Predict the next token, and the actual next token tells you if you were right. Mask a patch of an image, and the original pixels are the answer. Suddenly “labeled data” is infinite, because every raw example is a labeled example.

The three ingredients that made the recipe explode. Pretrain-and-adapt didn’t win on cleverness alone. It won where three conditions all held:

1. Free labels via self-supervision. Next-token (language), masked-patch (vision), next-state (weather) — no human annotation needed, so pretraining data is effectively unlimited.
2. Abundant data. Web-scale text, billions of images, decades of gridded reanalysis weather fields. The fixed pretraining cost $P$ buys a representation only if there’s a mountain of data to learn it from.
3. Near-stationary structure. This is the quiet hero. Grammar today works like grammar next year. A cat’s edges and textures don’t mutate when you switch datasets. Atmospheric physics obeys the same equations forever. Because the data-generating process is stable, a representation learned during pretraining still describes the world at adaptation time — so it transfers.

Why stationarity is the load-bearing wall. Transfer is the entire premise of amortization: the pretrained representation only helps a new task if the new task looks like what was pretrained on. When the underlying rules don’t drift, “looks like” holds. Stationarity is what lets a representation learned yesterday be valid tomorrow. Pull that wall out and the whole structure — economics included — collapses.

Notice the first three rows share all three ingredients, and the fourth — markets — fails on the one that matters most. That single red cell is what the rest of this course is about.

When to use it

Reach for self-supervised pretrain-and-adapt when all three ingredients are present: free labels, abundant data, and a stable generating process so the representation transfers. Miss any one and the recipe weakens. Miss stationarity in particular and the economics still look attractive (the cost crossover is real) while the quality quietly evaporates (the transfer fails) — the most dangerous failure mode, because the spreadsheet says yes while the out-of-sample results say no.

The adaptation spectrum (a preview)

Once you have a pretrained model, “adapt it to a task” is a spectrum, not a single move. Three landmarks, one sentence each (lesson 3, Zero-Shot, Few-Shot & Scaling Laws, goes deep on all of them):

• Zero-shot — run the pretrained model on the new task with no task-specific examples at all; you lean entirely on what pretraining already learned. Cheapest, most demanding of transfer.
• Few-shot — show the model a handful of labeled examples (in-context or via light tuning) to nudge it toward the task. A middle ground.
• Full fine-tuning — update the model’s weights on a task-specific dataset, specializing it the most but risking overfitting on small, noisy financial data.

The further left you sit (zero-shot), the more you’re betting on raw transfer; the further right (full fine-tune), the more task data you need and the more you risk the data-hunger problem from the prior course. The adaptation cost $a$ in our economics grows as you move right.

The central tension with markets

Here’s where the bet meets its toughest table. Amortization buys you cheap models; it only buys you cheap good models if the representation transfers, and transfer needs the target task to resemble the pretraining distribution. Markets attack that resemblance from three directions at once.

1. Non-stationarity — the rules drift. The data-generating process of markets is not fixed. Volatility regimes shift, correlations break in crises, central-bank policy rewrites the playbook, market structure evolves (decimalization, high-frequency trading, new venues). A representation learned on 2010–2015 may describe a market that no longer exists by 2026. Unlike grammar or physics, the thing you pretrained on moved.

2. Adversarial reflexivity — your edge decays as others find it. Finance is the one domain where using a pattern destroys it. If a signal genuinely predicts returns, traders pile in, push prices, and arbitrage the edge away. This is efficient-market pressure made concrete: the act of exploiting structure feeds back and erases it. A cat does not learn to hide its edges because a vision model got good at finding them; a market absolutely learns to hide its alpha because everyone’s model got good at finding it.

Worked example — an edge crowding to death. Suppose a signal earns a gross excess return of $r_0 = 8\%$ per year when only you trade it. As capital crowds in, the edge decays geometrically — say it shrinks by a factor of $0.7$ each year as competitors discover it: $r_t = r_0 \cdot 0.7^{\,t}.$ Then year by year: $r_0 = 8.0\%$, $r_1 = 8 \times 0.7 = 5.6\%$, $r_2 = 8 \times 0.49 = 3.92\%$, $r_3 = 8 \times 0.343 = 2.74\%$, $r_4 = 8 \times 0.2401 = 1.92\%$. Now subtract a realistic all-in trading cost of, say, $2\%$ per year. The net edge is $r_t - 2\%$: it’s $+6.0\%$ in year 0, but by year 4 it’s $1.92\% - 2\% = -0.08\%$ — negative. The signal didn’t just fade; continuing to trade it now loses money. A foundation model that learned this pattern during pretraining will keep confidently emitting it long after the market has arbitraged it into a cost-bleeding liability. The pattern was real in the pretraining window and worthless — even harmful — by deployment.

3. Brutally low signal-to-noise. Even where structure exists, it’s buried. Daily returns are dominated by unforecastable noise; the predictable component is a sliver. A representation pretrained to “predict the next return” is mostly fitting noise, because noise is most of what’s there. This is the data-hunger and low-SNR problem from the prior deep-learning course, inherited wholesale.

So the bet’s geometry is clean and its premise is precisely what markets break. The cost lines still cross — pretrain-and-adapt is genuinely cheaper at scale. But “cheaper” only becomes “better” through transfer, and transfer needs a stable, non-adversarial, signal-rich target. Markets are non-stationary, adversarial, and low-SNR. That’s not a reason to abandon the bet — it’s the reason this course exists: to find where the bet does pay (it sometimes does) and to evaluate it honestly enough that we don’t fool ourselves.

When the pretrain-and-adapt bet is worth making vs. not

It comes down to a clean trade-off between task count + shared structure (which favor the bet) and non-stationarity + adversarial decay + low SNR (which sink it):

Worth making when you have many related tasks that share structure and data is cheap — e.g., forecasting many benchmark time series at once, where lots of series share seasonality and trend dynamics (precisely where TimesFM, Chronos, Moirai and Lag-Llama in lesson 2 actually shine). Doubtful when the target is a single, heavily-arbitraged, non-stationary return series and you’re betting the whole edge on a representation learned from a moving, adversarial past. The honest answer is “it depends, and you must measure it without fooling yourself” — which is why lessons 5 (leakage) and 6 (evaluation) exist.

Recap