Time-Series Foundation Models: TimesFM, Chronos, Moirai & Lag-Llama

Lesson 1 sold you the bet: pretrain once on a mountain of data, adapt cheaply to your problem. That’s how a single language model writes Python, French and limericks without being retrained for each. This lesson cashes the bet out for time series — the actual machinery that lets one frozen model forecast a series it has never seen, with no fitting at all.

The catch is that a series of returns is not a sentence. A transformer eats tokens — discrete things with embeddings — and a price chart is a stream of continuous reals. So before any of the foundation-model magic, somebody has to answer an annoyingly concrete question: how do you turn a wiggly line into tokens? Two camps answer it differently, and the rest of the lesson is built on that fork.

Let’s earn it.

From one-model-per-series to a forecasting foundation model

The analogy. The classical forecaster is a locksmith who files a brand-new key for every single lock — ARIMA coefficients for this series, a GARCH fit for that one, a fresh neural net for the next. Beautiful craftsmanship, and utterly un-scalable: 5,000 series means 5,000 fits, each needing enough history to estimate its own parameters. A foundation model is a master locksmith who has already opened millions of locks and learned what locks are. Hand it a new one and it opens it on sight — no new key filed.

Precise definition. The classical recipe (lesson 1’s “specialist”) estimates parameters $\theta_s$ separately for each series $s$: fit $\text{ARIMA}(p,d,q)$ here, $\text{GARCH}(1,1)$ there, a fresh net somewhere else. Each fit needs that series to carry enough signal to pin down its own $\theta_s$. A time-series foundation model instead learns one shared parameter set $\Theta$ by pretraining on a corpus of millions of series across many domains, then forecasts a new series $x_{1:T}$ in a single forward pass:

$\hat{x}_{T+1:T+h} = f_\Theta(x_{1:T}),$

with $\Theta$ frozen — no gradient step touches the new series. That’s zero-shot forecasting, the headline trick of the whole field (lesson 3 makes “zero-shot” precise).

Worked example — the fitting budget. Say each ARIMA fit needs at least 100 observations to estimate its handful of coefficients reliably, and you have 5,000 series.

• Classical: 5,000 separate estimations. A series with only 30 points of history? You can barely fit it — too few observations for its own parameters. Cold-start is fatal.
• Foundation model: 0 estimations on your data. The 30-point series gets the same forward pass as the 30,000-point one; the model brings the prior it learned from millions of other series, so short history is survivable.

The shift is from “every series teaches its own model from scratch” to “one model has already learned the shape of time series in general, and merely conditions on your history.”

When to use it

Reach for the one-model-per-series classical approach when each series is long, well-behaved, and you have time to babysit individual fits. Reach for a foundation model when you have many series, short histories, or a cold-start need (a brand-new SKU with three weeks of data) — exactly where per-series fitting starves. We’ll see the catch for finance specifically by the end of this lesson.

Tokenizing a continuous series — the core technical problem

The analogy. A transformer is a reader who only understands words. Hand it the raw acoustic waveform of someone speaking and it’s helpless — somebody has to first cut that continuous sound into discrete syllables it can look up. Tokenizing a time series is that cut: turning an unbroken line into a sequence of bite-sized units the attention mechanism can attend over. The two families below cut the line in two very different ways.

(a) Patching — chop into windows, project each to an embedding

Precise definition. Slice the context into contiguous, fixed-length windows called patches. A patch of length $P$ is a vector in $\mathbb{R}^P$; a learned linear map $W \in \mathbb{R}^{d \times P}$ projects it to a $d$-dimensional embedding — one token per patch. A context of length $L$ with patch length $P$ (non-overlapping) yields

$n_{\text{patches}} = \left\lceil \frac{L}{P} \right\rceil$

tokens. This is the PatchTST idea, adopted by TimesFM and Moirai. Patching does two jobs at once: it shrinks the sequence length the $O(n^2)$ attention has to chew (lesson 1’s cost worry), and it lets each token carry local shape instead of a single lonely point.

Worked example — a length-512 context, patch length 32. Number of patch tokens:

$n_{\text{patches}} = \frac{512}{32} = 16.$

So 512 raw observations collapse to 16 patch tokens. Suppose $d = 128$. Each patch is a vector in $\mathbb{R}^{32}$, and the projection $W \in \mathbb{R}^{128 \times 32}$ maps it to a token in $\mathbb{R}^{128}$. The attention then runs over 16 tokens, not 512 — the score matrix is $16 \times 16 = 256$ entries instead of $512 \times 512 = 262{,}144$. Same information, roughly $1{,}000\times$ less attention arithmetic. The model forecasts by predicting the next patch (a length-32 block), so it emits 32 future points per output step.

(b) Scaling + quantization into a vocabulary — the Chronos route

Precise definition. Instead of projecting real-valued patches, Chronos turns the series into literal discrete tokens so it can reuse an off-the-shelf language-model stack (a T5-style encoder-decoder). Two steps:

1. Mean-scale the context: divide every value by the mean absolute value of the context, $s = \tfrac{1}{L}\sum_{t=1}^{L} |x_t|$, giving scaled values $\tilde{x}_t = x_t / s$. This makes a series of millions and a series of decimals look comparable.
2. Quantize each scaled value into one of $B$ bins (Chronos uses $B = 4096$) spanning a fixed range. The bin index is the token — an integer in $\{1, \dots, B\}$ — exactly like a word ID in a vocabulary.

Now a price series is a sentence of integers, and a vanilla T5 language model can be trained on it with no architectural surgery. Forecasting samples next-token IDs, which you de-quantize (back to a bin centre) and un-scale (multiply by $s$).

Worked example — scale then bin a tiny window. Context $x = [2, 4, 6, 8]$. Mean absolute value:

$s = \frac{|2| + |4| + |6| + |8|}{4} = \frac{20}{4} = 5.$

Scaled context: $\tilde{x} = [0.4,\ 0.8,\ 1.2,\ 1.6]$. Now suppose the bin grid is uniform on $[-10, 10]$ with $B = 4096$ bins, so each bin has width

$w = \frac{10 - (-10)}{4096} = \frac{20}{4096} \approx 0.004883.$

The scaled value $1.2$ lands in bin index

$\left\lfloor \frac{1.2 - (-10)}{w} \right\rfloor + 1 = \left\lfloor \frac{11.2}{0.004883} \right\rfloor + 1 = \lfloor 2293.4 \rfloor + 1 = 2294.$

So the value $6$ becomes token 2294. The whole window becomes a short string of such integers, and the T5 stack treats it exactly like text.

When to use each

Patching keeps values continuous (no quantization error) and shrinks sequence length — great when you want fidelity and a purpose-built stack. Tokenizing into a vocabulary throws away some precision (every value is rounded to a bin centre) but lets you inherit the entire mature language-model toolchain for free. Patching trades engineering effort for accuracy; quantization trades accuracy for reuse.

The pretraining objective — autoregressive and probabilistic

The analogy. It’s the language-model trick, ported to numbers. A language model learns by covering the next word and guessing it; the word it covered is the answer key, so the text labels itself. A time-series foundation model covers the next patch (or next quantized token) and guesses it — the future of the window is its own free label. Train on millions of series this way and the model absorbs the grammar of time series: trend, seasonality, mean-reversion, level shifts.

Precise definition — autoregressive next-step. Factor the sequence likelihood left-to-right and maximize it:

$\log p_\Theta(x_{1:T}) = \sum_{t} \log p_\Theta(x_{t} \mid x_{<t}),$

where each “step” is a patch (TimesFM, Moirai) or a vocabulary token (Chronos). Because a position may only condition on the past $x_{<t}$, attention is causal — exactly the masked, lower-triangular attention from the deep-learning course. Each patch token attends only to earlier patch tokens.

Precise definition — probabilistic, not point. A good forecaster doesn’t just guess a number; it guesses a distribution, because “the next return is +0.3% give or take 2%” is far more useful than a bare “+0.3%”. Two ways foundation models deliver a distribution:

• Quantile heads (TimesFM): the model outputs several quantiles at once — e.g. the 10th, 50th and 90th percentiles — trained with the pinball (quantile) loss. For a quantile level $\tau$ and prediction $\hat{q}$, the pinball loss on outcome $y$ is

$\rho_\tau(y, \hat{q}) = \max\big(\tau(y - \hat{q}),\ (\tau - 1)(y - \hat{q})\big).$

• Sampled trajectories (Chronos): because the output is a probability distribution over the next token, you can sample many possible next tokens, roll each forward autoregressively, and read percentiles off the cloud of sampled futures — a Monte-Carlo predictive distribution. (Lag-Llama instead heads a parametric Student-t distribution, capturing fat tails directly.)

Worked example — pinball loss tells the median from the 90th. True outcome $y = 12$.

• Median forecast ($\tau = 0.5$), prediction $\hat{q} = 10$: error $y - \hat{q} = 2 > 0$, so $\rho_{0.5} = 0.5 \times 2 = 1.0$.
• 90th-percentile forecast ($\tau = 0.9$), same prediction $\hat{q} = 10$: $\rho_{0.9} = 0.9 \times 2 = 1.8$.

The high-$\tau$ head is punished harder for under-shooting (it’s supposed to sit above most outcomes), so training pushes the 90th-percentile prediction upward relative to the median. That asymmetry is exactly how one network learns a whole spread of quantiles instead of a single point — the spread is the uncertainty estimate.

When to use it

Lean on the probabilistic output whenever you need calibrated uncertainty — risk limits, scenario fans, anything where being wrong about how wrong you might be is expensive. The autoregressive structure is the right inductive bias when the future genuinely depends on the past in a causal, left-to-right way (almost all forecasting). The one regime where this objective struggles is precisely low-signal data: if “predict the next step” is near-impossible (efficient-market returns), the objective still trains happily — it just learns to output a wide, near-trivial distribution, which lessons 3 and 6 will show is often the honest answer.

The lineage — TimesFM, Chronos, Moirai, Lag-Llama

The analogy. Think of these four as rival house styles for the same instrument. They all play “forecast the future from the past,” but they differ in how they read the sheet music (tokenization), how the band is arranged (architecture), whether they play one voice or many at once (uni- vs multivariate), and whether they hand you a single note or a whole chord of possibilities (probabilistic output).

Precise comparison.

A few things to read off the table. TimesFM and Lag-Llama are both decoder-only (the GPT-style causal stack), but TimesFM patches the raw series while Lag-Llama feeds lagged features (the value at lag 1, 7, 30, …) — engineering seasonality straight into the input, then heading a Student-t to capture fat tails. Chronos is the odd one out: by quantizing into a vocabulary it gets to be a language model, inheriting that whole ecosystem, at the price of rounding error. Moirai is the multivariate specialist — it can attend across several variates at once and chooses among patch sizes, which matters when your series come at different frequencies.

Worked example — picking from the lineage. You must forecast 200 correlated commodity series jointly, where one series’ move informs another’s. Univariate TimesFM/Chronos/Lag-Llama would forecast each in isolation, ignoring the cross-series structure. Moirai’s any-variate design is built precisely to attend across the 200 series together — the natural pick. Flip the task to a single, very long, fat-tailed series where you care about tail risk, and Lag-Llama’s Student-t head earns its place.

When to use which

Use TimesFM as a strong general univariate baseline with clean quantiles; Chronos when you want to ride the language-model toolchain and tolerate quantization; Moirai when the problem is genuinely multivariate or multi-frequency; Lag-Llama when seasonality is strong and tails are fat and you want an open, Student-t model. None of these choices, note, changes the uncomfortable fact coming next: what they were trained on.

The pretraining corpora — billions of points, almost none of them market-like

The analogy. Imagine training a champion sommelier by having them taste millions of glasses of — water. Mineral water, sparkling water, tap water from forty cities. They’ll become extraordinary at distinguishing waters. Then you ask them to grade a Bordeaux. The skill they built is real and deep, but it was built on the wrong distribution. The foundation-model corpora are oceans of well-behaved series — and returns are the Bordeaux.

Precise statement. The pretraining corpora (think the LOTSA / Chronos-style mixtures) aggregate billions of time points across energy (electricity load), weather, traffic, retail/web demand, transport, and a sprinkling of finance. The defining property of most of these domains is high signal-to-noise: electricity load has a thundering daily-and-weekly seasonality you can forecast in your sleep; traffic and demand are strongly autocorrelated and regular. Financial returns are the opposite — near-efficient, dominated by unforecastable noise, non-stationary across regimes (the brutal-low-SNR regime from the deep-learning course). So two problems stack:

1. Coverage: market-like series are a tiny slice of the corpus, so the model spent almost none of its capacity learning their dynamics.
2. Even where present, low SNR: the financial fraction is the hardest possible data to extract a transferable pattern from — there often isn’t a stable pattern to learn.

Worked example — the mix is lopsided. Suppose a corpus holds 10 billion points, split roughly: energy 35%, traffic 20%, weather 15%, demand/web 25%, finance 5%. That’s $0.05 \times 10\text{B} = 500$ million financial points — sounds enormous. But “financial” here is mostly prices, much of it low-frequency, and the returns you actually care about carry almost no forecastable signal per point. So the effective market-relevant, signal-bearing fraction the model learned from is far smaller than 5% — and what little it saw may teach the wrong lessons (a trending crypto sample is not a stationary law). Contrast 3.5 billion points of metronomic electricity load, where every point reinforces a clean seasonal prior.

When the corpus mismatch matters most

If your target series resembles the corpus — a demand forecast, a load curve, a well-behaved seasonal business metric — the mismatch barely bites and zero-shot transfer can be excellent. If your target is a single adversarial return series, the corpus mismatch is maximal and the pretrained prior is on its weakest footing. The decision rule is blunt: the more your series looks like the training domains, the more you should trust the model; the more it looks like returns, the more you should test before believing anything.

When a time-series foundation model is the right tool

The trade-off, stated plainly. A time-series foundation model is a magnificent cold-start baseline: hand it a new, well-behaved series with little history and it produces a calibrated, probabilistic forecast in one forward pass, no fitting, no babysitting. That is genuinely useful and genuinely new. But the same machine, pointed at a single low-SNR return series, is leaning on a prior built from the wrong distribution — and “no fitting” now means “no chance to adapt to how unlike-the-corpus your series is.”

Worked example — choosing on two desks. Desk A forecasts next-week electricity demand for 5,000 substations, most with under two months of data. A foundation model is close to ideal: corpus-like, cold-start, many series — exactly its strengths from the first section. Desk B forecasts tomorrow’s return for one large-cap stock. Same model, opposite footing: one series, brutal SNR, a prior trained on water trying to grade a Bordeaux. Desk A should ship the zero-shot baseline today; Desk B should treat any impressive backtest as guilty-until-proven-innocent (lesson 5’s leakage minefield, lesson 6’s honest evaluation).

When to use it

Use a time-series foundation model as your first, cheap, zero-shot baseline whenever you have many series, short histories, or a cold-start — and especially when those series resemble the high-SNR, seasonal domains it was trained on. Be deeply skeptical of it as a standalone market-return forecaster on a single adversarial series; there the corpus mismatch and the noise floor conspire against the very transfer that makes it powerful elsewhere. The next lesson makes “zero-shot vs few-shot” precise and asks whether the scaling laws that lift these models on clean data lift them on markets at all.

Recap