LLMs as Research Tools & as Signals

Lessons 1 through 3 built the machinery: the pretrain-and-adapt bet (1), time-series foundation models (2), and how zero-shot, few-shot, and scaling laws actually behave on financial data (3). Now we point that machinery at a job. There are exactly two jobs an LLM can hold on a quant desk, and confusing them is how good teams ship bad research.

A language model can be a signal — you feed it text, it emits a number, and that number predicts returns. Or it can be a researcher — it reads the literature, proposes hypotheses, writes the backtest code, and critiques the results. Same model, radically different risk profiles. A signal lives or dies on out-of-sample predictive power; a researcher lives or dies on whether its tireless creativity is disciplined or merely prolific. This lesson works both roles, then spends most of its energy on the failure mode the researcher role unleashes: an idea generator that can run ten thousand strategies overnight is a multiple-testing bomb with a very short fuse.

Let’s sort the two jobs first, then go deep on each.

Two roles: the LLM as signal vs. as researcher

The analogy. Think of a newsroom. The reporter turns events into a single filed number — “sentiment: +0.7” — that goes straight onto the trading wire; if the reporter is wrong, one story is wrong. The editor-in-chief doesn’t file stories — they assign hundreds of investigations, draft the methodology, and decide which leads to chase. A reckless editor who greenlights ten thousand investigations will, by sheer volume, surface a few that look like Pulitzer material purely by chance. The reporter’s failure is a bad article; the editor’s failure is a newsroom full of confident nonsense. An LLM can play either part, and the second part is where careers quietly end.

Precise definition. In the signal role, the LLM is a function $g$ mapping unstructured text $x$ (a filing, a headline, a transcript) to a real-valued score $s = g(x)$ that enters a return forecast or a portfolio weight. It is evaluated like any alpha factor: information coefficient, out-of-sample Sharpe, turnover, capacity. In the researcher role, the LLM is an agent in a loop — propose a hypothesis $H$, generate feature code, run a backtest, read the result, revise — and it is evaluated on the quality and honesty of the research process, not on a single number.

When to use which

Use the signal role when the edge is reading text faster, cheaper, or more consistently than the market — a stable, repeatable extraction job. Use the researcher role when the bottleneck is idea throughput and engineering boilerplate — and only if you have the validation discipline to survive your own productivity. If you can’t deflate for the trials a researcher-agent runs, you are not ready to let it loose; you’ll just overfit faster.

LLM as signal #1 — sentiment & event extraction

The analogy. Sentiment alpha is fresh fish. Caught at dawn (the moment the news breaks), it’s valuable; by lunch it’s worth less; by tomorrow it stinks. The whole market is also at the dock buying the same fish, so the price of fresh collapses fast. You can make money trading fish, but only if you’re quick, you don’t overpay to transact, and you don’t try to move a tanker’s worth — the market is too small and too perishable for size.

Precise definition. A sentiment-and-event signal scores documents and aggregates them into a cross-sectional factor. For each document $d$ about asset $i$ on day $t$, the LLM emits a polarity $\sigma_d \in [-1, +1]$ (and optionally an event tag — guidance cut, lawsuit, buyback). Aggregate to a daily per-asset score by averaging:

$s_{i,t} = \frac{1}{N_{i,t}} \sum_{d \in D_{i,t}} \sigma_d,$

where $D_{i,t}$ is the set of documents about asset $i$ available on day $t$ and $N_{i,t}$ is its count. Then rank assets cross-sectionally and form a dollar-neutral long-short portfolio: long the top group, short the bottom.

Worked example — a tiny long-short. Five stocks, today’s averaged sentiment scores:

Go long the top two (A, B) and short the bottom two (D, E), each leg dollar-neutral. With $1 long and $1 short total, weight each long name $0.50 and each short name −$0.50. Suppose next-day returns come in at $r_A = +1.2\%$, $r_B = +0.3\%$, $r_D = -0.6\%$, $r_E = -1.4\%$. The long leg earns $0.5(1.2\%) + 0.5(0.3\%) = 0.75\%$. The short leg earns $-0.5(-0.6\%) - 0.5(-1.4\%) = +1.0\%$. Gross long-short return: $0.75\% + 1.0\% = 1.75\%$ for the day. Now subtract costs: if a round-trip on this portfolio costs, say, 20 basis points, net is $1.75\% - 0.20\% = 1.55\%$ — still positive here, but notice how a few extra basis points of cost and a slightly weaker day flip the sign. That razor-thin margin is the whole story of sentiment alpha.

The decay is the constraint. Because the edge concentrates at short horizons, you must re-trade often to keep capturing it — and re-trading repeatedly pays the round-trip cost again and again. Hold too briefly and costs eat you; hold too long and the signal is already stale. There is an optimal holding horizon where net-alpha-per-day peaks, and faster decay pushes that sweet spot earlier. The chart below makes the trade-off tangible: drag the decay-speed slider down (faster decay, like fresh news) and watch the optimal horizon collapse toward “trade now or not at all.”

When to use it

Reach for LLM sentiment when you have a genuine, costly-to-process stream of unstructured text (full earnings-call transcripts, dense filings, non-English news) where consistent machine reading beats the average human reader, and you operate at short horizons with low transaction costs and modest size. Skip it when the text is already trivially summarized by everyone (the headline number everyone reads in the same second), when your costs are high, or when you need a signal that holds size for weeks — the decay won’t let it.

LLM as signal #2 — multimodal price-plus-text fusion

The analogy. A good macro trader watches two screens at once: the price ticking on the left, the news flashing on the right, and the interaction between them — “the stock dropped 2% but the news is benign, so this is a flush, not a fundamental break.” A multimodal model tries to be that trader: one encoder reads the numbers, another reads the narrative, and a fusion layer learns to condition each on the other. The magic is in the joint reading. But a human trader can’t accidentally read tomorrow’s newspaper; a sloppy data pipeline can.

Precise definition. A multimodal price-plus-text model maintains two encoders. A time-series encoder $f_{\text{ts}}$ maps a window of price/volume/microstructure features to an embedding $z_{\text{ts}}$; a text encoder $f_{\text{txt}}$ (often an LLM) maps the relevant documents to an embedding $z_{\text{txt}}$. A fusion module $\phi$ combines them — concatenation, cross-attention, or gating — into a joint representation $z = \phi(z_{\text{ts}}, z_{\text{txt}})$, from which a head predicts the forward return. Cross-attention fusion lets the price embedding query the text (“which part of this filing explains the move I’m seeing?”) and vice versa.

Worked example — what alignment actually requires. Say you want to predict the return over the window $[t, t+1]$ (the close on day $t$ to the close on day $t+1$). Every input — every price bar and every document — must carry a timestamp $\le t$ (specifically, knowable at the day-$t$ close). Concretely:

• A news article timestamped 4:15 PM on day $t$, after the 4:00 PM close, cannot be used to predict the $[t, t+1]$ return labeled at the day-$t$ close — it wasn’t available yet.
• An 8-K filing that hit the wire at 11:00 AM on day $t$ can be used, because it was public before the close.
• A transcript that gets revised (typos fixed, redactions) the next morning must be ingested in its original, time-$t$ form, not the cleaned version — otherwise you’ve smuggled future edits backward.

If even 1% of your documents violate this, your backtest inflates and your live performance collapses. The fusion architecture is the easy part; the point-in-time timestamping is the part that decides whether the result is real.

When to use it

Fuse price and text when the interaction genuinely carries signal that neither stream holds alone — when “this move plus this narrative” means something different from either in isolation — and when you can enforce airtight point-in-time discipline on every document. If you can’t timestamp the text rigorously, don’t add it: a clean price-only model beats a leaky multimodal one every single time, even when the leaky one’s backtest looks spectacular.

LLM as researcher — hypothesis & feature generation

The analogy. The LLM researcher is the world’s most caffeinated intern. It has skimmed every finance paper ever written, it never sleeps, it’ll write a hundred backtest scripts before lunch, and it’ll explain each one with breezy confidence. It is also an intern who has never been wrong out loud — when it doesn’t know, it doesn’t hesitate; it invents. Used as a tireless first-drafter under close supervision, it’s a force multiplier. Used as an oracle whose output you trust, it’s a machine for generating confident garbage at scale.

Precise definition. The “AI quant” or “AI scientist” loop is an agentic pipeline: the LLM (1) proposes a hypothesis (“post-earnings-announcement drift is stronger for high-dispersion names”), (2) operationalizes it into a feature and writes the backtest code, (3) runs the backtest, (4) reads the result and either refines or proposes the next idea — iterating autonomously. The promise is throughput: hundreds of ideas explored per day instead of one per week.

What it’s good at vs. bad at.

The good column is real and valuable. The bad column is why the next section exists: an idea generator that is fast, fluent, and unbothered by statistical conscience is precisely the thing that detonates a data-snooping explosion.

When to use it

Let an LLM run the researcher loop when your bottleneck is genuinely idea and engineering throughput and you have industrial-strength validation downstream — pre-registered out-of-sample windows, multiple-testing deflation, and strict point-in-time data. Don’t use it as a final judge of what’s real, and never trade its output un-audited. The intern drafts; the disciplined process decides.

The data-snooping explosion

The analogy. Give one person a coin and ask for ten heads in a row — they’ll almost certainly fail. Hand the same task to ten thousand people in parallel, and a few will succeed on the first try, look like wizards, and write a book about their technique. Nothing about the coin changed; you just ran enough trials that pure luck produced a champion. An LLM idea-generator is the ten-thousand-people machine, except it does it before your coffee gets cold — and each “champion” arrives wrapped in a confident backtest.

Precise statement — best-of-N Sharpe inflation. Suppose you run $N$ strategies that have zero true edge — pure noise. Each produces an in-sample Sharpe ratio that’s roughly a standard normal draw. The expected maximum of $N$ such draws grows like

$\mathbb{E}[\max_{1 \le j \le N} Z_j] \approx \sqrt{2 \ln N},$

so even with no real signal anywhere, the best-looking strategy of the batch has an in-sample Sharpe that climbs with the log of how many you tried. This is why you must deflate: judge the winner against the threshold $\sqrt{2 \ln N}$ for the $N$ trials you actually ran, not against zero.

Worked example — compute the fluke threshold. Let’s tabulate $\sqrt{2 \ln N}$ for several trial counts:

Read the $N = 1{,}000$ row: try a thousand zero-edge strategies and the best one shows an in-sample Sharpe of about 3.7 — a number that, taken at face value, would get a strategy funded. It is entirely luck. Worked arithmetic for that row: $\ln(1000) = 6.908$, times $2$ is $13.816$, and $\sqrt{13.816} = 3.717$. So before you celebrate a backtest Sharpe of 3.7 from a search over a thousand ideas, recognize you’ve exactly reproduced the expected noise maximum — the strategy has cleared no bar at all.

Why the LLM makes it acute. A human grinds out maybe dozens of trials before exhaustion imposes a natural limit. An LLM agent has no such limiter — it’ll cheerfully test 10,000 variants overnight, and because it doesn’t track the count for you, the deflation threshold silently climbs to 4.3 while you’re still mentally comparing against zero. The tool that makes you productive is the same tool that makes your false-discovery rate explode.

When to use it

A large automated search is legitimate only when paired with honest deflation: log every trial the agent runs (including the ones it discarded), compare the winner against $\sqrt{2 \ln N}$ for the full $N$, and reserve a genuinely untouched holdout for final confirmation. If you can’t count your trials, you can’t deflate; if you can’t deflate, an LLM idea-firehose is not a research accelerator — it’s a false-discovery accelerator. Use breadth for generating leads, never for certifying them.

A leakage preview

There’s a leakage channel unique to pretrained models, and it deserves a flag now even though lesson 5 — The Leakage Minefield — handles it in full. An LLM’s training corpus has a cutoff date, and that date is usually later than the historical periods you backtest on. So when you ask the model to “read this 2019 filing and predict the next quarter,” it isn’t reasoning purely from the filing — it may be quietly drawing on its training knowledge of what actually happened next. The backtest looks prescient because the model has, in a real sense, already read the answer key. This temporal contamination is invisible in the code, doesn’t show up as a leaky timestamp, and can survive even a careful point-in-time data pipeline — because the leak is in the weights, not the data feed. Lesson 5 dissects how to detect and defend against it; for now, just know that “the LLM was unusually good at predicting the past” is a red flag, not a triumph.

When an LLM signal or tool earns its place

The synthesis. Every thread of this lesson converges on one disciplined verdict. An LLM is worth its risk when all of the following hold, and a liability when any fails:

• Genuine, costly-to-process text. The edge is reading unstructured material — dense filings, full transcripts, foreign-language news — that the market hasn’t already trivially priced. If everyone reads the same headline in the same second, there’s no edge to extract.
• A real availability timestamp. Every document is provably knowable at decision time, in its original form. No post-close articles predicting pre-close labels; no revised transcripts smuggled backward; no training-cutoff contamination.
• Short horizon, modest capacity. The fast decay of text-derived alpha (the sentiment section) means the signal lives in days and holds limited size. Size it to its half-life and its post-cost capacity, not to its in-sample Sharpe.
• Deflation for trials. Any signal from a search — especially an LLM-driven one — is deflated against $\sqrt{2 \ln N}$ for the full trial count and confirmed on an untouched holdout. The firehose generates leads; the discipline certifies them.
• Strict point-in-time discipline. The leakage rules (full treatment in lesson 5) are enforced end to end, in both the signal and researcher roles.

Treat the LLM as neither an oracle (a forecaster you trade blindly) nor an un-audited idea firehose (a generator whose backtests you believe). It is a fast, fluent, tireless assistant whose output is raw material for a disciplined process — valuable exactly to the degree that the discipline around it is real.

Recap