The Honest Audit: Leakage and Over-Smoothing

You have spent five lessons learning what graph neural networks can do: pass messages along edges, learn embeddings that respect who-connects-to-whom, model contagion through a financial system. This lesson is the cold shower. It is the GNN sibling of the “why deep learning struggles in finance” discipline — the one that taught you to deflate your Sharpe for the number of trials, to purge and embargo your cross-validation, and to trust an out-of-sample number roughly as far as you can throw it.

Here is the punchline up front: a graph is the most leakage-prone data structure you will ever model, because the entire point of message passing is to let a node consume its neighbours’ information. Bolt a careless train/test split onto that, and your “test” nodes have been quietly reading the training set’s answer key across every edge. You will get a gorgeous AUC, present it proudly, and watch it evaporate the moment it touches a node the graph never saw before.

Everything you respected in the deep-learning course bites harder here: leakage, because edges are conduits; non-stationarity, because the graph itself rewires in exactly the crisis you care about; and the deflated-Sharpe paranoia, because a graph project hides its overfitting in the plumbing where nobody audits. Let’s audit it.

Why graphs leak harder than tables

Analogy. Imagine sitting an exam in a room where the person next to you — who has already been graded — keeps whispering their answers. You will ace the exam. You have learned nothing. A GNN with a leaky split is exactly that exam room: every edge is a neighbour whispering, and the test node’s “independent” answer is half-copied from a classmate the grader already knows the score of.

Definition. In a graph neural network, the embedding of node $v$ after $K$ layers of message passing is a function of the features of every node within $K$ hops of $v$. Schematically, layer $k$ updates each node from itself and its neighbours:

$h_v^{(k)} = \phi\!\left(h_v^{(k-1)},\ \operatorname{AGG}\{\,h_u^{(k-1)} : u \in \mathcal{N}(v)\,\}\right).$

Unrolling $K$ of these layers means $h_v^{(K)}$ depends on the $K$-hop receptive field of $v$. So the leakage radius of a GNN is not “the node itself” — it is every node reachable in $K$ steps. Stack more layers, and the contamination front pushes one hop further out with each one.

Contrast that with a table. In tabular data, row $i$ is (you hope) independent of row $j$; a clean split puts row $i$ in train and row $j$ in test and they never touch. There is no mechanism by which the model’s prediction for test row $j$ reads training row $i$. A graph manufactures that mechanism on purpose. That is why graphs leak harder than any tabular dataset: the leakage is not an accident of preprocessing, it is the design.

Worked example — a 2-layer GNN and one train neighbour. Let test node $t$ have a single neighbour $a$, and let $a$ in turn neighbour a node $b$. Both $a$ and $b$ are in the training set; the model has already fit on their features and (for $a$) its label. Trace the two layers of message passing for $t$:

So $h_t^{(2)} = f(\,x_t,\ x_a,\ x_b\,)$ where $x_a$ and $x_b$ are training features. The “held-out” prediction for $t$ is, mathematically, a function of data the model has already seen and fit. The AUC you measure on $t$ is contaminated. With $K=2$ the reach was 2 hops; push to $K=4$ and a single training node four edges away still seeps in.

When to use it

Treat the $K$-hop receptive field as the unit of leakage before you choose a split — it is the blast radius your train/test boundary has to contain. Whenever you report a GNN metric, the first question is not “is the AUC high?” but “could any test node’s embedding have consumed a training node’s features or label across an edge?” If yes, the number is inflated until proven otherwise.

Topology-aware splitting

Analogy. Splitting graph nodes at random is like shuffling a deck where every card is glued to its neighbours by a thread — you cannot pull a “test” card out cleanly, because it drags three “training” cards along with it. You need scissors that cut along the threads (the edges) and along time, not a blind random grab.

Definition. Topology-aware splitting means you partition data respecting the graph structure and time, never i.i.d. The three workhorse schemes:

• (a) Temporal split — the finance default. Train on the graph as it existed before time $T$ (edges, nodes, labels up to $T$); test on what comes after. The future cannot whisper into the past because it does not exist yet at training time.
• (b) Inductive split. Hold out entire subgraphs or connected components that the model never saw during training, then ask it to generalize to brand-new nodes — the GraphSAGE setting. The test component shares no edges with training, so there is no conduit.
• (c) Edge masking for link prediction. When the task is predicting edges, you must hide the test edges during message passing. If edge $(u,v)$ is the thing you are predicting, you cannot let that edge carry messages between $u$ and $v$ — that is the model reading its own answer.

Carry the purge/embargo idea straight over from Machine Learning for Alpha: in a temporal split you do not just cut at $T$, you embargo edges whose effect straddles the boundary. If an edge formed at $T - 1$ but its label (say, a default) only resolves at $T + 5$, that edge’s information overlaps the test window — embargo it, exactly as you purged overlapping labels in the time-series course.

Worked example — random vs temporal split. Same fraud graph, same model, two splits:

The 0.94 is the whispering-classmate score. The 0.71 and 0.68 are what the model actually knows. A practitioner who ships the 0.94 has shipped a fantasy; one who reports the 0.71 has shipped a model. The gap between them is the leakage.

When to use each

Use a temporal split as the default for any deployed finance model — you trade in time, so you must validate in time. Use an inductive split when the real question is “will this generalize to entities we have never seen” (new customers, new counterparties, a new market) — it is the honest test of transfer. Use edge masking whenever the prediction target is an edge itself (will these two firms transact, will this loan link form), because without it the model trivially reads the answer off the very edge you asked about.

Over-smoothing

Analogy. Stir a drop of milk into your coffee and at first you can see swirls — distinct regions, structure, contrast. Keep stirring and every sip tastes identical. Message passing is stirring: a little blends useful neighbourhood context; too much homogenizes the whole cup until every node tastes the same beige.

Definition. Over-smoothing is the phenomenon where, as you stack many message-passing layers, every node’s embedding aggregates an ever-larger neighbourhood until all embeddings converge toward a nearly identical vector. Once two nodes have the same representation, no classifier can tell them apart — the model’s discriminative power collapses. This is the crucial way GNNs differ from CNNs: in a CNN, depth builds richer hierarchical features and deeper usually helps; in a GNN, depth past a small number of layers actively destroys information by averaging it away.

Worked example — repeated averaging. Take two nodes whose embeddings start far apart, say node $P$ at value $10$ and node $Q$ at value $0$, on a graph where each layer replaces a node’s value with the average of itself and its neighbour (a toy mean-aggregation). Watch them collapse toward the global mean of $5$:

Do the layer-1 arithmetic: $P$ becomes $(10 + 0)/2 \cdot$ blended with itself — in this toy the gap simply halves each layer, $10 \to 5 \to 2.5 \to 1.25 \to 0.31$. After five layers $P$ and $Q$ are within a third of a unit of each other; after ten they are indistinguishable. The discriminative signal that made $P$ and $Q$ different nodes has been averaged into mush. That is over-smoothing in one number: the gap $\to 0$.

Mitigations:

• Stay shallow. Most production GNNs use 2–3 layers. That is not a lack of ambition; it is the sweet spot before over-smoothing bites.
• Residual / skip connections (jumping knowledge). Let each layer keep a copy of the pre-aggregation embedding, so the original signal survives the stirring.
• Normalization. Techniques like PairNorm or per-layer normalization push embeddings apart to counteract the collapse.

When to use depth

Reach for extra layers only when the task genuinely needs information from far across the graph — a long-range dependency that a 2-hop neighbourhood provably misses — and you pair the depth with skip connections or normalization to fight the collapse. For the overwhelming majority of financial graph tasks (fraud rings, counterparty risk, sector clustering), the signal lives within 2–3 hops, and a shallow network both avoids over-smoothing and contains your leakage radius. Depth is a cost you pay only against a demonstrated long-range need.

Non-stationary topology

Analogy. You learn the road map of a city in summer, with its quiet side streets and predictable flow. Then a flood hits: roads close, traffic reroutes, everyone funnels onto the same three bridges. Your beautiful summer map is now actively misleading — and the flood is the exact day you needed directions most. A financial graph floods in a crisis.

Definition. Non-stationary topology means the graph’s structure — which edges exist, how dense it is, which nodes are central — drifts over time rather than staying fixed. Loans roll off and new ones originate; supply chains re-route around a disruption; and in a correlation network, the edges themselves are statistical estimates that spike in a crash. A model trained on one regime’s topology can fail in another, and the regime that differs most is usually the stress regime you built the model to handle.

Worked example — correlation network densifying in a crash. Build edges between assets whenever their pairwise correlation exceeds a threshold, say $\rho > 0.5$. In a calm regime, average pairwise correlation might sit around $0.2$, so few pairs clear the bar and the graph is sparse. In a crash, “everything falls together” — average correlation jumps toward $0.8$, and almost every pair clears the threshold:

The crash graph is nearly complete — almost every node connects to almost every other. Now recall over-smoothing: on a near-complete graph, a single layer of message passing already averages over almost the entire network, so embeddings collapse toward the global mean in one step. The cruelest twist: the topology over-smooths exactly when it matters most. The model trained on the sparse calm graph meets a dense crisis graph and both the structure it learned and its ability to discriminate dissolve at once. This ties directly to the temporal GNNs of lesson 3 — modelling the graph as a sequence of time-stamped snapshots is the response to topology that will not hold still, and it mirrors the non-stationarity theme that haunted the deep-learning course.

When to use a static vs temporal graph

Use a static graph only when the relationships are genuinely slow-moving and structural — legal ownership hierarchies, fixed supply-chain contracts, settled corporate filings — where the topology barely drifts over your horizon. Use a temporal / dynamic graph (snapshots or a streaming temporal GNN) whenever the edges are themselves regime-sensitive estimates — correlation networks, transaction flows, interbank exposures — because those rewire precisely in the stress you are modelling, and a static snapshot will be most wrong when being right matters most.

The data-plumbing tax

Analogy. A GNN model is the tip of an iceberg poking proudly above the water. The 90% underwater — the part that sinks ships — is the pipeline that built the graph: who is a node, what counts as an edge, when did each edge exist, and can you assemble it at scale without a look-ahead. Everyone photographs the tip. The cold dark mass below is where projects die.

Definition / reality. The data-plumbing tax is the (usually dominant) share of a GNN project spent not on modelling but on constructing and maintaining the graph itself:

• Entity resolution. Is “JP Morgan”, “JPMorgan Chase”, “J.P. Morgan & Co.”, and ticker “JPM” one node or four? Get this wrong and your graph’s topology is fiction.
• Edge-definition thresholds. Which correlation counts as an edge — $\rho > 0.3$, $0.5$, $0.7$? Each threshold produces a different graph and a different model. The choice is a modelling decision masquerading as a data-cleaning step.
• Point-in-time timestamping. Every edge must carry when it became known, or you commit the same look-ahead bias the prerequisite courses drilled into you: using an edge in a prediction before that edge existed is leakage through time.
• Scale. Financial transaction graphs run to billions of edges; building, storing, and message-passing over them is an engineering problem before it is a modelling one.

Worked example — where the effort actually goes. A representative breakdown of engineering effort on a deployed financial GNN:

Roughly 85% of the work is not the architecture. The 15% everyone fights about (GCN vs GAT) sits between a 60% plumbing block that decides whether the graph is even real and a 15% validation block that decides whether the result is honest. Spend your scarce attention accordingly.

When to use a GNN at all

Reach for a graph neural network only when both conditions hold: the relational signal is strong enough to clear the steep plumbing cost (a plain tabular model on node features alone genuinely leaves money on the table), and the edge that signal lives on survives a topology-aware, point-in-time split. If a gradient-boosted tree on per-node features matches your GNN on an honest temporal split, the graph added cost, not edge — ship the tree. The GNN earns its keep only when relational structure is both real and leakage-free.

A pre-flight checklist

Analogy. A pilot does not trust a plane because it looks shiny on the tarmac; they run a fixed pre-flight checklist every single time, because the failure modes are known and the cost of skipping is a smoking crater. A GNN result deserves the same ritual. Below is the checklist — the GNN sibling of the deflated-Sharpe discipline you already carry from Machine Learning for Alpha. Run it before you trust any graph result, including your own.

When to use it

Run this checklist on every graph result you produce and every one you are asked to believe — before deployment, before a write-up, before a pitch. It is cheap relative to a model that fails silently in production. The moment a GNN’s reported metric cannot survive all six questions, treat it the way you treat an un-deflated Sharpe from a thousand-trial search: interesting, unproven, and not yet tradeable.

Recap

You came in able to build a graph neural network. You leave able to audit one. The four ways a GNN lies to you are now familiar: it leaks across edges harder than any table because message passing is leakage by design; it demands a topology-aware, temporal, embargoed split or its AUC is fiction; it collapses into over-smoothing if you mistake depth for power; and it is trained on a topology that rewires in the exact crisis it is meant to survive. Behind all of it sits the plumbing tax — the unglamorous 85% of the work that actually decides the outcome, while everyone argues about the architecture that decides the least.