Models, Ensembles & Feature Importance

In most of machine learning, the data is generous. Show a model a million labelled cat photos and the signal-to-noise ratio (SNR) — how much of the data is real, repeatable pattern versus random junk — is enormous: a cat is a cat, the pixels don’t lie, and a hungry enough neural net memorizes the whole feline universe and still generalizes. Financial returns are the opposite kind of data. Tomorrow’s stock return is maybe 1% signal and 99% noise, and the signal isn’t even stationary — it decays, mutates, and gets arbitraged away the moment too many people find it.

That single fact rewrites every modelling instinct you imported from computer vision. The flashy, ultra-flexible model that wins Kaggle leaderboards on clean data will, on returns, lovingly memorize the noise and faceplant out-of-sample. This lesson is about picking models that survive that environment: when to prefer a humble linear model over a tree forest, why regularization and ensembling stop being optional, and how to read which features matter without fooling yourself. Simpler, here, is not a compromise — it’s the whole strategy.

Linear vs tree models on low-signal data

Analogy. Imagine two students cramming for an exam from a textbook riddled with typos. The first student learns the concepts — they get the textbook’s main ideas but shrug off the typos. The second student memorizes the book verbatim, typos and all, and aces a quiz drawn from those exact pages. Then the real exam arrives, drawn from a fresh, correctly-typed source. The concept-learner does fine; the memorizer, who learned the typos as if they were facts, bombs. Financial data is the typo-ridden textbook, the typos are noise, and the memorizer is your overfit model.

The bias–variance trade-off (recap). Every model’s expected out-of-sample error decomposes into three pieces:

$\mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\big(\text{Bias}[\hat{f}]\big)^2}_{\text{too rigid}} + \underbrace{\text{Var}[\hat{f}]}_{\text{too twitchy}} + \underbrace{\sigma^2_{\varepsilon}}_{\text{irreducible noise}}$

• Bias — error from the model being too simple to capture the true relationship (a straight line trying to trace a curve). High bias = underfitting.
• Variance — how much the fitted model would jump around if you re-trained it on a different sample. A flexible model chases the noise, so a new sample yields a wildly different fit. High variance = overfitting.
• Irreducible noise $\sigma^2_{\varepsilon}$ — the part you can never predict. On returns, this term is enormous.

More flexibility (deeper trees, bigger nets, more features) lowers bias but raises variance, and there’s a sweet spot. Where it sits depends entirely on the SNR. On clean, high-SNR data the irreducible noise is small, so flexibility pays — drive bias to zero and the variance you pick up is worth it. On returns, where $\sigma^2_{\varepsilon}$ dwarfs the signal, any variance you add is almost pure noise-fitting, so the optimum slides hard toward the simple, high-bias, low-variance end.

Definitions — the two model families.

• Linear model. Predicts a weighted sum of features: $\hat{y} = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p$. Rigid (it can only draw hyperplanes, never bends), so it has high bias but naturally low variance. Few parameters to wiggle means little room to memorize noise.
• Decision tree. Repeatedly splits the data on feature thresholds (“if momentum > 0.3 and volatility < 0.2, predict +0.4%”). It carves the feature space into boxes, so it captures nonlinearity (curved relationships) and interactions (feature A only matters when feature B is high) for free. But a deep tree can isolate every training point into its own leaf — zero bias, catastrophic variance. Trees overfit at the drop of a hat.

Worked example — over-deep tree vs ridge. You have 500 days of data and one genuinely useful feature (last week’s mean reversion), buried in noise. You fit two models.

The tree’s 0.98 in-sample R² is the trap. It didn’t learn the market; it learned those exact 500 days, typos and all. The ridge model’s unglamorous 0.02 out-of-sample is worth infinitely more than the tree’s glamorous −0.04 — because in finance you only ever get paid on out-of-sample. Trees can win on returns, but only when heavily restrained (shallow, ensembled, regularized — the rest of this lesson).

Regularization — taxing complexity

Analogy. Regularization is a complexity tax. Left untaxed, a model will spend its coefficients freely, buying an elaborate mansion of wiggles to fit every data point — including the noise. Slap a tax on coefficient size and suddenly the model gets frugal: it only “spends” on features that genuinely pull their weight, and shrinks the rest toward zero. You give up a little fit (bias) and get back a lot of stability (variance). On noisy data that’s a fantastic trade.

Definition / objective. Ordinary least squares minimizes squared error alone. Regularization adds a penalty on the size of the coefficients $\beta$:

• Ridge (L2). Penalizes the sum of squared coefficients. Shrinks every coefficient smoothly toward zero, but never exactly to zero.

$\hat{\beta}^{\text{ridge}} = \arg\min_{\beta}\ \sum_{i=1}^{n}\big(y_i - x_i^\top\beta\big)^2 + \lambda \sum_{j=1}^{p}\beta_j^2$

• Lasso (L1). Penalizes the sum of absolute coefficients. Shrinks and selects — it drives weak coefficients exactly to zero, performing automatic feature selection.

$\hat{\beta}^{\text{lasso}} = \arg\min_{\beta}\ \sum_{i=1}^{n}\big(y_i - x_i^\top\beta\big)^2 + \lambda \sum_{j=1}^{p}|\beta_j|$

• Elastic net. A blend of both penalties ($\lambda_1\sum|\beta_j| + \lambda_2\sum\beta_j^2$) — gets lasso’s selection plus ridge’s stability when features are correlated (lasso alone arbitrarily picks one of a correlated group; the L2 term shares the load).

The knob $\lambda$ (lambda) sets the tax rate. $\lambda = 0$ is plain OLS (no tax, full overfit risk); $\lambda \to \infty$ crushes every coefficient to zero (pure bias, ignores the data). You tune $\lambda$ by cross-validation — and on returns, the cross-validated optimum is usually aggressive.

The tree analogue. Trees don’t have coefficients, so you regularize them structurally — limit how complex any single tree can get:

• max_depth — cap how many splits deep a tree may go (shallow trees can’t isolate single points).
• min_samples_leaf — require each leaf to hold at least, say, 50 samples, so no leaf is fit to one lucky day.
• max_features — only let each split consider a random subset of features (also the key to random forests, below).
• Early stopping — in boosting, halt training once validation error stops improving, before it starts fitting noise.

Worked example — ridge shrinking a noisy coefficient. Suppose feature $x_3$ is pure noise, but in your 500-day sample it happened to correlate with returns, so OLS hands it a coefficient of $\beta_3 = 0.40$. Out-of-sample, that 0.40 is a liability — $x_3$ has no real predictive power, so every time $x_3$ moves the model makes a confident, wrong adjustment. Ridge with a moderate $\lambda$ shrinks it toward zero:

By taxing $\beta_3$ down from 0.40 to 0.05, ridge stops the model betting on a fluke. The real signal $\beta_1$ takes a small haircut too (0.30 → 0.26) — that’s the bias you pay — but the variance you save on the noise coefficient more than compensates, and out-of-sample error drops.

Ensembling — bagging (kill variance)

Analogy. Ask one slightly-tipsy expert to guess the number of jellybeans in a jar and you’ll get a wild answer. Ask five hundred independent guessers and average them, and the crowd’s average is eerily accurate — the individual errors, being random and uncorrelated, cancel out. Bagging (bootstrap aggregating) is exactly this: build many high-variance models, each a bit wrong in its own random direction, and average them. The shared signal reinforces; the independent noise cancels.

Definition / formula. Train $B$ models, each on a different bootstrap sample (a random resample-with-replacement of your data), then average their predictions. If each model has variance $\sigma^2$ and the models are mutually uncorrelated, the variance of the average is:

$\text{Var}\Big(\tfrac{1}{B}\sum_{b=1}^{B}\hat{f}_b\Big) = \frac{\sigma^2}{B}$

So the standard deviation of the ensemble falls like $\sigma/\sqrt{B}$ — the exact same $1/\sqrt{N}$ diversification law that powers a breadth-driven stat-arb book. More de-correlated learners → less variance, with no extra bias. You’re diversifying across models instead of across trades, but it’s the same math.

The catch — and the random-forest fix. That clean $\sigma^2/B$ only holds if the models are uncorrelated. Bootstrap samples overlap heavily, so trees trained on them come out similar — strongly correlated — and correlated learners barely diversify (just like correlated bets in a portfolio). With average pairwise correlation $\rho$ between the $B$ trees, the ensemble variance is actually:

$\text{Var}(\text{ensemble}) = \rho\,\sigma^2 + \frac{1-\rho}{B}\,\sigma^2$

As $B \to \infty$ the second term vanishes but the first, $\rho\sigma^2$, does not — correlation puts a hard floor under how much variance you can cancel. Random forests attack that floor directly: at every split, each tree may only consider a random subset of features (max_features). This forces different trees to rely on different signals, mechanically de-correlating them — driving $\rho$ down so the $\frac{1-\rho}{B}\sigma^2$ term actually shrinks. The de-correlation, not the bootstrap, is the real magic.

Worked example — variance of an average. Each tree has variance $\sigma^2 = 1.0$. Average $B = 100$ of them.

Read the rows: plain bagging’s correlated trees ($\rho = 0.30$) hit a wall — even with infinite trees, variance can’t drop below 0.30. The random forest’s feature randomization slashes $\rho$ to 0.05, dropping the floor to near 0.06 and recovering most of the diversification. Correlated trees don’t diversify — that’s why a forest randomizes features, not just rows.

Ensembling — boosting (kill bias)

Analogy. If bagging is five hundred independent guessers voting at once, boosting is a relay of specialists, each one studying the mistakes of the previous runner. Model 1 makes a rough prediction. Model 2 doesn’t re-predict from scratch — it predicts model 1’s errors (residuals), patching them. Model 3 patches what’s left, and so on. Each learner is weak and shallow, but the team sequentially chips away at the bias the simple base learner couldn’t capture alone.

Definition / formula. Boosting fits an additive model sequentially. Start with a constant, then at each round $m$ fit a small tree $h_m$ to the current residuals (in gradient boosting, to the negative gradient of the loss) and add a shrunken slice of it:

$F_m(x) = F_{m-1}(x) + \nu\, h_m(x)$

where $\nu$ (nu) is the learning rate — a shrinkage factor, typically 0.01–0.1, that says “only take a small step toward fixing the residuals each round.” XGBoost, LightGBM, and friends are highly optimized gradient-boosting implementations that add their own regularization. Because boosting attacks residuals, it primarily reduces bias — it can model relationships a single shallow tree never could.

The danger. Bias-reduction is a double-edged sword on noisy data. Keep fitting residuals long enough and boosting will start “fixing” the noise — the residuals that were never signal in the first place. Boosting is also notoriously sensitive to mislabeled or noisy data: a bad label looks like a big residual, so the next learner obsesses over it. The defenses are all about restraint:

• Learning rate $\nu$ — small steps (0.01–0.05) so no single round overcommits to noise; pair with more rounds.
• Subsampling — fit each round on a random fraction of rows and/or columns (stochastic gradient boosting), which both de-correlates and regularizes.
• Early stopping — watch validation error and halt the moment it stops improving, before the noise-fitting begins.
• Shallow base learners — depth-2 to depth-4 trees (“stumps”-ish), so each round adds only a little flexibility.

Bagging vs boosting — the contrast.

On clean data, well-tuned boosting usually beats bagging. On noisy financial data the verdict is closer, and boosting’s fragility to noise is a real cost — which is why many quant shops lean on random forests or heavily-regularized, early-stopped, low-learning-rate boosting rather than a maxed-out XGBoost.

Feature importance — MDI vs MDA

You’ve trained a model that works. Now the inevitable question: which features are actually driving it? You want this for intuition, for pruning dead features, and for the risk team. Two methods dominate for tree models, and they disagree often enough that knowing the difference is a survival skill.

MDI — Mean Decrease in Impurity (a.k.a. Gini importance).

Every split in a tree reduces “impurity” (variance for regression, Gini/entropy for classification). MDI tallies, for each feature, the total impurity reduction it produced across all splits in all trees, weighted by how many samples passed through. It’s the default feature_importances_ in scikit-learn.

• Pros: essentially free — it falls out of training, no extra computation.
• Cons (serious): it’s computed in-sample (on training data), so a feature that overfits gets credit for it. Worse, it is biased toward high-cardinality and continuous features — a feature with many distinct values (a price, a continuous ratio) offers more possible split points, so it’ll grab importance by sheer chance even if it’s noise. A binary flag, with one possible split, is structurally handicapped. MDI systematically over-rates continuous noise.

MDA — Mean Decrease in Accuracy (a.k.a. permutation importance).

Take your trained model and a held-out set. Record the baseline out-of-sample score. Now shuffle one feature’s values (permute the column, breaking its link to the target) and re-score. The drop in performance is that feature’s importance — if scrambling it barely hurts, the model wasn’t really using it.

• Pros: measured out-of-sample, on the real performance metric, so a feature only scores if it genuinely helps the model generalize. Honest. Model-agnostic (works on any fitted model, not just trees).
• Cons: slower (re-score once per feature, often repeated for stability), and — crucially — it inherits the substitution effect when features are correlated.

The substitution effect (the trap that fools everyone). Suppose two features are highly correlated — say momentum_10d and momentum_12d, near-twins. Permutation importance shuffles momentum_10d: the model barely flinches, because momentum_12d carries the same information and covers for it, so momentum_10d looks unimportant. Shuffle momentum_12d instead and momentum_10d covers. Both twins score near zero, even though momentum as a concept is the single most important driver. The importance got diluted/shared across the correlated group, making a critical signal look worthless.

Worked example — two correlated features splitting importance. A model is genuinely driven by momentum. You feed it two near-identical momentum features (ρ ≈ 0.95) plus an irrelevant noise feature.

Look at the disaster: the two momentum features (the actual engine of the model) score 0.04–0.05, barely above the useless noise feature at 0.01. A naive read — “momentum doesn’t matter, let’s drop it” — would gut the strategy. The fix is to test correlated features as a group (permute both twins together: the drop is large, revealing momentum’s true importance) or to cluster/decorrelate features before measuring (e.g. López de Prado’s clustered MDA/MDI).

Two more pitfalls that apply to both methods:

• Importance ≠ causation. A feature can be “important” to the model purely by being correlated with the true driver, or with the label leakage you accidentally left in. High importance tells you the model uses it, not that it causes returns.
• Importance is conditional on the model. MDI/MDA describe this fitted model’s reliance on a feature, not the feature’s universal worth. A different model (or the same model on a different sample) can rank features completely differently. Never report importance as if it’s a property of the feature alone.

Putting it together

Modelling low-signal-to-noise data is a discipline of restraint. First, respect the bias–variance trade-off: because returns are 99% noise, the irreducible-noise term dominates, the optimum slides toward simple, and a regularized linear model or shallow ensemble routinely beats a deep tree that memorized the past. Second, regularize — tax complexity (ridge shrinks, lasso shrinks-and-selects, elastic net blends; trees use depth/leaf/feature caps and early stopping) to trade a little bias for a lot less variance. Third, ensemble with the right tool: bag (average de-correlated trees, random-forest-style, to crush variance via $1/\sqrt{B}$) when you’re overfitting, and boost (sequentially fit residuals to cut bias) — carefully, with a small learning rate and early stopping — when you’re underfitting. Finally, read feature importance with deep suspicion: prefer out-of-sample MDA over in-sample, cardinality-biased MDI, watch the substitution effect dilute correlated features, and never confuse “the model uses it” with “it causes returns.”

You can now pick a model that survives the noise, regularize and ensemble it without fooling yourself, and read its feature importances with the right amount of paranoia. But a model that predicts well is still not a portfolio — a vector of return forecasts isn’t positions, and the gap between “my model says these 200 stocks are attractive” and “here is exactly how much of each to hold” is where a surprising amount of alpha is won or lost. That translation — from raw ML signal to sized, risk-controlled, cost-aware positions — is the subject of the next lesson, From ML Signal to Portfolio.