Stacking (ML)
Stacking, or stacked generalization, is an ensemble learning technique that trains a meta-model to combine the predictions of several diverse base models, learning how to best weigh or blend their outputs rather than using a fixed rule…
Definition
Stacking, or stacked generalization, is an ensemble learning technique that trains a meta-model to combine the predictions of several diverse base models, learning how to best weigh or blend their outputs rather than using a fixed rule like averaging or voting.
Overview
Stacking differs from bagging and boosting in that it typically combines heterogeneous base models — for example, a gradient boosted tree, a neural network, and a support vector machine trained on the same task — rather than many copies of the same model type. The core idea is that different model types tend to make different kinds of errors, and a meta-model (also called a blender or second-level model) can learn to combine their predictions more intelligently than a simple average, by discovering, for example, that one base model is more reliable for a certain subset of the input space. The standard stacking procedure uses cross-validation to avoid overfitting the meta-model to the base models' own training performance. The training data is split into folds; for each fold, the base models are trained on the remaining folds and used to generate out-of-fold predictions on the held-out fold. This produces a new dataset where each row's features are the base models' predictions (rather than the original raw features), and the meta-model is trained on this new dataset to predict the true target. At inference time, the base models generate predictions on new data, which are then fed into the trained meta-model to produce the final prediction. Stacking can be extended to multiple levels, where the outputs of one layer of meta-models feed into another meta-model, though in practice most stacking ensembles use just one or two levels, since deeper stacks add complexity and overfitting risk with diminishing returns. Choosing diverse, complementary base models — ones that make different kinds of errors — is more important to stacking's success than simply adding more models of similar type. Because of its added complexity, longer training pipelines, and higher risk of overfitting if cross-validation is not used correctly, stacking is most commonly seen in machine learning competitions where squeezing out marginal performance gains matters more than simplicity, interpretability, or inference latency, and less often in production systems where a single well-tuned model or simpler ensemble is preferred.
Key Concepts
- Combines predictions from diverse, heterogeneous base models
- Trains a meta-model to learn how to best blend base model outputs
- Uses cross-validation to generate out-of-fold predictions and avoid overfitting
- Can be extended to multiple stacking levels, though rarely beyond two
- Benefits most from diverse base models that make different kinds of errors
- More complex to implement and maintain than bagging or simple boosting
- Commonly used in machine learning competitions for marginal performance gains
- Less common in production due to added latency and complexity
Use Cases
Frequently Asked Questions
From the Blog
Understanding Python decorators from scratch
Write decorators you actually understand. A decorator is a function that takes a function and returns a replacement, and every confusing variant — arguments, stacking, class decorators — is that one idea with an extra layer wrapped around it.
Read More ProgrammingHow to write parametrised tests in pytest
Parametrisation collapses near-identical tests into one data-driven case list — but only when the cases differ purely in data. This covers the basic form, readable IDs so a failure names the case, stacking for combinations, per-case marks, and the point at which a separate named test is clearer.
Read More