Ensemble Methods

Take a biased coin that comes up heads 51% of the time. After ten thousand independent flips, the heads ratio is 51% ± 0.5% almost certainly. The variance shrinks as $1/\sqrt{n}$, the bias stays the same.

Replace "coin flip" with "classifier vote". Suppose you have $N$ classifiers that each correctly classify a given example with probability $p > 0.5$, and that their errors are independent. The majority vote is wrong only if more than $N/2$ of them are wrong. The Hoeffding bound — the statement that a sample mean of $N$ independent bounded variables sits within $\gamma$ of the true mean with probability at least $1 - 2\exp(-2\gamma^2 N)$ — pins that failure probability down: it drops exponentially in $N$.

With $p = 0.51$ and $N = 1000$, the majority-vote accuracy is over 75%. With $N = 10000$, it is over 98%.

The catch is "independent". Real classifiers trained on the same data are not independent. They make correlated errors. The whole game of ensemble methods is generating diversity: making the individual classifiers disagree as much as possible while each remaining better than chance.

# extra-code from Géron Ch 7, Figure 7-3, ported
import numpy as np

heads_proba = 0.51
np.random.seed(42)
coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
ratios = coin_tosses.cumsum(axis=0) / np.arange(1, 10001).reshape(-1, 1)
# ratios at the end converge to ~0.51 with tiny variance.
# This is what an ensemble of independent classifiers does.

VotingClassifier takes a list of trained models and predicts by aggregating their outputs. Two modes.

Hard voting: each classifier predicts a class label, the majority vote wins.

Soft voting: each classifier outputs class probabilities, the probabilities are averaged, the argmax of the average wins.

Soft voting is usually better because confident-but-wrong classifiers get downweighted and confident-and-right classifiers dominate. Hard voting throws away the confidence information.

from sklearn.datasets import make_moons
from sklearn.ensemble import VotingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

voting_clf = VotingClassifier(
    estimators=[
        ("lr", LogisticRegression(random_state=42)),
        ("rf", RandomForestClassifier(random_state=42)),
        ("svc", SVC(probability=True, random_state=42)),
    ],
    voting="soft",
)
voting_clf.fit(X_train, y_train)
print(voting_clf.score(X_test, y_test))
# 0.92  — typically a few percent better than any individual base model.

For soft voting with SVC, you must pass probability=True, which fits Platt scaling internally. This is slow. For soft voting in production, prefer CalibratedClassifierCV(SVC(...)) or just use models whose predict_proba is native.

The cheaper way to generate ensemble diversity: train each classifier on a different subsample of the training data.

Bagging (bootstrap aggregating): sample $m$ examples with replacement from the training set, train a classifier on that bootstrap sample, repeat $N$ times.

Pasting: same but without replacement.

Bagging is the more common choice. Each bootstrap sample contains about 63% of unique training examples (the others are missing because of duplicates). The 37% that are missing become out-of-bag (OOB) examples for that classifier, which gives you a free validation set per tree.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

bag = BaggingClassifier(
    DecisionTreeClassifier(random_state=42),
    n_estimators=500,
    max_samples=100,
    bootstrap=True,
    n_jobs=-1,
    random_state=42,
)
bag.fit(X_train, y_train)
print(bag.score(X_test, y_test))

From-scratch path:

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from collections import Counter

def fit_bag(X: np.ndarray, y: np.ndarray, n_estimators: int = 100,
            max_samples: int = None, random_state: int = 42) -> list:
    """Train n_estimators decision trees on bootstrap samples."""
    rng = np.random.default_rng(random_state)
    m = len(y)
    max_samples = max_samples or m
    trees = []
    for i in range(n_estimators):
        idx = rng.integers(0, m, size=max_samples)   # with replacement
        tree = DecisionTreeClassifier(random_state=i)
        tree.fit(X[idx], y[idx])
        trees.append(tree)
    return trees

def predict_bag(trees: list, X: np.ndarray) -> np.ndarray:
    """Majority vote across trees."""
    votes = np.array([t.predict(X) for t in trees])
    return np.array([Counter(votes[:, j]).most_common(1)[0][0]
                     for j in range(X.shape[0])])

That is bagging in 15 lines. The sklearn version handles parallelism, the OOB tracking, and the predict_proba case, but the core loop is what is above.

Note on OOB: since each tree is missing 37% of the data, you can compute its accuracy on that 37% and average across all trees. BaggingClassifier(oob_score=True) does this. The OOB score is an unbiased estimate of generalization error, often better-behaved than a single held-out validation set.

A random forest is bagging applied to decision trees with one additional twist: at each split, consider only a random subset of features (typically $\sqrt{d}$ for classification, $d/3$ for regression). This further decorrelates the trees.

Why decorrelate? Because if two trees always pick the same root split, they will make highly correlated errors and the averaging benefit is small. Random feature selection forces different trees to use different features, which produces stronger diversity than bagging alone.

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=500,
    max_leaf_nodes=16,        # implicit depth control
    max_features="sqrt",      # default for classification
    n_jobs=-1,
    random_state=42,
    oob_score=True,
)
rf.fit(X_train, y_train)
print(f"Test acc: {rf.score(X_test, y_test):.3f}")
print(f"OOB acc:  {rf.oob_score_:.3f}")

Feature importances. rf.feature_importances_ gives a number per feature: the average decrease in Gini impurity at splits that use that feature, weighted by the number of points routed through those splits, averaged across all trees in the forest. Sum to 1.

import pandas as pd
from sklearn.datasets import load_iris

iris = load_iris(as_frame=True)
rf = RandomForestClassifier(n_estimators=500, random_state=42)
rf.fit(iris.data, iris.target)
importances = pd.Series(rf.feature_importances_, index=iris.feature_names)
print(importances.sort_values(ascending=False))
# petal length (cm)    0.44...
# petal width (cm)     0.42...
# sepal length (cm)    0.10...
# sepal width (cm)     0.02...

Petal length and petal width together explain 86% of the model's splits. That matches the actual structure of Iris (the two petal features separate the three species; the sepal features barely help).

Extremely Randomized Trees (ExtraTreesClassifier) take random forests one step further. At each split, instead of finding the best threshold for each candidate feature, pick a random threshold for each candidate feature, then take the best among those random thresholds.

This sounds wrong. It works. The randomness is yet another decorrelation lever, and the savings in computation are real (no sort over feature values needed). On many tabular datasets, ExtraTrees is within a percent of RF accuracy at half the training time.

from sklearn.ensemble import ExtraTreesClassifier

et = ExtraTreesClassifier(n_estimators=500, random_state=42, n_jobs=-1)
et.fit(X_train, y_train)
print(et.score(X_test, y_test))

Bagging makes trees diverse by perturbing the data. Boosting makes them diverse by perturbing the loss: each successive tree focuses on the examples the previous trees got wrong.

The first boosting algorithm is AdaBoost (Freund and Schapire, 1996). The recipe:

1. Initialize per-example weights $w_i = 1/m$.
2. For $t = 1, \ldots, T$:

• Train a weak classifier $h_t$ on the data, weighted by $w_i$.
• Compute weighted error $\epsilon_t = \sum_i w_i \mathbb{1}[h_t(x_i) \neq y_i]$.
• Compute classifier weight $\alpha_t = \frac{1}{2} \log \frac{1 - \epsilon_t}{\epsilon_t}$.
• Reweight examples: $w_i \leftarrow w_i \exp(-\alpha_t y_i h_t(x_i))$, normalize.

3. Predict: $\hat{y}(x) = \text{sign}\left( \sum_t \alpha_t h_t(x) \right)$.

The intuition: examples that the current classifier gets wrong get bigger weights, so the next classifier pays more attention to them. The classifier weights $\alpha_t$ are larger for more accurate classifiers.

# sklearn >=1.6 only supports discrete SAMME (the 'algorithm' arg is deprecated/no-op)
ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),  # decision stumps
    n_estimators=100, learning_rate=1.0, random_state=42)
ada.fit(X, y)

def fit_adaboost(X, y, T=100):
    """Train T decision stumps via AdaBoost. y in {-1, +1}."""
    m = len(y)
    w = np.ones(m) / m                       # uniform example weights
    classifiers, alphas = [], []
    for t in range(T):
        stump = DecisionTreeClassifier(max_depth=1)
        stump.fit(X, y, sample_weight=w)     # weak learner on weighted data
        pred = stump.predict(X)
        err = np.sum(w * (pred != y)) / np.sum(w)        # weighted error
        err = np.clip(err, 1e-10, 1 - 1e-10)             # dodge log(0)
        alpha = 0.5 * np.log((1 - err) / err)            # classifier weight
        w = w * np.exp(-alpha * y * pred)                # up-weight mistakes
        w = w / np.sum(w)                                # renormalize
        classifiers.append(stump)
        alphas.append(alpha)
    return classifiers, np.array(alphas)

def predict_adaboost(classifiers, alphas, X):
    scores = sum(a * c.predict(X) for a, c in zip(alphas, classifiers))
    return np.sign(scores)

AdaBoost is one specific way to combine weak learners. Gradient boosting generalizes it. Treat the ensemble's output as a function $F(x)$. At each round, compute the gradient of the loss with respect to $F$, fit a weak learner to that gradient, and add the weak learner's output (scaled by a learning rate) to $F$.

That is gradient descent. In function space. Each step moves $F$ slightly toward the direction that reduces loss the fastest, and each step is parameterized as "fit a tree to the negative gradient".

For regression with squared loss, the negative gradient at example $i$ is just the residual $y_i - F(x_i)$. So gradient boosting for regression is: fit a tree to the current residuals, add it (scaled), update residuals, repeat. This is one of the cleanest algorithms in ML.

gb = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1,
                               max_depth=3, random_state=42)
gb.fit(X, y)
preds = gb.predict(X_test)

def fit(self, X, y):
    y = y.astype(float)                       # int labels would truncate the boosting
    self.initial = float(np.mean(y))          # F_0 = mean(y), the MSE-optimal constant
    F = np.full(y.shape, self.initial, dtype=float)
    self.trees = []
    for t in range(self.n_estimators):
        residuals = y - F                     # = -gradient of 0.5*(y-F)^2 w.r.t. F
        tree = DecisionTreeRegressor(max_depth=self.max_depth,
                                     random_state=self.random_state + t)
        tree.fit(X, residuals)                # weak learner fits the negative gradient
        F = F + self.learning_rate * tree.predict(X)   # shrunk additive step
        self.trees.append(tree)
    return self

def predict(self, X):
    F = np.full(X.shape[0], self.initial, dtype=float)
    for tree in self.trees:
        F = F + self.learning_rate * tree.predict(X)
    return F

40 lines. That is the algorithm. Everything else (XGBoost, LightGBM, CatBoost) is engineering: better split-finding, better regularization, better handling of categorical features, GPU acceleration. The core is what is above.

Hyperparameters that matter:

• learning_rate (also called shrinkage): typically 0.01 to 0.3. Smaller = more trees needed but better generalization.
• n_estimators: number of boosting rounds. Tune with early stopping.
• max_depth: 3-8. Shallower than random forests, because boosting builds the ensemble additively.

In 2026, "gradient boosting" almost always means one of three libraries.

XGBoost (Chen and Guestrin, 2016). The first production-grade GBM library. Adds: regularized objective ($L_1$ and $L_2$ penalties on leaf values), second-order optimization (uses the Hessian — the curvature of the loss, recall — and not just the gradient, so each split is chosen from a closer approximation of the loss), efficient tree construction (sparse-aware, approximate split-finding for huge data), GPU support.

LightGBM (Microsoft, 2017). Optimized for speed on large datasets. Adds: histogram-based split finding (much faster than exact on large data), leaf-wise tree growth (deeper trees with fewer splits), gradient-based one-side sampling (GOSS), exclusive feature bundling.

CatBoost (Yandex, 2017). Specializes in categorical features. Adds: ordered boosting (avoids target leakage when encoding categoricals), symmetric trees (fast inference), built-in handling of categorical columns without one-hot encoding.

The named additions above (histogram split-finding, leaf-wise growth, GOSS, exclusive feature bundling, ordered boosting, symmetric trees) are speed-and-accuracy engineering on top of the same residual-fitting loop from §7; you can use these libraries well without knowing each one, so treat them as a glossary to grow into rather than prerequisites.

import xgboost as xgb
from sklearn.metrics import accuracy_score

# XGBoost
xgb_clf = xgb.XGBClassifier(
    n_estimators=500,
    max_depth=6,
    learning_rate=0.1,
    objective="binary:logistic",
    n_jobs=-1,
    random_state=42,
    early_stopping_rounds=20,
)
xgb_clf.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)
print(accuracy_score(y_test, xgb_clf.predict(X_test)))

# LightGBM
import lightgbm as lgb
lgb_clf = lgb.LGBMClassifier(
    n_estimators=500,
    learning_rate=0.05,
    num_leaves=31,
    random_state=42,
)
lgb_clf.fit(X_train, y_train, eval_set=[(X_test, y_test)],
             callbacks=[lgb.early_stopping(stopping_rounds=20)])

Practical defaults for XGBoost on a new tabular problem:

xgb.XGBClassifier(
    n_estimators=1000,
    max_depth=6,
    learning_rate=0.05,
    subsample=0.8,            # row sampling per tree
    colsample_bytree=0.8,     # feature sampling per tree
    reg_alpha=0.0,            # L1 on leaf values
    reg_lambda=1.0,           # L2 on leaf values
    early_stopping_rounds=50,
)

Run that on most tabular benchmarks and you have a competitive baseline before you tune anything. The 2026 reality: on Kaggle's tabular competitions, the winning solution is almost always an XGBoost + LightGBM blend with elaborate feature engineering. Deep learning on tabular data (TabNet, FT-Transformer, etc.) exists but rarely beats well-tuned GBMs.

Voting averages predictions. Stacking trains a model that learns how to average them.

The setup:

1. Train $N$ base models on the training set.
2. Get each base model's predictions on a held-out set (or via cross-validation).
3. Use those predictions as features for a meta-learner (typically a simple logistic regression).
4. Train the meta-learner to map base-model predictions to the target.

For inference: pass new data through all $N$ base models, feed their outputs to the meta-learner, get the final prediction.

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression

stacking = StackingClassifier(
    estimators=[
        ("rf", RandomForestClassifier(random_state=42)),
        ("xgb", xgb.XGBClassifier(random_state=42, eval_metric="logloss")),
        ("svc", SVC(probability=True, random_state=42)),
    ],
    final_estimator=LogisticRegression(),
    cv=5,
)
stacking.fit(X_train, y_train)

Bagging reduces variance. Boosting reduces bias. That is the high-level story.

Bagging. Each tree has roughly the same bias as a single tree. The variance is reduced by averaging $N$ partially-correlated trees: $\text{Var}(\bar{f}) = \rho \sigma^2 + (1 - \rho) \sigma^2 / N$, where $\rho$ is the average correlation between trees and $\sigma^2$ is the variance of a single tree. As $N$ grows, only the correlated component remains. Random forests reduce $\rho$ via feature randomization, which makes bagging more effective.

Boosting. Each weak learner is high-bias, low-variance (depth-3 trees barely fit anything). The ensemble combines them additively, which reduces bias over rounds. AdaBoost and gradient boosting do not reduce variance much per round, but they decrease bias dramatically. The risk is overfitting (variance creeping back in if boosting too many rounds), which is why early stopping is mandatory.

The textbook example: bagging 1000 depth-30 trees gives a low-variance, low-bias ensemble (because depth-30 trees already have low bias, and bagging fixes the variance). Boosting 1000 depth-3 stumps gives a low-bias, low-variance ensemble (because boosting fixes the bias, and depth-3 trees have low variance natively). Both routes get to the same destination through different mechanics.

The honest 2026 picture: ensembles dominate tabular ML, but a single deep model wins almost everywhere else.

Where ensembles still win:

• Tabular data with mixed categorical and numerical features
• Small datasets (< 100k rows)
• Heterogeneous features with sharp discontinuities (e.g., "zip code matters above price threshold X but not below")
• Problems where feature importance matters for the deployment (interpretability, regulatory)

Where a single deep model wins:

• Images (CNNs, ViTs)
• Text and code (transformers)
• Audio (transformers, conv-stacks)
• Video and multimodal (transformers)
• Any problem with > 1M training examples and complex non-tabular structure

The pattern: ensembles thrive on the inductive biases of trees (axis-aligned splits, piecewise-constant predictions, native handling of feature interactions). Those biases match tabular data and mismatch perceptual data. A CNN can see edges and textures; a tree cannot. A transformer can compose long-range token relationships; a tree cannot.

A practical heuristic: if your data fits in a Pandas DataFrame with named columns, try XGBoost first. If your data is pixels, tokens, or waveforms, train a small neural network first. The decision tree of decisions about which model to use is itself a useful diagnostic.

Pull back. Voting, bagging, boosting, stacking — all four are different strategies for searching the space of functions $F: \mathcal{X} \to \mathcal{Y}$.

• Voting picks a fixed weighted sum of pre-trained functions.
• Bagging averages many functions drawn from a randomized training procedure.
• Boosting does coordinate descent in function space, one weak learner per step.
• Stacking trains a meta-function whose inputs are the outputs of base functions.

The unifying frame is functional gradient descent. Treat the loss as a functional $L[F]$ over functions, and ask which direction in function space reduces $L$ fastest. The answer is the negative gradient evaluated at the training points: $-\partial L / \partial F(x_i)$. Boosting approximates this gradient with a tree; bagging averages over independent runs of the procedure; stacking adds a learned post-aggregation.

You will see this view again in deep learning. Each layer of a neural network is essentially a basis function chosen by gradient descent, and the network's prediction is the composition (not the sum, but close enough at conceptual level) of those basis functions. Boosting is a shallow neural network with a peculiar training procedure. Gradient descent is gradient descent. The connections matter because they tell you when one technique should win over another.