Classification

Given training data $\{(x_i, y_i)\}_{i=1}^N$ with $y_i \in \{1, 2, \ldots, K\}$, learn a function $f: \mathcal{X} \to \{1, \ldots, K\}$. Sometimes $f$ also outputs class probabilities $p(y = k \mid x)$, and the discrete prediction is $\arg\max_k p(y = k \mid x)$.

Three flavors:

• Binary classification: $K = 2$. Spam vs. not spam, cancer vs. healthy, click vs. no-click.
• Multi-class classification: $K > 2$, each example has exactly one true class. MNIST (digits 0-9), CIFAR (10 categories), language-model next-token (vocab size $K$).
• Multi-label classification: each example can have multiple true classes. Tagging an image with all objects it contains. Categorizing a news article into multiple topics.

A fourth thing that is often conflated: multi-output classification, where the model outputs multiple independent class predictions per example (e.g. predicting both digit identity and writer demographic). This is just stacking multiple classifiers; nothing new conceptually.

The MNIST dataset is the canonical sandbox. 70,000 28x28 grayscale images of handwritten digits, with the integer label. We'll use it for almost every example in this chapter.

from sklearn.datasets import fetch_openml
import numpy as np

mnist = fetch_openml('mnist_784', as_frame=False, parser='auto')
X, y = mnist.data, mnist.target.astype(np.uint8)
print(X.shape, y.shape)   # (70000, 784), (70000,)

# The standard train/test split: first 60k train, last 10k test
X_train, X_test = X[:60000], X[60000:]
y_train, y_test = y[:60000], y[60000:]

Start with the simplest version of the problem. Build a binary classifier for "is this digit a 5". The target becomes:

y_train_5 = (y_train == 5)
y_test_5 = (y_test == 5)
print(y_train_5.sum(), "fives in training set out of", len(y_train_5))
# 5421 fives out of 60000 -> ~9% positive class

This is an imbalanced binary problem. Random guessing would predict "not 5" for everything and achieve 91% accuracy. This is the trap that motivates every other metric in the chapter.

Library path with SGDClassifier:

from sklearn.linear_model import SGDClassifier

sgd = SGDClassifier(loss='hinge', random_state=42, max_iter=1000)
sgd.fit(X_train, y_train_5)

# Predict on one example
print(sgd.predict([X_train[0]]))   # array([ True]) or [False]

# Accuracy (warning: misleading on imbalanced data)
from sklearn.metrics import accuracy_score
print("Train accuracy:", accuracy_score(y_train_5, sgd.predict(X_train)))
print("Test accuracy:", accuracy_score(y_test_5, sgd.predict(X_test)))

The SGDClassifier with loss='hinge' is a linear SVM trained with stochastic gradient descent. A linear support vector machine (SVM) is a linear classifier that places the decision boundary to maximize the margin — the distance from the boundary to the nearest training points of each class (the support vectors). The hinge loss $\max(0, 1 - y \cdot s)$, where $s = w^\top x + b$ is the score and $y \in \{-1, +1\}$, is what enforces that margin: it charges nothing once a point is on the correct side by a safe distance, and grows linearly as points cross into or past the boundary. SVMs get a full treatment in a later chapter; for now, treat this as a fast linear classifier whose raw output is a signed score, not a probability.

For the "is this a 5" problem, accuracy is misleading because the classes are imbalanced. A dummy classifier that always predicts "not 5" gets 91% accuracy and learns nothing. To see this:

from sklearn.dummy import DummyClassifier
dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train_5)
print("Dummy accuracy:", accuracy_score(y_test_5, dummy.predict(X_test)))   # ~0.91

To meaningfully compare your classifier to the dummy, you need a metric that penalizes the "always negative" strategy. The first such metric is balanced accuracy:

A dummy classifier gets balanced accuracy = 0.5 (it catches all of one class, none of the other). A useful classifier gets > 0.5.

But balanced accuracy still throws away information. The full picture is the confusion matrix.

The confusion matrix is the four-number summary of binary classification performance.

from sklearn.metrics import confusion_matrix
from sklearn.model_selection import cross_val_predict

# Get cross-validated predictions (so we don't peek at val data)
y_train_pred = cross_val_predict(sgd, X_train, y_train_5, cv=3)
cm = confusion_matrix(y_train_5, y_train_pred)
print(cm)
# [[TN, FP],
#  [FN, TP]]

A typical output:

[[53892,   687],     # actual not-5: 53892 correctly classified, 687 wrong
 [ 1891,  3530]]     # actual 5:     1891 missed (FN), 3530 caught (TP)

From this:

from sklearn.metrics import precision_score, recall_score, f1_score
print(f"Precision: {precision_score(y_train_5, y_train_pred):.3f}")
print(f"Recall:    {recall_score(y_train_5, y_train_pred):.3f}")
print(f"F1:        {f1_score(y_train_5, y_train_pred):.3f}")

The four metrics each measure something different. Which one you optimize depends on the cost of the two error types.

For a binary classifier with a probability output, you choose a threshold $\tau$. Predict positive if $p(y = 1 \mid x) > \tau$. As you raise $\tau$, you get fewer positive predictions, so precision goes up and recall goes down. As you lower $\tau$, recall goes up and precision goes down. You cannot have both unless your classifier is perfect.

An SVM doesn't output probabilities, but it does output a score: decision_function returns $s = w^\top x + b$ for each example, a signed value that is positive on the "is a 5" side of the boundary and negative on the other, with magnitude growing the farther the point sits from the boundary. The default predict simply thresholds this score at zero. To trade precision against recall we threshold the score ourselves instead.

# Get decision function scores (signed distance to decision boundary, not calibrated probability)
y_scores = cross_val_predict(sgd, X_train, y_train_5, cv=3, method="decision_function")

from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)

# Find the threshold that achieves precision >= 0.90
idx = np.argmax(precisions >= 0.90)
threshold_90 = thresholds[idx]
print(f"Threshold for 90% precision: {threshold_90:.3f}")
print(f"Recall at that threshold: {recalls[idx]:.3f}")

Plot precision and recall against the threshold:

import matplotlib.pyplot as plt
plt.plot(thresholds, precisions[:-1], 'b-', label='Precision')
plt.plot(thresholds, recalls[:-1], 'g-', label='Recall')
plt.axvline(threshold_90, color='red', linestyle='--')
plt.xlabel('Threshold')
plt.legend()
plt.grid(True)
plt.show()

Or plot recall on x-axis vs precision on y-axis (the PR curve):

plt.plot(recalls, precisions)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.grid(True)
plt.show()

The PR curve summarizes the trade-off. The area under it (PR-AUC, sometimes called average precision) is one threshold-free metric:

from sklearn.metrics import average_precision_score
print(f"PR-AUC (average precision): {average_precision_score(y_train_5, y_scores):.3f}")

ROC (Receiver Operating Characteristic) plots TPR (recall) on the y-axis against FPR (false positive rate) on the x-axis, varying the threshold.

A perfect classifier reaches (0, 1) in the top-left corner. A random classifier traces the diagonal. ROC-AUC is the area under this curve. AUC = 0.5 is random; AUC = 1.0 is perfect.

from sklearn.metrics import roc_curve, roc_auc_score

fpr, tpr, thresholds = roc_curve(y_train_5, y_scores)
print(f"ROC-AUC: {roc_auc_score(y_train_5, y_scores):.3f}")

plt.plot(fpr, tpr)
plt.plot([0, 1], [0, 1], 'k--')   # diagonal
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.grid(True)
plt.show()

When to use PR vs ROC. Both summarize the threshold-vs-quality trade-off. They differ in what they emphasize:

• ROC treats positives and negatives symmetrically (TPR vs FPR). Useful when classes are roughly balanced and both kinds of error matter.
• PR focuses on the positive class (precision vs recall). More informative when positives are rare (imbalanced data) or when you care specifically about the positive class.

For the MNIST-5 problem (9% positive), PR-AUC is the better summary. For balanced problems, use ROC.

SGDClassifier(loss='hinge') gives you an SVM. To get probability outputs, you want logistic regression: a linear model trained to maximize the likelihood under a sigmoid-of-linear model.

The loss is binary cross-entropy:

This is NLL under the Bernoulli model The gradient with respect to $w$ has a clean form: $\frac{1}{N} \sum_i (\sigma(w^\top x_i + b) - y_i) x_i = \frac{1}{N} X^\top (\hat{p} - y)$. The "predicted minus actual" times input. Recognize it.

clf = LogisticRegression(penalty=None, fit_intercept=True, max_iter=1000)  # penalty=None matches the unregularized scratch math; default is penalty='l2', C=1.0
clf.fit(X, y)                    # X: (N, d), y in {0, 1}
p = clf.predict_proba(X)[:, 1]   # P(y=1 | x)
yhat = clf.predict(X)            # thresholds p at 0.5
# fitted params: clf.coef_ (1, d) ~ w ,  clf.intercept_ (1,) ~ b

def sigmoid(z):
    # Stable form: for negative z use exp(z) / (1 + exp(z))
    out = np.empty_like(z, dtype=np.float64)
    pos = z >= 0
    out[pos] = 1.0 / (1.0 + np.exp(-z[pos]))
    expz = np.exp(z[~pos])
    out[~pos] = expz / (1.0 + expz)
    return out

def fit_logistic_regression(X, y, *, lr=0.1, n_steps=1000, l2=0.0):
    X = np.asarray(X, np.float64); y = np.asarray(y, np.float64)
    N, d = X.shape
    w = np.zeros(d); b = 0.0
    for step in range(n_steps):
        z = X @ w + b
        p = sigmoid(z)
        dw = X.T @ (p - y) / N + l2 * w     # grad of BCE (+ L2 on w)
        db = float((p - y).mean())
        w -= lr * dw
        b -= lr * db
    return w, b

def predict_proba(X, w, b):
    return sigmoid(np.asarray(X, np.float64) @ w + b)

def predict(X, w, b, threshold=0.5):
    return (predict_proba(X, w, b) >= threshold).astype(int)

This is the smallest probabilistic classifier and the foundation of every more complex model in deep learning. another chapter will derive it from maximum likelihood; another chapter will turn it into a single-layer neural network. Same equations.

Generalize logistic regression to $K$ classes. The model produces $K$ logits and a softmax turns them into a probability distribution:

This is the softmax regression (also called multinomial logistic regression) classifier. The loss is categorical cross-entropy:

The gradient with respect to logits $z_k$ for example $i$ is $p_k^{(i)} - \mathbf{1}[y_i = k]$. Predicted minus one-hot target. Same shape as logistic regression's $\hat{p} - y$.

from sklearn.linear_model import LogisticRegression

# 'multinomial' uses softmax cross-entropy (one model, K classes)
multi = LogisticRegression(multi_class='multinomial', solver='lbfgs', max_iter=500)
multi.fit(X_train[:10000], y_train[:10000])
print("Test accuracy:", multi.score(X_test, y_test))   # ~0.91

One-vs-rest and one-vs-one decompositions. Older multi-class strategies that build a $K$-class predictor out of binary classifiers, which matters when your base classifier is inherently binary (like an SVM). One-vs-rest (OvR) trains one "class $k$ vs. everything else" classifier per class and predicts the class whose classifier scores highest; one-vs-one (OvO) trains one classifier per pair of classes and predicts by majority vote. For most modern models, just use the multinomial formulation.

from sklearn.multiclass import OneVsRestClassifier
ovr = OneVsRestClassifier(SGDClassifier())
ovr.fit(X_train[:10000], y_train[:10000])
print("OvR accuracy:", ovr.score(X_test, y_test))

The multi-class confusion matrix is a $K \times K$ array where entry $(i, j)$ is the count of examples with true class $i$ predicted as $j$.

from sklearn.metrics import confusion_matrix
y_pred = multi.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
print(cm)
# 10x10 matrix; diagonal entries are correct, off-diagonal are errors

# Normalize each row by class size to see the per-class error pattern
cm_norm = cm / cm.sum(axis=1, keepdims=True)
plt.imshow(cm_norm, cmap='Blues')
plt.colorbar()
plt.xlabel('predicted')
plt.ylabel('true')
plt.show()

The error analysis workflow:

1. Find the largest off-diagonal entries. These are the most confused class pairs.
2. For each confused pair, look at a sample of misclassified images. What do they have in common?
3. Hypothesize what feature the model is missing or what training data is underrepresented.

# Find the worst confusion
errors = cm.copy()
np.fill_diagonal(errors, 0)
worst_pairs = []
for _ in range(5):
    i, j = np.unravel_index(np.argmax(errors), errors.shape)
    worst_pairs.append((i, j, errors[i, j]))
    errors[i, j] = 0
print("Worst confusions (true -> predicted):", worst_pairs)
# typical: (5, 3, ~50), (3, 5, ~45), (8, 5, ~40), (4, 9, ~35), (9, 4, ~30)

# Look at sample errors for the worst pair
true_class, pred_class, _ = worst_pairs[0]
mask = (y_test == true_class) & (y_pred == pred_class)
errors_X = X_test[mask][:9]
fig, axes = plt.subplots(3, 3, figsize=(6, 6))
for ax, img in zip(axes.flat, errors_X):
    ax.imshow(img.reshape(28, 28), cmap='gray_r')
    ax.axis('off')
plt.suptitle(f'true {true_class}, predicted {pred_class}')
plt.show()

Looking at the worst MNIST confusion (often 3 vs 5 or 5 vs 8), you'll see digits with ambiguous loops, partial strokes, or stylistic variations the linear model can't disambiguate. This points you to: try a CNN (another chapter), or augment data with rotations and small distortions.

Some examples have multiple true labels. A picture might contain both "cat" and "dog". A news article might be tagged "politics" and "economy". The target becomes a length-$K$ binary vector $y \in \{0, 1\}^K$ where $y_k = 1$ if class $k$ applies.

The model output is $K$ independent sigmoid outputs (not one softmax). Loss is the sum of $K$ binary cross-entropies. This is not the same as multi-class.

# Synthetic multi-label MNIST: predict (is_large, is_odd) for each digit
y_train_large = (y_train >= 7)
y_train_odd = (y_train % 2 == 1)
y_multilabel = np.c_[y_train_large, y_train_odd]   # shape (N, 2)

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier()
knn.fit(X_train, y_multilabel)
preds = knn.predict(X_test[:5])
print(preds)
# Each row is [is_large, is_odd] prediction

In PyTorch, the loss is F.binary_cross_entropy_with_logits applied to a $K$-dim sigmoid output. This is the loss for image tagging, multi-topic classification, and many recommender system formulations.

Metric for multi-label. Per-class precision/recall/F1, then averaged. Three averaging strategies:

• macro: unweighted average across classes. Each class counts equally.
• micro: pool TP/FP/FN counts across classes, then compute. Each example counts equally.
• weighted: average across classes, weighted by class frequency.

from sklearn.metrics import f1_score
print(f"Macro F1: {f1_score(y_test_multilabel, preds, average='macro'):.3f}")
print(f"Micro F1: {f1_score(y_test_multilabel, preds, average='micro'):.3f}")

Use macro when each class is equally important. Use micro when each example is equally important. Weighted is a compromise.

In the MNIST-5 problem, positives are 9% of the data. In real-world problems like fraud detection or rare disease screening, the positive class can be 0.1% or less. Standard classifiers trained on imbalanced data tend to:

1. Predict the majority class for almost everything.
2. Have high accuracy but very low recall on the minority class.
3. Produce miscalibrated probabilities skewed toward the majority class.

Three families of fixes:

Class weights. Re-weight the loss so each class contributes equally to the gradient regardless of frequency. sklearn supports this via class_weight='balanced':

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(class_weight='balanced', max_iter=1000)
clf.fit(X_train, y_train_5)

In PyTorch: F.cross_entropy(..., weight=class_weights) where class_weights is a vector with one entry per class.

Resampling. Rebalance the training set before fitting: oversample the minority class so it appears more often, or undersample the majority class so it appears less. The imblearn library packages these — RandomOverSampler duplicates minority examples, SMOTE instead synthesizes new ones by interpolating between nearby minority points. Trade-off: oversampling causes overfitting, undersampling discards data.

Threshold tuning. Train a normal classifier, then pick the decision threshold on validation data to optimize the metric you care about (often F1 or recall at a precision floor). This is often the cleanest fix and gets ignored.

# Find the threshold maximizing F1
from sklearn.metrics import f1_score
probs = clf.predict_proba(X_val)[:, 1]
best_f1, best_t = 0, 0.5
for t in np.linspace(0.05, 0.95, 19):
    preds = (probs > t).astype(int)
    f1 = f1_score(y_val_5, preds)
    if f1 > best_f1:
        best_f1, best_t = f1, t
print(f"Best F1 {best_f1:.3f} at threshold {best_t:.2f}")

A classifier is calibrated if when it says "80% probability of class 1", the true frequency in cases where it said that is also 80%. Most classifiers, especially neural networks, are miscalibrated. They are overconfident.

from sklearn.calibration import calibration_curve

probs_test = pipe.predict_proba(X_test)[:, 1]
frac_pos, mean_pred = calibration_curve(y_test_5, probs_test, n_bins=10)

plt.plot(mean_pred, frac_pos, marker='o')
plt.plot([0, 1], [0, 1], 'k--')
plt.xlabel('Mean predicted probability')
plt.ylabel('Fraction of positives')
plt.title('Calibration curve')
plt.grid(True)
plt.show()

This plot — predicted probability on the x-axis, observed frequency on the y-axis — is the calibration curve, also called a reliability diagram. A perfectly calibrated classifier lies on the diagonal. Overconfident classifiers fall below it.

Two fixes:

Platt scaling. Fit a logistic regression on the validation set with the classifier's scores as the only feature. Use the fitted sigmoid to recalibrate predictions.

Isotonic regression. A non-parametric monotonic mapping from scores to calibrated probabilities. More flexible than Platt but needs more data.

sklearn.calibration.CalibratedClassifierCV does both:

from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(sgd, cv=3, method='isotonic')
calibrated.fit(X_train, y_train_5)
probs_calib = calibrated.predict_proba(X_test)[:, 1]

This matters in production. If you use predicted probabilities as inputs to downstream business logic (rank items, decide thresholds, compute expected costs), miscalibrated probabilities corrupt everything downstream. Always check calibration on a held-out set before deploying.

Putting it all together: a stratified, cross-validated, calibrated, class-weight-aware classification pipeline.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import classification_report, confusion_matrix

# Stratified CV preserves class proportions across folds
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

base = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(class_weight='balanced', max_iter=1000)),
])

calibrated = CalibratedClassifierCV(base, cv=skf, method='isotonic')

# Cross-validated metrics
scores = cross_val_score(calibrated, X_train, y_train_5, cv=skf, scoring='f1')
print(f"CV F1: {scores.mean():.3f} ± {scores.std():.3f}")

# Final fit on all training data
calibrated.fit(X_train, y_train_5)

# Test (once)
y_test_pred = calibrated.predict(X_test)
print(classification_report(y_test_5, y_test_pred))
print(confusion_matrix(y_test_5, y_test_pred))

This is a strong starting template for many tabular classification problems. Pipeline composition, stratified CV, and metrics beyond accuracy are broadly useful. Class weights and probability calibration are conditional: use them when the class/cost structure or downstream probability decisions justify them, and validate the choice on held-out data.