The ML Landscape

Tom Mitchell's 1997 definition is the one researchers actually use:

A computer program is said to learn from experience $E$ with respect to some task $T$ and performance measure $P$, if its performance on tasks in $T$, as measured by $P$, improves with experience $E$.

That's three things: a task ($T$), data ($E$), a metric ($P$). The reason this matters: when someone says "we trained a model", the right follow-up is always "on what data, with what metric, for what task?". Without those three, the claim is incomplete.

Géron sharpens it further: ML is useful when (a) traditional rule-based programming requires long lists of hand-tuned rules, (b) the rules change over time, or (c) you have data with structure but no human-articulable model of it. Spam filtering is the canonical example. The 1990s solution was rules ("contains the word VIAGRA, score 5"). The modern solution is to let a classifier learn the rules from labeled examples, and to retrain it weekly as spammers adapt.

ML algorithms split, first and most importantly, by what their training data looks like.

Supervised learning. Every training example is a pair $(x_i, y_i)$. You're given the inputs and the desired outputs. The model learns a function $f: X \to Y$. Classification (discrete $Y$) and regression (continuous $Y$) are the two flavors. ImageNet classification, sentiment analysis, house-price prediction, named-entity recognition: all supervised. The dominant paradigm. The reason ML has a "data labeling" industry.

Unsupervised learning. Training data is $\{x_i\}$ only. No labels. The model has to find structure on its own. Clustering (group similar examples), density estimation (model $p(X)$), dimensionality reduction (find a low-dim representation that preserves what matters). Word embeddings are unsupervised. Self-supervised pretraining (predict the next word) is technically supervised on a synthetic label derived from the data itself; people call it unsupervised anyway.

Reinforcement learning. The model (called an agent) interacts with an environment, takes actions, receives rewards, learns a policy that maximizes expected reward. No labels in the supervised sense; the reward signal is sparser and arrives after a delay. AlphaGo, robot control, recommendation-system bandits, RLHF on top of a pretrained LLM.

There are mixed regimes worth knowing the names of:

• Semi-supervised: a little labeled data plus a lot of unlabeled data.
• Self-supervised: derive labels from the data structure itself (BERT's masked language modeling).
• Active learning: the model picks which examples should be labeled next.
• Transfer learning: train on task A, fine-tune on task B.

Modern LLM training is a stack of these. Pretraining is self-supervised. Instruction-tuning is supervised. RLHF is reinforcement learning. Each stage builds on the last.

Orthogonal to the supervision axis: how the data arrives.

Batch (offline) learning. You have a fixed dataset. You train once. You deploy the trained model. To update, you collect new data, retrain from scratch (or from a checkpoint), redeploy. This is what every textbook example does. It is also what most production ML systems do, on a weekly or daily retraining cadence.

Online learning. The model updates as new data arrives. One example, or a small mini-batch, at a time. The model is always "live". Used for streaming systems where retraining from scratch is too slow (high-frequency trading), where storing all historical data is impossible (sensors), or where the data distribution changes fast enough that recent examples matter much more than old ones (recommendation systems on rapidly-shifting catalogs).

The hidden cost of online learning: it is much easier to break. A burst of bad data poisons the model immediately. Batch learning gives you a chance to review the dataset before training. Online learning does not. Most production "online" systems are actually mini-batch online with a short delay and a separate validation gate before the model goes live.

A third independent axis: how the model represents what it learned.

Instance-based. The model is the training data. To make a prediction, find the most similar training examples and copy or interpolate from their labels. k-Nearest Neighbors is the canonical case. Pros: zero training time, easy to update (just add new examples). Cons: prediction is slow ($O(N)$ per query unless you index), memory-hungry, sensitive to feature scaling (the features are the attributes of each input $x_i$ the model sees — the columns of your data), no compression of the data into knowledge.

Model-based. The training procedure produces a compact set of parameters $\theta$ that summarizes the training data. To predict, you evaluate $f_\theta(x)$, which is fast and constant-time. Linear regression, logistic regression, decision trees, neural networks: all model-based. Pros: fast inference, parameters are interpretable (sometimes), the model captures structure beyond memorization. Cons: training is slow, capacity is limited by the model class, you need to choose the model class up front.

Almost everything in this curriculum is model-based. kNN shows up in another chapter as a baseline and again in another chapter (RAG) as the "retrieval" half.

This is the central explanation of why models fail.

Imagine your model's prediction error decomposed:

Bias is the error from your model being too simple to capture the true function. A linear model fitting a clearly nonlinear pattern has high bias.

Variance is the error from your model being too sensitive to the particular training set you used. A high-degree polynomial that wiggles to hit every training point but produces wildly different predictions on different training sets has high variance.

Irreducible noise is everything else: noise in the labels, missing features, fundamentally random aspects of the world.

The trade-off: simpler models have higher bias, lower variance. More complex models have lower bias, higher variance. The sweet spot is the model class where the total error is minimized. This is why "throw more parameters at it" doesn't always work, and why "use a simpler model" doesn't always work either.

In practice:

• High bias = underfitting. Train loss and val loss are both high. Both are close together.
• High variance = overfitting. Train loss is low. Val loss is much higher.

Both have specific fixes (more capacity for underfitting, more regularization or data for overfitting). Regularization means adding a penalty for model complexity to the loss, so the optimizer is pushed toward simpler functions that are less able to memorize the training set; another chapter goes through the specific forms in detail.

You never use the same data to train a model and to estimate its real-world performance. The reason is simple: training optimizes performance on the training data, which makes training-set performance an upper bound on generalization, not an estimate of it.

The standard split:

• Train set (~70-80%): used to fit parameters via gradient descent or whatever your optimizer is.
• Validation set (~10-15%): used to pick hyperparameters (learning rate, regularization strength, model architecture) — the settings you choose for the learning algorithm, as opposed to the parameters gradient descent fits for you. You look at it many times.
• Test set (~10-15%): used to estimate real-world performance. You look at it once, at the very end, after every hyperparameter is fixed.

The test set is the most-violated rule in ML practice. Every time someone reports a final test accuracy and then says "let me tweak the model and try again", they are leaking the test set into the model selection process. The test number becomes an overestimate of real-world performance. Multiply that across thousands of papers and you get the reproducibility crisis.

For small datasets, k-fold cross-validation generalizes the train/val split: divide the data into $k$ folds, train $k$ times each holding one fold as validation, average the validation scores. sklearn.model_selection.KFold is the standard tool.

from sklearn.model_selection import train_test_split

# Single split (good for large datasets)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.15, random_state=42)

# K-fold cross-validation (good for small datasets)
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
print(f"CV accuracy: {scores.mean():.3f} +/- {scores.std():.3f}")

Data leakage is what happens when information from outside the training set sneaks into the training process. The model looks great on validation. It fails in production. The textbook examples:

• Future leakage: a feature in the training set is computed using data from the future relative to the prediction time. Example: predicting customer churn next month using "total revenue this year" as a feature, when "this year" includes future months.
• Target leakage: a feature is a proxy for the target. Example: predicting whether a patient has a disease using "hospital department they were admitted to", when the diagnostic department is highly correlated with the diagnosis.
• Train-test contamination: the same example, or a near-duplicate, appears in both train and test. Common with image datasets that have multiple crops of the same scene, or with web-scraped datasets where the same Wikipedia article appears multiple times.
• Preprocessing leakage: you compute normalization statistics on the full dataset before splitting. The mean and std of your features now depend on the test set. Fix: fit the scaler on train only, then transform train, val, and test with the same fitted scaler.

from sklearn.preprocessing import StandardScaler

# WRONG: leaks test statistics
X_all = scaler.fit_transform(X_all)   # then split

# RIGHT: fit on train only
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_val = scaler.transform(X_val)
X_test = scaler.transform(X_test)

The pipeline pattern (sklearn.pipeline.Pipeline) exists specifically to prevent this kind of leakage by composing preprocessing and modeling into one fittable object.

Accuracy looks like an obvious choice. It is often the wrong one.

For binary classification with class imbalance, accuracy is misleading. If 99% of your data is class 0, predicting class 0 always gives 99% accuracy. The model has learned nothing. You need the full confusion matrix:

From which:

• Precision = $\frac{TP}{TP + FP}$ — of the things I predicted positive, how many really are?
• Recall = $\frac{TP}{TP + FN}$ — of the things that really are positive, how many did I catch?
• F1 = $\frac{2 \cdot P \cdot R}{P + R}$ — the harmonic mean of P and R, useful when both matter.
• ROC-AUC = the area under the receiver operating characteristic curve. Measures the model's ability to rank positives above negatives at varying thresholds. Threshold-free.

Which to use. It depends on the cost of each error.

• Cancer screening: false negatives are catastrophic, false positives are merely inconvenient. Optimize recall.
• Spam filtering: false positives (real emails marked spam) are worse than false negatives. Optimize precision.
• Information retrieval: rank quality matters. Optimize AUC or precision@k.
• Fraud detection: depends. Sometimes you need a precision-recall curve at multiple operating points.

For regression: MSE (mean squared error) penalizes large errors quadratically. MAE (mean absolute error) is robust to outliers. R² is the fraction of variance explained, normalized so a constant predictor scores 0 and a perfect predictor scores 1.

another chapter will go through all of these in code.

Géron's checklist (06-practice/geron-ml-project-checklist) lists eight steps that every ML project follows. Memorize them.

1. Frame the problem. What is the business or research goal? What does "good" look like? What is the current baseline (heuristic, manual process, simpler model)? If you cannot answer these three questions, you are not yet ready to write code.

2. Get the data. Find it, download it, document its provenance. Check licenses. Save a snapshot of the exact version you used (raw bytes, not "we used the Kaggle dataset").

3. Explore the data. Plot histograms of every feature. Plot pairwise scatter for the most important features against the target. Look at the labels. Compute summary statistics. Find the weirdness. This is where you discover that 5% of your "income" feature is the value -999, used as a missing-value indicator.

4. Prepare the data. Clean. Impute missing. Encode categoricals. Scale numerics. Engineer features. Pipeline everything so it's reproducible.

5. Try several models. Pick three or four candidate model families. Train each. Compare on validation set. Do not yet tune.

6. Fine-tune the best model. Hyperparameter search (grid, random, Bayesian). Look at errors. Engineer more features. Iterate.

7. Present the solution. Write down what you did, what worked, what didn't. Sanity-check on the test set. Once.

8. Launch, monitor, maintain. Deploy. Set up monitoring for input distribution shift, output distribution shift, and downstream metric degradation. Plan for retraining cadence.

another chapter walks through all eight on a real housing-price dataset.

A trained model is one component of an ML system, not the whole thing. The system also contains:

• Data pipeline: how raw data becomes training-ready features (ETL, feature stores, schema validation).
• Training infrastructure: where the model trains (Colab notebook, SageMaker, internal GPU cluster), how runs are tracked (MLflow, Weights & Biases, plain CSVs).
• Serving infrastructure: where the model runs at inference (REST API, batch job, on-device).
• Monitoring: dashboards for distribution shift, latency, error rates.
• Retraining loop: when and how the model gets updated.

The Made-With-ML curriculum splits these out explicitly. Huyen Chip's Designing ML Systems and Eugene Yan's posts hammer on the same point: in production, the model is 10% of the code and 5% of the bugs.

another chapter (MLOps) covers all of this. For now, the lesson is just: when you read about an ML system in the news, the headline is the model, but most of the actual engineering work is elsewhere.

The model trains fine. The val accuracy is good. You ship it. Then:

1. Distribution shift: production data looks different from training data. Maybe seasons changed. Maybe a new product was launched. Maybe COVID happened. The model is fine; the world moved.

2. Concept drift: the relationship between inputs and labels changes. Spammers adapt. Fashion changes. Users learn the system and game it.

3. Feedback loops: the model's predictions influence the data it will see next. A recommender that pushes popular items creates more popularity for those items, which the next training run sees as "users love popular items". Self-reinforcing bias.

4. Adversarial inputs: someone discovers inputs your model handles badly, on purpose. Image classifiers fooled by sticker patches. LLMs jailbroken by adversarial suffixes. Once the existence of an attack is public, the attack becomes part of your real input distribution.

5. Label drift: the labelers' standards change. The 2023 hate-speech labeling guidelines differ from the 2024 ones. Your training set is now stale.

6. Underspecification: the training objective is consistent with many possible models, only some of which generalize well. You shipped the version of the model that happened to memorize a spurious correlation.

7. Operational silent failure: the model returns reasonable-looking outputs on garbage inputs. Nobody notices. This is the most common failure mode. The fix: alerting on input distribution, not just output distribution.

8. Ethical failure: the model performs unequally across demographic groups. ImageNet's "wedding" class was overwhelmingly Western. Face recognition systems had higher error rates on darker skin. These are not abstract; they have shipped, and they have caused harm. another chapter covers this in depth.

The list is from the fastbook ethics chapter (13-fastbook/03_ethics) merged with Huyen Chip's production-failures post and Eugene Yan's "what could go wrong" framework.

A pragmatic flowchart. Use it for ≈80% of new problems before reaching for anything fancier.

Tabular data, classification or regression.

• Start: sklearn.ensemble.GradientBoostingClassifier or xgboost.XGBClassifier or lightgbm.LGBMClassifier. Trees on tabular data are still state of the art.
• Baseline: sklearn.linear_model.LogisticRegression (classification) or LinearRegression (regression). If gradient boosting doesn't crush this baseline, your features need work.
• Iterate from there.

Images.

• Start: a pretrained vision backbone (torchvision.models.resnet50 or a ViT) with a new classifier head. Fine-tune.
• Baseline: a simple CNN trained from scratch. If it beats pretrained, something's weird.
• another chapter.

Text.

• Start: a pretrained transformer (transformers.AutoModel) with a task-specific head. For classification, embed + logistic regression often works.
• Baseline: TF-IDF + logistic regression. Frequently competitive. 15.

Sequences / time series.

• Start: gradient boosting on engineered features (lag features, rolling stats).
• Then: LSTM or Transformer with explicit time encoding.
• another chapter.

Tiny dataset.

• k-NN, naive Bayes, or a heavily-regularized linear model. Stop. More data first.

No labels.

• Cluster (k-means, DBSCAN) and inspect the clusters with a human. Or train a self-supervised representation and use those embeddings downstream.
• another chapter.

Everything I've written is true for classical ML. Deep learning changes three things, not all of them.

It changes the function family: from "linear in features" or "tree" to "deep neural network with millions or billions of parameters". This lets you skip feature engineering for images, text, and speech, because the network learns features.

It changes the data regime: deep models need a lot more labeled data, or a lot of cheap unlabeled data plus a clever pretraining objective.

It does not change the workflow. You still split train/val/test. You still pick a loss. You still track training and validation curves. You still measure error analysis on held-out examples. The fastbook chapter 13-fastbook/01_intro calls this out explicitly: deep learning is not magic, it is the same supervised learning loop with a bigger function family.

The downstream chapters take this for granted. By another chapter you'll be training neural networks. The reason it's worth stating up front: the vocabulary in this chapter (loss, gradient, train/val/test, overfitting, metric) survives unchanged into the deep learning chapters. You are not learning a separate framework. You are learning more of the same.

A practical skill that nobody teaches explicitly. When you open a new ML repository, here is where to look, in order.

1. README.md — find the one paragraph that explains what the model does and what dataset it trains on.
2. The script that trains the model. Often train.py or main.py. Find the line that calls loss.backward. The model class is upstream of that line. The dataset is upstream of that line.
3. The model class. Read __init__ for what the parameters are. Read forward for the function family.
4. The data loader. Read __getitem__ for what one training example looks like. Read the transforms applied to it.
5. The loss function. Often hidden inside forward or computed in train.py. Find the F.cross_entropy call or equivalent.
6. The hyperparameters. Usually in a config.yaml or argparse defaults. Note: learning rate, batch size, total steps, warmup steps, weight decay. These are the five hyperparameters that determine 80% of training outcomes.

Karpathy's nanoGPT (12-karpathy-code/nanoGPT-master-model + nanoGPT-master-train) is laid out exactly this way and is worth opening as a reference point. By another chapter you'll be reading it line by line.