MLOps & Observability

MLOps is the practice of running ML systems in production. It is not "DevOps for ML"; it is DevOps plus the parts of running ML that have no DevOps analog. The parts that have no analog:

• The artifact is learned from data, so the data is part of the source code. Any bug in the data is a bug in the model.
• The artifact is probabilistic, so "correct" is replaced by "good enough on the relevant distribution", and the relevant distribution drifts.
• The artifact is expensive to recompute, so rollbacks, hotfixes, and A/B tests are all colored by the cost of a training run.
• The artifact's behavior on inputs you have not seen is not knowable in advance, so production monitoring is not optional; it is the only way you find out.

Chip Huyen's "Real-time machine learning" and "Why data distribution shifts matter" (24-founder-blogs/huyenchip-*) make this case at length. Eugene Yan's "Design patterns for ML systems" (24-founder-blogs/eugeneyan-*) is the practical companion. The combined point: most of the production failures of ML systems happen at the data layer, not at the model layer. Most ML team time, post-launch, is data work.

The mental model that worked best for me, lifted from 06-practice/madewithml-mlops-design: an ML system is a directed graph from raw events through derived features through training through evaluation through deployment through monitoring back to retraining. Every edge in that graph is a place a bug can live. MLOps is the discipline of instrumenting every edge.

No code path here. Vocabulary section. Citations:

Memorize this sequence. Every section in this chapter hangs off it:

1. Problem framing. What decision does the model support? Who is the user? What is the success metric, in business units?
2. Data collection. From where, with what schema, at what frequency, with what privacy posture.
3. Data labeling. When required. Eugene Yan's "Bootstrapping data labels" essay is the canonical entry point.
4. EDA + feature engineering. Distributions, correlations, leakage checks, transformations.
5. Modeling + training. The part you have been doing for the last twenty chapters.
6. Evaluation. Holdout, cross-validation, slice analysis, fairness analysis, calibration.
7. Deployment. Pick a serving pattern (batch, online, streaming), pick infrastructure, write the service.
8. Monitoring. Inputs (feature drift), outputs (prediction drift), and outcomes (eventual ground truth) all measured against trained baselines.
9. Retraining trigger. Time-based, drift-based, performance-based.
10. Retirement. Eventually, you turn the model off. Do so deliberately.

This loop is not waterfall; the practical version cycles dozens of times per quarter. Géron's "ML Project Checklist" (06-practice/geron-ml-project-checklist) is the canonical printable version, and you should print it.

Library path (a single Python module that asserts the lifecycle has been followed for every shipped model):

# lifecycle_gate.py — gates a shipping model on lifecycle artifacts being present
from pathlib import Path
import json, hashlib

REQUIRED = [
    "problem_framing.md",          # one-pager: decision, user, success metric
    "data_schema.json",            # data versioning lineage
    "eda.ipynb",                   # the EDA artifact
    "training_config.yaml",        # hyperparameters, seeds, code SHA
    "eval_report.json",            # holdout + slice metrics
    "model_card.md",               # intended use, limitations, training data
    "monitoring_plan.yaml",        # what to track, thresholds, alert routes
    "rollback_plan.md",            # how to disable in <5 minutes
]

def assert_ready(model_dir: Path):
    missing = [r for r in REQUIRED if not (model_dir / r).exists()]
    assert not missing, f"missing lifecycle artifacts: {missing}"

Code without version control is amateur hour. Data without version control is the same thing, with worse failure modes. The unit of versioning for an ML system is (code, data, config, environment) → model. If any of those is unpinned, you cannot reproduce.

DVC is the most-used tool. It stores small metadata files in git that point at large data blobs in an object store (S3, GCS, local). The workflow is dvc add data/train.parquet (which produces data/train.parquet.dvc you commit), dvc push (which uploads the blob), and dvc pull (which downloads what dvc.lock says you need). Pipelines (dvc.yaml) define stages with inputs, outputs, and commands; DVC tracks the DAG and re-runs stale stages.

Pachyderm does the same thing as Kubernetes-native object pipelines, with stronger lineage guarantees, at higher operational cost.

lakeFS treats your data lake itself like a git repo with branches and atomic commits. Good when "I want to branch the data" is a frequent operation.

For most teams, dvc + git + S3 is the right floor. Add Pachyderm or lakeFS only when you have hit a wall.

Library path (a minimal dvc pipeline):

# dvc.yaml
stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - data/raw.csv
      - src/prepare.py
    outs:
      - data/prepared.parquet
  train:
    cmd: python src/train.py
    deps:
      - data/prepared.parquet
      - src/train.py
      - config.yaml
    outs:
      - models/model.pt
    metrics:
      - metrics.json:
          cache: false

# Initialize and run
dvc init
dvc remote add -d s3 s3://bucket/path
dvc add data/raw.csv
git add data/raw.csv.dvc .gitignore
git commit -m "track raw data"
dvc push
dvc repro                    # runs only stale stages
git add dvc.lock metrics.json
git commit -m "trained run-001"

From-scratch path (a 30-line content-addressed store, for understanding):

import hashlib, shutil, json
from pathlib import Path

class CAS:
    """Content-addressed store. The 'manifest' is a JSON file mapping logical
    names to content hashes. To check out a version, look up the hash and
    copy the blob to the working path."""
    def __init__(self, store: Path):
        self.store = store
        self.store.mkdir(parents=True, exist_ok=True)

    def hash_file(self, path: Path) -> str:
        h = hashlib.sha256()
        with open(path, "rb") as f:
            for chunk in iter(lambda: f.read(1 << 20), b""):
                h.update(chunk)
        return h.hexdigest()

    def put(self, path: Path) -> str:
        h = self.hash_file(path)
        dest = self.store / h[:2] / h[2:]
        if not dest.exists():
            dest.parent.mkdir(parents=True, exist_ok=True)
            shutil.copy(path, dest)
        return h

    def checkout(self, manifest: Path, into: Path):
        m = json.loads(manifest.read_text())
        for name, h in m.items():
            shutil.copy(self.store / h[:2] / h[2:], into / name)

A feature store is a system that computes features once and serves them to both training and inference, ensuring the same feature has the same value in both places. The thing it prevents: training-serving skew, the most-common silent ML bug.

Eugene Yan's "Feature stores: a hierarchy of needs" (24-founder-blogs/eugeneyan-eugeneyan-com-writing-feature-stores) is the canonical taxonomy. The three layers:

1. Offline store: historical features for training. Usually a data warehouse — a queryable store for large analytical tables (Snowflake, BigQuery, Redshift) — or a data lake: cheap bulk file storage (an object store like Amazon S3 or Google Cloud Storage, GCS) holding columnar parquet files.
2. Online store: low-latency features for inference. A key-value store you can read in single-digit milliseconds (Redis, DynamoDB, Cassandra).
3. Feature transformation: the logic that produces both. Defined once; executed in two places (batch backfill into offline store, streaming or request-time compute into online store).

Feast is the open-source canonical. You define entities (user, item), features (last-7-day clicks), and sources (a Snowflake table). Feast handles backfills, retrieval for training, and online lookup at inference. Tecton is the commercial best-in-class. Home-grown is fine for many teams; the discipline is what matters, not the tool.

Library path (a minimal Feast definition):

from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64
from datetime import timedelta

user = Entity(name="user_id", join_keys=["user_id"])

source = FileSource(
    path="s3://bucket/user_features.parquet",
    timestamp_field="event_timestamp",
)

user_clicks_7d = FeatureView(
    name="user_clicks_7d",
    entities=[user],
    ttl=timedelta(days=1),
    schema=[
        Field(name="clicks_7d", dtype=Int64),
        Field(name="ctr_7d", dtype=Float32),
    ],
    source=source,
)

# At training time
from feast import FeatureStore
fs = FeatureStore(repo_path=".")
training_df = fs.get_historical_features(
    entity_df=entity_df_with_user_id_and_ts,
    features=["user_clicks_7d:clicks_7d", "user_clicks_7d:ctr_7d"],
).to_df()

# At inference time
features = fs.get_online_features(
    features=["user_clicks_7d:clicks_7d", "user_clicks_7d:ctr_7d"],
    entity_rows=[{"user_id": 12345}],
).to_dict()

The minimum: every training run logs its hyperparameters, its metrics over time, its config, the code SHA, the seed, and the produced artifacts. Without this you cannot answer "what was the best hyperparameter setting from last quarter" and you cannot diff two runs.

Weights & Biases is the dominant SaaS. Two lines of code in your training loop give you a hosted dashboard with metric curves, system metrics (GPU, memory), config diffs, artifact lineage, hyperparameter sweeps, and a model registry. The free tier is generous; the enterprise tier handles on-prem.

MLflow is the dominant open-source. Same primitives, self-hosted, more friction to set up, no vendor lock-in. Many teams run MLflow because the ops team is comfortable with it.

The decision is rarely about features; it is about who is going to operate the tracking server.

Library path (W&B):

import wandb
import torch

run = wandb.init(
    project="ch25-mlops",
    config={
        "lr": 3e-4,
        "batch_size": 64,
        "model": "resnet50",
        "seed": 0,
    },
)
config = wandb.config

# ... train loop ...
for step, (x, y) in enumerate(dataloader):
    loss = train_step(model, x, y, optimizer)
    if step % 100 == 0:
        wandb.log({"train/loss": loss.item(), "train/lr": optimizer.param_groups[0]["lr"]}, step=step)

# Log the trained model as an artifact for later promotion to the registry
artifact = wandb.Artifact("resnet50-classifier", type="model")
artifact.add_file("model.pt")
run.log_artifact(artifact)
run.finish()

Library path (MLflow — same idea, different API):

import mlflow

mlflow.set_experiment("ch25-mlops")
with mlflow.start_run():
    mlflow.log_params({"lr": 3e-4, "batch_size": 64, "model": "resnet50", "seed": 0})
    for step in range(num_steps):
        loss = train_step(...)
        mlflow.log_metric("train_loss", loss, step=step)
    mlflow.pytorch.log_model(model, artifact_path="model")

Hyperparameter sweeps are where W&B earns its keep. Define a sweep config; the agent process pulls suggestions from the W&B Bayesian-optimization server; you run many agents in parallel; W&B updates the parallel-coordinate plot in real time.

# sweep.yaml
program: train.py
method: bayes
metric:
  name: val/accuracy
  goal: maximize
parameters:
  lr:
    distribution: log_uniform_values
    min: 1e-5
    max: 1e-2
  weight_decay:
    distribution: log_uniform_values
    min: 1e-6
    max: 1e-2
  dropout:
    distribution: uniform
    min: 0.0
    max: 0.5

wandb sweep sweep.yaml          # registers the sweep, prints SWEEP_ID
wandb agent <SWEEP_ID>          # run one or more in parallel

After tracking, you need a place to say "this version is the production version". A model registry is a small database of model artifacts annotated with stage (development, staging, production), version, lineage (which run produced it, which data), and approvals.

W&B Artifacts, MLflow Model Registry, Vertex AI Model Registry, and SageMaker Model Registry all implement the same primitives:

• Versioned model artifacts — every registered model has a (name, version) pair.
• Stages — Development, Staging, Production, Archived per version, transitionable by RBAC-protected actions.
• Aliases — prod and canary aliases that point at versions and can be flipped atomically.
• Lineage — every version links back to the run that produced it and the data version it trained on.

The discipline: serving code references models by alias, not version. The prod alias is what model_loader.py resolves. Promotion is a single API call. Rollback is the same API call to the previous version.

Library path:

import wandb
api = wandb.Api()
# Promote a registered model
model_artifact = api.artifact("team/resnet50-classifier:v17")
model_artifact.aliases.append("prod")
model_artifact.save()

# In serving code:
def load_prod_model():
    api = wandb.Api()
    artifact = api.artifact("team/resnet50-classifier:prod")
    artifact.download(root="/tmp/model")
    return torch.load("/tmp/model/model.pt")

A single GPU gets you far. For most teams, single-GPU training is the right answer; if you need more, you usually need more efficient single-GPU code first. When you actually need multi-GPU or multi-node, the landscape is:

PyTorch native: torch.distributed + torchrun. Lowest abstraction. The pattern is distributed data-parallel (DDP): one process per GPU, each holding a full copy of the model, each fed a different slice of the batch, with gradients averaged across them every step. You write the rank logic (each process gets a rank, its integer ID in the group), manage the process group (the set of processes that coordinate, set up by init_process_group), and place the model; torchrun launches one process per GPU, possibly across multiple machines (multi-node). Works. Painful.

Lightning / Accelerate: thin wrappers around torch.distributed that handle the rank logic so you write a single-machine-shaped training loop and run it on any cluster.

Ray Train: the higher-level framework. Define a training function; Ray schedules it across a cluster. Works for PyTorch, TensorFlow, XGBoost, LightGBM. Pairs with Ray Tune for hyperparameter search and Ray Data for distributed data loading.

Modal: serverless GPU compute — you never provision machines; the platform's autoscaler starts containers when requests arrive, scales them to zero when idle, and bills per second of actual use. Functions you decorate run in containers — isolated processes packaged with their own filesystem and dependencies, built from an image (the frozen recipe of OS, libraries, and code) — which Modal spins up on demand. Two-second cold starts (the delay while a fresh container boots from its image) on small images. Free tier exists. The "I don't want to manage a cluster" answer for most non-frontier teams.

SLURM / Kubernetes: the on-prem alternatives. SLURM if your shop is HPC-origin; K8s if your shop is cloud-native.

Library path (Ray Train, distributed PyTorch in 30 lines):

import ray
from ray.train.torch import TorchTrainer
from ray.train import ScalingConfig

def train_loop_per_worker(config):
    import torch, torch.nn as nn
    from ray import train
    from ray.train.torch import prepare_model, prepare_data_loader
    model = nn.Linear(100, 10)
    model = prepare_model(model)   # wraps in DDP across the cluster
    optimizer = torch.optim.AdamW(model.parameters(), lr=config["lr"])
    loader = prepare_data_loader(get_loader(config["batch_size"]))
    for epoch in range(config["epochs"]):
        for x, y in loader:
            loss = nn.functional.cross_entropy(model(x), y)
            optimizer.zero_grad(); loss.backward(); optimizer.step()
        train.report({"epoch": epoch, "loss": loss.item()})

trainer = TorchTrainer(
    train_loop_per_worker=train_loop_per_worker,
    train_loop_config={"lr": 1e-3, "batch_size": 64, "epochs": 10},
    scaling_config=ScalingConfig(num_workers=4, use_gpu=True),
)
result = trainer.fit()

Library path (Modal — serverless GPU function):

import modal

app = modal.App("ch25-train")
image = (
    modal.Image.debian_slim()
    .pip_install("torch", "torchvision", "wandb")
)

@app.function(image=image, gpu="A10G", timeout=3600)
def train(lr: float, epochs: int) -> dict:
    import torch
    # ... your training code ...
    return {"final_loss": 0.42}

@app.local_entrypoint()
def main():
    result = train.remote(lr=3e-4, epochs=10)
    print(result)

Run modal run train.py from your laptop. Modal builds the image, schedules an A10G GPU, runs the function, returns the result. You pay seconds.

Three ways to serve predictions, three latency/freshness trade-offs.

Batch. You score everything once a day. Cheap, simple, no online-store complexity. The right answer for daily emails, weekly recommendations, churn-risk lists. Limits: predictions are stale by up to the batch interval, you cannot react to events that happened after the batch ran.

Online (request-response). A service accepts a request, computes features (from an online store), runs the model, returns a prediction. Latency budget typically <100ms. The default for "user is about to see a thing, score it now": ranking, fraud detection, real-time recommendations.

Streaming. Predictions are emitted in response to events flowing through a stream transport — a system that stores and forwards a continuous log of events as they happen, the way Kafka, Kinesis, or Pulsar do. Used when "an event happened and a downstream system needs the model's response asap": real-time anomaly detection, real-time bidding, real-time fraud.

The decision is downstream of "what does the consumer of the prediction need". Huyen's "Real-time machine learning" (24-founder-blogs/huyenchip-huyenchip-com-2020-12-27-real-time-machine-learning-html) is the canonical taxonomy.

Library path (online inference with FastAPI + Modal):

import modal

app = modal.App("ch25-serving")
image = modal.Image.debian_slim().pip_install("torch", "transformers", "fastapi")

@app.cls(image=image, gpu="T4", min_containers=1, max_containers=10)
class Classifier:
    @modal.enter()
    def load(self):
        import torch
        self.model = torch.load("/models/model.pt")
        self.model.eval()

    @modal.method()
    def predict(self, x: list[float]) -> dict:
        import torch
        with torch.no_grad():
            t = torch.tensor(x).unsqueeze(0)
            logits = self.model(t)
            probs = torch.softmax(logits, dim=-1).squeeze().tolist()
        return {"probs": probs, "argmax": int(max(range(len(probs)), key=probs.__getitem__))}

@app.function(image=image)
@modal.fastapi_endpoint(method="POST")
def predict(payload: dict) -> dict:
    return Classifier().predict.remote(payload["features"])

FastAPI is a Python web framework: it turns a function into an HTTP endpoint — a named URL (here POST /predict, meaning a request that sends data to the /predict path) that any client can call over the network and get a JSON reply. Deploy: modal deploy serve.py. You get an HTTPS endpoint, autoscaling from min_containers=1 (keep one container warm to avoid cold starts) to max_containers=10, and per-second billing.

Library path (batch inference with Ray Data):

import ray

ds = ray.data.read_parquet("s3://bucket/features/2026-05-15/*.parquet")

class Predictor:
    def __init__(self):
        import torch
        self.model = torch.load("model.pt").eval()
    def __call__(self, batch):
        import torch
        with torch.no_grad():
            x = torch.tensor(batch["features"])
            batch["prediction"] = self.model(x).argmax(-1).tolist()
        return batch

predictions = ds.map_batches(Predictor, concurrency=8, num_gpus=1)
predictions.write_parquet("s3://bucket/predictions/2026-05-15/")

Online serving lives inside a latency budget. SLAs are written on percentiles, not averages: p99 is the latency that 99% of requests come in under (so only the slowest 1% are worse), and you target the tail because a fast average hides the users who time out. A budget like "p99 < 100ms" decomposes into: feature fetch + preprocessing + model forward + postprocessing + network. The model forward is rarely more than half. If your p99 is 200ms and your model forward is 30ms, you are not optimizing the model.

The components, with rough order-of-magnitude numbers for a typical recommender serving 8B parameters at fp16 on an A100:

Optimization order (cheap → expensive):

1. Batch. Most servers underuse the GPU. Dynamic batching with a 5-10ms window often doubles throughput at sub-1% latency cost.
2. Quantize. fp16 → int8 → int4. Each step is roughly 2× faster with manageable accuracy loss. AWQ, GPTQ, BitsAndBytes are the canonical libraries.
3. Speculative decoding for LLMs. A small draft model proposes tokens; the big model verifies. ~2× speedup at parity.
4. KV-cache management. For LLM serving, the KV cache is most of the memory and most of the per-token cost. vLLM's PagedAttention is the canonical implementation.
5. Distillation. Train a smaller student against the production model's outputs.
6. Architecture change. MoE, sliding-window attention, linear attention — only after exhausting the above.

Lilian Weng's "Inference Optimization" survey (18-lilian-weng/2023-01-10-inference-optimization) is the canonical reading.

Library path (vLLM, the standard LLM serving runtime in 2026):

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    dtype="float16",
    gpu_memory_utilization=0.9,
    max_model_len=8192,
    enable_prefix_caching=True,            # shared prompt prefixes cached
)
params = SamplingParams(temperature=0.0, max_tokens=256)
outputs = llm.generate(["Hello, who are you?"], params)
print(outputs[0].outputs[0].text)

vLLM handles continuous batching (swapping finished sequences out of the running batch and new ones in every step, instead of waiting for the whole batch to finish), the paged-attention KV cache (storing the KV cache in fixed-size pages like OS virtual memory, so GPU memory is not fragmented or over-reserved), and prefix caching (reusing the cached KV of a shared prompt prefix across requests), and serves an OpenAI-compatible HTTP API by default. It is the closest thing to a default for self-hosted LLM serving in 2026.

The four monitoring questions, in order of how quickly the signal arrives:

1. Are my inputs the same as in training? Feature distributions. Detectable in seconds.
2. Are my predictions distributed the same as in training? Prediction histograms. Detectable in minutes.
3. Are my labels (when they arrive) the same as in training? Label distributions. Detectable in hours to days, depending on ground-truth latency.
4. Is my model still accurate? Per-slice accuracy on labeled data. Detectable only after labels arrive.

The metrics:

Population Stability Index (PSI). Bin a feature on the reference distribution; compare bin proportions in production. $\text{PSI} = \sum_i (p_i - q_i) \log(p_i / q_i)$. Industry rule of thumb: PSI < 0.1 is no drift, 0.1-0.25 is moderate, > 0.25 is significant. Loose, but useful as a first-pass alert.

KL divergence. $D_\text{KL}(P \| Q) = \sum_i p_i \log(p_i / q_i)$. PSI is symmetrized KL. Use KL when the asymmetry matters.

Kolmogorov-Smirnov test. For continuous features. Computes the maximum gap between the two empirical CDFs (a cumulative distribution function maps a value to the fraction of samples at or below it; the KS statistic is the largest vertical distance between the reference curve and the production curve). Outputs a p-value. Be careful with sample sizes: with millions of rows, any tiny shift is "statistically significant"; you want effect size, not p-value.

Wasserstein distance. Better-behaved than KL when distributions barely overlap. Slower to compute.

Chi-squared for categorical features.

edges = np.quantile(reference, np.linspace(0, 1, n_bins + 1))
edges[0] -= 1e-9; edges[-1] += 1e-9
rp = np.histogram(reference, edges)[0] / len(reference) + 1e-9
pp = np.histogram(production, edges)[0] / len(production) + 1e-9
psi = float(entropy(rp, pp) + entropy(pp, rp))  # symmetrized KL = PSI

def psi(reference: np.ndarray, production: np.ndarray, n_bins: int = 10) -> float:
    ref = np.asarray(reference, dtype=float)
    prod = np.asarray(production, dtype=float)
    edges = np.quantile(ref, np.linspace(0, 1, n_bins + 1))
    edges[0] -= 1e-9
    edges[-1] += 1e-9
    r, _ = np.histogram(ref, edges)
    p, _ = np.histogram(prod, edges)
    rp = r / r.sum() + 1e-9
    pp = p / p.sum() + 1e-9
    return float(np.sum((rp - pp) * np.log(rp / pp)))

Chip Huyen's "Data distribution shifts and monitoring" (24-founder-blogs/huyenchip-huyenchip-com-2022-02-07-data-distribution-shifts-and-monitoring-html) is the canonical taxonomy. It pins precise probability statements on the informal "distribution shift" ideas: write the inputs as $X$, the label as $Y$. Covariate shift is when the input distribution $P(X)$ changes but the input→label relationship $P(Y \mid X)$ holds (your users changed, the world did not); label shift is when the label distribution $P(Y)$ changes; concept drift is when $P(Y \mid X)$ itself changes (the same input now means a different answer). Each is detectable by different signals and addressed by different mitigations.

Traditional ML observability tools (Arize Phoenix, Fiddler) compute the metrics above against tabular features. LLM observability tools (LangSmith, Arize Phoenix's LLM tracing, OpenTelemetry GenAI semantic conventions) layer in:

• Trace of every span in a complex prompt chain. A trace is the full recording of one request's path through the system; a span is one timed step inside it — a tool call, a retrieval, a model call — so a trace is a tree of spans, each carrying its own timings, token counts, and full prompts/responses.
• Eval scores automatically computed against each trace: faithfulness, relevance, answer-quality, hallucination-flag.
• Cohort analysis: filter traces by user, by tool used, by output property. Find the 12 sessions where the agent hallucinated a SKU.
• Dataset builder: promote interesting traces into a regression-eval set.
• Prompt-version A/B: ship a new prompt, watch the eval-score delta in real time.

The most-used in 2026 is LangSmith (LangChain's). It is opinionated but deeply integrated with the LangChain ecosystem. Arize Phoenix is the open-source neutral alternative; it speaks OpenTelemetry GenAI traces, which is the emerging standard.

Library path (LangSmith for tracing an LLM call):

import os
from langsmith import traceable
from openai import OpenAI

os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_PROJECT"] = "ch25-mlops"

client = OpenAI()

@traceable(name="answer_question")
def answer(question: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": question}],
    )
    return response.choices[0].message.content

answer("What's the capital of Burkina Faso?")
# Trace shows up in LangSmith with full request/response, latency, token counts, cost.

Library path (OpenTelemetry GenAI tracing, vendor-neutral):

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
tracer = trace.get_tracer(__name__)

def answer(question: str) -> str:
    with tracer.start_as_current_span("llm.chat") as span:
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", "gpt-4o")
        span.set_attribute("gen_ai.prompt.0.content", question)
        # ... call the model ...
        response = "..."
        span.set_attribute("gen_ai.response.0.content", response)
        return response

The OTel-GenAI spec is converging across Arize, Honeycomb, Datadog, Grafana. Because every vendor speaks the same wire format, you point your app at one collector — a service that receives spans and forwards them to your chosen backend(s) — and the data flows everywhere.

The discipline LLM teams converge on, eventually. The skeleton:

1. Golden set. A frozen list of (input, expected behavior) pairs that captures the things you care about. 50-500 examples. Hand-curated. Maintained.
2. Scorer. A function that takes (input, output) and returns a score. May be exact-match, regex, LLM-as-judge with a rubric, or a downstream task metric.
3. CI gate. Every prompt or model change runs the golden set. The gate fails if the score drops below a threshold (usually "no regression on >5% of cases").
4. Online sampling. Production traces feed a queue. A weekly review session promotes interesting traces (failures, edge cases, novel inputs) into the golden set.
5. Regression archive. When a case breaks, it goes into the golden set and never leaves.

Eugene Yan's "Evals, the most important thing in LLM apps" (24-founder-blogs/eugeneyan-eugeneyan-com-writing-evals) is the manifesto. The reason eval-driven development matters more for LLMs than for traditional ML: LLMs have so many failure modes that no single accuracy number captures them. The eval suite is the spec.

Library path (a minimal eval harness with LangSmith):

from langsmith import Client
from langsmith.evaluation import evaluate

client = Client()

# Golden dataset, one-time
client.create_dataset("burkina-qa", description="Geography Q&A golden set")
client.create_examples(
    dataset_name="burkina-qa",
    inputs=[{"question": "Capital of Burkina Faso?"}, ...],
    outputs=[{"answer": "Ouagadougou"}, ...],
)

# Scorer
def correctness(run, example) -> dict:
    expected = example.outputs["answer"].lower()
    actual = run.outputs["output"].lower()
    return {"key": "correct", "score": int(expected in actual)}

# Run
result = evaluate(
    answer,                       # the function under test from §11
    data="burkina-qa",
    evaluators=[correctness],
    experiment_prefix="gpt-4o-baseline",
)
print(result)

From-scratch path (a 40-line eval harness with no dependencies):

import json, hashlib
from pathlib import Path

def eval_run(fn, golden_path: str, scorer) -> dict:
    """Run fn on every example in golden_path; score and aggregate."""
    examples = [json.loads(l) for l in open(golden_path)]
    results = []
    for ex in examples:
        out = fn(**ex["inputs"])
        s = scorer(out, ex["expected"])
        results.append({"id": ex.get("id"), "score": s})
    agg = sum(r["score"] for r in results) / len(results)
    return {"per_example": results, "aggregate": agg, "n": len(results)}

def regression_gate(current: dict, baseline: dict, threshold: float = 0.95) -> None:
    """Fail if the aggregate score dropped by more than (1-threshold)."""
    ratio = current["aggregate"] / max(baseline["aggregate"], 1e-9)
    assert ratio >= threshold, (
        f"Regression: aggregate score {current['aggregate']:.3f} is "
        f"{ratio:.1%} of baseline {baseline['aggregate']:.3f}"
    )

Once you have monitoring and evals, you need a person who looks at the alerts. Most teams skip this. Most teams' models silently degrade.

The minimum on-call discipline for an ML system:

• Page-worthy alerts only. If the alert fires more than once a week and is non-actionable, it is noise; tune it down.
• Runbook per alert. Each alert has a one-page runbook: what the alert means, how to diagnose, the three most likely causes, the rollback action.
• Rollback button. A single command (or a single button in your CI/CD) that reverts the prod alias to the previous model version. Pre-tested. Audited monthly.
• Blameless post-mortem. Every page produces a post-mortem within a week. The output is a) what happened, b) why the monitoring/eval did not catch it earlier, c) a new test that would have caught it.
• Game days. Quarterly, you intentionally break something in staging and run the rotation through diagnosing it. If you have never done one, schedule one.

The set of failures that an ML on-call rotation handles, in rough order of frequency:

1. Drift. Input distribution shifted; predictions look weird.
2. Upstream data corruption. A producer of features started emitting NaNs.
3. Inference latency spike. Backend dependency slow; KV cache thrashing; GPU contention.
4. Cost anomaly. Token spend up 10× because someone shipped a bad prompt.
5. Capability regression. Eval suite dropped on the latest model swap.
6. Adversarial input. The another chapter red-team alerts. (Yes, you should have one. Yes, on-call should triage it.)

Library path (PagerDuty integration via opsgenie/slack — same shape):

import requests

def page(severity: str, title: str, details: dict):
    """Page the on-call rotation. severity in {'critical', 'high', 'low'}."""
    payload = {
        "payload": {"summary": title, "severity": severity, "source": "ml-monitor",
                    "custom_details": details},
        "routing_key": os.environ["PAGERDUTY_ROUTING_KEY"],
        "event_action": "trigger",
    }
    r = requests.post("https://events.pagerduty.com/v2/enqueue", json=payload)
    r.raise_for_status()

# In your drift detector:
if psi_value > 0.25:
    page("high", f"PSI {psi_value:.2f} on feature ctr_7d",
         {"feature": "ctr_7d", "model": "ranker-v17", "psi": psi_value,
          "runbook": "https://wiki/runbooks/feature-drift"})

The CS-curriculum version of MLOps treats it as DevOps + a model artifact. The lived experience is that several of the hardest parts have no DevOps analog. The list:

Reproducibility is harder than determinism. In conventional software, a bug-free system produces identical outputs for identical inputs. In ML, the same code on the same data with the same seeds produces slightly different outputs across machines (different CUDA versions, different cuDNN heuristics, different hardware FP behavior). The discipline is to log every version of every component and accept a tolerance window in your equality checks.

Testing is harder than asserting. You cannot write a test that says "the model should return the correct answer for query Q" because you do not know the correct answer for most Qs. The eval suite is the closest analog. The hardest part is curating the suite so that it tracks the thing you care about.

The model is a function of training data you may no longer have. GDPR right-to-be-forgotten requests imply that any training row may need to be deleted. Strict implementations require retraining without that row. The infrastructure cost is real.

The model can be wrong in ways that are not bugs. A traditional bug is a deviation from spec. An ML failure may be the model doing exactly what it was trained to do, on a query it was never trained on. The fix may be to gather more data, not to change the code.

Costs are non-linear. A 10% larger model is not 10% more expensive to serve; it may be 30% slower per token, occupy 10% more GPU memory, and trigger autoscaling that 2× the fleet. The cost model has to be a first-class deployment concern.

Huyen's "Real-time machine learning challenges" (24-founder-blogs/huyenchip-huyenchip-com-2022-01-02-real-time-machine-learning-challenges-and-solutions-htm) and "ML systems design" (24-founder-blogs/huyenchip-huyenchip-com-2020-10-27-ml-systems-design-stanford-html) are the two essential reads here.

Most of the above generalizes to LLM applications, with five modifications:

1. Prompts are code. Version them, lint them, test them. Treat the prompt registry like a model registry.
2. The "training" step is the prompt-engineering step. Iteration is faster (seconds vs hours), the iteration loop is governed by the eval suite, not by loss curves.
3. The model is usually rented. OpenAI/Anthropic/Google models update without you. Re-run the eval suite on every API-side notification, plus a monthly cadence regardless.
4. Cost is metered per token, not per request. Cost monitoring needs per-request token attribution.
5. The output is unstructured. Output schemas (JSON mode, structured outputs) are part of the system. Schema-validation failures are a primary alert category.

Microsoft's "GenAI application lifecycle" (10-microsoft-lessons/genai-14-the-generative-ai-application-lifecycle) maps the same five phases as classical MLOps onto the LLM-rented-model world. The translation is mechanical once you have done it once.

Library path (a prompt registry — small, but the right shape):

# prompts.py — version-controlled, tested
from dataclasses import dataclass

@dataclass(frozen=True)
class Prompt:
    name: str
    version: str
    template: str
    description: str

CUSTOMER_SUPPORT_V1 = Prompt(
    name="customer_support",
    version="v1.2.0",
    description="Triage incoming support tickets into one of {billing, technical, other}.",
    template=(
        "You are a triage agent. Read the ticket and return a JSON object with "
        "keys 'category' and 'confidence'. Categories: billing, technical, other.\n"
        "\nTicket:\n{ticket}\n"
    ),
)

def call(prompt: Prompt, **kwargs) -> str:
    return openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt.template.format(**kwargs)}],
    ).choices[0].message.content