Mechanistic Interpretability

Mechanistic interpretability is the project of reverse-engineering neural networks. Not "explain what the model does on average" (that is behavioral evaluation). Not "highlight which input tokens mattered" (that is feature attribution). The thing mech interp tries to produce is a program: a description of the computation the model performs, in terms of variables and operations that a human can audit and predict from.

The historical motivation has two strands. The vision-interp strand (Olah, Mordvintsev, the Distill circuits thread, 2015-2020) found that individual convolutional neurons in InceptionV1 detected specific visual concepts (curves, dog faces, car wheels), and that pairs of layers wired together implemented recognizable algorithms (a curve detector feeding into an oriented edge detector). The Distill articles "Zoom In: An Introduction to Circuits" and "Visualizing Weights" formalized this view: a circuit is a connected subgraph of the network's compute graph that implements a coherent function.

The transformer-interp strand picked this up around 2021. Elhage et al.'s "A Mathematical Framework for Transformer Circuits" introduced the residual-stream view: instead of treating the transformer as alternating attention/MLP blocks, treat it as a sequence of writes and reads to a high-dimensional vector (the residual stream) that every layer can both read from and add to. This reframe is the foundation of everything in this chapter.

The blunt fact that motivated all of this: looking at single neurons in a transformer's MLP does not work. Neurons are polysemantic: a single neuron's activations span many unrelated concepts. The toy-models-of-superposition paper (Elhage et al. 2022) explained why: a network can pack more features than it has neurons by representing each feature as a sparse combination of neurons, accepting interference, and using nonlinearities to denoise. Polysemanticity is the symptom; superposition is the cause. Sparse autoencoders are the current best response. We get to them in §10.

Recall the residual-stream picture introduced in Ch 15 §9: the activation x flowing through a transformer block is not transformed but added to. Every attention head and every MLP layer reads from x (after LayerNorm) and writes a delta back. The output logits are unembed(ln_final(x)), one final read.

Here, it is the technical foundation of every mech-interp tool in this chapter. The reframe matters because every component, every head, every neuron, every layer, has a direct linear contribution to the final logits. You can write:

Substituting and ignoring the LayerNorm for a moment (we will pay for that later), the logits are a sum of contributions: one from the token embedding, one from the positional embedding, one from each attention head, one from each MLP. Each contribution is a vector in residual-stream space, projected by $W_U$ to vocab space.

This decomposition is the whole point. If you want to know why the model said "John" instead of "Mary", you compute the contribution of each component to the logit("John") - logit("Mary") difference, and see who wrote what. That technique has a name: Direct Logit Attribution. We use it in §4.

The tool that makes this practical is TransformerLens (Nanda's library), which loads a GPT-2-style model into one consistent, interpretability-friendly architecture so that every internal activation has a stable name you can read or edit. You load a model with HookedTransformer.from_pretrained("gpt2-small"); you run it and grab all the internal activations at once with logits, cache = model.run_with_cache(tokens), which returns an ActivationCache; and you index that cache by name — cache["resid_post", l] (a shorthand for the full cache["blocks.{l}.hook_resid_post"]) pulls out the residual stream after layer l. The weight matrices are exposed under the same naming, e.g. model.W_U for the unembed and model.W_Q for the per-head query projection. We use this library from here on; the chapter's labs deliberately rebuild the same operations by hand so you never depend on a black box you cannot reconstruct.

# Cache the residual stream at each layer's output
import torch
from transformer_lens import HookedTransformer

model = HookedTransformer.from_pretrained("gpt2-small")
tokens = model.to_tokens("The Eiffel Tower is in")
logits, cache = model.run_with_cache(tokens)

# Residual stream after each layer
for l in range(model.cfg.n_layers):
    resid = cache["resid_post", l]   # (batch, seq, d_model)
    print(l, resid.shape, resid.norm().item())

The residual stream view is what TransformerLens's hookpoints are organized around. resid_pre, resid_mid, resid_post, attn_out, mlp_out are all named views into this single vector.

TransformerLens exposes every internal tensor as a named hook. The list is finite and worth memorizing. For each layer l:

• hook_embed: token embedding output, shape (batch, seq, d_model). Once, at the bottom.
• hook_pos_embed: positional embedding output. Once.
• blocks.{l}.hook_resid_pre: residual stream just before this layer's attention block.
• blocks.{l}.attn.hook_q, hook_k, hook_v: post-projection Q, K, V. Shape (batch, seq, n_heads, head_dim).
• blocks.{l}.attn.hook_attn_scores: pre-softmax attention pattern.
• blocks.{l}.attn.hook_pattern: post-softmax attention pattern.
• blocks.{l}.attn.hook_z: per-head attention output. Shape (batch, seq, n_heads, head_dim). This is the per-head pre-output-projection vector.
• blocks.{l}.hook_attn_out: post-output-projection attention output. The thing added to the residual stream.
• blocks.{l}.hook_resid_mid: residual stream after attention, before MLP.
• blocks.{l}.mlp.hook_pre, hook_post: MLP pre-activation and post-activation neuron values.
• blocks.{l}.hook_mlp_out: MLP output. The thing added to the residual stream.
• blocks.{l}.hook_resid_post: residual stream after this whole layer.
• ln_final.hook_normalized: input to the unembed, after final LayerNorm.

Two operations on hooks. Read: register a hook that copies activations into a cache. Intervene: register a hook that modifies activations on the way through. Reading is what run_with_cache does. Intervention is what enables activation patching. You intervene by calling model.run_with_hooks(tokens, fwd_hooks=[(name, fn)]), where each fn(activation, hook) receives the activation tensor and returns the (possibly edited) replacement.

import torch
from transformer_lens.hook_points import HookPoint

def zero_head(activation: torch.Tensor, hook: HookPoint, head_idx: int) -> torch.Tensor:
    """Zero out one head's z output. activation shape: (batch, seq, n_heads, head_dim)."""
    activation[:, :, head_idx, :] = 0.0
    return activation

# Run the model with head 5 of layer 3 ablated
hook_name = "blocks.3.attn.hook_z"
logits = model.run_with_hooks(
    tokens,
    fwd_hooks=[(hook_name, lambda act, hook: zero_head(act, hook, head_idx=5))],
)

The hook-based intervention pattern is the foundation of every causal experiment that follows. Every paper you read in §15 is some clever choice of which hook to read from and what to write into another hook.

DLA is the trick of computing each component's projection onto the logit difference between two specific tokens.

The recipe. Pick a prompt and two candidate next tokens, $t_{\text{correct}}$ and $t_{\text{incorrect}}$. The "logit difference" is $\text{logits}[t_{\text{correct}}] - \text{logits}[t_{\text{incorrect}}]$. By the linearity of the unembed, this equals the projection of the final-layer residual onto the logit-difference direction: $u = W_U[t_{\text{correct}}] - W_U[t_{\text{incorrect}}] \in \mathbb{R}^{d_{\text{model}}}$.

Now decompose the final residual. Every component (embedding, each layer's attn_out, each layer's mlp_out) contributes one vector $c_k$. The total is $x_{\text{final}} = \sum_k c_k$. So the logit difference is $\sum_k \langle c_k, u \rangle$. Each component has a single scalar contribution.

Caveat 1: LayerNorm. The real model has $x_{\text{final}}$ passed through LayerNorm before the unembed. LN is approximately linear when you fold its scaling factor into $u$, but only approximately. For careful DLA, apply the logit lens — the technique of taking a residual stream from partway up the network, passing it through the final LayerNorm and unembed, and reading off the resulting token distribution as if that layer were the last (Nostalgebraist 2020). The fix here is the same arithmetic: multiply each component by the LN scale at the position of interest before projecting. TransformerLens has a model.unembed and a helper apply_ln_to_stack that handles this.

Caveat 2: DLA is a direct attribution. It does not tell you about indirect effects, where a component contributes a vector that subsequent components use, or compose with, to produce a logit. For indirect effects you need activation patching (§5) or path patching (§6).

def direct_logit_attribution(cache, model, correct_token: int, incorrect_token: int, pos: int):
    """Returns per-component projections onto the logit-diff direction at `pos`."""
    W_U = model.W_U          # (d_model, vocab)
    u = W_U[:, correct_token] - W_U[:, incorrect_token]   # (d_model,)
    # accumulate per-layer attn and mlp contributions
    contributions = {}
    for l in range(model.cfg.n_layers):
        attn_out = cache["attn_out", l][0, pos]   # (d_model,) for batch=0
        mlp_out = cache["mlp_out", l][0, pos]
        contributions[f"L{l}.attn"] = (attn_out @ u).item()
        contributions[f"L{l}.mlp"] = (mlp_out @ u).item()
    return contributions

For an IOI-style prompt ("When John and Mary went to the store, John gave a drink to"), correct = " Mary", incorrect = " John". Run DLA on this and you will see Layer 9 head 9 (a "name-mover" head) contributing strongly positively to Mary's logit, while Layer 10 head 7 and Layer 11 head 10 (the "negative name-mover" heads) contribute negatively — they push the logit toward John. The head names — name-mover, negative name-mover, S-inhibition, and the rest — are the vocabulary of the IOI circuit and are defined in §11; for now read them as labels for specific heads this example will name.

Direct logit attribution tells you who correlates with the answer. To tell who causes it, you need to intervene. Activation patching is the simplest intervention: replace one activation on the way through with the activation from a different run, and see how the output changes.

The setup. Pick two prompts: a clean prompt the model gets right, and a corrupted prompt where you have changed one thing (often a token or two) such that the answer flips. Cache activations on both runs. Now, for each candidate site $h$ (a specific hookpoint and position), run the model on the corrupted prompt but patch in the clean activation at site $h$. Measure: does the output recover toward the clean answer?

If patching site $h$ recovers a lot of the clean behavior, then site $h$ carries information that matters for the task. If patching site $h$ does nothing, then $h$ is irrelevant. Sweep across all sites; the heatmap is your causal map.

The metric. Define $P(\text{site}) = \frac{\text{logit-diff}_\text{patched at site} - \text{logit-diff}_\text{corrupt}}{\text{logit-diff}_\text{clean} - \text{logit-diff}_\text{corrupt}}$. $P = 0$ means no recovery (site irrelevant). $P = 1$ means full recovery (site carries all the relevant info). Values above 1 happen (overpatching) and are themselves informative.

The choice of site matters. Patching resid_pre at every layer-position tells you when in the network the information lives. Patching hook_z at every (layer, head, position) tells you which heads carry it. Patching mlp_out tells you whether MLPs are involved.

import torch
from transformer_lens import HookedTransformer

def patch_resid_pre(corrupt_act, hook, clean_cache, layer, pos):
    corrupt_act[:, pos, :] = clean_cache[f"blocks.{layer}.hook_resid_pre"][:, pos, :]
    return corrupt_act

def activation_patching_sweep(model, clean_tokens, corrupt_tokens, logit_diff_metric):
    """Returns a (n_layers, seq_len) heatmap of patching effect."""
    _, clean_cache = model.run_with_cache(clean_tokens)
    n_layers, seq_len = model.cfg.n_layers, clean_tokens.shape[-1]
    out = torch.zeros(n_layers, seq_len)
    for l in range(n_layers):
        for p in range(seq_len):
            hook = (f"blocks.{l}.hook_resid_pre",
                    lambda a, h, layer=l, pos=p: patch_resid_pre(a, h, clean_cache, layer, pos))
            patched_logits = model.run_with_hooks(corrupt_tokens, fwd_hooks=[hook])
            out[l, p] = logit_diff_metric(patched_logits)
    return out

Caveat: activation patching is not a clean causal test. The model's later layers are conditioned on what the earlier activations would have been, so patching introduces an off-manifold input. In practice, the result is still strongly informative; in theory, you should worry about the patched activation being out-of-distribution for downstream layers. Path patching (§6) addresses this by also patching the downstream "freeze" path.

Path patching (Goldowsky-Dill et al. 2023) is activation patching's careful sibling. Instead of patching one site and letting all downstream sites recompute, path patching freezes the downstream activations to their corrupt-run values, except along the one path you want to attribute.

The intuition. In a normal activation patch, the patch's effect propagates through every downstream layer. Some of that effect is direct (the patched activation feeds the unembed). Some is indirect (the patched activation changes layer 5's output, which changes layer 6's input, etc.). Path patching lets you isolate one of those paths.

The implementation is fiddly. You do three forward passes:

1. Clean run, cache all activations.
2. Corrupt run, cache all activations.
3. Patched run: at the source site, use the clean activation. At every other site downstream of the source, use the corrupt activation, except along the path you want to keep "live".

The "path" is specified as a sequence of components: e.g., "L5 H3's output, through L7 H8's query input, through the unembed". You patch the L5 H3 output to clean values, force the rest of L5-L7 to corrupt, but allow L7 H8 to read clean L5 H3 output through its query projection.

This is the technique that produced the IOI circuit diagram (Wang et al. 2023). Without path patching, you cannot tell apart "this head writes to the logits directly" from "this head writes something that another head reads and uses to write to the logits".

In practice, for first-pass analysis, activation patching is usually enough. Reach for path patching when you have a hypothesis like "head A's output is read by head B, not by the unembed directly" and you want to test it.

A linear probe is the simplest "what does this layer know about X?" experiment. Train a linear classifier on top of a frozen layer's activations to predict some target (e.g., "is this token a noun?"). If the probe's accuracy is high, the layer's activations linearly encode the target.

The catch is that linear probes are correlational. Probe accuracy tells you the information is decodable. It does not tell you the model uses it. The probe could be picking up an epiphenomenon the model represents but never reads.

Two probe variants that mitigate this. Trained probes (the default): fit a linear classifier with logistic regression on activations. Causal probes: instead of fitting a classifier, intervene on the probed direction (add a scaled version to the residual stream) and see if the behavior changes. The OthelloGPT work (Li et al. 2022; Nanda 2023) is the canonical example: a non-linear probe found that the model "knew" the board state, then a linear probe found a probably-linear representation, then a causal intervention confirmed the model uses it.

from sklearn.linear_model import LogisticRegression
import numpy as np

def probe_layer(cache, layer: int, labels: np.ndarray) -> float:
    """Train a linear probe on layer's residual stream. Returns held-out accuracy."""
    X = cache["resid_post", layer].reshape(-1, cache["resid_post", layer].shape[-1]).cpu().numpy()
    n = X.shape[0]
    idx = np.random.permutation(n)
    split = int(0.8 * n)
    train_idx, test_idx = idx[:split], idx[split:]
    clf = LogisticRegression(max_iter=1000)
    clf.fit(X[train_idx], labels[train_idx])
    return clf.score(X[test_idx], labels[test_idx])

When to reach for probes: as a fast first-pass diagnostic. When not to trust them: as evidence the model uses what they detect, without a confirming causal experiment.

If a linear probe finds a direction in residual stream that correlates with a behavior, you can sometimes just add that direction to the residual stream and steer the model. This is the steering vector trick, popularized by Turner et al. 2023 and Subramani et al. 2022 under "activation steering".

The recipe. Find or compute a direction $v \in \mathbb{R}^{d_{\text{model}}}$ associated with a behavior (positive sentiment, refusal, French language, etc.). Pick a layer and a coefficient $\alpha$. At inference time, after the chosen layer, add $\alpha v$ to the residual stream at every position. Sample as normal.

Empirical fact: this works some of the time. The "love minus hate" direction extracted from contrastive activations (Turner) steers GPT-2 to write affectionate text. The refusal direction (Arditi et al. 2024) steers Llama-2-chat to refuse or comply on harmful prompts. The "French" direction extracted from translation contrasts steers code-switching.

When it works and when it does not. It works when the underlying concept is approximately linear in the residual stream (which, per the linear representation hypothesis, many high-level concepts are). It fails when the concept is implemented by multiple directions, when the model has anti-feature redundancy, or when the steering coefficient pushes you off the data manifold and the model outputs garbage.

Function vectors (Todd et al. 2024) are the same trick applied to in-context-learning tasks: extract a vector that summarizes "the function this in-context demo is teaching", inject it on a zero-shot prompt, watch the model behave as if it had seen the demo.

def add_steering(activation, hook, vector, alpha):
    return activation + alpha * vector

def steer(model, prompt, steering_vec, layer, alpha):
    hook = (f"blocks.{layer}.hook_resid_post",
            lambda a, h: add_steering(a, h, steering_vec, alpha))
    return model.generate(prompt, fwd_hooks=[hook], max_new_tokens=50)

In the 2-layer attention-only transformer trained on language, there exists a specific pair of heads that together implement in-context learning of repeated patterns. This is the canonical mech-interp result, and it is the first circuit anyone reverse-engineered end-to-end. Olsson et al. 2022 ("In-context Learning and Induction Heads") is the paper.

The phenomenon. Feed the model a sequence like ... A B... A B... A. The model predicts B at the last position with high probability, much higher than chance. How does it do it without having seen A B during training as that exact pair? By looking back, finding the previous A, and copying what came after it.

The algorithm has two heads:

1. The previous-token head (Layer 0). For each position $t$, this head's attention pattern is essentially shifted by one: position $t$ attends to position $t-1$. The output written to the residual stream at position $t$ is a function of token at $t-1$. After Layer 0, the residual stream at each position contains "what token came before me".

2. The induction head (Layer 1). At the current position (last A), the head computes a query that is "find me positions whose 'previous-token info' equals the current token (A)". So it attends to the position right after the previous A, which is the position containing B. It then copies B to the output.

Together, two heads in different layers compose via the residual stream to implement copy-by-pattern. The composition happens through specific subspaces of the residual stream: head 1's $W_K$ reads from head 0's $W_O$, and head 1's $W_O$ writes to a direction the unembed reads. The math here is the OV circuit and QK circuit decomposition, which we cover in §16.

The bigger claim of the Olsson paper: induction heads are how transformers do in-context learning more generally. They emerge in training at the same phase transition where in-context learning ability jumps. They generalize from "copy the literal token" to "copy a semantically related token" in larger models.

Why this matters. (1) It is the existence proof. A real, useful behavior in a real model is implemented by a specific identifiable circuit. (2) It is the from-scratch lab in this chapter: you train a 2-layer transformer on a corpus, you find the head, you intervene on it, you confirm it does what it does. (3) It scaffolds all later circuit work: the methodology used here (look for a behavior, isolate the heads via ablation, decompose the circuit via QK and OV analysis) is the template.

# Detect induction heads by their characteristic attention pattern
import torch

def induction_score(attn_pattern: torch.Tensor) -> float:
    """attn_pattern: (n_heads, seq_len, seq_len). Sequence is rand tokens repeated 2x.
    An induction head attends from position t to position (t - seq_len/2 + 1)."""
    seq_len = attn_pattern.shape[-1]
    half = seq_len // 2
    # The induction offset: position t (in second half) attends to (t - half + 1)
    offsets = torch.arange(half, seq_len) - half + 1
    scores = attn_pattern[:, torch.arange(half, seq_len), offsets]
    return scores.mean(dim=-1)   # one score per head

If polysemanticity comes from superposition, an obvious fix is to find a basis where features are sparse, even if it means using more basis vectors than the residual stream has dimensions. That is the SAE.

The setup. Take a layer's activations $x \in \mathbb{R}^{d_{\text{model}}}$ from many forward passes. Train an autoencoder:

with overcomplete $f \in \mathbb{R}^{d_{\text{SAE}}}$, where $d_{\text{SAE}} = 8 d_{\text{model}}$ to $64 d_{\text{model}}$. The loss is:

The L1 penalty enforces sparsity. The reconstruction loss enforces faithfulness. With the right $\lambda$, most of $f(x)$'s entries are exactly zero on any given input, and the few non-zero entries correspond to interpretable features.

Empirical fact: this works. On Claude 3 Sonnet (Anthropic 2024 "Scaling Monosemanticity"), SAEs trained at the residual stream level recover features for "the Golden Gate Bridge", "code that handles edge cases", "deception in role-play", "sycophancy", and tens of millions more. Many of the features are monosemantic in the sense that activating only one of them produces a coherent shift in the model's output.

Variants and gotchas:

• TopK SAEs (Gao et al. 2024) replace the L1 with a hard top-k constraint: only keep the $k$ largest pre-activations. Eliminates the L1-coefficient tuning headache. Top-k is the default for new work.
• JumpReLU SAEs (Rajamanoharan et al. 2024) use a threshold-and-pass-through nonlinearity, which gives sharper feature on/off boundaries.
• Gated SAEs (Rajamanoharan et al. 2024) split the encoder into a gating path and a magnitude path; better reconstruction at the same sparsity.
• Dead neurons: a fraction of SAE features never activate after training. Resampling and the "auxiliary loss" trick fix it partially.
• Feature splitting: when you train a bigger SAE on the same data, single features sometimes split into multiple finer features. There is no "true" feature count; the SAE basis is a function of the SAE's width.

import torch
import torch.nn as nn

class SAE(nn.Module):
    def __init__(self, d_model: int, d_sae: int):
        super().__init__()
        self.W_enc = nn.Parameter(torch.randn(d_model, d_sae) / d_model**0.5)
        self.b_enc = nn.Parameter(torch.zeros(d_sae))
        self.W_dec = nn.Parameter(torch.randn(d_sae, d_model) / d_sae**0.5)
        self.b_dec = nn.Parameter(torch.zeros(d_model))
    
    def forward(self, x):
        f = torch.relu((x - self.b_dec) @ self.W_enc + self.b_enc)
        x_hat = f @ self.W_dec + self.b_dec
        return f, x_hat

def sae_loss(x, x_hat, f, l1_coeff=1e-3):
    return ((x - x_hat) ** 2).mean() + l1_coeff * f.abs().sum(dim=-1).mean()

The Indirect Object Identification circuit (Wang et al. 2023) is the second canonical circuit, and the first one in a real (not toy) model. GPT-2 small, on the task: given "When John and Mary went to the store, John gave a drink to", predict the indirect object (Mary).

The decomposed algorithm:

1. Duplicate Token Heads (Layers 0-3, e.g., 0.1, 3.0): detect that "John" appears twice in the prompt. Write a "this is a duplicate" signal to the residual stream at the second John's position.
2. S-Inhibition Heads (Layers 7-8, e.g., 7.3, 8.6): read the duplicate signal. Write a negative signal that suppresses attention to John in later layers.
3. Name Mover Heads (Layers 9-10, e.g., 9.6, 9.9, 10.0): attend to the indirect-object position (Mary's position), copy Mary's name to the output position.
4. Backup Name Movers (Layers 10-11): redundant copies of name movers. When you ablate the primary name movers, these step up. (This is a fascinating finding: the circuit has built-in redundancy. There is also a phenomenon called "negative name movers" that suppress the correct answer; we still do not fully understand why.)

The full reverse-engineering required dozens of activation-patching and path-patching experiments. The diagram you usually see (the one with arrows from each head class to the next) is the cleaned-up summary; the full paper has roughly 30 components in the loose definition of the circuit.

What IOI taught the field:

• Circuits in real models are not clean. They are messy, with redundancy, with components that fire in confusing ways, with "backup" copies of capabilities.
• Reverse-engineering a single behavior in GPT-2 small (124M params) takes months. Scaling this approach to GPT-4 is not a matter of more compute; it requires fundamentally different methods. That is why the field is now investing in attribution graphs (§12) and SAEs (§10).
• The circuit is task-specific. The same heads do other things on other tasks. Polysemanticity at the head level is real. The IOI circuit is "what these heads do for IOI prompts".

In 2024-2026, the SAE-and-circuit research moved from "find features in one layer" to "trace the computation across layers". Three techniques:

• Transcoders (Templeton et al. 2024; Ge et al. 2024). A transcoder is an SAE whose input is one layer's pre-MLP activation and whose output reconstructs that layer's post-MLP activation. It tries to replace the dense MLP with a sparse, interpretable approximation. Once trained, you can compute "which transcoder features in layer $l$ contribute to which transcoder features in layer $l+1$", giving a sparse computation graph.

• Crosscoders (Anthropic 2024). A crosscoder is an SAE trained jointly on activations from multiple layers (or multiple models, like a base and chat variant of the same architecture). It identifies features that are shared across layers/models versus features unique to one. The model-diffing application (compare what changed between base and chat) is striking: many features are exactly aligned, a small subset shift in interesting ways, and that subset disproportionately includes the "refusal", "RLHF compliance", and "persona" features.

• Attribution graphs (Anthropic 2025, "On the Biology of a Large Language Model"). Combine transcoders with attribution analysis to produce a graph of feature-to-feature influences for a specific prompt. Each node is a feature firing at a specific (layer, position). Each edge is the (linear, attributable) influence of one feature on another. The resulting graph is the closest thing the field has to "this is how the model thought about this prompt".

The biology paper is the closest thing to a worked example in modern interp. It walks through 8 cases (poetry, multilingual translation, reasoning, hallucination, refusal, math, etc.) and shows the attribution graph for each. The graphs are large and weird; they often include features that are not obviously related to the task and that the model nonetheless uses. The paper does not claim full understanding; it claims a concrete, intervenable hypothesis of what the model is doing.

Caveats. Attribution graphs are still expensive (one transcoder training run per layer, one attribution pass per prompt). The features are stable across small perturbations but not across changes in prompt format. The "biology" framing acknowledges this: we are reading a complicated system, not deriving it from first principles.

The "toy models" thread (Elhage et al. 2022, "Toy Models of Superposition") and the grokking thread (Power et al. 2022; Nanda et al. 2023) are the two places mech interp has produced complete understanding of a model's algorithm.

Toy models of superposition: train a tiny model to reconstruct sparse high-dimensional inputs from a low-dimensional bottleneck. Sweep sparsity. Observe phase transitions: at low sparsity, the model uses one bottleneck neuron per input feature. At high sparsity, it packs multiple features per neuron in superposition, arranging them at specific geometric angles (regular polygons, pentagons, etc.). The geometry can be derived analytically. This is the cleanest existence proof of superposition.

Grokking: train a small transformer on modular addition ($a + b \mod 113$). The model first overfits the training set (memorization), then, much later, "groks" the algorithm and generalizes. By analyzing the weights at the grokked state, Nanda et al. showed the model is computing a Fourier transform of the inputs: it represents $a$ and $b$ as combinations of $\sin(k \pi a / 113)$ for various $k$, multiplies them in the MLP using trig identities, and reads off the result. The complete algorithm fits in a paragraph and can be unit-tested.

Why these matter. They are the only examples where the mech-interp community can write down, end-to-end, the exact computation a model performs. They are smaller than any practical model, but they are real models doing nontrivial computations. They calibrate intuition: this is what a "fully reverse-engineered model" looks like.

The practical entry point for SAE work is Neuronpedia (neuronpedia.org). It hosts SAE features from Pythia, GPT-2, Gemma 2, Llama 3, and others. For each feature, you get the top-activating examples (the dataset texts that fire the feature most strongly), neighboring features in the SAE basis, the feature's direction, and an LLM-generated description.

How to use it for research. Pick a model and an SAE release. Search for a concept ("refusal", "Python code", "the word 'however'") and inspect the top features. Click into a feature, scroll the top-activating examples, decide if the LLM-generated description matches what you see. The interpretive eye is the limiting reagent.

Caveat: Neuronpedia is a tool, not a truth. The auto-descriptions are LLM outputs and inherit LLM-style errors (confident, plausible, sometimes wrong). The "interpretability" of a feature is the researcher's call. The current best practice is to (a) inspect top-activating examples, (b) hypothesize a description, (c) construct a held-out test set, (d) check if the feature fires on the held-out positives and not the negatives. Few people do all four.

For a project: pick an open-weights chat model that has a public SAE release. Find five features that, in your interpretation, encode something safety-relevant (refusal, sycophancy, deception, persona, instruction-following). Verify each with a held-out test. Write up. That is a credible mech-interp project portfolio piece.

The discipline is honest about its limits. As of 2026, mech interp can:

• Reverse-engineer small (≤2-layer) attention-only transformers end-to-end. (Toy models, induction heads.)
• Reverse-engineer single behaviors in mid-sized models (124M-1B). (IOI.)
• Find interpretable features at scale via SAEs in frontier models. (Scaling Monosemanticity, Gemma Scope.)
• Trace influence between features within a single prompt via attribution graphs. (Biology paper.)
• Steer model behavior along simple linear directions, sometimes. (Refusal, persona.)

It cannot:

• Reverse-engineer a frontier model end-to-end. The combinatorial scale is wrong.
• Reliably detect deception. There are papers that find "deception features", but their predictive validity on held-out adversarial settings is shaky.
• Predict generalization. Knowing the algorithm a model uses on the training distribution does not tell you what it will do off-distribution.
• Distinguish "model represents X" from "model uses X". Probes find the first; causal experiments confirm the second; many published claims conflate them.
• Find the "true" feature basis. SAE features depend on SAE width, training data, and seed. The same model trained with different SAE hyperparameters yields different feature sets that all look reasonable.

The optimistic case: mech interp is the field where, ten years from now, "did we audit the model" is a question with a real answer rather than a wave at evaluations. The pessimistic case: feature interpretability scales sublinearly with model size and the methods top out below frontier scale. Both can be true.

This is the math that makes the induction-head analysis work, and the math that the rest of the chapter has been implicitly using. Take a single attention head with weights $W_Q, W_K, W_V, W_O$. Given residual stream $x$, the head computes:

Define two derived matrices:

• QK circuit: $W_Q W_K^T \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$. This determines which positions attend to which. It is the "decide which thing to copy" matrix.
• OV circuit: $W_V W_O \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$. This determines what gets copied (transformed) when a position is attended to. It is the "what to write" matrix.

You can study a head by studying these two matrices independently. For an induction head, the OV circuit approximately implements the identity (it copies whatever it reads), and the QK circuit implements "find positions whose representation matches my current token's previous-position info".

For a previous-token head, the QK circuit is essentially a shift operator (each position attends to the prior position), and the OV circuit is a copy of the value vector.

The decomposition generalizes. For any head, plotting $W_Q W_K^T$ against $W_V W_O$ in some interpretable basis can reveal what the head does. TransformerLens has a FactoredMatrix class that stores these as factored products, allowing efficient SVD and inspection.

from transformer_lens import FactoredMatrix

W_QK = FactoredMatrix(model.W_Q[0, 0], model.W_K[0, 0].T)   # (d_model, d_model)
W_OV = FactoredMatrix(model.W_V[0, 0], model.W_O[0, 0])
print(W_QK.svd().S[:5])   # top singular values reveal the rank of the head's selection

The technique that taught the field what circuits feel like came from the vision side. Olah and Schubert's "Feature Visualization" (Distill 2017) showed that you can ask "what input maximally activates this neuron" by gradient ascent in pixel space, and the result is interpretable: curve detectors, dog-face detectors, car-wheel detectors. "Activation Atlas" (2019) extended this to whole layers: project the activations of millions of images into 2D and tile the resulting map with feature-vis renderings.

Why this still matters for transformer interp:

• Feature visualization is the original argument that features are real, that neurons in trained networks encode specific, namable concepts. Without that prior, the SAE story in transformers reads as wishful.
• Activation atlases are the visual template for Neuronpedia-style feature dashboards.
• The polysemantic-neuron phenomenon was first identified in vision (Olah et al. 2020 "Zoom In") and carried over to language.
• The "circuit" terminology (curve detector + edge detector → contour detector) came from vision and was generalized to transformer-block compositions.

If you want intuition for what a "feature" is, the vision-interp Distill articles are the best starting point. They predate everything in this chapter and they will make the abstractions feel concrete.