Multimodal Transformers

In 2020, Dosovitskiy et al. shipped "An Image Is Worth 16x16 Words" and the title is the paper. Take an image of shape (3, 224, 224). Cut it into 14×14 non-overlapping patches of size (3, 16, 16). Flatten each patch to a 768-dimensional vector. You now have 196 tokens of d_model=768. Prepend a learned [CLS] token — an extra embedding with no pixels behind it, whose only job is to accumulate a summary of the whole image as it flows up the stack, so you classify off its final vector (the trick BERT, the bidirectional text model, used for sentence classification). Add a learned positional embedding to each of the 197 positions. Feed this sequence into a stack of 12 transformer encoder blocks — the same another chapter block you built, minus the causal mask, so every token attends to every other token instead of only to its past — identical to the ones in BERT. Read the final [CLS] token's embedding, project it to your number of classes, train with cross-entropy on ImageNet. That is ViT-B/16.

Two things are surprising about this once you accept it works. First, the "patch embedding" can be implemented as a single 2D convolution with stride equal to patch size: nn.Conv2d(3, 768, kernel_size=16, stride=16). The Conv2d is mathematically equivalent to "flatten each patch and apply a linear projection". This is the only conv in the whole network. Second, ViT does worse than ResNet on ImageNet at small data scales. It needs JFT-300M (300 million images) to overtake ResNet. With less data, the inductive bias of convolutions (locality, translation equivariance) wins. With enough data, the transformer just learns the bias from scratch. This is the lesson that generalizes across modalities: transformers replace inductive bias with data.

processor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224')
model = ViTModel.from_pretrained('google/vit-base-patch16-224')
inputs = processor(images=img, return_tensors='pt')
hidden = model(**inputs).last_hidden_state  # (1, 197, 768): CLS + 196 patch tokens
# the patchifier underneath is literally:
# nn.Conv2d(3, 768, kernel_size=16, stride=16) then flatten(2).transpose(1, 2)

self.patch_embed = nn.Conv2d(1, d_model, kernel_size=patch_size, stride=patch_size)
self.cls_token = nn.Parameter(torch.zeros(1, 1, d_model))
self.pos_embed = nn.Parameter(torch.zeros(1, 17, d_model))   # 16 patches + 1 CLS
...
patches = self.patch_embed(img)              # (B, d_model, 4, 4)
patches = patches.flatten(2).transpose(1, 2) # (B, 16, d_model)
cls = self.cls_token.expand(B, 1, -1)
x = torch.cat([cls, patches], dim=1)         # (B, 17, d_model)
x = x + self.pos_embed                        # learned 1D positions
for blk in self.blocks:
    x = blk(x)
x = self.ln(x[:, 0])                          # read the CLS token
x = self.proj(x)                              # (B, embed_dim)

The Block class is the same one. ViT is decoder-shaped without the causal mask. Note: no triangular mask here. Every patch attends to every other patch. Vision is non-autoregressive.

Patches have a 2D grid structure. Position (3, 5) is one row above (4, 5) and one column right of (3, 4). The original ViT throws this information away: it uses a 1D learned position embedding, one per patch index in row-major order. The model is supposed to figure out the 2D structure from data.

It does. But several follow-up papers show you can do better.

Sinusoidal 2D. Pre-compute a fixed positional encoding that is the concatenation of sin/cos features for the row index and sin/cos features for the column index. SimpleViT (the cleanup of original ViT by some of the same authors) uses this and trains faster.

RoPE on 2D. Apply rotary embeddings independently along the row and column axes. Common in DiT (the diffusion transformer that powers Sora's predecessors) and in NaViT.

No position encoding. Some recent ViT variants drop the position encoding entirely, on the basis that the attention pattern itself can encode locality if you let it. This works at scale but trains slower.

The empirical finding from the ViT-22B paper and others: positional encoding matters less than you would expect. The model recovers locality from data, especially with enough patches. But for small data, 2D positional information helps.

ViT trained on ImageNet (1.3M labels) is fine. ViT trained on JFT-300M (300M labels) is better. ViT trained on JFT-3B is better still. But labels do not scale to billions. The interesting question is whether you can train a competitive ViT with zero labels.

DINO (Caron et al. 2021) showed you can. The recipe:

1. Take two random crops of the same image, one large (the global view), one small (the local view).
2. Pass each through a teacher network and a student network. Both are ViT.
3. The teacher's weights are an exponential moving average of the student's. No gradient updates flow to the teacher.
4. The student's outputs (after a small MLP head and softmax) should match the teacher's outputs for the same image. Loss: cross-entropy between the two distributions.
5. To prevent collapse (all images get the same output), the teacher's outputs are centered by subtracting a running mean and sharpened with a low softmax temperature. This combination is the part that nobody understood properly for two years.

The surprise: after training DINO, the [CLS] token's self-attention map on a held-out image segments objects. Foreground vs background. Cars vs people. No segmentation labels were used. The attention found object boundaries because that is what makes the two-crop matching work.

DINOv2 (Oquab et al. 2023) scales this up with bigger ViTs and a curated 142M-image dataset, and the embeddings transfer to dense prediction tasks (depth, segmentation) competitively with supervised models. This is the part of vision-transformer research that quietly replaced ImageNet pretraining for most downstream tasks.

Library path:

import torch
# Load a pretrained DINOv2 backbone — no head, just features
backbone = torch.hub.load("facebookresearch/dinov2", "dinov2_vitb14")
backbone.eval()

# Embed an image
img = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    features = backbone(img)   # (1, 768) — global feature for the image

You can use those features directly in a kNN classifier and get ~80% on ImageNet without ever training a linear layer. The kNN-on-DINO-features baseline is what every new self-supervised method now compares against.

Before ViT, "transformer for vision" mostly meant attaching a transformer to a CNN feature map and doing classification. DETR (Carion et al. 2020) was the paper that made a transformer the whole detector and broke the dependence on hand-tuned NMS and anchor boxes — the two staples of the older CNN detector stack: anchor boxes are a fixed grid of candidate box shapes the model scores and refines, and NMS (non-maximum suppression) is the post-processing pass that, when several boxes fire on one object, keeps the highest-confidence one and drops the rest by overlap.

The architecture: a ResNet backbone extracts a feature map of shape (C, H/32, W/32). Flatten to a sequence of length $H \cdot W / 1024$. Feed into a transformer encoder. The decoder takes $N$ learned "object queries" (typically $N=100$) and cross-attends to the encoded features. Each query outputs one (bbox, class) prediction. Loss: bipartite matching between predictions and ground-truth boxes, then per-match L1 and classification losses.

The trick is the bipartite matching. With $N=100$ queries and (say) 5 ground-truth objects, you solve a $100 \times 5$ assignment problem at training time using the Hungarian algorithm. Each ground-truth gets one query. The other 95 queries are trained to predict "no object". This eliminates NMS because the model is now trained to directly predict the set, not a redundant set of candidates.

DETR trains slowly (500 epochs to convergence on COCO). Deformable DETR (Zhu et al. 2021) fixed this with sparse attention on a sparse set of sampling points, reducing it to ~50 epochs. The lineage continues through DETR-style detectors (DINO, RT-DETR) which are now competitive with YOLOv8 and similar at real-time inference.

CLIP is the most influential multimodal architecture of the past five years. Built by Radford et al. at OpenAI in 2021, the recipe is comically simple:

1. Scrape 400 million (image, caption) pairs from the web.
2. Train an image encoder (ViT or ResNet variant) and a text encoder (small transformer).
3. For a batch of $N$ pairs, compute image embeddings $\mathbf{I} \in \mathbb{R}^{N \times d}$ and text embeddings $\mathbf{T} \in \mathbb{R}^{N \times d}$. Both L2-normalized.
4. Compute the similarity matrix $S = \mathbf{I} \mathbf{T}^T / \tau$, where $\tau$ is a learned temperature.
5. Loss: symmetric softmax cross-entropy. The image-to-text loss is cross-entropy where the correct label for row $i$ is column $i$. The text-to-image loss is cross-entropy on the transposed matrix.

That is the entire training objective. There is no caption generation, no masked language modeling, no auxiliary loss. Two encoders, a dot product, a symmetric cross-entropy. Train for a long time on a lot of data.

What you get: an embedding space where "a photo of a cat" and an image of a cat land at nearby points. The downstream consequence is that you can do zero-shot classification. For ImageNet, encode the prompt "a photo of a {class}" for each of the 1000 classes, encode your test image, find the closest class embedding by cosine similarity (the dot product after L2-normalizing both vectors, so it measures direction agreement and ignores magnitude). CLIP scores ~76% top-1 on ImageNet this way, comparable to a fully supervised ResNet-50.

loss_fn = open_clip.loss.ClipLoss()
# image_features, text_features: (N, d) L2-normalized; logit_scale = log_temp.exp()
loss = loss_fn(image_features, text_features, logit_scale)
# equivalently, the whole thing is just:
# logits = logit_scale * image_features @ text_features.T
# labels = torch.arange(N)
# loss = 0.5 * (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels))

img_feat = self.image_encoder(images)        # (B, d), L2-normalized
txt_feat = self.text_encoder(texts)          # (B, d), L2-normalized
logits = self.log_temperature.exp() * img_feat @ txt_feat.T   # (B, B)
B = logits.shape[0]
labels = torch.arange(B, device=logits.device)
loss = 0.5 * (F.cross_entropy(logits, labels)
              + F.cross_entropy(logits.T, labels))
return loss, logits

# log_temperature init = 2.65  (= log(1/0.07)), clamped to [0, 4.6] each step
# in train_step: model.log_temperature.clamp_(0.0, 4.6)

That is CLIP's training loss in 8 lines. The diagonal of the similarity matrix is the "correct match" for each row and each column, and cross-entropy says "the diagonal should be highest". The temperature $\tau = 1 / \text{logit\_scale}$ is initialized at 0.07 and learned. Models with sharper similarities (high logit_scale) generalize better, up to a point, after which they collapse.

The CLIP loss is a specific instance of InfoNCE (Information Noise Contrastive Estimation), the workhorse of self-supervised learning. The InfoNCE objective for one positive pair $(x, x^+)$ and $N-1$ negatives $\{x^-_i\}$ is:

This is precisely the softmax cross-entropy you computed in clip_loss. The numerator is the diagonal entry of the similarity matrix. The denominator is the row sum. CLIP uses every other row's image as a negative for the current row's text and vice versa, so each batch of $N$ pairs gives $2(N-1)$ negatives per positive. Larger batch = more negatives = better gradient signal. CLIP was trained at batch size 32,768. This is one reason multimodal pretraining is expensive: it is bottlenecked by negative-sample quality, which scales with batch size, which scales with GPU memory.

InfoNCE has a beautiful information-theoretic justification: its negative is a lower bound on the mutual information $I(x; x^+)$ — the number of bits that knowing one view (the image) tells you about the other (its caption), zero when they are independent. So minimizing the loss (it is a negative-log term, like the cross-entropy) raises that bound, pushing up the shared information between the two views, which is a sensible thing to want. See 18-lilian-weng/2021-05-31-contrastive §infonce for the proof.

Practical caveat: if your batch contains semantically similar pairs (two different images of cats with caption "a cat"), they will appear as false negatives. CLIP-style training is robust to this because of the sheer volume of data, but if you train on a small clean dataset, you may need to deduplicate or use a hard-negative mining strategy. SimCLR-style augmentations (random crops, color jitter) are a different way to generate "obvious" negatives.

You have a pretrained CLIP. You have a list of class names you want to classify into. You have no labeled training data for those classes. What do you do?

# Pseudo-code; see the library path above for the full version
class_names = ["cat", "dog", "car", "plane"]
prompts = [f"a photo of a {c}" for c in class_names]
text_features = clip.encode_text(tokenize(prompts))   # (4, d), normalized

def classify(image):
    img_feat = clip.encode_image(preprocess(image))
    img_feat = img_feat / img_feat.norm(dim=-1, keepdim=True)
    sims = img_feat @ text_features.T  # (1, 4)
    return class_names[sims.argmax().item()]

The text encoder is now your classifier. Each class is represented by a 512-dimensional vector (the average of the text embeddings of however many prompts you chose). You can ensemble multiple prompts per class ("a photo of a {c}", "a sketch of a {c}", "a painting of a {c}") and average the resulting features to get a more robust class vector. This is called "prompt ensembling" and is what the OpenAI CLIP paper used to push ImageNet zero-shot from 73% to 76%.

This is the moment most people realize CLIP changed something. You can classify into any category nameable in English without training a model. You can describe a novel concept in a sentence and the model will find images that match. The space of useful classifiers became open-ended.

When you want a model that takes both an image and text and generates text (caption, answer to a question, dialogue), there are three architectural patterns. Each one is a different answer to the question: how do the image tokens reach the language model's residual stream?

Family 1: Concatenation as prefix tokens. Embed the image as a sequence of tokens (with a ViT or a CLIP image encoder), project them into the LM's embedding space, prepend them to the text tokens. The LM treats them as a long prefix. The classic example is Frozen (Tsimpoukelli et al. 2021) and the modern example is Fuyu-8B (Adept, 2023), which uses ViT-style patch tokens projected by a single linear layer with no separate vision tower. Fuyu is the maximally simple multimodal architecture: it has only a language model, and the image is treated as exotic tokens.

Family 2: Cross-attention into LM layers. Keep the LM frozen. Add new cross-attention layers between the LM's blocks that read from a separate vision encoder's outputs. Train only the new cross-attention layers. The classic example is Flamingo (Alayrac et al. 2022). The cross-attention layers are inserted with gating (a learned tanh gate initialized at zero, so at initialization the new layer is a no-op and the LM is preserved). The vision-encoder outputs are first compressed by a Perceiver resampler — a small transformer with a fixed number of learned query vectors — that maps an arbitrary-length sequence of image features down to a fixed 64 visual tokens per image. This is what lets Flamingo handle interleaved sequences of images and text.

Family 3: Frozen vision encoder + projector + frozen LM. A simple MLP (or two-layer projector) maps image features into the LM's embedding space, then the LM sees them as prefix tokens (so it's a hybrid of Family 1's interface and a separate vision tower). The vision encoder is pretrained (typically CLIP-ViT) and frozen. The LM is pretrained (typically Llama or Mistral) and either frozen or LoRA-tuned — LoRA (Low-Rank Adaptation) freezes the pretrained weights and trains only small low-rank matrices injected into the attention layers, cutting trainable parameters by ~90% so you adapt the LM without paying for a full fine-tune. Only the projector is trained from scratch on image-caption data. LLaVA (Liu et al. 2023) is the canonical example. It is the cheapest credible multimodal architecture: you can train LLaVA-1.5 on 8 A100s in a day. It is also the most copied: Qwen-VL, MiniGPT-4, InstructBLIP, and the entire long tail of open multimodal models all use some variant of "frozen vision + projector + LM".

Choosing between them, very roughly:

• Lots of paired data + lots of compute: Family 1 (concatenation) or Family 3 (projector) trained end-to-end. Most production multimodal LMs in 2024.
• Limited data, want to preserve LM: Family 2 (cross-attention) with frozen LM. Flamingo, Otter.
• No compute, just want VLM: Family 3 with everything frozen except a tiny projector. LLaVA's original recipe.

LLaVA is the model most people will start with when building their own multimodal system, so it deserves its own section. The architecture:

1. A pretrained CLIP ViT-L/14 image encoder. Frozen. Outputs a sequence of 256 patch features (after global average pooling is not applied) of dimension 1024.
2. A 2-layer MLP projector. Input dim 1024, hidden dim 4096, output dim 4096 (the LM's embedding dim). Trained from scratch. GELU activation between the layers.
3. A pretrained Llama / Vicuna / Mistral LM, e.g. Llama-2-7B-Chat. Either frozen or LoRA-tuned.

At inference, you tokenize the prompt with a special <image> placeholder. Encode the image with CLIP, project with the MLP, get 256 image tokens. Replace <image> in the embedding sequence with those 256 tokens. The LM then sees a sequence of [text-tokens-before, 256-image-tokens, text-tokens-after] and generates the response autoregressively.

Training is in two stages.

Stage 1 — feature alignment. Train only the MLP projector on 558K image-caption pairs (LAION-CC-SBU subset). The vision encoder and LM are both frozen. The MLP learns to project CLIP features into the LM's embedding space. About 4 hours on 8 A100s.

Stage 2 — instruction tuning. Train the MLP and the LM (with LoRA, typically) on 158K image-text instruction pairs generated by GPT-4 from image captions and bounding boxes. About 8 hours on 8 A100s.

The result is a model that takes an image and a question and produces a coherent answer. It is not state-of-the-art on benchmarks. It is good enough to be useful, and most importantly it is reproducible on a small lab's compute budget.

# Pseudo-code for LLaVA's forward pass
def llava_forward(image, text_tokens, image_placeholder_id):
    img_feats = clip_vit(image)              # (1, 256, 1024)
    img_embeds = mlp_projector(img_feats)    # (1, 256, 4096)
    text_embeds = llm.embed(text_tokens)     # (1, T, 4096)
    # Replace the placeholder with image embeds
    embeds = splice_image_into_text(
        text_embeds, img_embeds, image_placeholder_id
    )                                        # (1, T + 256 - 1, 4096)
    return llm.forward_from_embeds(embeds)

The deeper lesson: a 2-layer MLP is enough projection capacity if the vision encoder is already aligned with language (which CLIP's pretraining does for free). When the vision encoder is not aligned (e.g. DINO features that have never seen text), you need a heavier projector — often a Q-Former.

Q-Former (BLIP-2; Li et al. 2023) is a small transformer that takes a fixed set of learned query vectors and cross-attends to the vision encoder's outputs. It serves as a learned compression + alignment layer between vision and language. With Q-Former, you can use a non-aligned vision encoder (e.g. EVA-CLIP, or DINO) and still get good text generation. The cost is a heavier model (the Q-Former has its own parameters and forward pass). BLIP-2 with a Q-Former and FlanT5 was state-of-the-art VLM in mid-2023 before LLaVA simplified the approach.

Flamingo's design solves a problem LLaVA-style models duck: how do you handle multiple images interleaved with text in a single context? "Here's image A, here's image B, which is bigger?" requires the model to attend to both images and reason about them together.

The Flamingo trick is the Perceiver resampler. The vision encoder produces a variable-length sequence of features per image (sometimes a long sequence, e.g. for video). The Perceiver resampler takes those features and compresses them to a fixed number of latent tokens (64 in the original Flamingo) using cross-attention with learned query vectors.

import torch
import torch.nn as nn

class PerceiverResampler(nn.Module):
    def __init__(self, d_model: int, n_latents: int = 64, n_layers: int = 6):
        super().__init__()
        self.latents = nn.Parameter(torch.randn(n_latents, d_model))
        self.layers = nn.ModuleList([
            PerceiverBlock(d_model) for _ in range(n_layers)
        ])

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: (B, T_visual, d_model) — the encoder's output, variable length
        # Expand learned latents to batch
        latents = self.latents.unsqueeze(0).expand(x.shape[0], -1, -1)
        for layer in self.layers:
            latents = layer(latents, x)  # cross-attend latents -> x, plus FFN
        return latents  # (B, n_latents, d_model) — fixed length

Each PerceiverBlock does cross-attention (queries from latents, keys+values from $x$) plus self-attention among the latents plus an FFN. After the resampler, every image becomes exactly 64 tokens regardless of resolution, video length, or anything else. This is what makes Flamingo's interleaved-context training work: each image slot has predictable cost.

Flamingo's cross-attention layers (the gated layers inserted between Llama blocks) then attend to the latent visual tokens, not to the raw vision-encoder features. Two-stage compression. The masking is also designed so each text token only sees the visual tokens of the last preceding image in the interleaved sequence — a constraint that turned out to work better than allowing global attention across all preceding images.

The downside of Flamingo: the cross-attention layers add ~10% parameters to a frozen LM, and the gating-initialized-at-zero trick means early in training the visual signal is suppressed and the model basically ignores images until the gate opens. This makes training data-hungry. Flamingo was trained on 43M webpages with interleaved images (M3W) plus standard image-text pair datasets. Most groups can't reproduce this.

The Perceiver/Perceiver IO architecture (Jaegle et al. 2021) generalizes the resampler idea: a fixed-size latent bottleneck that you can cross-attend to from any modality. Used in DeepMind's RT-2 robotics model and several recent multimodal architectures. Worth knowing as a pattern even if you never use Flamingo specifically.

CLIP's contrastive loss is great for retrieval and zero-shot classification. It is bad for generation — CLIP can't write a caption. The next generation of multimodal models add a captioning loss to the contrastive loss.

CoCa (Contrastive Captioner; Yu & Wang et al. 2022) is the textbook example. The architecture is a unimodal text decoder (causal self-attention only) on the bottom, a multimodal text decoder (cross-attends to image features) on top. Total loss is a weighted sum:

The contrastive loss aligns image and text representations like CLIP. The captioning loss makes the model predict the next text token given the image. The shared encoder produces representations that work for both retrieval and generation.

BLIP-2 (Li et al. 2023) does something similar but with the Q-Former as the bridge. Two-stage training:

1. Vision-language representation learning: train Q-Former with three losses: image-text contrastive (like CLIP), image-text matching (binary classifier on whether image-text pair is matched), and image-grounded text generation.
2. Vision-to-language generative learning: freeze the vision encoder and the LM (typically FlanT5 or OPT), train only the Q-Former to produce features that the LM can use for text generation.

The output: a model that does retrieval, classification, captioning, and VQA all reasonably well, at modest training cost.

The takeaway pattern: contrastive loss alone gives you alignment. Captioning loss alone gives you generation. Combining them gives you both, and the joint training doesn't hurt either task much. Modern multimodal pretraining recipes (PaLI, CoCa, BLIP-2, InternVL) all do some flavor of this.

Image generation from text is structurally different from VLMs. The model produces pixels, not tokens. But the multimodal trick is the same: cross-attention reaches from one modality into another.

Stable Diffusion's pipeline:

1. Text encoder. A frozen CLIP text encoder (ViT-L for SD 1.x, OpenCLIP H/14 for SD 2.x) maps the prompt to a sequence of 77 token embeddings of dim 768 or 1024.
2. VAE encoder. A pretrained variational autoencoder (VAE) — an autoencoder whose bottleneck compresses an image down to a small latent code you can later decode back into pixels (covered properly in another chapter) — compresses a 512×512 RGB image to a 64×64×4 latent: a low-resolution, channels-deep tensor that stands in for the full image, so all the heavy work happens on 64×64×4 numbers instead of 512×512×3 pixels. (At inference, you start from random noise of this shape; the VAE is only used to decode at the end.)
3. UNet. The denoiser. Takes (noisy latent, timestep, text embeddings). Outputs predicted noise. This is the heart of diffusion: training adds Gaussian noise to a latent in small steps, and the UNet learns to predict the noise that was added so you can subtract it; at generation time you start from pure noise and run the UNet ~50 times, each pass predicting and removing a slice of noise so the latent edges toward a clean image (the diffusion mechanics are another chapter). This is the network that runs 50 times during inference, once per denoising step.
4. VAE decoder. Takes the final denoised latent and produces the 512×512 image.

The text reaches the UNet through cross-attention layers interleaved with the convolutional ResNet blocks. The UNet at each spatial resolution applies:

x = conv_block(x)               # spatial conv, no text
x = cross_attention(q=x, k=text_emb, v=text_emb)   # text reaches in here
x = self_attention(x)           # spatial self-attention

The cross-attention's queries come from the image latents (flattened across spatial axes); the keys and values come from the text embeddings. The attention pattern says "this spatial position should pay attention to these text tokens". After many forward passes through these layers, the noise prediction is conditioned on the text.

The "text encoder choice matters" finding (from the Imagen paper) is important here: scaling the text encoder gives a bigger generation-quality lift than scaling the UNet. SD 2.x switched to a larger OpenCLIP for exactly this reason. Modern models (SDXL, Stable Cascade) use even bigger text encoders, sometimes two of them concatenated.

The structural lesson: cross-attention is the universal fusion primitive across multimodal architectures. Family 2 above uses it to fuse vision into LMs. Stable Diffusion uses it to fuse text into image generation. The signature pattern is Q ← modality A, K, V ← modality B. Once you know to look for it, you see it everywhere.

DALL-E (the original, 2021) and Stable Diffusion (2022) reached similar results from different architectural starting points. The contrast is instructive.

DALL-E (v1) is a single autoregressive transformer over a joint token sequence: text tokens followed by image tokens. The image tokens come from a discrete VAE (dVAE) that encodes a 256×256 image to a 32×32 grid of 8192-way categorical tokens (1024 image tokens total). The transformer (12B params in the original) is trained to predict the next token given the previous. Sample: type a prompt, generate 1024 image tokens autoregressively, decode with the dVAE.

Stable Diffusion is a denoising diffusion model on continuous latent representations, conditioned on text via cross-attention. No tokenization of images. No autoregressive sampling. Iterative denoising of a continuous tensor instead.

DALL-E's autoregressive image generation is slow at inference (1024 tokens of sequential generation per image) and the discrete VAE creates a hard quality ceiling (you can't generate finer detail than the dVAE codebook supports). Stable Diffusion's diffusion-on-latents is fast (40-50 steps, each doing one UNet forward pass), and the continuous latents have no codebook limit, but training requires a complex schedule and the model is harder to fine-tune cleanly.

DALL-E 2 and DALL-E 3 switched to diffusion. The autoregressive line lived on in Parti (Google, 2022) and Muse (Google, 2023, with parallel non-autoregressive masked generation). The diffusion line continued through SDXL, Stable Cascade, FLUX, and so on. As of writing this chapter, diffusion is dominant for image generation. Autoregressive is competitive for video (where the sequential structure maps onto frames) and is making a comeback via flow matching and consistency models that combine diffusion's quality with faster sampling.

The lesson is that the modality of the output (continuous pixels vs discrete tokens) and the generation strategy (autoregressive vs diffusion vs masked) are largely separable design choices. You can mix them: discrete-token diffusion exists (D3PM), continuous-pixel autoregression has been tried (Image GPT), and so on.

The Distill "Multimodal Neurons" paper (Goh et al. 2021) opened the inside of CLIP and found something striking: individual neurons in the image encoder respond to concepts, not just visual features. There is a "Spider-Man neuron" that fires on photographs of Spider-Man, drawings of Spider-Man, the word "spider" written in text inside an image, and Halloween costumes. Same neuron, multiple modalities of evidence.

The interpretation: CLIP's contrastive training pressure makes the image encoder learn features that are aligned with linguistic concepts. Anything you can name in a caption tends to become a feature, because that's what makes the contrastive loss low. This is mechanistically similar to what Anthropic's sparse autoencoder work later found in language model residual streams: the model encodes a sparse dictionary of concepts, and many of them are interpretable.

The famous failure mode discovered in the same paper: typographic attacks. The Spider-Man neuron fires on the text "Spider-Man" written on a piece of paper. So you can fool CLIP into misclassifying an apple as an iPod by writing "iPod" on the apple. The model reads text in images, treats the text as evidence about content, and so a hostile label written on an object overrides the actual visual evidence. This is not a bug — it is what the training data taught the model to do, because in the LAION corpus, text in images is often a strong signal about content (book covers, store signs, etc.). It is just exploitable.

More recent work has scaled this up. Sparse autoencoders on multimodal models (Anthropic's 2024 scaling-monosemanticity paper trained SAEs on Claude 3 Sonnet, which is multimodal) found features that activate on both visual and textual evidence of the same concept: "Golden Gate Bridge" features fire on photographs of the bridge and on text mentioning it. The features are conceptual, not modality-specific. This is the same finding as the 2021 multimodal neurons paper, scaled up.

Brutal summary of when to use what, given the modeling landscape as of 2025-2026:

• Image classification with a fixed label set, lots of labels: fine-tuned ViT or supervised CLIP-style model. ConvNeXt is also competitive if you don't have the compute for a ViT.
• Image classification with an open label set, no labels: zero-shot CLIP. Use OpenCLIP-ViT-L/14 or bigger.
• Object detection: DETR-family (RT-DETR, DINO) for the modern stack. YOLO if you need real-time edge inference.
• Semantic segmentation: SAM-family (Segment Anything) for promptable segmentation. Mask2Former-on-ViT for fixed-vocabulary segmentation.
• Image features for retrieval / kNN / downstream tasks: DINOv2 if labels are unavailable, CLIP if you also need text alignment.
• Multimodal LM that takes images and text in, generates text: LLaVA-style (frozen CLIP-ViT + projector + Llama-family LM) is the cheap default. Move to Qwen2-VL or InternVL-style models with native multi-resolution handling if you need video or high-resolution images.
• Text-to-image generation: latent diffusion (SDXL, FLUX) for general use. Specialized models (consistency, distilled) for fast inference.
• Building your own from scratch: do not. Use a pretrained CLIP for vision, a pretrained LM for language, train a projector. Multimodal pretraining from scratch is a $1M+ undertaking.

Most production multimodal stacks in 2026 are some variant of "CLIP image encoder + projector + Llama-family LM + LoRA fine-tuning on task-specific data". The architecture has converged. The interesting work is on data, evals, and serving (which is another chapter).