Generative Models

An autoencoder is two functions stacked back-to-back. An encoder $g_\phi : \mathbb{R}^d \to \mathbb{R}^k$ with $k < d$, and a decoder $f_\theta : \mathbb{R}^k \to \mathbb{R}^d$. You train both jointly to make $f_\theta(g_\phi(\mathbf{x})) \approx \mathbf{x}$, with squared error or cross-entropy as the reconstruction loss. The interesting thing is the middle. The $k$-dimensional bottleneck is forced to retain whatever information about $\mathbf{x}$ is necessary to reconstruct it, and discard everything else. After training, the bottleneck activations $\mathbf{z} = g_\phi(\mathbf{x})$ are a learned representation.

This is dimensionality reduction the way PCA is dimensionality reduction. PCA finds the linear subspace that captures the most variance. An autoencoder finds a nonlinear manifold that captures the most reconstructable structure. PCA is a special case where the encoder and decoder are both linear and tied. The autoencoder generalizes the framing.

Two things to remember. First, a deep enough autoencoder with an unconstrained bottleneck just learns the identity function and is useless. The bottleneck width is the regularizer. Second, a vanilla autoencoder is not a generative model. The latent space has no probability structure. If you sample a random $\mathbf{z}$ and decode it, you get garbage almost always. The VAE fixes this. We will get there.

Library path (PyTorch):

import torch
import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, d_in: int = 784, d_latent: int = 32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(d_in, 256), nn.ReLU(),
            nn.Linear(256, d_latent),
        )
        self.decoder = nn.Sequential(
            nn.Linear(d_latent, 256), nn.ReLU(),
            nn.Linear(256, d_in), nn.Sigmoid(),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.decoder(self.encoder(x))

# Training loop sketch (MNIST, flattened to 784)
model = Autoencoder()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for x, _ in loader:
    x = x.view(-1, 784)
    recon = model(x)
    loss = nn.functional.mse_loss(recon, x)
    opt.zero_grad(); loss.backward(); opt.step()

From-scratch path (NumPy, just to see that nothing is hidden):

import numpy as np

def relu(x): return np.maximum(0, x)
def sigmoid(x): return 1.0 / (1.0 + np.exp(-x))

class AE:
    def __init__(self, d_in=784, d_h=256, d_z=32, seed=0):
        rng = np.random.default_rng(seed)
        scale = lambda a, b: rng.standard_normal((a, b)) / np.sqrt(a)
        self.W1, self.W2 = scale(d_in, d_h), scale(d_h, d_z)
        self.W3, self.W4 = scale(d_z, d_h), scale(d_h, d_in)

    def forward(self, x):
        h1 = relu(x @ self.W1)
        z  = h1 @ self.W2
        h2 = relu(z @ self.W3)
        return sigmoid(h2 @ self.W4), z

You can write the gradients by hand if you want to see why backprop through a bottleneck is well-conditioned. Most people use PyTorch.

A denoising autoencoder is the same architecture trained with a different objective. You corrupt $\mathbf{x}$ to get $\tilde{\mathbf{x}}$ (mask some pixels, add Gaussian noise, set some values to zero) and train the network to reconstruct the clean $\mathbf{x}$ from the corrupted input. The loss is now $\|\mathbf{x} - f_\theta(g_\phi(\tilde{\mathbf{x}}))\|^2$. The intuition Vincent et al. give in the 2008 paper: to repair a partially destroyed input, the network has to learn the joint structure of the input dimensions. It cannot just copy values across the bottleneck. It has to know that if those pixels are missing, the neighborhood looks like an eye, so the missing pixels are probably skin.

This is more than a regularization trick. The denoising objective is mathematically connected to score matching — training a network to estimate the score $\nabla_{\mathbf{x}} \log q(\mathbf{x})$, the gradient of the log data-density, by the detour of adding a little noise and learning to predict it away — which is in turn connected to diffusion models, where the same noise-then-predict move is run at many noise levels. We come back to this connection in the diffusion section. For now, two practical points. First, the denoising autoencoder pre-dates dropout by four years and is morally the same idea (Lilian Weng makes this observation; it shows up in every careful history of the area). Second, the corruption can be salt-and-pepper, masking, Gaussian, or learned. Each gives a different inductive bias.

Library path:

# Same Autoencoder as before, plus a corruption op in the loop
for x, _ in loader:
    x = x.view(-1, 784)
    x_noisy = x + 0.3 * torch.randn_like(x)
    x_noisy = x_noisy.clamp(0, 1)
    recon = model(x_noisy)
    loss = nn.functional.mse_loss(recon, x)   # target is clean x
    opt.zero_grad(); loss.backward(); opt.step()

A sparse autoencoder adds an L1 penalty (or a KL-to-Bernoulli penalty in the original 2008 framing) on the hidden activations to force most units to be near zero on any given input. The model is encouraged to use a small set of features per input, even when the hidden layer is much wider than the input. This is the trick that makes the modern mech-interp dashboard possible. If you train a sparse autoencoder on the residual stream of a transformer and the L1 penalty is set right, the features the SAE learns tend to be approximately monosemantic. One feature lights up on "function calls in Python," another on "tokens followed by a date," and so on. Anthropic's Towards Monosemanticity and Scaling Monosemanticity hang the entire post-2023 interpretability program on this hook.

We will not belabor this here, because another chapter is the whole story. The point for this chapter is that the sparse autoencoder is a generative-model technique that turned out to be the right tool for a non-generative problem. The same encoder-bottleneck-decoder schema, plus an L1 on the bottleneck, plus enough scale, gives you features. The first time I saw the dashboard I thought it was magic. It is not magic. It is L1 plus scale.

Library path:

class SparseAE(nn.Module):
    def __init__(self, d_in: int, d_features: int, l1_coeff: float = 1e-3):
        super().__init__()
        self.encoder = nn.Linear(d_in, d_features)
        self.decoder = nn.Linear(d_features, d_in, bias=False)
        self.l1_coeff = l1_coeff

    def forward(self, x: torch.Tensor):
        z = torch.relu(self.encoder(x))
        x_hat = self.decoder(z)
        recon_loss = (x_hat - x).pow(2).mean()
        sparsity_loss = z.abs().mean()
        return x_hat, z, recon_loss + self.l1_coeff * sparsity_loss

Note: in modern SAE training the decoder weights are normalized per-feature, and the sparsity penalty is sometimes a top-k or JumpReLU activation rather than L1. The principle is the same.

A variational autoencoder turns the encoder into a probability distribution over latents, not a single point. Instead of $\mathbf{z} = g_\phi(\mathbf{x})$, you have $q_\phi(\mathbf{z} \mid \mathbf{x}) = \mathcal{N}(\mathbf{z}; \boldsymbol{\mu}_\phi(\mathbf{x}), \boldsymbol{\sigma}_\phi^2(\mathbf{x}) \mathbf{I})$, where the encoder network outputs a mean and a log-variance per latent dimension. To decode, you sample $\mathbf{z} \sim q_\phi(\mathbf{z} \mid \mathbf{x})$ and pass it through the decoder.

The training objective is the evidence lower bound (ELBO):

The first term is reconstruction: how well does the decoder reproduce $\mathbf{x}$ when given a sample $\mathbf{z}$ from the encoder. The second term pulls the encoder distribution $q_\phi(\mathbf{z}\mid\mathbf{x})$ toward the prior $p(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$. Without the KL, the encoder collapses to delta-function distributions and you have an ordinary autoencoder. With the KL too strong, the encoder ignores $\mathbf{x}$ and the decoder learns nothing. The balance is the whole game.

The reparameterization trick. Sampling $\mathbf{z} \sim \mathcal{N}(\mu, \sigma^2)$ is not differentiable in $\mu, \sigma$. The trick: rewrite $\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon}$ where $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$. Now the stochasticity is in $\boldsymbol{\epsilon}$, which is parameter-free, and $\mathbf{z}$ is a deterministic function of $\mu, \sigma, \epsilon$. Gradients flow through $\mu$ and $\sigma$. This is the single trick that made VAEs trainable in 2013. Every neural generative model that involves a sampled latent uses some version of it.

dist = Normal(mu, torch.exp(0.5 * logvar))
z = dist.rsample()                              # rsample = reparameterized, gradients flow
prior = Normal(0.0, 1.0)
kld = kl_divergence(dist, prior).sum()
# reconstruction term is still your own BCE/MSE

def reparameterize(self, mu, logvar):
    std = torch.exp(0.5 * logvar)
    eps = torch.randn_like(std)
    return mu + eps * std

def vae_loss(recon, x, mu, logvar):
    bce = F.binary_cross_entropy(recon, x, reduction="sum")
    kld = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return bce + kld

The closed-form KL between a diagonal Gaussian and the standard normal is one of the most copy-pasted formulas in the field. Memorize it once.

β-VAE (Higgins et al., 2017) is one number changed. Multiply the KL term by a constant $\beta > 1$:

The claim is that for the right $\beta$, the latent dimensions become disentangled: each axis of $\mathbf{z}$ becomes responsible for one independent generative factor of the data (rotation, color, scale, etc.). The mechanism is information-theoretic: a tighter KL forces the encoder to be more efficient, and the efficient code happens to use independent axes for independent factors.

Caveat: the disentanglement claim is delicate. Locatello et al. (2019) showed that unsupervised disentanglement is impossible without inductive bias or supervision. β-VAE works empirically on some datasets and not others, and the precise hyperparameters matter more than the authors' framing implied. The lesson is more pragmatic: the KL coefficient is a knob that trades off reconstruction sharpness against latent regularity, and "what $\beta$ should I use" is mostly an empirical question.

For practical generation, $\beta < 1$ also makes sense and is often what people use under the name "warmup": start with a small KL weight so the decoder learns something, then ramp up. The HuggingFace diffusers library uses this pattern internally for some of its VAE pre-training.

VQ-VAE (van den Oord et al., 2017) makes the latent space discrete. Instead of a continuous $\mathbf{z}$, the encoder output is snapped to the nearest entry in a learned codebook $\mathbf{e} \in \mathbb{R}^{K \times D}$. The decoder sees the codebook vector. Because argmin is not differentiable, gradients are passed through with a straight-through estimator (the gradient of the decoder input is copied to the encoder output). The codebook is updated with an EMA or a separate loss term.

Why bother? Discrete latents are a natural fit for autoregressive priors. Once you have a VQ-VAE with $K$ codebook entries, you can train an autoregressive model (a PixelCNN, a Transformer) over the discrete latent grid. Sampling becomes: autoregressively sample latent codes, then decode. This is how DALL-E (the original 2021 version) worked and how most modern audio generative models work. The codebook structure shows up again in modern image and audio tokenizers, and it is the reason "image tokens" are a coherent phrase.

A Generative Adversarial Network is two networks trained against each other. A generator $G_\theta : \mathbb{R}^k \to \mathbb{R}^d$ maps noise $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ to fake samples. A discriminator $D_\phi : \mathbb{R}^d \to [0, 1]$ outputs a probability that an input is real. The training objective is a saddle point:

The discriminator maximizes log-likelihood that it correctly classifies real vs fake. The generator minimizes the discriminator's ability to do so. At the Nash equilibrium of this game, $p_{G} = p_\text{data}$ and $D = 1/2$ everywhere. Goodfellow's 2014 paper proves the equilibrium exists under idealized assumptions. Whether you can find it with SGD is a different question.

In practice, the generator's gradient at the start of training is poor when $\log(1 - D(G(\mathbf{z})))$ saturates. The non-saturating loss replaces the minimization with $\max_\theta \mathbb{E}_\mathbf{z}[\log D_\phi(G_\theta(\mathbf{z}))]$. It is the same equilibrium, with better gradients near the start. This is the loss used in basically every GAN you have heard of.

Library path:

class Generator(nn.Module):
    def __init__(self, d_z: int = 100, d_out: int = 784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_z, 256), nn.LeakyReLU(0.2),
            nn.Linear(256, 512), nn.LeakyReLU(0.2),
            nn.Linear(512, d_out), nn.Tanh(),
        )
    def forward(self, z): return self.net(z)

class Discriminator(nn.Module):
    def __init__(self, d_in: int = 784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_in, 512), nn.LeakyReLU(0.2),
            nn.Linear(512, 256), nn.LeakyReLU(0.2),
            nn.Linear(256, 1),
        )
    def forward(self, x): return self.net(x)

def train_step(G, D, x_real, opt_G, opt_D, d_z=100):
    bs = x_real.size(0)
    z = torch.randn(bs, d_z)
    x_fake = G(z)

    # D step
    d_real = D(x_real); d_fake = D(x_fake.detach())
    loss_D = (nn.functional.binary_cross_entropy_with_logits(d_real, torch.ones_like(d_real))
            + nn.functional.binary_cross_entropy_with_logits(d_fake, torch.zeros_like(d_fake)))
    opt_D.zero_grad(); loss_D.backward(); opt_D.step()

    # G step (non-saturating)
    d_fake = D(x_fake)
    loss_G = nn.functional.binary_cross_entropy_with_logits(d_fake, torch.ones_like(d_fake))
    opt_G.zero_grad(); loss_G.backward(); opt_G.step()
    return loss_D.item(), loss_G.item()

Mode collapse is when the generator finds one mode of the data distribution that fools the discriminator and stays there. You train on faces and every face looks the same. You train on digits and every sample is a 3. The generator is stuck producing a slice of the data because the discriminator's gradient field has a local minimum, and the discriminator chases the generator to that mode rather than spreading out.

A second pathology: the generator and discriminator oscillate. The discriminator wins, the generator's loss explodes, the generator updates and outpaces the discriminator, and the cycle repeats. The samples never improve.

Three diagnostic habits. First, look at samples, not at the loss curves. GAN losses are uninformative because of the adversarial dynamics. Second, track FID (Fréchet Inception Distance) or KID (Kernel Inception Distance) across training: both summarize a batch of generated images by the statistics of their features under a fixed pretrained Inception network and report a distance to the same statistics computed on real images, so lower means the two image distributions are closer (the precise distance — Fréchet for FID, a kernel two-sample test for KID — is not derived here). These are the only loss-like signals that correlate with subjective sample quality. Third, plot a 2D projection of generated samples against real samples. If the generated points cluster in a sub-region, you have mode collapse. The ganlab playground from Polo Club is the best in-browser way to see this happen on toy data.

Wasserstein GAN (Arjovsky et al., 2017) replaces the Jensen-Shannon divergence with the Wasserstein-1 (earth mover's) distance. JS is the symmetric, $[0,1]$-bounded divergence the §7 value function is implicitly minimizing when the discriminator is optimal; its problem is that for two distributions with no overlap (common early in training) it sits flat at $\log 2$ and gives no usable gradient, whereas the Wasserstein-1 distance (informally, the minimum cost of moving one pile of probability mass to match the shape of the other) stays smooth even then. The Kantorovich-Rubinstein duality rewrites that intractable transport cost as

a supremum over all 1-Lipschitz functions $f$ (functions whose output changes by at most $|x_1 - x_2|$ between any two inputs, so the slope is bounded by 1 everywhere). The discriminator (now called a critic) outputs a real-valued score, not a probability, and approximates the supremum. Training minimizes $W_1(p_\text{data}, p_G)$ as estimated by the critic.

The original WGAN enforced the Lipschitz constraint by weight clipping (bound every critic parameter to $[-0.01, 0.01]$). This works but is crude. WGAN-GP (Gulrajani et al., 2017) replaces clipping with a gradient penalty: at random interpolated points $\hat{\mathbf{x}} = \alpha \mathbf{x} + (1-\alpha)\hat{\mathbf{x}}_\text{fake}$, penalize $(\|\nabla_{\hat{\mathbf{x}}} D(\hat{\mathbf{x}})\| - 1)^2$. The 1-Lipschitz constraint is enforced softly via the penalty.

WGAN-GP is the workhorse of stable GAN training. It still mode-collapses sometimes. It still requires hyperparameter tuning. It is much more forgiving than the original GAN, and it is the baseline you should reach for if you have to train a GAN from scratch in 2026.

Library path (the critical bits):

def gradient_penalty(D, x_real, x_fake, device):
    alpha = torch.rand(x_real.size(0), 1, device=device)
    x_hat = (alpha * x_real + (1 - alpha) * x_fake).requires_grad_(True)
    d_hat = D(x_hat)
    grads = torch.autograd.grad(
        outputs=d_hat, inputs=x_hat,
        grad_outputs=torch.ones_like(d_hat),
        create_graph=True, retain_graph=True,
    )[0]
    return ((grads.norm(2, dim=1) - 1) ** 2).mean()

# WGAN-GP critic step
loss_D = D(x_fake).mean() - D(x_real).mean() + 10.0 * gradient_penalty(D, x_real, x_fake, device)

StyleGAN (Karras et al., 2019, NVIDIA) is the architectural high-water mark of pre-diffusion image generation. Two ideas worth carrying forward. First, the mapping network: instead of feeding the noise $\mathbf{z}$ directly into the convolutional generator, pass it through an 8-layer MLP to produce a "style" vector $\mathbf{w}$ first. The style vector is then used to modulate the convolutional layers via AdaIN (adaptive instance normalization). This decouples the noise prior from the generator's input distribution and improves disentanglement empirically.

Second, progressive growing (StyleGAN1) and style-mixing regularization (StyleGAN2+). Growing trains the model at 4x4, then 8x8, then 16x16, doubling resolution until 1024x1024. Style mixing samples two latents during training and applies them to different resolution blocks of the generator, which encourages independent control across scales.

You will not train a StyleGAN today, because diffusion replaces it. But the pattern of "decouple the latent from the generator input via a learned mapping" reappears everywhere. Text-to-image diffusion models do the same thing with CLIP embeddings.

A diffusion model is a sequence of $T$ steps. The forward process takes a clean image $\mathbf{x}_0$ and progressively adds Gaussian noise:

where $\{\beta_t\}_{t=1}^T$ is a variance schedule fixed ahead of time. Typical $T = 1000$, $\beta_1 = 10^{-4}$, $\beta_T = 0.02$, linearly interpolated. The forward process is not learned. It is a known stochastic recipe for destroying data.

The key trick: you can sample $\mathbf{x}_t$ directly from $\mathbf{x}_0$ in closed form. Define $\alpha_t = 1 - \beta_t$ and $\bar\alpha_t = \prod_{s=1}^t \alpha_s$. Then

This is reparameterization again. You do not have to actually simulate $T$ steps to get $\mathbf{x}_t$. One Gaussian draw and a scaling does it. This is what makes diffusion training tractable: at training time, you pick a random $t$, sample $\mathbf{x}_t$ in one shot, and ask the network to denoise it.

As $t \to T$, $\bar\alpha_t \to 0$ and $\mathbf{x}_T$ becomes indistinguishable from $\mathcal{N}(\mathbf{0}, \mathbf{I})$. The forward process has erased everything about $\mathbf{x}_0$. The reverse process is going to learn to climb back up the same chain.

sched = DDPMScheduler(num_train_timesteps=1000, beta_start=1e-4, beta_end=0.02, beta_schedule="linear")
t = torch.randint(0, sched.config.num_train_timesteps, (x0.size(0),))
noise = torch.randn_like(x0)
x_t = sched.add_noise(x0, noise, t)   # = sqrt(bar_alpha_t)*x0 + sqrt(1-bar_alpha_t)*noise

class Schedule:
    def __init__(self, T=1000, beta_min=1e-4, beta_max=0.02):
        self.T = T
        self.betas = torch.linspace(beta_min, beta_max, T)
        self.alphas = 1.0 - self.betas
        self.bar_alphas = torch.cumprod(self.alphas, dim=0)

    def q_sample(self, x0, t, noise):
        bar = self.bar_alphas.to(x0.device).gather(0, t)
        bar = bar.view(-1, 1, 1, 1)
        return torch.sqrt(bar) * x0 + torch.sqrt(1 - bar) * noise

The reverse process is a learned Markov chain $p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \Sigma_\theta(\mathbf{x}_t, t))$. The network has to predict the parameters of the Gaussian that walks one step back up the chain. Ho et al. (2020) showed two simplifying moves that make this practical.

First, fix the variance $\Sigma_\theta$ to a constant schedule (either $\beta_t$ or $\tilde\beta_t$). Don't learn it. The samples are almost as good and the training is much more stable.

Second, reparameterize the mean prediction. Instead of predicting $\boldsymbol{\mu}_\theta$ directly, predict the noise $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$ that was added at step $t$. The mean is then recovered by

And the training loss collapses to a stunningly simple form:

Read that loss carefully. You sample a random $t$, a random clean image $\mathbf{x}_0$, and a random Gaussian noise $\boldsymbol{\epsilon}$. You construct $\mathbf{x}_t$ by the closed-form forward sample. You ask a network to look at $\mathbf{x}_t$ and $t$ and predict the noise you added. It is a regression problem. The network is usually a U-Net with timestep conditioning via sinusoidal embeddings added to the residual blocks. There is no adversarial training. There is no KL constraint on a latent. The only loss is MSE between the actual noise and the predicted noise. This is why diffusion training is so much more stable than GAN training.

t = torch.randint(0, sched.config.num_train_timesteps, (x0.size(0),), device=x0.device)
noise = torch.randn_like(x0)
x_t = sched.add_noise(x0, noise, t)
noise_pred = model(x_t, t).sample        # UNet2DModel returns .sample
loss = F.mse_loss(noise_pred, noise)
opt.zero_grad(); loss.backward(); opt.step()

def train_step(model, schedule, x0, optimizer):
    B = x0.size(0)
    t = torch.randint(0, schedule.T, (B,), device=x0.device)
    eps = torch.randn_like(x0)
    x_t = schedule.q_sample(x0, t, eps)
    eps_hat = model(x_t, t)
    loss = F.mse_loss(eps_hat, eps)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    return float(loss.item())

The U-Net model here takes $(\mathbf{x}_t, t)$ and outputs a tensor of the same shape as $\mathbf{x}_t$, interpreted as the noise prediction $\boldsymbol{\epsilon}_\theta$.

Sampling from a trained diffusion model means walking the reverse chain. You start with $\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and iterate

for $t = T, T-1, \dots, 1$. The stochastic term $\sigma_t \mathbf{z}$ is what makes this DDPM. With $T = 1000$ steps, a single sample takes 1000 forward passes through the network. That is too slow for any serious application.

DDIM (Song et al., 2020) is a deterministic sampler that uses the same trained model but skips steps. Set $\sigma_t = 0$ and parameterize the update as

where $\hat{\mathbf{x}}_0$ is the network's implicit estimate of the clean image at step $t$. You can subsample the timestep schedule (use 50 steps instead of 1000) and quality holds up surprisingly well. DDIM is the default sampler in most production diffusion code paths because it is the simplest sub-100-step option. Modern competitors (DPM-Solver, UniPC, Heun's method, Euler-ancestral) get to 20-step sampling at similar quality. The bookkeeping varies. The principle is the same: solve the underlying probability-flow ODE more accurately, take fewer steps.

Diffusion on raw pixels is expensive: 512x512x3 inputs at 1000 steps with a U-Net. Latent Diffusion (Rombach et al., 2022) does the diffusion process in the latent space of a separately-trained VAE. The pipeline:

1. A VAE encodes 512x512x3 images into a 64x64x4 latent (a factor of 48x compression).
2. The diffusion U-Net is trained to denoise in this 64x64x4 latent space, not in pixel space.
3. At inference, sample a latent via the reverse process, then decode through the VAE decoder.

The compression makes the whole thing tractable on a single GPU. Stable Diffusion is the open-weights instantiation of this design (latent diffusion + CLIP text conditioning + the VAE that was trained on LAION). The U-Net has cross-attention layers that attend to text embeddings produced by a frozen CLIP text encoder. This is how you go from a prompt to a sample.

Three production details that matter. First, the VAE is not particularly good as a standalone generative model. Its job is to be a high-fidelity, low-distortion compressor. You sample in latent space, not from the VAE prior. Second, the text conditioning is injected through cross-attention at multiple resolutions of the U-Net, not just at the start. This is what gives the model fine-grained control. Third, the VAE decoder is the source of most of the "Stable Diffusion looks" texture. Different VAEs (the original "EMA" VAE, the "MSE" VAE, the SDXL VAE) produce visibly different outputs from the same latent.

Classifier-free guidance (Ho & Salimans, 2022) is the single trick that makes text-to-image diffusion follow prompts well. Train the diffusion model jointly on conditional (text-conditioned) and unconditional inputs by dropping the text condition with some probability $p \approx 0.1$ during training. At inference time, run the network twice per step: once with the prompt $c$ to get $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, c)$ and once with no condition (or a null prompt) to get $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \varnothing)$. Combine:

The guidance scale $w$ is the CFG knob. $w = 1$ recovers the conditional model. $w > 1$ amplifies the conditional direction. Stable Diffusion's de-facto default is $w = 7.5$.

The high-CFG samples are visibly more "prompt-faithful" but also more saturated, more contrasty, and lose diversity. Very high CFG ($w > 15$) produces samples that look burnt. There is no theoretical reason $w = 7.5$ is the right number. It is what the community converged on through eyeballing.

Library path (the single sampling step):

def classifier_free_guidance_step(model, x_t, t, c_pos, c_null, guidance_scale=7.5):
    eps_uncond = model(x_t, t, c_null)
    eps_cond   = model(x_t, t, c_pos)
    return eps_uncond + guidance_scale * (eps_cond - eps_uncond)

There are several equivalent parameterizations of what the network predicts at each step. The original DDPM predicts $\boldsymbol{\epsilon}$ (the noise). v-prediction (Salimans & Ho, 2022) predicts a velocity-like quantity $\mathbf{v}_t = \sqrt{\bar\alpha_t}\boldsymbol{\epsilon} - \sqrt{1 - \bar\alpha_t}\mathbf{x}_0$. The loss surface is conditioned differently and v-prediction is more numerically stable at the endpoints of the noise schedule. Imagen-Video, Stable Diffusion XL, and several others use v-prediction.

Flow matching (Lipman et al., 2023; Esser et al., 2024) is the 2024+ replacement direction. The idea: instead of a Markov chain of noisy variables, parameterize a continuous-time flow between data and noise. The training objective is to predict the conditional vector field that pushes a noise sample toward a data sample along a straight line. The sampling step is a single ODE integration. Rectified flow (the variant used in Stable Diffusion 3 and FLUX) makes the conditional flow straight by design. Empirically, flow models train more stably, sample with fewer steps, and scale better.

The takeaway: the "diffusion" framing of 2020 is one parameterization of a broader family. The math is the same up to substitutions. If you are starting a new generative project in 2026, look at flow matching first. The reason is empirical: the loss surface is nicer.