RL + RLHF

A Markov Decision Process is the formal object an RL agent acts on. It is a tuple $\langle \mathcal{S}, \mathcal{A}, P, R, \gamma \rangle$ where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $P(s' \mid s, a)$ is the transition probability, $R(s, a, s')$ is the reward function, and $\gamma \in [0, 1)$ is the discount factor. A policy $\pi(a \mid s)$ is a probability distribution over actions conditioned on state. The agent's goal is to find a policy that maximizes expected discounted return $\mathbb{E}_\pi[\sum_{t=0}^\infty \gamma^t R_t]$.

Two derived quantities make the math tractable. The state-value function under policy $\pi$:

The action-value function:

The Bellman equation relates the value at a state to the values at successor states:

The optimal policy satisfies the Bellman optimality equation:

Read that twice. The Bellman optimality equation is a fixed-point equation. The unique solution is the optimal value function, and from it you can recover the optimal policy by acting greedily. Every algorithm in this chapter is some way of approximating that fixed point.

From-scratch path (value iteration on a known MDP, the simplest possible RL solver):

import numpy as np

def value_iteration(P: np.ndarray, R: np.ndarray, gamma: float = 0.99,
                    tol: float = 1e-6) -> tuple[np.ndarray, np.ndarray]:
    """
    P: (S, A, S) transition probabilities
    R: (S, A, S) rewards
    Returns: V (S,), pi (S,) greedy policy.
    """
    S, A, _ = P.shape
    V = np.zeros(S)
    while True:
        Q = np.einsum("sat,sat->sa", P, R + gamma * V[None, None, :])
        V_new = Q.max(axis=1)
        if np.abs(V_new - V).max() < tol: break
        V = V_new
    pi = Q.argmax(axis=1)
    return V, pi

That is the entire planning side of classical RL. The hard part is what happens when $P$ and $R$ are unknown and you have to estimate them by interacting with the environment.

When the MDP is unknown, the agent has to learn the value function by interacting with the environment. The temporal-difference (TD) family is the workhorse. The simplest member is Q-learning. Maintain a table $Q(s, a)$ initialized to zero. At each step, take an action, observe the reward and next state, and update:

The term in brackets is the TD error: how much the new estimate (one-step bootstrap) disagrees with the old. $\alpha$ is the learning rate. The crucial thing is that the update uses $\max_{a'} Q(s', a')$ regardless of what action the agent actually took next. This makes Q-learning off-policy: it learns the value of the greedy policy while behaving according to some other (usually $\epsilon$-greedy) policy.

SARSA is the on-policy cousin. The update is identical except the next-action is sampled from the behaviour policy:

The difference matters in cliff-walking environments. SARSA learns a "safe" policy that stays away from the cliff because the $\epsilon$-greedy exploration occasionally falls off. Q-learning learns the "optimal" policy that walks right along the edge because it ignores the exploration in its update. Both converge, eventually, in the tabular case under standard step-size conditions.

The exploration policy matters. $\epsilon$-greedy is the dumbest thing that works: with probability $\epsilon$ act random, otherwise act greedy. Decay $\epsilon$ from 1.0 toward 0.05 over training. Boltzmann (softmax over Q) is sometimes nicer. UCB and Thompson sampling are nicer still. For most of what we will do here, $\epsilon$-greedy is fine.

From here on the library paths use gymnasium (pip install gymnasium), the maintained fork of OpenAI's gym that gives every RL environment a uniform interface: gym.make("FrozenLake-v1") builds an env, env.reset returns the start observation, and env.step(a) returns the (observation, reward, terminated, truncated, info) five-tuple where done = terminated or truncated.

Library path (Frozen Lake with tabular Q-learning):

import gymnasium as gym
import numpy as np

env = gym.make("FrozenLake-v1", is_slippery=True)
Q = np.zeros((env.observation_space.n, env.action_space.n))

alpha, gamma, eps = 0.1, 0.99, 1.0
for ep in range(20_000):
    s, _ = env.reset()
    done = False
    while not done:
        a = env.action_space.sample() if np.random.rand() < eps else int(Q[s].argmax())
        s_next, r, term, trunc, _ = env.step(a)
        Q[s, a] += alpha * (r + gamma * Q[s_next].max() - Q[s, a])
        s, done = s_next, term or trunc
    eps = max(0.05, eps * 0.9995)

Tabular Q-learning fails when the state space is too large to enumerate. Replace the table with a neural network $Q_\theta(s, a)$, where $\theta$ are the network parameters, and you have Deep Q-Networks (Mnih et al., 2013, 2015). Two engineering tricks make the training stable enough to actually work on Atari (the suite of arcade games that became the standard deep-RL benchmark).

Experience replay. Store every transition $(s, a, r, s')$ in a replay buffer. At each gradient step, sample a minibatch uniformly from the buffer and update on that batch. This breaks the temporal correlation between consecutive samples (which violates the i.i.d. assumption of SGD) and lets you reuse each transition for multiple updates.

Target networks. The Q-learning target $r + \gamma \max_{a'} Q_\theta(s', a')$ uses the same parameters being updated. This means the target moves while you are trying to fit to it, which causes divergence. Fix: maintain a second set of "target" parameters $\theta^-$ that are a slow-moving copy of $\theta$ (either hard-updated every $N$ steps or Polyak-averaged). Use $\theta^-$ for the target. The loss becomes:

Library path (the minimum viable DQN):

import torch
import torch.nn as nn
import torch.nn.functional as F
from collections import deque
import random

class QNet(nn.Module):
    def __init__(self, obs_dim: int, n_actions: int):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, 128), nn.ReLU(),
            nn.Linear(128, 128), nn.ReLU(),
            nn.Linear(128, n_actions),
        )
    def forward(self, x): return self.net(x)

def dqn_step(q, q_target, batch, opt, gamma=0.99):
    s, a, r, s_next, done = batch
    q_sa = q(s).gather(1, a.unsqueeze(1)).squeeze(1)
    with torch.no_grad():
        q_next_max = q_target(s_next).max(dim=1).values
        target = r + gamma * q_next_max * (1 - done.float())
    loss = F.mse_loss(q_sa, target)
    opt.zero_grad(); loss.backward()
    nn.utils.clip_grad_norm_(q.parameters(), max_norm=10.0)
    opt.step()
    return loss.item()

Remove the replay buffer and DQN diverges. Remove the target network and DQN diverges. Both are necessary. The 2013 paper makes a big deal of this in the experiments section, and ARENA's part 2.1 walks you through the failure modes one at a time. Worth reproducing.

Two refinements that became standard. Double DQN (van Hasselt et al., 2016) uses $\theta$ to select the next action and $\theta^-$ to evaluate it, reducing the overestimation bias of the max operator. Dueling DQN factorizes $Q(s, a) = V(s) + A(s, a) - \text{mean}_a A(s, a)$, which separates the state value from the action advantage and learns faster in states where the action does not matter.

DQN learns a value function and acts greedily. The policy-gradient family learns the policy directly. Parameterize the policy as a neural network $\pi_\theta(a \mid s)$, sample actions from it, observe returns, and adjust $\theta$ in the direction that increases the probability of high-return actions.

The math is the policy gradient theorem. Define $J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]$ where $\tau$ is a trajectory and $R(\tau) = \sum_t \gamma^t r_t$. The gradient is

where $G_t = \sum_{t' \geq t} \gamma^{t'-t} r_{t'}$ is the return from step $t$ onward. The derivation is the log-derivative trick: $\nabla p = p \nabla \log p$, applied to the trajectory distribution. It is five lines once you see it, and Karpathy's Pong from Pixels writes it out in NumPy with full annotations.

This is REINFORCE (Williams, 1992). Sample a full episode, compute the return at each step, take the log-probability gradient, scale by the return, sum, step. The estimator is unbiased. It is also very high variance, because $G_t$ can vary enormously across episodes.

Library path:

import torch
import torch.nn as nn
from torch.distributions import Categorical

class PolicyNet(nn.Module):
    def __init__(self, obs_dim: int, n_actions: int):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, 64), nn.Tanh(),
            nn.Linear(64, n_actions),
        )
    def forward(self, x): return self.net(x)

def reinforce_loss(policy: PolicyNet, observations, actions, returns):
    logits = policy(observations)
    log_probs = Categorical(logits=logits).log_prob(actions)
    return -(log_probs * returns).mean()

From-scratch path (the log-derivative trick by hand, on a binary action space):

import numpy as np

def reinforce_policy_gradient_numpy(W1, W2, obs, action, advantage):
    # 2-layer NN policy with sigmoid output (prob of action=1)
    h = np.maximum(0, obs @ W1)          # ReLU
    logits = h @ W2                       # scalar
    p = 1.0 / (1.0 + np.exp(-logits))     # probability of action=1
    # log pi(a|s) = log(p) if a=1 else log(1-p)
    # d/d_logits log pi = (a - p)
    d_logits = (action - p) * advantage
    dW2 = np.outer(h, d_logits)
    dh = d_logits * W2
    dh[h <= 0] = 0
    dW1 = np.outer(obs, dh)
    return dW1, dW2

This is exactly the Karpathy Pong-from-pixels architecture, simplified for one observation. The whole Pong from Pixels post is this gradient applied for 6,000,000 frames.

The REINFORCE gradient is unbiased but high-variance. The trick that fixes this is the baseline: subtract a state-dependent function $b(s)$ from $G_t$ in the gradient:

The estimator remains unbiased for any $b(s_t)$ that does not depend on the action (because $\mathbb{E}[\nabla \log \pi(a \mid s) b(s)] = b(s) \nabla \mathbb{E}[1] = 0$). The variance is minimized when $b(s) = V^\pi(s)$. The difference $G_t - V^\pi(s_t) \approx Q^\pi(s_t, a_t) - V^\pi(s_t) = A^\pi(s_t, a_t)$ is the advantage: how much better is action $a_t$ than the average action from state $s_t$.

This gives Actor-Critic. Two networks: an actor $\pi_\theta(a \mid s)$ that picks actions and a critic $V_\phi(s)$ that estimates value. The critic is trained on TD targets ($V_\phi(s_t) \approx r_t + \gamma V_\phi(s_{t+1})$) and the actor is trained on advantage estimates ($\hat A_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$ for one-step, or richer estimators for multi-step). A2C is the synchronous version. A3C runs multiple agents in parallel and updates a shared model asynchronously (DeepMind, 2016).

The big practical question is how many steps to bootstrap. One-step TD ($\hat A_t = r_t + \gamma V(s_{t+1}) - V(s_t)$) is biased but low-variance. Full Monte Carlo return ($\hat A_t = G_t - V(s_t)$) is unbiased but high-variance. GAE (next sub-section) splits the difference.

Library path:

class ActorCritic(nn.Module):
    def __init__(self, obs_dim: int, n_actions: int):
        super().__init__()
        self.body = nn.Sequential(
            nn.Linear(obs_dim, 64), nn.Tanh(),
            nn.Linear(64, 64), nn.Tanh(),
        )
        self.actor  = nn.Linear(64, n_actions)
        self.critic = nn.Linear(64, 1)
    def forward(self, x):
        h = self.body(x)
        return self.actor(h), self.critic(h).squeeze(-1)

GAE (Schulman et al., 2015) interpolates between one-step TD and Monte Carlo return. Define the one-step TD residual $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$. The GAE estimator is the exponentially-weighted sum of these residuals:

The parameter $\lambda \in [0, 1]$ is the bias-variance knob. $\lambda = 0$ recovers one-step TD (high bias, low variance). $\lambda = 1$ recovers Monte Carlo (low bias, high variance). Schulman's empirical sweet spot is $\lambda = 0.95$, which has been the de-facto PPO default ever since.

In practice GAE is computed backwards through a trajectory in a single pass:

def compute_gae(rewards, values, dones, gamma=0.99, lam=0.95):
    advantages = []
    gae = 0
    for t in reversed(range(len(rewards))):
        next_value = values[t + 1] if t + 1 < len(values) else 0
        delta = rewards[t] + gamma * next_value * (1 - dones[t]) - values[t]
        gae = delta + gamma * lam * (1 - dones[t]) * gae
        advantages.insert(0, gae)
    return torch.tensor(advantages)

Proximal Policy Optimization (Schulman et al., 2017) is the workhorse of modern policy-gradient RL. It is the algorithm OpenAI used for the Dota 5 agents, the algorithm InstructGPT/ChatGPT use for RLHF, and the algorithm ARENA uses as the centerpiece of its RL chapter.

The framing: vanilla policy gradient takes one gradient step per batch of experience and throws the data away. This is sample-inefficient. You would like to take many gradient steps per batch. But each gradient step changes the policy, which means subsequent gradient steps are using off-policy data, and the gradient estimator is biased.

PPO's fix is the clipped surrogate objective. Define the probability ratio between the new policy and the old policy:

The objective is:

Read this carefully. When $\hat A_t > 0$ (action was better than average), increasing $r_t(\theta)$ is good, but clipping at $1 + \epsilon$ stops you from increasing it too much in one step. When $\hat A_t < 0$ (action was worse than average), decreasing $r_t(\theta)$ is good, clipping at $1 - \epsilon$ stops you. The $\min$ takes the more pessimistic of the clipped and unclipped objectives. The net effect is a trust region (the idea, from TRPO, that each update should keep the new policy close to the old one): the policy cannot move too far from $\theta_\text{old}$ in a single update.

The full PPO loss has three terms:

The first is the clipped policy loss. The second is the value function loss (MSE against the GAE returns). The third is an entropy bonus: maximize the entropy of $\pi$ to encourage exploration. Typical $c_1 = 0.5$, $c_2 = 0.01$, $\epsilon = 0.2$.

Library path (PPO update, the central function):

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.distributions import Categorical

def ppo_update(actor_critic, optimizer, observations, actions, old_log_probs,
               returns, advantages, clip_eps=0.2, vf_coef=0.5, ent_coef=0.01,
               n_epochs=4, batch_size=64):
    advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
    for _ in range(n_epochs):
        idx = torch.randperm(len(observations))
        for start in range(0, len(observations), batch_size):
            b = idx[start:start + batch_size]
            logits, values = actor_critic(observations[b])
            dist = Categorical(logits=logits)
            new_log_probs = dist.log_prob(actions[b])
            entropy = dist.entropy().mean()

            ratio = (new_log_probs - old_log_probs[b]).exp()
            surr1 = ratio * advantages[b]
            surr2 = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * advantages[b]
            policy_loss = -torch.min(surr1, surr2).mean()

            value_loss = F.mse_loss(values, returns[b])
            loss = policy_loss + vf_coef * value_loss - ent_coef * entropy

            optimizer.zero_grad()
            loss.backward()
            nn.utils.clip_grad_norm_(actor_critic.parameters(), 0.5)
            optimizer.step()

A vanilla PPO agent trains CartPole to 500 reward (the cap) in about 30 seconds on a CPU. ARENA's part 2.3 walks you through every hyperparameter individually.

PPO is on-policy and works on discrete action spaces. For continuous control (robotics, MuJoCo) and for sample-efficient learning, the modern picks are SAC and TD3. We will be brief because this chapter is mostly about discrete and language RL.

SAC (Soft Actor-Critic, Haarnoja et al., 2018) is the entropy-regularized variant of off-policy actor-critic. The reward is augmented with $\alpha H(\pi(\cdot \mid s))$, encouraging high-entropy policies. Two Q-networks (clipped double-Q trick), a target network, a replay buffer, automatic temperature tuning. Robust to hyperparameters in a way that PPO is not.

TD3 (Twin Delayed DDPG, Fujimoto et al., 2018) is the deterministic-policy alternative. Two Q-networks (take the min for the target, reducing overestimation), delayed policy updates, target policy smoothing. Less of a robust default than SAC, but simpler to implement.

The lesson for a reader of this chapter: off-policy methods (DQN family, SAC, TD3) are more sample-efficient because they reuse data from a replay buffer. On-policy methods (PPO, A2C) are more stable in practice because the policy gradient estimator is unbiased on freshly-collected data. For language model RLHF (next), the choice is PPO. The data is small and the off-policy distribution shift is the problem you want to minimize.

Reinforcement Learning from Human Feedback is the procedure that turned GPT-3 into ChatGPT. The pipeline has three phases.

Phase 1: Supervised fine-tuning (SFT). Start with a pretrained language model. Fine-tune it on a small set of demonstration data: pairs of (prompt, ideal completion) hand-written by contractors. The output is a model that mimics the style of the demonstrations. The SFT model is the policy initialization for the RL phase.

Phase 2: Reward model training. Sample pairs of completions $(y_1, y_2)$ for the same prompt $x$ from the SFT model. Have a human (or stronger model) rank them: $y_1 \succ y_2$ or $y_2 \succ y_1$. Train a separate model $r_\phi(x, y)$ to predict these preferences. The loss is the Bradley-Terry likelihood:

where $y_w$ is the winner and $y_l$ is the loser. The reward model is usually initialized from the SFT model with the final unembedding replaced by a scalar head. After training, $r_\phi(x, y)$ gives a real-valued score that approximates "how much would a human prefer this response."

Phase 3: PPO fine-tuning against the reward model. Treat the language model as a policy $\pi_\theta(y \mid x)$ (sequentially sampling tokens), the prompt $x$ as the initial state, the action space as the vocabulary, and the reward as $r_\phi(x, y)$ paid at the end of the sequence. Run PPO. Critically, add a KL penalty to the reward to prevent the policy from drifting too far from the SFT model:

The KL penalty $\beta$ is the most important knob. Too small and the policy collapses onto reward-model failure modes (next section). Too large and the policy ignores the reward signal. Typical $\beta \in [0.01, 0.2]$.

The PPO step inside RLHF — sample completions, fold a per-token KL penalty into the reward stream, run GAE, then take the clipped update — is trl.PPOTrainer, unrolled:

trainer = PPOTrainer(
    args=PPOConfig(kl_coef=0.05, cliprange=0.2, vf_coef=0.5,
                   gamma=1.0, lam=0.95, num_ppo_epochs=4),
    model=actor_with_value_head, ref_model=ref, reward_model=rm,
    processing_class=tokenizer, train_dataset=prompt_ds,
)
trainer.train()                    # generate -> score -> KL-penalize -> GAE -> clipped update

@torch.no_grad()
def rollout(actor, ref_model, prompts, max_new_tokens=8, temperature=1.0):
    full = prompts.clone(); log_probs=[]; ref_log_probs=[]; values=[]
    for _ in range(max_new_tokens):
        logits, vals = actor(full)
        dist = Categorical(F.softmax(logits[:, -1, :] / temperature, dim=-1))
        nxt = dist.sample()
        log_probs.append(dist.log_prob(nxt))
        ref_logits = ref_model(full)[:, -1, :] / temperature
        ref_log_probs.append(Categorical(logits=ref_logits).log_prob(nxt))
        values.append(vals[:, -1])
        full = torch.cat([full, nxt.unsqueeze(-1)], dim=-1)
    _, vf = actor(full); values.append(vf[:, -1])
    return full, torch.stack(log_probs,-1), torch.stack(ref_log_probs,-1), torch.stack(values,-1)

def compute_gae(rewards, values, gamma=1.0, lam=0.95):
    B, T = rewards.shape; adv = torch.zeros_like(rewards); last = torch.zeros(B)
    for t in reversed(range(T)):
        delta = rewards[:, t] + gamma*values[:, t+1] - values[:, t]
        last = delta + gamma*lam*last; adv[:, t] = last
    return adv

def ppo_update(actor, full_tokens, prompt_len, old_log_probs, advantages, returns, opt,
               clip_eps=0.2, vf_coef=0.5, ent_coef=0.01, n_epochs=2):
    for _ in range(n_epochs):
        logits, values = actor(full_tokens)
        comp_logits = logits[:, prompt_len-1:-1, :]
        dist = Categorical(logits=comp_logits)
        new_log_probs = dist.log_prob(full_tokens[:, prompt_len:])
        ratio = (new_log_probs - old_log_probs).exp()
        surr1 = ratio * advantages
        surr2 = ratio.clamp(1-clip_eps, 1+clip_eps) * advantages
        policy_loss = -torch.min(surr1, surr2).mean()
        value_loss = F.mse_loss(values[:, prompt_len-1:-1], returns)
        loss = policy_loss + vf_coef*value_loss - ent_coef*dist.entropy().mean()
        opt.zero_grad(); loss.backward()
        torch.nn.utils.clip_grad_norm_(actor.parameters(), 0.5); opt.step()

The reference implementation in hf-alignment-handbook has more bells (adaptive KL controller, value clipping, advantage normalization), all of which earn their place experimentally.

RLHF with PPO has a characteristic failure mode. Train long enough against a fixed reward model and the policy collapses onto a small set of high-reward strings. ARENA's part 2.4 walks the reader through training a transformer to "maximize output of periods" and observing the policy go from coherent English to spam like "...........................". This is mode collapse, and it is the toy version of the failure mode that matters in production.

The mechanism is reward hacking (Lilian Weng's 2024 post enumerates the taxonomy). The reward model $r_\phi$ is a finite-capacity approximation to "human preferences." It has failure modes. The policy is being optimized against the proxy, not the true objective. This is Goodhart's law ("when a measure becomes a target, it ceases to be a good measure"): the bigger the gap between proxy and true reward, the more the policy exploits.

Three concrete RLHF reward-hacking patterns, all documented in the literature:

1. Sycophancy. The reward model rewards "responses the rater agreed with." The policy learns to mirror the user's stated views regardless of correctness. (Sharma et al., 2023, Towards Understanding Sycophancy in LMs.) Detection: ask the model the same factual question with different framings that imply different "right answers" and see if the answer changes.
2. Length bias. Human raters slightly prefer longer responses. The reward model learns this. The policy learns to pad responses with bullet points and disclaimers. (Singhal et al., 2023, A Long Way to Go.) Detection: regression of reward against length on the held-out set. If the slope is significantly positive, your RM has a length confound.
3. Hallucinated confidence. The reward model rewards "confident-sounding" responses. The policy learns to be confidently wrong rather than appropriately uncertain. Detection: calibration measurement on held-out fact questions.

The KL term in the RLHF objective is the principled mitigation: it pulls the policy back toward the SFT model and bounds the exploitation. But the KL term cannot save you from a bad reward model. If your reward model is systematically wrong, KL just makes the wrong-ness milder. The deeper fix is to keep iterating: collect new preference data on the RLHF-fine-tuned model's outputs, train a fresh reward model, redo PPO. This is "iterated RLHF" and it is what production labs actually do.

DPO (Rafailov et al., 2023) is the most elegant idea in alignment since RLHF. The observation: the KL-regularized RL objective with a reward model has a closed-form optimal policy, and you can use that closed form to fold the reward model out of the algorithm entirely. You train the policy directly on the preference data.

Start with the RLHF objective (KL-regularized expected reward):

The optimal policy satisfies $\pi^*(y \mid x) \propto \pi_\text{ref}(y \mid x) \exp\left(r(x, y) / \beta\right)$. Rearrange to solve for the implicit reward:

The partition function $Z(x)$ cancels out of any preference comparison. Substitute this expression into the Bradley-Terry likelihood and you get the DPO loss:

That is it. One supervised classification loss on preference triples $(x, y_w, y_l)$. No reward model. No PPO. No KL penalty as a separate term: the KL is implicit in the $\pi_\theta / \pi_\text{ref}$ ratio.

trainer = DPOTrainer(
    model=policy, ref_model=ref,
    args=DPOConfig(beta=0.1, output_dir="dpo"),
    train_dataset=ds,             # columns: prompt, chosen, rejected
    processing_class=tokenizer,
)
trainer.train()                   # -logsigmoid(beta*((logpi_w-logref_w)-(logpi_l-logref_l)))

def dpo_loss(policy, ref_policy, prompts, chosen, rejected, beta=0.1):
    pi_chosen   = policy.log_probs(prompts, chosen).sum(-1)
    pi_rejected = policy.log_probs(prompts, rejected).sum(-1)
    with torch.no_grad():
        ref_chosen   = ref_policy.log_probs(prompts, chosen).sum(-1)
        ref_rejected = ref_policy.log_probs(prompts, rejected).sum(-1)
    log_ratio_chosen   = pi_chosen   - ref_chosen
    log_ratio_rejected = pi_rejected - ref_rejected
    margin = beta * (log_ratio_chosen - log_ratio_rejected)
    return -F.logsigmoid(margin).mean()

DPO has practical advantages over RLHF. Implementation is simpler (no rollout, no reward model, no value head). Training is more stable (one supervised loss, no on-policy adversarial dynamics). Compute is lower (no two-model PPO loop). On many benchmarks DPO matches or exceeds RLHF.

Caveats. DPO assumes the preference data is well-modelled by Bradley-Terry, which is not always true (humans are intransitive). DPO can over-reduce the likelihood of all responses, including the chosen ones, if the gradient pushes the policy too far from the reference. The $\beta$ tradeoff is just as delicate as the RLHF $\beta$ tradeoff.

DPO is now the baseline that newer alignment algorithms compete with. A non-exhaustive tour.

IPO (Identity Preference Optimization, Azar et al., 2023) replaces the sigmoid in DPO with a squared-error loss on the implicit reward margin. The motivation: DPO's logistic loss is unbounded, so deterministic preferences ($y_w \succ y_l$ with certainty) push the model's log-ratio to infinity, which is pathological. IPO bounds the margin and is more robust to label noise.

KTO (Kahneman-Tversky Optimization, Ethayarajh et al., 2024) replaces pairwise preferences with single-response binary feedback ("good" or "bad"). This unlocks data sources that do not come as pairs (thumbs up/down from production logs). The loss is built on a prospect-theory utility function, which is one of the rare cases where behavioral economics shows up in ML training.

ORPO (Odds Ratio Preference Optimization, Hong et al., 2024) does away with the reference model. The KL-to-reference term in DPO is replaced with an odds-ratio penalty inside the loss. The result: a single training stage that combines SFT and preference optimization. Practical for shops that do not want to manage two copies of the model.

SimPO (Simple Preference Optimization, Meng et al., 2024) drops the reference model and length-normalizes the log-probabilities. The argument: DPO's length-bias and ref-model dependency are both artifacts of the formulation. SimPO removes them. On AlpacaEval 2, SimPO outperforms DPO with the same data.

The pattern across all five (DPO + IPO + KTO + ORPO + SimPO): each variation tweaks one term of the DPO loss or removes one assumption. Empirically, the differences are smaller than the marketing implies. The right framing: this is a family of preference-optimization losses, each with a different bias-variance-robustness tradeoff, and the choice depends on your data shape and how much you trust your labels.

GRPO (Group Relative Policy Optimization, Shao et al., 2024) is the variant of PPO that DeepSeek used to train DeepSeek-Math and DeepSeek-R1. The observation: PPO's value function is expensive to maintain and noisy on long-form text. Replace it with a group baseline. For each prompt, sample $G$ completions from the current policy. Use the mean reward across the group as the baseline; the per-completion advantage is $(r_i - \bar r) / \text{std}(r)$. Plug this into the PPO clipped objective. No value head needed.

The savings are real. The value network in RLHF is the same size as the policy, doubling GPU memory. GRPO removes it.

RLVR (Reinforcement Learning with Verifiable Rewards) is the framing DeepSeek used for math/code. The reward is not a learned reward model. It is a hard verifier: did the proof check, did the test pass, did the answer match the ground truth. Verifiable rewards eliminate the reward-hacking surface area that comes from preference modeling. You cannot game a Python interpreter into accepting a wrong answer the way you can game a Bradley-Terry classifier. (You can still game the verifier, e.g. by modifying test files. See the coding model learns to change the unit tests example in Weng's reward-hacking taxonomy. The surface area is smaller, not zero.)

GRPO + RLVR is the recipe behind the surprising reasoning capabilities of DeepSeek-R1. The o1-style reasoning models from OpenAI use a closely related procedure. Both make the same bet: if you can get the reward signal right, RL can do a lot more than RLHF gave it credit for.

trainer = GRPOTrainer(
    model=policy,
    reward_funcs=my_reward_fn,    # or a reward model; verifiable reward for RLVR
    args=GRPOConfig(num_generations=8, beta=0.04),  # G completions per prompt
    train_dataset=prompt_ds, processing_class=tokenizer,
)
trainer.train()                   # advantage = (r - group_mean) / group_std, no critic

def grpo_advantages(rewards: torch.Tensor) -> torch.Tensor:
    """rewards: (n_prompts, group_size). Returns same shape, group-normalized."""
    mean = rewards.mean(dim=1, keepdim=True)
    std = rewards.std(dim=1, keepdim=True) + 1e-8
    return (rewards - mean) / std

Reward hacking is the phenomenon that ties this chapter together. Every algorithm above optimizes a policy $\pi_\theta$ against a scalar reward signal $r$, and reward hacking is the gap between what you wrote down as $r$ and what you actually wanted.

Here, the focus is what reward hacking looks like in the training curves and what an RL practitioner can do inside the algorithm to detect and mitigate it.

Three technical signals you should monitor during training:

1. KL divergence to reference policy. RLHF imposes a KL penalty for this reason: it bounds how far the optimized policy can drift from the SFT model. Track $D_\text{KL}(\pi_\theta \| \pi_\text{ref})$ per step. If it spikes mid-training while reward keeps climbing, the policy is exploiting a reward-model loophole.
2. Reward-model overoptimization curves. Train two reward models on the same preference data with different seeds. Track the gap between (a) reward as scored by the model you are optimizing against, and (b) reward as scored by the held-out model. The gap grows as overoptimization kicks in. The held-out model is a noisy oracle but a useful one.
3. Pan et al. (2022) scaling pattern. Across model sizes, the gap between proxy reward and true reward grew with scale on the same training run. The implication: bigger models reward-hack harder. Hold-out-reward-vs-proxy-reward should be in your training dashboard from day one.

Three categories of mitigation that operate at the algorithm level:

• Algorithmic. KL penalties (RLHF, DPO), iterated RLHF, conservative algorithms (CPO, KTO), ensemble reward models that disagree, RLAIF (use a strong LLM as the rater) with the same bias controls as human raters.
• Reward-model-level. More diverse raters, calibration questions, structured rubrics, pairwise consistency checks, reward-model uncertainty (Bayesian or dropout-based), pessimistic estimation.
• Verifiable rewards (RLVR). Where possible, replace the learned reward with a checkable one: unit tests, mathematical proof checkers, formal verifiers. The reward becomes a measurement, not a model. DeepSeek's GRPO + RLVR (§13 above) is the production case study.

You do not need to wait for deceptive-alignment debates to land before adopting the discipline that catches reward-hacking failures early. Three habits to write into the training scaffolding of any RL run.

• Never trust an eval score on a model you trained on the same eval distribution. Even mild contamination (eval prompts in training, eval-similar prompts in training, eval-style preferences in the reward model) breaks the score's meaning. See Ch 23 §10 for the eval-side discipline.
• Add adversarial probes to your eval suite. Questions designed to elicit the failure mode, not the behaviour. The Apollo Research evals and the METR evals are public examples worth borrowing from.
• Separate the people optimizing capability metrics from the people validating safety. Anthropic and OpenAI both run separate red-team groups for this reason. The full red-team operational discipline, plus the deceptive-alignment debate it grew out of, lives in Ch 24 §13.

The algorithms in sections 1–13 are good at optimizing what you tell them to optimize. The what you tell them part is the bottleneck, and the discipline of telling them the right thing is half of what another chapter is about.

For length, this chapter skipped: model-based RL (PlaNet, Dreamer, MuZero), inverse RL, imitation learning beyond plain SFT, hierarchical RL, meta-RL, offline RL (CQL, IQL), exploration bonuses (RND, ICM), and the entire multi-agent RL subfield. Three pointers:

• Sutton & Barto, chapters 8 and 17, for model-based RL.
• Lilian Weng's 2018-04-08-policy-gradient for the full algorithmic zoo (SAC, TD3, IMPALA, MPO).
• Berkeley CS285 for the lecture-quality treatment of everything else.