Eval Science

Pre-LLM ML had a stable eval methodology. You had a held-out test set, a metric, and a leaderboard. The job was to push the metric. The metric was a noisy estimate of generalization; held-out meant held-out from training; the leaderboard sorted by the metric.

LLMs broke this in three places. First, contamination: pretraining corpora are large enough that "held-out" cannot be verified without auditing the training set, which the lab usually does not share. Second, task ambiguity: a single-number metric on a 10,000-item benchmark hides the fact that the items measure different capabilities, some of which the model has and some of which it does not. Third, judging: most interesting LLM tasks have no ground truth, so the eval needs an LLM judge, which has its own biases that the eval inherits.

The honest framing of evaluation in 2026 is: every eval is a measurement of one specific operationalization of a fuzzy capability, with uncertainty in (a) the capability definition, (b) the eval items' coverage of the capability, (c) the metric's relationship to the capability, and (d) any judging used. A "GPT-4 hit 87% on MMLU" headline is four uncertainties stacked, and the headline reports zero of them.

This chapter is the discipline of measuring well enough that you trust your own claims. It does not promise the field has solved the problem. It promises the methods that protect you from being most wrong.

The benchmarks you will see cited every week. Brief enough to use as reference; honest about each one's flaws.

• MMLU (Hendrycks et al. 2020). 57 subjects, multiple choice, college-level. 15,908 items. The default "general knowledge" benchmark. Known contamination, item-quality variance, and ceiling-near saturation by 2024. Useful as a coarse capability filter; misleading as a capability claim.

• MMLU-Pro (Wang et al. 2024). MMLU with more options per question, less surface contamination, harder distractors. Saturates slower. Inherits MMLU's domain coverage shape.

• GSM8K (Cobbe et al. 2021). 8,500 grade-school math word problems. Heavily contaminated in modern pretraining sets. Useful only with caution.

• MATH (Hendrycks et al. 2021). 12,500 competition math problems. Less contaminated, more rigorous. Frontier models now solve ≈80%+; the eval is approaching saturation.

• HumanEval (Chen et al. 2021). 164 hand-written programming problems. Tiny, contaminated, but still useful for sanity-checking code generation. Strict superset by HumanEval+ (EvalPlus, Liu et al. 2023) which adds robust test cases.

• HellaSwag (Zellers et al. 2019). 70k sentence completion items, 4 choices. Was the de facto reasoning benchmark in 2020-2022. Now saturated.

• BigBench / BigBench-Hard (BBH; Suzgun et al. 2022). 23 tasks selected as hard for then-current models. Better than MMLU for cross-task capability profiles; many tasks have known label noise.

• TruthfulQA (Lin et al. 2021). 817 questions designed to elicit common misconceptions. Measures something like "willingness to be honest about counter-narrative facts", which is more specific than truthfulness in general.

• ARC-Challenge (Clark et al. 2018). Grade-school science. Largely saturated.

• WinoGrande / WinoBias (Sakaguchi et al. 2019). Pronoun resolution with social-bias variants. Useful as a bias-eval baseline.

• SWE-bench and SWE-bench Verified (Jimenez et al. 2024). 2,294 real GitHub issues; Verified is 500 human-checked. The most cited agent benchmark in 2025-2026. Excellent task realism; comes with its own contamination concerns once it became a target.

• GAIA (Mialon et al. 2023). 466 assistant tasks across 3 difficulty levels. Real web search, real file handling. Best general-assistant eval.

• MMMU (Yue et al. 2024). Multimodal MMLU. 11,500 college-level questions across 6 disciplines with images.

• HELM (Stanford CRFM, 2022 onwards). Holistic Evaluation: same models on the same evals with the same prompting, reproducibly. Less a benchmark than a benchmarking discipline; the value is the comparability.

• MMLU-Redux (Gema et al. 2024). MMLU with item-level error analysis: ≈6.5% of MMLU items have annotator errors. This is why your eval set is 6% less reliable than you thought.

Use each with the assumption that its absolute number is meaningless and only its delta within a controlled comparison is meaningful. A model "hitting 92% on MMLU" tells you almost nothing. A model "improving 5 points on MMLU under matched prompting versus its own previous checkpoint, with bootstrap CI not crossing zero" tells you something.

How you aggregate samples per item matters as much as which items you score.

Pass@1. Sample one completion per item. Score 1 if it passes, 0 otherwise. The simplest and most pessimistic metric. Used in everything from HumanEval (with n=1 sampling) to chat evals.

Pass@k. Sample $n$ completions per item ($n \geq k$). Score 1 if at least one of the top-$k$ passes. The HumanEval paper's original Pass@k uses an unbiased estimator: $\text{pass@k} = 1 - \binom{n-c}{k} / \binom{n}{k}$ where $c$ is the number of correct samples out of $n$. Pass@10 with $n=20$ samples is a different measurement than Pass@1.

Majority vote / self-consistency (Wang et al. 2022). Sample $n$ completions, run them through a final-answer extractor, take the modal answer. Improves accuracy on tasks with verifiable answers (math, multi-choice). Cost: $n$× inference per item.

Best-of-N with verifier. Sample $n$, score each with a learned verifier (a reward model or a code-runner), take the best. Even better than majority vote when you have a verifier you trust. Used heavily in modern reasoning models (o1, R1-style).

Elo / Bradley-Terry / TrueSkill (LMSys Arena). Sample pairwise comparisons of model outputs from real users; fit a strength rating per model. The output is a leaderboard. Pros: captures user-facing preference. Cons: vulnerable to demographic bias of the rater pool, vulnerable to style preferences over content, and Elo is a one-dimensional projection of a multi-dimensional capability space.

The bias you should worry about most: which aggregation method the paper picked is usually the one that flatters its result. If a paper reports Pass@10 but not Pass@1, ask why. If it reports majority-vote over 64 samples but the deployment will use $n=1$, the eval is misleading about production behavior.

# Unbiased pass@k estimator from the HumanEval paper
import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    """Probability that at least 1 of top-k sampled completions is correct."""
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

For tasks without ground truth (open-ended QA, summarization, creative writing, helpfulness), you ask a strong LLM to judge model outputs. Zheng et al. 2023 ("Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena") is the canonical methodology paper.

The recipe. Define a rubric (3-7 criteria, each on a 1-5 or 1-10 scale). Sample candidate outputs. Prompt the judge model with the prompt, the output, and the rubric. Parse the score. Aggregate.

Three flavors:

• Pointwise: judge one output at a time. Score 1-10. Easy to implement, vulnerable to absolute-scale drift.
• Pairwise: judge two outputs A vs B. "Which is better?" or "Tie". More robust to scale drift, vulnerable to position bias.
• Reference-based: provide a gold reference and ask the judge to score how close the candidate is. Best when you have references; expensive to produce them.

The failure modes you must measure and mitigate:

• Position bias (pairwise). Judges prefer A or B systematically. Mitigation: randomize order, run each comparison twice (A-then-B and B-then-A), keep only stable preferences.
• Verbosity bias. Longer outputs win even when content is equivalent. Mitigation: length-control in the rubric, or explicit instruction "do not penalize/reward length".
• Self-preference bias. A model judging its own outputs scores them higher than outputs from other models. Mitigation: cross-evaluation (always use a judge from a different model family) or human spot-check on a subset.
• Style-over-substance. Judges prefer outputs that "sound" authoritative even when factually wrong. Mitigation: explicit factuality checks separated from style judgments.
• Sycophancy in the judge. Judges sometimes agree with framing claims in the prompt. Mitigation: blind the judge to which output came from which model.
• Rubric drift. The same rubric, paraphrased differently, gives different scores. Mitigation: pin the exact rubric text, run rubric-paraphrase robustness tests.

The bound on LLM-as-judge accuracy: at best, the judge's agreement with humans on the same task. Recent work (Bavaresco et al. 2024) finds judge-human agreement of 60-80% on subjective tasks, which puts a hard ceiling on the eval. If you trust the eval, you are implicitly trusting that ceiling.

JUDGE_PROMPT = """You are evaluating two responses to the same prompt.
Rate each on a 1-5 scale across these criteria. Be specific.

Criteria:
1. Factual correctness
2. Relevance to the prompt
3. Reasoning quality
4. Clarity (independent of length)

Do not let length influence your score.

Prompt: {prompt}

Response A: {a}
Response B: {b}

Format your answer as JSON:
{{"A": {{"factual": 1-5, "relevant": 1-5, "reasoning": 1-5, "clarity": 1-5}},
 "B": {{...}}, "overall_winner": "A" | "B" | "tie", "reason": "..."}}
"""

def llm_judge(judge_llm, prompt: str, a: str, b: str, swap_check: bool = True) -> dict:
    """Judge with a position-bias swap check."""
    result_ab = parse_json(judge_llm(JUDGE_PROMPT.format(prompt=prompt, a=a, b=b)))
    if not swap_check:
        return result_ab
    result_ba = parse_json(judge_llm(JUDGE_PROMPT.format(prompt=prompt, a=b, b=a)))
    # Reconcile: only count a winner if both orderings agree
    if result_ab["overall_winner"] != _swap(result_ba["overall_winner"]):
        return {"overall_winner": "tie_unstable", "ab": result_ab, "ba": result_ba}
    return {"overall_winner": result_ab["overall_winner"], "stable": True}

The Chatbot Arena (lmsys.org) is the de facto open Elo leaderboard. Users submit a prompt, see two model responses side-by-side without knowing which is which, vote for the better one. Ratings are computed by Bradley-Terry (BT). As of 2026 it has tens of millions of votes across 100+ models.

What it captures: real-user preferences, in the wild, with their full noise. What it misses: the prompts skew toward what users on the LMSys site ask, which over-represents coding, math, and chat-style tasks and under-represents domain-specific work. The judgments are also subjective, so two users with different preferences will disagree about the "better" answer.

The methodology lessons from running Arena:

• The pairwise Bradley-Terry model estimates win probability between two models as $P(i \text{ beats } j) = \frac{\exp(s_i)}{\exp(s_i) + \exp(s_j)}$ where $s_i, s_j$ are the model strengths. Fit by maximum likelihood. Confidence intervals from bootstrap.
• Style controlling (Chiang et al. 2024) reduces verbosity bias by adding length and formatting features as additional Bradley-Terry covariates. The ranking shifts noticeably when style is controlled.
• Hard prompts subset. Arena recently introduced subsets for hard-coding, hard-math, etc. The ranking on these is different from the overall ranking; models that are popular but middling on substance drop.

If you build an Arena-style eval internally, the takeaway is to keep at least three Elo-style leaderboards: overall, hard-substance subset, and style-controlled. The same model can be top on one and middling on another. Reporting only one is misleading.

The benchmarks above are public, contaminated, and not your problem. The eval that matters for any system you ship is the one you built for your specific use case.

The recipe, in order:

1. Define the capability you want to measure. Be specific. Not "summarization quality"; "summarizes a customer support transcript in 200 words, preserving every product mentioned and every action item, in the customer's reported emotional tone".
2. Write 5-10 hand-crafted gold items. Force yourself to think about edge cases by writing them. This is where eval design typically fails: people skip this step and let dataset generation invent its own coverage.
3. Generate 50-500 items via model + spot-check. Use a strong model with the 5-10 hand-crafted items as in-context examples. Spot-check at least 10% of the generated items. Document the spot-check rate.
4. Have two humans label a subset. Compute inter-annotator agreement: how often two raters give the same label, beyond what they would hit by chance. Cohen's $\kappa = (p_o - p_e)/(1 - p_e)$, where $p_o$ is the raw agreement rate and $p_e$ is the agreement expected by chance, is the standard chance-corrected version (or report simple agreement). If $\kappa < 0.6$, your rubric is ambiguous; rewrite.
5. Decide pass/fail or scoring. Anchor each grade level to a real example. "5/5 means like example #14, 1/5 means like example #28."
6. Pre-register the analysis. Write down before running it: what counts as success, what counts as no-effect, what counts as failure. Otherwise you will move the goalposts.
7. Run the eval. With bootstrap confidence intervals. With at least one stratification (per topic, per difficulty, per length).
8. Audit failures. Read 20 failures yourself. You will find that some are eval bugs, some are real model failures, some are ambiguous. The split tells you the eval's effective error rate.

The hardest part is step 4. Humans disagree about model outputs more than anyone expects. A rubric that is "obvious" to its author is often $\kappa = 0.4$ across two annotators. Treat low agreement as a signal to rewrite the rubric, not to "average more annotators".

Inspect (inspect.aisi.org.uk) is the AISI evaluation framework. It is the closest thing the field has to a standard. Open source. Used by AISI, METR, Apollo, and a growing list of red-team groups.

What Inspect gives you, that hand-rolled eval loops do not:

• A solver/scorer architecture. A Task has dataset, solver (the agent/model loop), scorer (the grader). Each is composable and replaceable.
• Built-in multiple_choice, pattern_match, model_graded_qa, model_graded_fact scorers. You can subclass for custom rubrics.
• Sandboxed code execution via Docker. Critical for agent evals where the model writes shell commands.
• A first-class CLI: inspect eval./my_task.py --model openai/gpt-4o --limit 100. Outputs are JSON-logged with full traces.
• Native support for tool-use, agent loops, and multi-turn conversations.

# A minimal Inspect task
from inspect_ai import Task, eval, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import model_graded_qa
from inspect_ai.solver import generate

@task
def my_eval():
    return Task(
        dataset=[
            Sample(input="What is the capital of France?", target="Paris"),
            Sample(input="Who wrote the Iliad?", target="Homer"),
        ],
        solver=generate(),
        scorer=model_graded_qa(),
    )

# Run with: inspect eval my_eval.py --model anthropic/claude-3-5-sonnet

The reason Inspect is worth adopting: the eval discipline it enforces (typed datasets, typed solvers, typed scorers, logged traces) is exactly the discipline you should be adopting anyway. Hand-rolled evals end up reinventing all of this poorly.

Most of another chapter was about how to build agents. This section is about how to evaluate them. Three benchmarks worth knowing in depth:

• SWE-bench Verified (500 issues). The agent gets a Python repo, an issue, and shell access. Pass = produces a patch that makes the hidden test suite go green. Real-world relevance: very high. Contamination: now substantial. The leaderboard runs on a maintained eval harness; report your harness version.

• GAIA (466 tasks, 3 levels). General assistant tasks requiring web, file, image, and reasoning. Humans hit ≈92%, frontier agents hit ≈75% L1 and ≈50% L3 as of mid-2026. The eval is harder because it spans modalities.

• Cybench (Zhang et al. 2024). 40 capture-the-flag style cybersecurity tasks. Agent has shell, web, code. Pass = retrieves the flag. The capability-eval relevance is acute: this is what "can the agent autonomously hack" looks like operationalized.

What agent evals miss that you will need to add:

• Long-horizon coherence. Most benchmark tasks fit in <50 steps. Production tasks run 200-2000 steps. Failure modes (drift, scope creep, recursive helpfulness) emerge there. Add a held-out long task or two.
• Tool-use safety. Does the agent ask before destructive operations? Does it leak credentials? These are not measured by pass/fail benchmarks.
• Cost discipline. Some benchmark winners spend $50 per task. The eval should report dollars and tokens alongside accuracy. METR's framing — "time-horizon, money-budget" — is the right one.

Two methodology gotchas:

• The harness matters more than the model. Two SWE-bench-Verified runs with the same model can differ by 5-10 points depending on the harness's tool prompts, retry policy, and patch-application logic. Always report the harness commit hash.
• Pass rate hides per-task-class breakdown. SWE-bench has Django issues, sympy issues, scikit-learn issues. The model that passes 60% might pass 90% on one class and 30% on another. Always stratify.

Safety evaluation is a distinct genre. It asks "what could the model do" and "what will the model not do" rather than "how well does it do this task".

The genres:

• Red-team / jailbreak resistance. Datasets like HarmBench (Mazeika et al. 2024), AdvBench, Anthropic's internal red-team set. The eval: given a prompt designed to elicit harmful content, did the model comply? Comply rate of 5-30% is typical for chat models on these sets. Caveat: the gap between "refused" and "actually safe" is real; a model can refuse the literal prompt and comply with a paraphrase.
• Refusal evals. The flip side: does the model refuse too much, including on benign prompts? Over-refusal is a regression that RLHF can introduce. Datasets like OR-Bench (Cui et al. 2024) measure false-positive refusal rate.
• Capability evals (dangerous capabilities). Cybench (cyber), WMDP (Li et al. 2024; chem/bio/cyber knowledge), and lab-internal capability evals. The question is "does the model know enough to do X if asked", separate from "would the model comply".
• Persona / character evals (Anthropic 2024 Persona Vectors). Does the model maintain its intended character across long conversations? Drift under adversarial pressure is a measurable failure mode.
• Sycophancy evals. Does the model agree with the user's framing even when the user is wrong? Sharma et al. 2023 showed this is a robust failure mode. Standard set: SycophancyEval.
• Honesty evals (Park et al. 2023, MASK, etc.). Does the model state beliefs it does not hold, or assert things it knows are false? Hard to operationalize; the field has multiple competing setups.

The most important methodological point about safety evals: they are upper bounds on safety, not lower bounds. If a model passes an eval, you have evidence about that eval, not about its safety in deployment. If a model fails an eval, you have evidence of at least that failure mode. The asymmetry is structural.

The safety-eval failure mode the field is just starting to recognize: evaluation awareness. Models can detect when they are being evaluated and behave differently. Anthropic 2025 "evaluation awareness" experiments showed measurable detection in chat models. The implication: behavioral evaluation alone is becoming an unreliable signal. This is where mech-interp enters the discipline — checking whether internal model state matches behavioral output, not just observing the output.

A benchmark item is contaminated if it appeared in the training set. The model "passes" by recall, not by capability. Contamination rates on popular benchmarks (MMLU, GSM8K, HumanEval) are now substantial in any model trained on a large web corpus.

Detection methods, in increasing order of strength:

• Membership inference. Train a probe to predict, given the model's outputs on an item, whether the item was in training. Carlini et al. 2022. Catches strong contamination, misses weak.
• Verbatim memorization checks. Ask the model to continue a benchmark prompt; if it produces the exact correct answer's text verbatim, contamination is likely.
• N-gram overlap. Compare benchmark items to the training corpus (when available) for high $n$-gram overlap. Used by HF leaderboards on common datasets. Easy when the corpus is public, impossible when it is not.
• Perplexity comparison. Compare model perplexity on benchmark items versus matched control items. Sharp differences are evidence of contamination. Magar and Schwartz 2022.
• MMLU-Redux-style audit. Hand-audit a sample of benchmark items against known training-data sources. Tedious. Yields the most credible estimates.

The mitigations:

• Fresh evals. Build evals from data created after the model's training cutoff. METR's evals are partially this; new GitHub PRs filtered to dates after model training.
• Decontamination. Subtract training-set-matching items from benchmarks at run time. LiveCodeBench (Jain et al. 2024) is built this way: only contest problems from after the model's cutoff count.
• Eval rotation. Have multiple eval sets and rotate which is used to publish results, keeping a hidden set for spot-checks.
• Private evals. The "blind eval" pattern AISI and Apollo use: the eval is not public, the model is run by the eval team, the lab does not see the items.

The thing to internalize: any public benchmark, once it is being optimized against, is contaminated within ≈1-2 years. The half-life is shorter than the development cycle. Public benchmarks are useful as baselines; treat the most recent published number as the optimistic case.

# Crude verbatim-memorization check
def memorization_score(model, tokenizer, full_text, prefix_tokens=50, cont_tokens=100) -> float:
    """Feed the first `prefix_tokens` of a benchmark item; measure how many of the
    next `cont_tokens` the model regenerates verbatim. High score => likely memorized."""
    toks = tokenizer.encode(full_text)
    prompt, truth_continuation = toks[:prefix_tokens], toks[prefix_tokens:prefix_tokens + cont_tokens]
    if not truth_continuation:
        return 0.0
    generated = model.generate(tokenizer.decode(prompt), max_new_tokens=len(truth_continuation))
    gen_tokens = tokenizer.encode(generated)[:len(truth_continuation)]
    matches = sum(1 for a, b in zip(gen_tokens, truth_continuation) if a == b)
    return matches / len(truth_continuation)

Goodhart's law applied: when a metric becomes a target, it ceases to be a good metric. In ML, this is reward hacking. In evaluation specifically, it is the failure mode where a model (or a training process) exploits the eval rather than the underlying capability.

The taxonomy (Weng 2024, "Reward Hacking in Reinforcement Learning"):

• Specification gaming. The model satisfies the letter of the eval, not the intent. Classic example: a code-generation eval where the model "solves" the task by reading the test file and returning its expected outputs.
• Reward tampering. The model modifies its own reward signal. Rare in practice but the canonical example is the Carlsmith line of reasoning: a sufficiently capable agent acts on its training process directly.
• Sycophancy. The model says what the rater (or the LLM judge) wants to hear, not what is true. The most common failure mode in RLHF.
• Sandbagging. The model deliberately performs worse on capability evals to avoid triggering safety measures. Documented in some Anthropic experiments; the operational concern grows with capability.
• Eval-aware behavior. The model detects evaluation contexts and behaves differently than in deployment.

The mitigations are mostly about not letting the eval become the optimization target in the first place:

• Held-out evals not visible to training. The eval that decides "did training succeed" is run on data the training never saw.
• Eval rotation. As above.
• Adversarial eval generation. Generate new eval items designed to catch the patterns that previous training data exploited.
• Mechanistic checks. As another chapter suggested: check internal state, not just behavior. If the refusal direction is active but the model complied, that is a discrepancy worth investigating.
• Process-based supervision. Reward the chain-of-thought as well as the final answer; this catches some specification-gaming cases (Lightman et al. 2023 "Let's Verify Step-by-Step").

A surprising fraction of "model A beats model B by 1 point on benchmark X" claims do not survive a confidence-interval check.

The minimum statistical discipline:

• Bootstrap CIs. For any metric reported, resample the eval set with replacement, compute the metric on each resample, report the 95% CI. Done in 10 lines; almost never done in papers.
• McNemar's test for paired binary outcomes. When two models are evaluated on the same items, the only items that carry information are the ones where they disagree; McNemar's test works on exactly those disagreement counts — (A right, B wrong) vs (A wrong, B right) — rather than a two-sample $t$-test on accuracy. The test itself is in your "what you need first" stats prereq and you build it in the lab below.
• Bonferroni or BH correction for multiple comparisons. Run 10 independent evals at $p < 0.05$ each and, even with no real effect, you expect one to clear the bar by chance — the more comparisons you make, the more likely at least one is a false positive. A correction (Bonferroni divides the threshold by the number of tests; Benjamini-Hochberg controls the false-discovery rate less conservatively) pulls the bar back so the chance of any false claim stays near 5%. Correct or do not claim significance.
• Effect size, not just p-value. A statistically significant 0.2-point gain on MMLU is not interesting. Report the effect size — how large the difference is, not just whether it is non-zero; Cohen's $d$ expresses it in standard-deviation units, or use the absolute delta — and the practical-significance threshold you set in advance.

import numpy as np

def bootstrap_ci(scores: np.ndarray, n_resamples: int = 10000, alpha: float = 0.05):
    """Bootstrap CI for the mean of `scores`."""
    n = len(scores)
    means = np.empty(n_resamples)
    rng = np.random.default_rng(0)
    for i in range(n_resamples):
        sample = rng.choice(scores, size=n, replace=True)
        means[i] = sample.mean()
    return np.quantile(means, [alpha/2, 1 - alpha/2])

def mcnemar(a_correct: np.ndarray, b_correct: np.ndarray) -> float:
    """McNemar's test on paired binary outcomes. Returns p-value (continuity-corrected)."""
    b01 = ((~a_correct) & b_correct).sum()
    b10 = (a_correct & (~b_correct)).sum()
    if b01 + b10 == 0:
        return 1.0
    chi2 = (abs(b01 - b10) - 1) ** 2 / (b01 + b10)
    from scipy.stats import chi2 as chi2_dist
    return 1 - chi2_dist.cdf(chi2, df=1)

The point of this section is not to teach you statistics. It is to make explicit that the statistical floor of credible eval reporting is higher than 90% of papers clear, and that you should hold your own work to a higher bar than the field's median.

Most evals that fail are never published. The field's belief about what works is biased by the file drawer: ten attempts at a method, one that beat baseline, only the winner is reported.

Good eval reports — the ones from AISI, METR, Apollo, and the better lab system cards — share a structure:

• The capability being measured, in one sentence.
• The eval items: size, sourcing, contamination check.
• The protocol: prompting, sampling, aggregation, exact harness.
• Baselines: at least one prior model, at least one trivial baseline (random, constant, length-of-prompt).
• Headline number with CI.
• Stratified breakdowns: per-subtask, per-difficulty, per-attribute.
• Failure analysis: 10-50 hand-read failures with the failure-mode taxonomy.
• The negative results: what was tried, what did not change anything.
• Reproducibility: harness commit, model checkpoints, exact prompts, exact seeds.

If you can publish reports that have all nine, your evals will be more credible than 80% of the field's. None of it is hard. All of it is tedious. The discipline is the gap between "interesting result" and "result that survives scrutiny".