{ "cells": [ { "cell_type": "markdown", "id": "c9277478", "metadata": {}, "source": [ "# Ch 23 — Eval Science (notebook)\n", "\n", "`[← 22 mechanistic-interpretability]` · **this notebook** · `[24 safety-and-red-team →]`\n", "\n", "Runs top-to-bottom in ~1 min on free Colab CPU. Last verified 2026-06-11.\n", "\n", "**What you'll build**\n", "- A synthetic eval where model A beats model B by 5 points, then a paired bootstrap confidence interval that eats the gap: the CI on the difference crosses zero, and the \"+5 leaderboard win\" turns out to be noise.\n", "- The unbiased pass@k estimator from the HumanEval paper, and a demonstration that pass@10 and pass@1 are different measurements of the same samples.\n", "- A mock LLM judge with a planted position bias, and the swap check that catches it: keep only the preferences that survive showing A first *and* second.\n", "- A deliberate failure where the headline metric improves while held-out loss gets worse, the loss-vs-metric divergence that ships a worse model.\n", "- Cohen's kappa on a custom eval, McNemar's test for paired model comparisons, and an experiment ledger that records every change with its CI.\n", "\n", "**How this notebook works.** Code cells with a `# TODO` are yours to fill in. Run the cell to grade yourself: `[ ok ]` passed, `[FAIL]` shows what went wrong, `[ -- ]` means not attempted yet. Every exercise has a hint ladder (open only as many as you need) and a folded solution below it. The notebook runs top-to-bottom even if you fill in nothing, because the folded solutions redefine the pieces the later cells need. There is no API key and no network call anywhere: the \"LLM judge\" is a small deterministic Python function whose biases we control, so every claim about judging is reproducible. See Ch 00 for the full protocol.\n" ] }, { "cell_type": "markdown", "id": "d032ca8f", "metadata": {}, "source": [ "## Before you start\n", "\n", "Three probes. The answers are in the dropdowns; they set up the three ideas this notebook leans on hardest. They are answerable after Ch 02 (where the bootstrap test-set discipline was introduced) and Ch 03 (metrics).\n", "\n", "1. Model A scores 87% and model B scores 82% on the same 200-item benchmark. Is A better than B?
AnswerYou cannot tell yet. 87% on 200 items is `174/200`; 82% is `164/200`. A ten-item gap on 200 items is well inside the sampling noise of either estimate. The whole point of Part 2 is that the honest answer is \"report a confidence interval on the *difference* and see whether it crosses zero\", and on a set this small a 5-point gap usually does. The leaderboard shows you a point estimate and hides the uncertainty.
\n", "2. You ask an LLM judge \"which is better, response A or response B?\" and it says A. You then swap the order and ask again about the same two responses; now it says B. What does that tell you?
AnswerThe judge has *position bias*: its preference is driven by *where* the response sits, not by its content. The single most important mitigation for pairwise LLM-as-judge is to run every comparison in both orders and keep only the preferences that are stable across the swap. A \"win\" that flips when you swap the order is not a win; it is an artifact of the prompt layout.
\n", "3. Predict before you run: your training loss is going down every epoch, but the metric you actually care about (held-out task accuracy) starts going *down* too after some point. Which number do you trust, and what is happening?
AnswerTrust the held-out metric, not the loss. A falling training loss with a falling held-out metric is the signature of overfitting: the model is getting better at the *proxy* (the loss on the data it sees) and worse at the *target* (the capability on data it does not). The loss is a surrogate; the metric is the thing. Part 5 stages this divergence and shows how selecting on the loss ships the worse model.
\n" ] }, { "cell_type": "markdown", "id": "3f04671e", "metadata": {}, "source": [ "## Setup\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "2abb1b35", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:41.373958Z", "iopub.status.busy": "2026-06-10T20:48:41.373866Z", "iopub.status.idle": "2026-06-10T20:48:42.208584Z", "shell.execute_reply": "2026-06-10T20:48:42.208132Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "numpy 2.2.6 · scipy 1.15.3\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "torch 2.12.0+cpu (used only for the optional loss-vs-metric demo)\n" ] } ], "source": [ "import numpy as np\n", "import scipy\n", "import matplotlib.pyplot as plt\n", "print(f\"numpy {np.__version__} · scipy {scipy.__version__}\")\n", "if np.__version__ < \"2.0\":\n", " print(\"WARN: written for NumPy 2.x; older versions may shift the last digit or two\")\n", "try:\n", " import torch\n", " HAVE_TORCH = True\n", " print(f\"torch {torch.__version__} (used only for the optional loss-vs-metric demo)\")\n", "except Exception:\n", " HAVE_TORCH = False\n", " print(\"torch not found: the one optional torch cell will print-and-skip; nothing else needs it\")" ] }, { "cell_type": "code", "execution_count": 2, "id": "604045c6", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.209877Z", "iopub.status.busy": "2026-06-10T20:48:42.209738Z", "iopub.status.idle": "2026-06-10T20:48:42.217704Z", "shell.execute_reply": "2026-06-10T20:48:42.217362Z" } }, "outputs": [], "source": [ "import os, math, random\n", "SEED = 0\n", "FAST = bool(os.environ.get('NB_FAST')) # CI smoke mode: ~10x fewer resamples, same code paths\n", "N_BOOT = 2000 if FAST else 20000 # bootstrap resamples; CI width is stable for both (see ledger)\n", "N_JUDGE = 60 if FAST else 200 # mock-judge comparisons per condition\n", "rng = np.random.default_rng(SEED)\n", "random.seed(SEED)\n", "\n", "# ── house self-check harness (identical across all chapter notebooks) ──\n", "import numpy as _np\n", "\n", "def check(label, test_fn, required=False):\n", " \"\"\"Run one self-check. test_fn raises AssertionError (with a teaching\n", " message) on failure, NotImplementedError if the stub is unfilled.\n", " required=True is used only in solution cells; it is what CI grades.\"\"\"\n", " try:\n", " test_fn()\n", " except NotImplementedError:\n", " if required:\n", " raise AssertionError(f\"{label}: reference solution incomplete\")\n", " print(f\"[ -- ] {label}: not attempted yet — fill in the TODO above, then re-run.\")\n", " return False\n", " except AssertionError as e:\n", " if required:\n", " raise\n", " print(f\"[FAIL] {label}: {e}\")\n", " return False\n", " print(f\"[ ok ] {label}\")\n", " return True\n", "\n", "def attempted(*vals):\n", " \"\"\"Treat None placeholders as 'not attempted'.\"\"\"\n", " if any(v is None for v in vals):\n", " raise NotImplementedError\n", "\n", "def check_shape(x, want):\n", " assert tuple(x.shape) == tuple(want), \\\n", " f\"shape {tuple(x.shape)}, expected {tuple(want)} — check your reshape/transpose order\"\n", "\n", "def check_close(got, want, atol=1e-5, rtol=1e-4, msg=\"\"):\n", " g, w = _np.asarray(got, dtype=float), _np.asarray(want, dtype=float)\n", " assert g.shape == w.shape, f\"shape {g.shape} vs expected {w.shape}. {msg}\"\n", " bad = ~_np.isclose(g, w, atol=atol, rtol=rtol)\n", " assert not bad.any(), \\\n", " f\"{bad.mean():.2%} of values wrong (max diff {abs(g - w).max():.3g}). {msg}\"\n", "\n", "# ── small house helpers (defined here, never imported) ──\n", "def fmt_ci(lo, hi):\n", " \"\"\"Render a confidence interval, flagging whether it crosses zero.\"\"\"\n", " crosses = ' (crosses 0 -> not significant)' if lo <= 0 <= hi else ''\n", " return f'[{lo:+.3f}, {hi:+.3f}]{crosses}'\n", "\n", "def err_bars(ax, xs, means, los, his, **kw):\n", " # viz helper: point estimate with asymmetric CI error bars\n", " lo_err = np.asarray(means) - np.asarray(los)\n", " hi_err = np.asarray(his) - np.asarray(means)\n", " ax.errorbar(xs, means, yerr=[lo_err, hi_err], fmt='o', capsize=5, **kw)" ] }, { "cell_type": "markdown", "id": "093327a3", "metadata": {}, "source": [ "> **Note:** seeds make this notebook's printed numbers reproduce on CPU. Library versions and BLAS threading can shift the last digit or two; quoted numbers hold for the pinned environment. If a CI is `[-0.013, 0.114]` and the page says `[-0.012, 0.115]`, you did nothing wrong. The *structural* claims, that the +5 gap's CI crosses zero, that the swap check flags the biased judge, are robust by construction, not by a magic threshold.\n" ] }, { "cell_type": "markdown", "id": "2ea08f20", "metadata": {}, "source": [ "## The map\n", "\n", "> **Part 1 — The number that lies.** Generate two models' per-item eval scores on a shared set. Read the raw means: A beats B by 5 points. Hold that number; we spend the rest of the notebook deciding whether to believe it.\n", "> **Part 2 — The bootstrap CI that eats the gap.** Build the percentile bootstrap from scratch, check it against `scipy`, then put a CI on the *paired difference* A − B and watch it cross zero. Confirm with McNemar's test.\n", "> **Part 3 — Pass@k and the metric you choose.** Implement the unbiased pass@k estimator; show that pass@10 and pass@1 rank the same samples differently, and that the metric a report picks is usually the one that flatters it.\n", "> **Part 4 — LLM-as-judge, and position bias.** Build a deterministic mock judge with a planted position bias, measure the bias, and implement the swap check that keeps only order-stable preferences.\n", "> **Part 5 — Loss is not the metric (a deliberate failure).** Stage a run where training loss falls while the held-out metric falls too; select on the loss and ship the worse model; then select on the metric and fix it.\n", "> **Part 6 — Custom evals: kappa and the ledger.** Cohen's kappa for inter-annotator agreement, and an experiment ledger that records every change with its CI, the artifact that makes an eval defensible.\n" ] }, { "cell_type": "markdown", "id": "924fe5fd", "metadata": {}, "source": [ "## Part 1 — The number that lies\n", "\n", "> **Objectives.** Generate per-item scores for two models on one shared eval set, with a *known* ground-truth difference baked in, and read the raw leaderboard number. The synthetic data has a real answer (we set the true accuracies ourselves), so \"is the gap real?\" is a question we can grade.\n", "\n", "A leaderboard shows you a point estimate: model A, 87.0; model B, 82.0; A wins by 5. What the leaderboard hides is that each of those numbers is an *estimate* from a finite eval set, and the gap between two estimates has its own uncertainty. We start by building the data the leaderboard summarizes: for each of `N_ITEMS` eval items, a 0/1 score for model A and a 0/1 score for model B.\n", "\n", "We set the *true* per-item success probabilities ourselves. The honest experiment is one where A and B are genuinely close: true 0.85 vs true 0.83, a real 2-point edge for A. A finite eval will, by luck, sometimes show a much larger gap. That is the whole lesson.\n", "\n", "> **Predict:** with 200 items and true accuracies 0.85 and 0.83, roughly how wide is the 95% sampling interval for *one* model's measured accuracy?
AnswerAbout ±0.05. The standard error of a proportion is $\\sqrt{p(1-p)/n} \\approx \\sqrt{0.85 \\cdot 0.15 / 200} \\approx 0.025$, and the 95% interval is roughly ±2 SE ≈ ±0.05. So each model's measured number wobbles by ±5 points just from which 200 items you happened to draw, which is *larger* than the 2-point true gap. A 5-point measured gap is well within reach of pure noise.
\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "e0ebb6bf", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.218665Z", "iopub.status.busy": "2026-06-10T20:48:42.218540Z", "iopub.status.idle": "2026-06-10T20:48:42.483481Z", "shell.execute_reply": "2026-06-10T20:48:42.482721Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "eval set: 200 items, shared difficulty (a hard item tends to fail BOTH models)\n", "model A measured accuracy: 0.860 (true 0.85)\n", "model B measured accuracy: 0.805 (true 0.83)\n", "measured leaderboard gap (A - B): +0.055 <- 'A beats B by 5'\n" ] } ], "source": [ "# synthetic anchor data: two models scored on ONE shared eval set of N_ITEMS items.\n", "# We set the TRUE per-item success rates, so the ground-truth gap is known (a leaderboard never knows it).\n", "from scipy.stats import norm\n", "N_ITEMS = 200\n", "TRUE_ACC_A = 0.85 # model A's true success rate; the real edge for A is exactly 2 points\n", "TRUE_ACC_B = 0.83 # model B's true rate\n", "RHO = 0.6 # how much a SHARED item difficulty couples the two models (the pairing)\n", "gen = np.random.default_rng(SEED)\n", "# Gaussian-copula construction: a shared latent z (item difficulty) plus per-model noise.\n", "# Thresholding a standard-normal latent at qnorm(true_acc) makes each model's MARGINAL rate exact,\n", "# while the shared z makes a hard item tend to fail BOTH models (paired, not independent).\n", "z = gen.standard_normal(N_ITEMS) # shared item difficulty\n", "lat_A = RHO * z + math.sqrt(1 - RHO**2) * gen.standard_normal(N_ITEMS)\n", "lat_B = RHO * z + math.sqrt(1 - RHO**2) * gen.standard_normal(N_ITEMS)\n", "scores_A = (lat_A < norm.ppf(TRUE_ACC_A)).astype(int) # P = TRUE_ACC_A by construction\n", "scores_B = (lat_B < norm.ppf(TRUE_ACC_B)).astype(int)\n", "print(f\"eval set: {N_ITEMS} items, shared difficulty (a hard item tends to fail BOTH models)\")\n", "print(f\"model A measured accuracy: {scores_A.mean():.3f} (true {TRUE_ACC_A})\")\n", "print(f\"model B measured accuracy: {scores_B.mean():.3f} (true {TRUE_ACC_B})\")\n", "print(f\"measured leaderboard gap (A - B): {scores_A.mean() - scores_B.mean():+.3f} <- 'A beats B by 5'\")" ] }, { "cell_type": "markdown", "id": "1b2066da", "metadata": {}, "source": [ "> **Interpretation.** The measured gap is far larger than the 2-point truth we baked in. With this seed the eval *happens* to flatter A. A leaderboard would print this gap and move on. The number is not wrong, it is just an estimate with uncertainty the leaderboard never showed you. The rest of Part 1 and all of Part 2 is the work of attaching that uncertainty.\n", "\n", "> **Common confusion:** \"the data is synthetic, so isn't the 5-point gap fake?\" The *scores* are simulated, but the statistical situation is exactly the one a real eval faces: a finite sample of items, a true difference smaller than the noise, and a measured gap that overstates it. Synthetic data is what lets us *know* the truth (2 points) and watch the method recover it, which you can never do on a real benchmark.\n" ] }, { "cell_type": "markdown", "id": "bd48c986", "metadata": {}, "source": [ "### Exercise 23.1 — The point estimate and its standard error\n", "`Difficulty 1/5 · ~8 min`\n", "\n", "Before any bootstrap, get the textbook standard error of a proportion. Fill in `acc_and_se(scores)`: return `(accuracy, standard_error)` where `accuracy` is the mean of the 0/1 scores and the standard error of that mean is $\\sqrt{p(1-p)/n}$. This is the cheapest possible uncertainty estimate, and the bootstrap in Part 2 should roughly agree with it.\n", "\n", "The checks verify the accuracy against a hand-computed toy value and the SE against the closed form on the real `scores_A`.\n" ] }, { "cell_type": "code", "execution_count": 4, "id": "32b1db95", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.484385Z", "iopub.status.busy": "2026-06-10T20:48:42.484259Z", "iopub.status.idle": "2026-06-10T20:48:42.488973Z", "shell.execute_reply": "2026-06-10T20:48:42.488629Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ -- ] 23.1 acc + SE (toy): not attempted yet — fill in the TODO above, then re-run.\n", "[ -- ] 23.1 acc + SE (real): not attempted yet — fill in the TODO above, then re-run.\n" ] }, { "data": { "text/plain": [ "False" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def acc_and_se(scores):\n", " \"\"\"scores: array of 0/1. Return (accuracy, standard_error_of_the_mean).\"\"\"\n", " scores = np.asarray(scores, dtype=float)\n", " n = len(scores)\n", " # TODO 1: accuracy = mean of the 0/1 scores\n", " p = None\n", " # TODO 2: standard error of a proportion = sqrt(p*(1-p)/n)\n", " se = None\n", " attempted(p, se)\n", " return float(p), float(se)\n", "\n", "# self-checks (run this cell)\n", "def _acc_se_toy():\n", " # 3 correct out of 4 -> p = 0.75; se = sqrt(0.75*0.25/4) = sqrt(0.046875) = 0.21651...\n", " p, se = acc_and_se([1, 1, 1, 0])\n", " check_close(p, 0.75, msg=\"accuracy of [1,1,1,0] is 3/4 = 0.75\")\n", " check_close(se, math.sqrt(0.75 * 0.25 / 4), msg=\"SE of a proportion is sqrt(p(1-p)/n)\")\n", "\n", "def _acc_se_real():\n", " p, se = acc_and_se(scores_A)\n", " assert abs(p - scores_A.mean()) < 1e-9, \"accuracy must equal scores_A.mean()\"\n", " assert 0.0 < se < 0.1, f\"SE {se:.3f} should be a few percent on {N_ITEMS} items\"\n", "\n", "check(\"23.1 acc + SE (toy)\", _acc_se_toy)\n", "check(\"23.1 acc + SE (real)\", _acc_se_real)" ] }, { "cell_type": "markdown", "id": "89af3633", "metadata": {}, "source": [ "
Hint 1 (conceptual)The accuracy is `scores.mean()`. The standard error of a proportion is the standard deviation of the estimate of the mean: `sqrt(p*(1-p)/n)`, where `p` is the accuracy and `n` the number of items.
\n", "\n", "
Hint 2 (the lines)\n", "\n", "```python\n", "p = scores.mean()\n", "se = math.sqrt(p * (1 - p) / n)\n", "```\n", "
\n", "\n", "
Help — \"SE looks way too big or is nan\"If the SE is `nan`, you may have taken `sqrt` of a negative number; check the formula is `p*(1-p)`, not `p*(1+p)`. If it is order-1 instead of a few percent, you likely forgot to divide by `n` inside the square root.
\n" ] }, { "cell_type": "code", "execution_count": 5, "id": "0d5c62da", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.489688Z", "iopub.status.busy": "2026-06-10T20:48:42.489622Z", "iopub.status.idle": "2026-06-10T20:48:42.492404Z", "shell.execute_reply": "2026-06-10T20:48:42.492049Z" }, "collapsed": true, "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ ok ] 23.1 acc + SE (toy)\n", "[ ok ] 23.1 acc + SE (real)\n", "A: 0.860 ± 0.048 (95% normal interval ±2 SE)\n", "B: 0.805 ± 0.055\n", "the two intervals OVERLAP: 0.812..0.908 vs 0.750..0.860\n" ] } ], "source": [ "#@title Solution { display-mode: \"form\" }\n", "# solution: redefines acc_and_se; the checks below re-verify the reference.\n", "def acc_and_se(scores):\n", " scores = np.asarray(scores, dtype=float)\n", " n = len(scores)\n", " p = scores.mean()\n", " se = math.sqrt(p * (1 - p) / n)\n", " return float(p), float(se)\n", "\n", "check(\"23.1 acc + SE (toy)\", _acc_se_toy, required=True)\n", "check(\"23.1 acc + SE (real)\", _acc_se_real, required=True)\n", "pA, seA = acc_and_se(scores_A)\n", "pB, seB = acc_and_se(scores_B)\n", "print(f\"A: {pA:.3f} ± {1.96*seA:.3f} (95% normal interval ±2 SE)\")\n", "print(f\"B: {pB:.3f} ± {1.96*seB:.3f}\")\n", "print(f\"the two intervals OVERLAP: {pA-1.96*seA:.3f}..{pA+1.96*seA:.3f} vs \"\n", " f\"{pB-1.96*seB:.3f}..{pB+1.96*seB:.3f}\")" ] }, { "cell_type": "markdown", "id": "c578a965", "metadata": {}, "source": [ "> **Caveat:** overlapping individual CIs is *suggestive* that the difference is not significant, but it is not the correct test. The right object is a CI on the *difference itself*, computed with the pairing (each item scored by both models) preserved. That is exactly what Part 2 builds. The closed-form SE here is the sanity check the bootstrap must roughly reproduce.\n", "\n", "> **Key takeaways.** A leaderboard reports point estimates and hides their uncertainty. Each accuracy on `N_ITEMS` items has a standard error of a few percent, often larger than the true gap between two close models. Overlapping single-model intervals hint at non-significance, but the honest object is a CI on the paired difference.\n" ] }, { "cell_type": "markdown", "id": "14084277", "metadata": {}, "source": [ "## Part 2 — The bootstrap CI that eats the gap\n", "\n", "> **Objectives.** Build the percentile bootstrap from scratch, check it against `scipy.stats.bootstrap`, then put a CI on the *paired* difference A − B and watch it cross zero. Cross-check with McNemar's exact test for paired binary outcomes. This is the center of the chapter.\n", "\n", "The bootstrap is the most useful ten lines of statistics you can carry. The idea: you have one eval set, but you want to know how much your metric would wobble if you had drawn a *different* eval set of the same size. You cannot draw new items, so you resample *with replacement* from the items you have, thousands of times, recompute the metric each time, and read the spread of those numbers off as your uncertainty. The 95% CI is the 2.5th and 97.5th percentiles of the resampled metrics.\n", "\n", "$$\\text{CI}_{95} = \\left[\\, Q_{2.5}\\big(\\{m(\\text{resample}_b)\\}_b\\big),\\; Q_{97.5}\\big(\\{m(\\text{resample}_b)\\}_b\\big) \\,\\right]$$\n", "\n", "where each $\\text{resample}_b$ draws `n` items with replacement and $m$ is your metric. We will resample `N_BOOT` times.\n" ] }, { "cell_type": "markdown", "id": "fd3dc2cb", "metadata": {}, "source": [ "### Exercise 23.2 — Percentile bootstrap, from scratch\n", "`Difficulty 2/5 · ~12 min`\n", "\n", "Fill in `bootstrap_ci(scores, n_resamples, alpha, seed)`: resample `scores` with replacement `n_resamples` times, take the mean of each resample, and return the `(alpha/2, 1-alpha/2)` quantiles of those means as `(lo, hi)`. Seed an internal `np.random.default_rng(seed)` so the CI is reproducible.\n", "\n", "The checks verify (a) the CI brackets the true mean on a known stream, (b) the CI *shrinks* as `n` grows (the $1/\\sqrt{n}$ law), and (c) your CI agrees with `scipy.stats.bootstrap` to a tolerance.\n" ] }, { "cell_type": "code", "execution_count": 6, "id": "e84ea4d8", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.493198Z", "iopub.status.busy": "2026-06-10T20:48:42.493128Z", "iopub.status.idle": "2026-06-10T20:48:42.498702Z", "shell.execute_reply": "2026-06-10T20:48:42.498415Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ -- ] 23.2 bootstrap brackets mean: not attempted yet — fill in the TODO above, then re-run.\n", "[ -- ] 23.2 bootstrap shrinks with n: not attempted yet — fill in the TODO above, then re-run.\n", "[ -- ] 23.2 bootstrap vs scipy: not attempted yet — fill in the TODO above, then re-run.\n" ] }, { "data": { "text/plain": [ "False" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def bootstrap_ci(scores, n_resamples=10000, alpha=0.05, seed=0):\n", " \"\"\"Percentile bootstrap CI for the MEAN of `scores`.\n", " Resample with replacement n_resamples times, return (lo, hi) quantiles.\"\"\"\n", " scores = np.asarray(scores, dtype=float)\n", " n = len(scores)\n", " brng = np.random.default_rng(seed)\n", " # TODO 1: draw an (n_resamples, n) array of random indices in [0, n) with replacement.\n", " # brng.integers(0, n, size=(n_resamples, n)) is one way.\n", " idx = None\n", " # TODO 2: index `scores` with idx to get an (n_resamples, n) matrix, take the mean of\n", " # each ROW (axis=1) -> one bootstrap mean per resample.\n", " boot_means = None\n", " attempted(idx, boot_means)\n", " lo = float(np.quantile(boot_means, alpha / 2))\n", " hi = float(np.quantile(boot_means, 1 - alpha / 2))\n", " return lo, hi\n", "\n", "# self-checks (run this cell)\n", "def _boot_brackets_mean():\n", " s = (np.random.default_rng(1).uniform(0, 1, 400) < 0.7).astype(float)\n", " lo, hi = bootstrap_ci(s, n_resamples=2000, seed=0)\n", " assert lo < s.mean() < hi, f\"CI [{lo:.3f}, {hi:.3f}] must bracket the sample mean {s.mean():.3f}\"\n", "\n", "def _boot_shrinks_with_n():\n", " g = np.random.default_rng(2)\n", " small = (g.uniform(0, 1, 100) < 0.7).astype(float)\n", " large = (g.uniform(0, 1, 2000) < 0.7).astype(float)\n", " los, his = bootstrap_ci(small, n_resamples=2000, seed=0)\n", " lol, hil = bootstrap_ci(large, n_resamples=2000, seed=0)\n", " assert (his - los) > (hil - lol), \\\n", " f\"CI on 100 items (width {his-los:.3f}) should be WIDER than on 2000 (width {hil-lol:.3f})\"\n", "\n", "def _boot_vs_scipy():\n", " from scipy.stats import bootstrap as scipy_bootstrap\n", " s = (np.random.default_rng(3).uniform(0, 1, 300) < 0.6).astype(float)\n", " lo, hi = bootstrap_ci(s, n_resamples=5000, seed=0)\n", " res = scipy_bootstrap((s,), np.mean, n_resamples=5000,\n", " confidence_level=0.95, method=\"percentile\",\n", " random_state=np.random.default_rng(0))\n", " assert abs(lo - res.confidence_interval.low) < 0.02, \\\n", " f\"lo {lo:.3f} disagrees with scipy {res.confidence_interval.low:.3f} (>0.02)\"\n", " assert abs(hi - res.confidence_interval.high) < 0.02, \\\n", " f\"hi {hi:.3f} disagrees with scipy {res.confidence_interval.high:.3f} (>0.02)\"\n", "\n", "check(\"23.2 bootstrap brackets mean\", _boot_brackets_mean)\n", "check(\"23.2 bootstrap shrinks with n\", _boot_shrinks_with_n)\n", "check(\"23.2 bootstrap vs scipy\", _boot_vs_scipy)" ] }, { "cell_type": "markdown", "id": "6e982751", "metadata": {}, "source": [ "
Hint 1 (conceptual)One resample is `n` indices drawn uniformly from `0..n-1` with replacement. Do all `n_resamples` at once with a 2D index array of shape `(n_resamples, n)`, then `scores[idx]` is a 2D matrix you mean over `axis=1`.
\n", "\n", "
Hint 2 (the lines)\n", "\n", "```python\n", "idx = brng.integers(0, n, size=(n_resamples, n))\n", "boot_means = scores[idx].mean(axis=1)\n", "```\n", "
\n", "\n", "
Help — \"CI is way too wide / nearly the whole [0,1]\"You probably meaned over the wrong axis. `scores[idx]` is `(n_resamples, n)`; you want one mean *per resample*, so `axis=1`. If you mean over `axis=0` you collapse the resamples and get back roughly the original sample, which is not what you want.
\n" ] }, { "cell_type": "code", "execution_count": 7, "id": "d74fce10", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.499458Z", "iopub.status.busy": "2026-06-10T20:48:42.499394Z", "iopub.status.idle": "2026-06-10T20:48:42.620559Z", "shell.execute_reply": "2026-06-10T20:48:42.620167Z" }, "collapsed": true, "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ ok ] 23.2 bootstrap brackets mean\n", "[ ok ] 23.2 bootstrap shrinks with n\n", "[ ok ] 23.2 bootstrap vs scipy\n", "A accuracy CI: [0.810, 0.905]\n", "B accuracy CI: [0.750, 0.860]\n" ] } ], "source": [ "#@title Solution { display-mode: \"form\" }\n", "# solution: redefines bootstrap_ci; the checks below re-verify the reference.\n", "def bootstrap_ci(scores, n_resamples=10000, alpha=0.05, seed=0):\n", " scores = np.asarray(scores, dtype=float)\n", " n = len(scores)\n", " brng = np.random.default_rng(seed)\n", " idx = brng.integers(0, n, size=(n_resamples, n))\n", " boot_means = scores[idx].mean(axis=1)\n", " lo = float(np.quantile(boot_means, alpha / 2))\n", " hi = float(np.quantile(boot_means, 1 - alpha / 2))\n", " return lo, hi\n", "\n", "check(\"23.2 bootstrap brackets mean\", _boot_brackets_mean, required=True)\n", "check(\"23.2 bootstrap shrinks with n\", _boot_shrinks_with_n, required=True)\n", "check(\"23.2 bootstrap vs scipy\", _boot_vs_scipy, required=True)\n", "loA, hiA = bootstrap_ci(scores_A, n_resamples=N_BOOT, seed=SEED)\n", "loB, hiB = bootstrap_ci(scores_B, n_resamples=N_BOOT, seed=SEED)\n", "print(f\"A accuracy CI: [{loA:.3f}, {hiA:.3f}]\")\n", "print(f\"B accuracy CI: [{loB:.3f}, {hiB:.3f}]\")" ] }, { "cell_type": "markdown", "id": "0dc82cff", "metadata": {}, "source": [ "> **Interpretation.** The bootstrap CI on each model roughly matches the closed-form `±2 SE` from Part 1, as it must, the bootstrap is rediscovering the same uncertainty without the proportion formula. The reason we bother with the bootstrap rather than the formula: it works for *any* metric (median, F1, AUC, a custom rubric score), not just proportions, and it handles the *paired difference* we build next without any new math.\n" ] }, { "cell_type": "markdown", "id": "09cad28f", "metadata": {}, "source": [ "### The paired difference is what the leaderboard should show\n", "\n", "The two single-model CIs overlap, but overlapping CIs are not the correct test. The right object is a CI on the *difference* `A − B`, and crucially it must respect the *pairing*: both models were scored on the *same* items, so on each bootstrap resample we draw a set of item indices and score *both* models on that same set. Pairing removes the shared item-difficulty noise (a hard item drags both models down together), which tightens the difference CI, and it is the only honest way to compare two models on a shared eval.\n", "\n", "> **Predict:** the per-model CIs were each about ±0.05 wide. Will the CI on the *paired difference* be wider or narrower than you would get by treating the models as independent?
AnswerNarrower. Pairing cancels the shared component: when a hard item knocks A down, it usually knocks B down too, so the *difference* on that item is more stable than either score alone. This is exactly why paired tests (paired bootstrap, McNemar, paired t) are more powerful than their unpaired cousins, and why throwing away the pairing, comparing two independently-bootstrapped numbers, is a real methodological error you will see in published reports.
\n" ] }, { "cell_type": "code", "execution_count": 8, "id": "88c0b649", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.621812Z", "iopub.status.busy": "2026-06-10T20:48:42.621738Z", "iopub.status.idle": "2026-06-10T20:48:42.663098Z", "shell.execute_reply": "2026-06-10T20:48:42.662720Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "measured gap A - B : +0.055\n", "95% CI on the gap : [-0.010, +0.120] (crosses 0 -> not significant)\n" ] } ], "source": [ "# paired bootstrap on the DIFFERENCE A - B: resample ITEM INDICES, score both models on them.\n", "def paired_diff_ci(a, b, n_resamples=N_BOOT, alpha=0.05, seed=SEED):\n", " a = np.asarray(a, dtype=float); b = np.asarray(b, dtype=float)\n", " n = len(a)\n", " brng = np.random.default_rng(seed)\n", " idx = brng.integers(0, n, size=(n_resamples, n)) # (n_resamples, n) shared item indices\n", " diffs = a[idx].mean(axis=1) - b[idx].mean(axis=1) # difference on the SAME resampled items\n", " lo = float(np.quantile(diffs, alpha / 2))\n", " hi = float(np.quantile(diffs, 1 - alpha / 2))\n", " return scores_A.mean() - scores_B.mean(), lo, hi\n", "\n", "point, dlo, dhi = paired_diff_ci(scores_A, scores_B)\n", "print(f\"measured gap A - B : {point:+.3f}\")\n", "print(f\"95% CI on the gap : {fmt_ci(dlo, dhi)}\")" ] }, { "cell_type": "markdown", "id": "39fe8a29", "metadata": {}, "source": [ "> **Interpretation.** This is the moment the chapter is named for. The leaderboard's +5 gap comes with a 95% CI that *crosses zero*: the data are consistent with A and B being equal, or even with B being slightly better. The \"+5 win\" is not significant at this eval size. We baked in a true 2-point edge for A, and the correct procedure refuses to claim even that from 200 items, which is the right call: 200 items cannot resolve a 2-point difference. The number was real; the *conclusion* drawn from it was not.\n" ] }, { "cell_type": "code", "execution_count": 9, "id": "d4c953e3", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.664234Z", "iopub.status.busy": "2026-06-10T20:48:42.664131Z", "iopub.status.idle": "2026-06-10T20:48:42.718646Z", "shell.execute_reply": "2026-06-10T20:48:42.718295Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAADlCAYAAAClHXjxAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQbdJREFUeJzt3XlcVNX/P/DXMMAMO4jIqoC7KIqikChSauFuLomoCWblJyO1j5b2a9HSzNJSc9cMyzIxLTU1TQ3cc8ctUVFwQQREZZGdOb8/+DIfhxlgBhgG8fV8POahc+65577PmZk7b+49945ECCFARERERJUyMnQARERERE8LJk5EREREWmLiRERERKQlJk5EREREWmLiRERERKQlJk5EREREWmLiRERERKQlJk5EREREWmLiRERERKQlJk56IpFIEBERYegw9CImJgYSiQQxMTG1vu1169ZBIpEgMTGxRtt9/vnn0a5duxptU1/0NQZPk8TEREgkEqxbt07rugsWLKjStgz5fq8NJ0+eREBAACwsLCCRSBAbG4tZs2ZBIpFotb5EIsGsWbP0G2QtqC/9qEx9fz/XBiZO1XD06FHMmjULjx49MnQoOvPw8IBEIlF7/Oc//zF0aERVsmvXrlr74tuwYQMWLVpUK9vSp8LCQrzyyit48OABFi5ciPXr18Pd3d3QYT115s6di61btxo6DKolxoYO4Gl29OhRfPrppwgPD4etra2hw9GZj48Ppk6dqlLWsmVLA0VDpD13d3fk5ubCxMREWbZr1y4sW7asxpOnHj16IDc3F6ampsqyDRs24OLFi5gyZUqNbqu2Xb9+HTdv3sSaNWvw+uuvK8s/+ugjzJgxw4CR1b7c3FwYG1ftK3Hu3LkYPnw4Xn755ZoNiuokJk71TExMDF544QUkJCTAw8Ojwrqurq4YM2ZM7QRWRz1+/BgWFhaGDkOFQqFAQUEB5HK5oUOpsyQSSa2Nj5GRUb19LVJTUwFA7Q8/Y2PjKicRT6u69hrn5eXB1NQURkY8MVTX8BWpolmzZuG9994DAHh6eipPdZWdd7J161a0a9cOMpkMbdu2xe7du9XaSkpKwmuvvQZHR0dlve+//742uoGCggI8fvy4Rto6fvw4+vTpAxsbG5ibmyMoKAhHjhxRqXPz5k1MnDgRrVq1gpmZGezt7fHKK69onK9z6dIl9OzZE2ZmZnBzc8OcOXOgUCg0bvvPP/9EYGAgLCwsYGVlhf79++PSpUsqdcLDw2FpaYnr16+jX79+sLKywujRo1XqnD59GgEBATAzM4OnpydWrlyptq3U1FSMHz8ejo6OkMvl6NChA3744Qe1egsWLEBAQADs7e1hZmYGX19fbN68Wa1e6Xy4n3/+GW3btoVMJlO+T3QZA01+/fVXeHl5QS6Xo127dvj9998RHh6ullRXJdZWrVpBLpfD19cXBw8e1DqmJ/33v/+Fvb09hBDKsnfeeQcSiQTffvutsiwlJQUSiQQrVqwAoD7HKTw8HMuWLVPGWPooa/Xq1WjWrBlkMhm6dOmCkydPVhpj2Tkhzz//PHbu3ImbN28qt/PkeC5ZsgRt27aFubk57Ozs0LlzZ2zYsKHS7eTl5WHWrFlo2bIl5HI5nJ2dMXToUFy/fl1Z5/Hjx5g6dSoaN24MmUyGVq1aYcGCBSrjVzoGERERFe5/wsPDERQUBAB45ZVXIJFI8PzzzwOAxjlO+fn5ePfdd+Hg4AArKysMGjQId+7c0dgXbfZppeO6adMmfP7553Bzc4NcLkevXr0QHx+v1ubx48fRr18/2NnZwcLCAu3bt8fixYtV6sTFxWH48OFo0KAB5HI5OnfujO3bt1cy8v8bsyePVpaOQXx8vPKsgo2NDcaNG4ecnByV9R4/fowffvhB+X4IDw+v0lhs3LgRH330EVxdXWFubo4zZ85AIpFo3L/s2bMHEokEO3bsAKDbvpWq59n6k6IGDR06FFevXsUvv/yChQsXomHDhgAABwcHZZ3Dhw/jt99+w8SJE2FlZYVvv/0Ww4YNw61bt2Bvbw+g5AvhueeeU+7oHBwc8Oeff2L8+PHIzMzU66mAv//+G+bm5iguLoa7uzveffddTJ48ucpt9e3bF76+vpg5cyaMjIwQGRmJnj174tChQ/Dz8wNQMhH16NGjGDlyJNzc3JCYmIgVK1bg+eefx7///gtzc3MAwL179/DCCy+gqKgIM2bMgIWFBVavXg0zMzO1ba9fvx5hYWEIDg7Gl19+iZycHKxYsQLdu3fH2bNnVb7UioqKEBwcjO7du2PBggXK7QHAw4cP0a9fP4wYMQKhoaHYtGkT3nrrLZiamuK1114DUHI4//nnn0d8fDwiIiLg6emJX3/9FeHh4Xj06JHK+C1evBiDBg3C6NGjUVBQgI0bN+KVV17Bjh070L9/f7Xx27RpEyIiItCwYUN4eHjoNAaa7Ny5EyEhIfD29sYXX3yBhw8fYvz48XB1dVWrq0usBw4cQFRUFCZNmgSZTIbly5ejT58+OHHihM4T7AMDA7Fw4UJcunRJue6hQ4dgZGSEQ4cOYdKkScoyoOS0mSYTJkzA3bt3sXfvXqxfv15jnQ0bNiArKwsTJkyARCLBV199haFDh+LGjRsqp/wq8+GHHyIjIwN37tzBwoULAQCWlpYAgDVr1mDSpEkYPnw4Jk+ejLy8PJw/fx7Hjx/HqFGjym2zuLgYAwYMwP79+zFy5EhMnjwZWVlZ2Lt3Ly5evIhmzZpBCIFBgwYhOjoa48ePh4+PD/bs2YP33nsPSUlJylhKVbb/mTBhAlxdXTF37lxMmjQJXbp0gaOjY7kxvv766/jpp58watQoBAQE4O+//1Z7bwC679PmzZsHIyMjTJs2DRkZGfjqq68wevRoHD9+XFln7969GDBgAJydnTF58mQ4OTnh8uXL2LFjh/Izd+nSJXTr1g2urq7Kz8umTZvw8ssvY8uWLRgyZEjFL2w5RowYAU9PT3zxxRc4c+YMvvvuOzRq1AhffvklgJL9z+uvvw4/Pz+8+eabAIBmzZpVaSxmz54NU1NTTJs2Dfn5+fDy8kLTpk2xadMmhIWFqdSNioqCnZ0dgoODAWi/b6UaIKjK5s+fLwCIhIQEtWUAhKmpqYiPj1eWnTt3TgAQS5YsUZaNHz9eODs7i/v376usP3LkSGFjYyNycnJ0iik6OrrcmJ40cOBA8eWXX4qtW7eKtWvXisDAQAFAvP/++1pvIzo6WgghhEKhEC1atBDBwcFCoVAo6+Xk5AhPT0/x4osvqpSVdezYMQFA/Pjjj8qyKVOmCADi+PHjyrLU1FRhY2Oj0r+srCxha2sr3njjDZU27927J2xsbFTKw8LCBAAxY8YMtRiCgoIEAPH1118ry/Lz84WPj49o1KiRKCgoEEIIsWjRIgFA/PTTT8p6BQUFomvXrsLS0lJkZmaW29eCggLRrl070bNnT5VyAMLIyEhcunRJpVzbMSiPt7e3cHNzE1lZWcqymJgYAUC4u7ur1NUlVgDi1KlTyrKbN28KuVwuhgwZUmE8mqSmpgoAYvny5UIIIR49eiSMjIzEK6+8IhwdHZX1Jk2aJBo0aKB8fyUkJAgAIjIyUlnn7bffFpp2aaV17e3txYMHD5Tl27ZtEwDEH3/8UWGMZd/vQgjRv39/tTEUQojBgweLtm3batN1Fd9//70AIL755hu1ZaV93rp1qwAg5syZo7J8+PDhQiKRqOxrtN3/lPbt119/VWlz5syZKmMZGxsrAIiJEyeq1Bs1apQAIGbOnKks03afVrrtNm3aiPz8fGW9xYsXCwDiwoULQgghioqKhKenp3B3dxcPHz7UODZCCNGrVy/h7e0t8vLyVJYHBASIFi1aiMqU7UfpGLz22msq9YYMGSLs7e1VyiwsLERYWJham7qORdOmTdU+ix988IEwMTFRee/m5+cLW1tbldi03bdqej+TbniqTo969+6t/MsDANq3bw9ra2vcuHEDACCEwJYtWzBw4EAIIXD//n3lIzg4GBkZGThz5kyF28jIyFBZLyMjA0DJ0ZMny7Ozs1XW2759O95//30MHjwYr732Gg4cOIDg4GB888035R5+L09sbCyuXbuGUaNGIT09XbnNx48fo1evXjh48KDy9NKTR0sKCwuRnp6O5s2bw9bWVqWvu3btwnPPPac8UgWUHM0re2pt7969ePToEUJDQ1X6K5VK4e/vj+joaLV433rrLY39MDY2xoQJE5TPTU1NMWHCBKSmpuL06dPKuJycnBAaGqqsZ2JigkmTJiE7OxsHDhxQlj/Z14cPHyIjIwOBgYEaX9OgoCB4eXmplGk7BprcvXsXFy5cwNixY5VHQ0q34+3trVZfl1i7du0KX19f5fMmTZpg8ODB2LNnD4qLiyuN7UkODg5o3bq18lTfkSNHIJVK8d577yElJQXXrl0DUHLEqXv37lpfIq9JSEgI7OzslM8DAwMBQPl5rAm2tra4c+eOVqcAn7RlyxY0bNgQ77zzjtqy0j7v2rULUqlUeRSu1NSpUyGEwJ9//qlSXtn+Rxe7du0CALVtlz1iUpV92rhx41Qm3pd9Xc6ePYuEhARMmTJFbS5W6dg8ePAAf//9N0aMGIGsrCzlNtPT0xEcHIxr164hKSlJ534DULvSODAwEOnp6cjMzKxwvaqMRVhYmNoR5ZCQEBQWFuK3335Tlv3111949OgRQkJClGXa7lup+niqTo+aNGmiVmZnZ4eHDx8CANLS0vDo0SOsXr0aq1ev1thG6eTN8gwePFjly7pUp06dVJ6HhYVVeM8biUSCd999F3v27EFMTIxOk8ZLv9zKHkp+UkZGBuzs7JCbm4svvvgCkZGRSEpKUpmbUZr0ASXn6/39/dXaadWqlcZt9+zZU+N2ra2tVZ4bGxvDzc1NY10XFxe1ieKlVxkmJibiueeew82bN9GiRQu1CZtt2rRRxl1qx44dmDNnDmJjY5Gfn68s1/Tl7+npqVam7RhoUhpH8+bN1ZY1b95cbUeqS6wtWrRQK2vZsiVycnKQlpYGJyenSuN7UmBgoPKL+dChQ+jcuTM6d+6MBg0a4NChQ3B0dMS5c+cqPNWljbKfx9IkqvTzWBOmT5+Offv2wc/PD82bN8dLL72EUaNGoVu3bhWud/36dbRq1arCCdk3b96Ei4sLrKysVMo1vfeAyvc/urh58yaMjIxUEjFA/b1YlX1aZa9L6Ryvik4Dx8fHQwiBjz/+GB9//HG529V0mroyFcVXdv/ypKqMhab9QIcOHdC6dWtERUVh/PjxAEpO0zVs2FBlv6ftvpWqj4mTHkmlUo3lpW/o0qMwY8aMKTfpaN++fYXb+Prrr1V2hOfOncO0adPw008/qcxXcHFxqTTexo0bAyj5600Xpf2YP38+fHx8NNYpPerxzjvvIDIyElOmTEHXrl1hY2MDiUSCkSNH6jTpuey2169fr/ELu+wXkUwmq5WrVA4dOoRBgwahR48eWL58OZydnWFiYoLIyEiNE4W1nbekD7rGWtO6d++ONWvW4MaNGzh06BACAwMhkUjQvXt3HDp0CC4uLlAoFMojEVVV2eexJrRp0wZXrlzBjh07sHv3bmzZsgXLly/HJ598gk8//bTGtqON2uhvWVXZp9VEnKXbnTZtmnLOT1ma/ojQRlXjq8pYlLcfCAkJweeff4779+/DysoK27dvR2hoqMr+rab3rVQ+Jk7VUJ3TBgCUV6cUFxejd+/eVWrjyVMmwP8ShW7dulV6O4KySg+NPznBXRulf4VaW1tX2o/NmzcjLCwMX3/9tbIsLy9P7Sai7u7uyqNJT7py5YrGbTdq1KjKY1jq7t27arcnuHr1KgAox9Ld3R3nz5+HQqFQScDi4uKUy4GSUy9yuRx79uyBTCZT1ouMjNQ6Hm3HoLx1AWi8Oqlsma6xaorp6tWrMDc31/m9A/zv1MzevXtx8uRJ5f2DevTogRUrViiPBJZ9r5dV3c+jLiraloWFBUJCQhASEoKCggIMHToUn3/+OT744INyL3lv1qwZjh8/jsLCwnInqru7u2Pfvn3IyspSOepU9r2nD+7u7lAoFMojY6XKvhdrYp9WVuln/OLFi+W22bRpUwAlp81raru60PR+qMmxCAkJwaeffootW7bA0dERmZmZGDlypEodbfetVH2c41QNpV+wVX1jSqVSDBs2DFu2bMHFixfVlqelpVUnvHI9ePBAbS5KYWEh5s2bB1NTU7zwwgs6tefr64tmzZphwYIFanOpANV+SKVStb/UlixZohZPv3798M8//+DEiRMq7fz8888q9YKDg2FtbY25c+eisLCwwm1XpqioCKtWrVI+LygowKpVq+Dg4KD80u7Xrx/u3buHqKgolfWWLFkCS0tL5eXdUqkUEolEpV+JiYk63V1Y2zHQxMXFBe3atcOPP/6o8pocOHAAFy5cUKmra6zHjh1TOdV3+/ZtbNu2DS+99FK5f51XxNPTE66urli4cCEKCwuVp7UCAwNx/fp1bN68Gc8991yl9xWq7udRFxYWFhpPf6Snp6s8NzU1hZeXF4QQGt+fpYYNG4b79+9j6dKlastKPy/9+vVDcXGxWp2FCxdCIpGgb9++VemKVkrbfvIWEQDU7p6uj31ap06d4OnpiUWLFqm9tqVj06hRIzz//PNYtWoVkpOTa2S7urCwsFCLrSbHok2bNvD29kZUVBSioqLg7OysdoWptvtWqj4ecaqG0i/TDz/8ECNHjoSJiQkGDhyo0w0V582bh+joaPj7++ONN96Al5cXHjx4gDNnzmDfvn06nzbTxvbt2zFnzhwMHz4cnp6eePDggfJOyHPnztV5joqRkRG+++479O3bF23btsW4cePg6uqKpKQkREdHw9raGn/88QcAYMCAAVi/fj1sbGzg5eWFY8eOYd++fcrbM5R6//33sX79evTp0weTJ09WXopfesSnlLW1NVasWIFXX30VnTp1wsiRI+Hg4IBbt25h586d6Natm8YvI01cXFzw5ZdfIjExES1btkRUVBRiY2OxevVq5VGAN998E6tWrUJ4eDhOnz4NDw8PbN68GUeOHMGiRYuURwL69++Pb775Bn369MGoUaOQmpqKZcuWoXnz5irxV0TbMSjP3LlzMXjwYHTr1g3jxo3Dw4cPsXTpUrRr104lmdI11nbt2iE4OFjldgQA1E5FSSQSBAUFafWbWIGBgdi4cSO8vb2Vc0g6deoECwsLXL16Vav5TaWfx0mTJiE4OBhSqVTtr/Ka4uvri6ioKPz3v/9Fly5dYGlpiYEDB+Kll16Ck5MTunXrBkdHR1y+fBlLly5F//791eYmPWns2LH48ccf8d///hcnTpxAYGAgHj9+jH379mHixIkYPHgwBg4ciBdeeAEffvghEhMT0aFDB/z111/Ytm0bpkyZojb/qCb5+PggNDQUy5cvR0ZGBgICArB//36NRzRrep9mZGSEFStWYODAgfDx8cG4cePg7OyMuLg4XLp0CXv27AEALFu2DN27d4e3tzfeeOMNNG3aFCkpKTh27Bju3LmDc+fO1chYaOLr64t9+/bhm2++gYuLCzw9PeHv71+jYxESEoJPPvkEcrkc48ePV5tyoO2+lWpA7V7EV//Mnj1buLq6CiMjI5VLxAGIt99+W62+u7u72mWrKSkp4u233xaNGzcWJiYmwsnJSfTq1UusXr1a53i0uR3BqVOnxMCBA4Wrq6swNTUVlpaWonv37mLTpk06baPs5axnz54VQ4cOFfb29kImkwl3d3cxYsQIsX//fmWdhw8finHjxomGDRsKS0tLERwcLOLi4jSOy/nz50VQUJCQy+XC1dVVzJ49W6xdu1Zj/6Kjo0VwcLCwsbERcrlcNGvWTISHh6tcNh8WFiYsLCw09ikoKEi0bdtWnDp1SnTt2lXI5XLh7u4uli5dqlY3JSVF2QdTU1Ph7e2tcll8qbVr14oWLVoImUwmWrduLSIjI9Uu8xai/PeKrmOgycaNG0Xr1q2FTCYT7dq1E9u3bxfDhg0TrVu3rlasP/30k7J+x44d1d4LWVlZAoAYOXJkpTEKIcSyZcsEAPHWW2+plPfu3VsAUHkPCaH5dgRFRUXinXfeEQ4ODkIikShjL607f/58te2izCXommh6v2dnZ4tRo0YJW1tblds7rFq1SvTo0UP5GWjWrJl47733REZGRqVjkJOTIz788EPh6emp3A8MHz5cXL9+XVknKytLvPvuu8LFxUWYmJiIFi1aiPnz56tcll/aL232P9rejkAIIXJzc8WkSZOEvb29sLCwEAMHDhS3b9/WOIba7NPK27am11YIIQ4fPixefPFFYWVlJSwsLET79u1Vbq0ghBDXr18XY8eOFU5OTsLExES4urqKAQMGiM2bN6uNRVll+1E6BmlpaSr1IiMj1T5/cXFxokePHsLMzEwAUBnj6ozFk65du6a8Hcjhw4fVlmu7b+XtCKpPIoQeZwoSUZ3j4+MDBwcH7N27V+d1JRIJ3n777UqP4u3atQsDBgzAuXPnNN7+gIjoacU5TkT1VGFhIYqKilTKYmJicO7cOeVPa+hLdHQ0Ro4cyaSJiOodznEiqqeSkpLQu3dvjBkzBi4uLoiLi8PKlSvh5OSkdlO/mjZ//ny9tk9EZChMnIjqKTs7O/j6+uK7775DWloaLCws0L9/f8ybN48TRomIqohznIiIiIi0xDlORERERFrS6lSdQqHA3bt3YWVlVat35yUiIiLSNyEEsrKy4OLiUunPcmmVON29e1f5O2ZERERE9dHt27fL/SH4UlolTqV3vL19+3aFvwZd3+VcvYorb7yBVmvWwLxlS0OHQ0RERDUgMzMTjRs3rvAO/6W0SpxKT89ZW1s/04mTsaUlLKVSWFtawvwZHgciIqL6SJvpSJwcTkRERKQlJk5EREREWuINMHUgtbJCg759IdXiHCgRET1dhBAoKipCcXGxoUMhPTExMYFUKq1WG0ycdCBzdYXn7NmGDoOIiGpYQUEBkpOTkZOTY+hQSI8kEgnc3NxgaWlZ5TaYOOlAkZ+PgtRUmDZqBCOZzNDhEBFRDVAoFEhISIBUKoWLiwtMTU15z8J6SAiBtLQ03LlzBy1atKjykScmTjrIS0jA5TFj0Oann2DeurWhwyEiohpQUFAAhUKBxo0bw9zc3NDhkB45ODggMTERhYWFVU6cODmciIgIqPSO0fT0q4kjiXyXEBEREWmJiRMREdEzYtq0aZg1axYAYOXKlZg/f75y2fjx4+Hl5YUhQ4ZofE4lOMeJiIjoGfSf//xH+f+UlBRs3LgRmZmZkEqlas+1pVAoANTv0571t2d6YN66NXxPneLEcCKiZ0Dh/fvIiYtTeeQnJQEoucq67LKcuDjlunmJiWrLijIydNq+RCLB3Llz4efnB09PT0RGRiqXnTp1CgEBAWjfvj38/Pxw5MgRjW0kJycjODgYXl5e6N27N+7cuaNcNmvWLEyZMgWPHj3CCy+8gLy8PPj6+mLevHlqzwFgwYIF8PPzQ6dOndCnTx/cvHlT2c6wYcMQHByMdu3aITk5GXv27EH37t3h6+sLPz8/REdHAwBiYmLQrl07TJw4ER06dEDbtm1x6tQpZUw7d+5Ely5d0KFDB/j4+OD48eMAgJMnT6Jnz57o3LkzOnbsiF9//VWnsaxJPOJERESkQdqWLUhes0alrEHfvvCcPRsFqam4PGaM2jq+/5cEJH76KR5fuKCyzOOzz2Dfr59OMchkMpw4cQJxcXHo0qULXn31VSgUCgwdOhRr1qxBcHAwDh8+jGHDhiE+Pl7t/kSTJk2Cn58f9uzZg6SkJPj4+KB1mT/+bW1tsWvXLvj4+CA2NhYAMHLkSJXnGzZswJUrV3Ds2DFIpVKsX78eEydOxM6dOwEAx44dw9mzZ+Ho6IgbN25g1qxZ2LNnD6ytrREfH4/AwEAkJiYCAOLi4rB27VosX74cK1euxIcffog9e/bg6tWrGDduHA4ePIjWrVujsLAQOTk5ePToEd58803s2rULzs7OuH//Pjp16oSAgAC4urrqNJ41gYmTDvISE5H46afwmDkTcg8PQ4dDRER65DBsGGyDglTKSn85wrRRI7T56ady1/WYOROKvDyVMlNnZ51jGD16NACgdevWMDY2xr179/Dw4UMYGRkhODgYANC9e3c4OjoiNjYW3bt3V1l///79WLBgAQDA1dUVgwYN0jkGANi6dStOnjwJX19fAFC7u3q/fv3g6OgIANi9ezfi4+PRo0cP5XIjIyPcunULANC8eXP4+/sDALp27aqMb+/evejTp48ysTMxMYGNjQ127dqFGzduoG/fvirbvHLlChOnuk6Rl4fHFy6ofRiIiKj+MWnYECYNG2pcZiSTVThto6b+uJbL5cr/S6VSFBUVaayn7WX2Vb0cXwiBDz74AG+++abG5U8e6RJC4MUXX8SGDRvU6iUlJWndpyfba9u2LY4ePVql2Gsa5zgRERE9RVq1agWFQoG9e/cCAI4ePYp79+7Bx8dHrW7v3r3x/fffAyiZ77R9+/YqbfPll1/GypUr8eDBAwBAYWEhzp49q7FucHAw9u3bh/PnzyvLTpw4Uek2goODsWfPHsT931yxwsJCZGRkICAgAAkJCdi3b5+ybmxsLAoKCqrUl+riESciIqKniKmpKX777TdMmjQJU6dOhVwux+bNmzX+/trixYsRHh4OLy8vuLq6omfPnlXa5ujRo5Geno4XXngBAFBUVITXXnsNHTt2VKvbvHlzbNiwARMmTEBOTg4KCgrQsWNHjUegyq4XGRmJMWPGKO/svXLlSvj5+WHnzp2YNm0apk6disLCQjRp0gRbt26tUl+qSyKEEJVVyszMhI2NDTIyMmBtbV0bcdVJOXFx/MkVIqJ6Ji8vDwkJCfD09FQ5jUT1T3mvtS55Dk/V6cDU2Rken31WpQl+RERE9PTjqTodGNvY6HwpKREREdUfPOKkg8KHD5G6aRMKHz40dChERFTLUtKBhetK/q3JuvR0YeKkg8KUFNz+6isUpqQYOhQiIqplqenAoh9K/q3JuvR0YeJEREREpCXOcSIiIqqi4mLgxIWSI0uN7AE/b0CH38SlpxATJyIioir48yDw6VIgOe1/Zc4OwMwIwM3JcHGRfvFUnQ6MzM1h/dxzMDI3N3QoRERkQH8eBN6aqZo0AcC9tJLyo2dqZjtZWVmwtLTE+PHja6bBJ3h4eKBVq1bw8fFBmzZtMGrUKDx+/Ljc+tOmTcPGjRsBAI8fP8a4cePg7e2N1q1bY8aMGSi9LWRMTAzMzMzg4+OjfOTm5gIATp06BR8fH3h5eeGHH35Qtv33339jwoQJFcabmpqKcePGoWnTpujYsSM6deqEuXPnAgDWrVuHl19+GQCQkpICPz+/Sn/KpaqYOOlA3qQJWixdCnmTJoYOhYiIDORxLjBrCaDp7tGlZauiamZbUVFR8PX1xW+//Ybs7OyaabRM+7Gxsbh06RIyMjKwbt06jfWSkpKwa9cuhISEAADmzp2L4uJinD9/HhcuXMC5c+ewefNmZf1WrVohNjZW+TAzMwMAzJs3D99++y1OnjyJTz/9FACQm5uLWbNm4csvvyw3ztzcXAQFBcHd3R3Xrl3D2bNncfjwYVhYWKjVdXR0REBAAH788ceqDkuFmDjpQBQXozg7G6LMr0ITEdGzI2QKcO9++csFgPRHNbOttWvXYvr06ejRoweiomooG9OgoKAAOTk5sLOz07j8+++/x7Bhw5Q/Enzu3Dn06dMHEokEJiYmePHFF7F+/fpKt2NiYoKcnBzk5eVB+n+TwWbNmoXJkyfD1ta23PU2bNgAKysrzJo1S7meubk5Jk+erLF+aGgoVq1aVWk8VcHESQe5164h9vnnkXvtmqFDISKieu7ff//F7du3ERwcjPHjx2Pt2rU1vo2QkBD4+PjAyckJRkZGGDFihMZ6MTEx8Pf3Vz739fXFr7/+ivz8fGRnZ2Pr1q1ITExULr9+/To6deqELl26YPny5cryTz75BHPnzsVLL72E+fPnIzY2Fjdu3MCwYcMqjPP06dPo2rWr1v3y9fXF+fPnkZmZqfU62mLiREREpINZ79TOdtauXYuxY8dCKpWiX79+SEhIwOXLlzXWDQkJQcOGDTU+jhw5Uu42Sk/V3b9/Hx4eHpg+fbrGenfu3IGjo6Py+YwZM9CkSRP4+/ujf//+8PPzg7FxyfVmnTp1wp07d3DmzBn8/vvvWLlyJTZt2gQAaNOmDQ4ePIjTp09j4MCBmDp1KhYvXoxffvkFw4YNw7hx4/CwBm4ybWxsDDs7O9y9e7fabZXFxImIiEgHHb1Krp6TlLNcAqCh5jNeWissLMT69evxww8/wMPDA82bN0dOTk65R52ioqJw//59jY9u3bpVuj1jY2MMGzYMu3fv1rjc3NwceXl5yudmZmZYvHgxYmNjceDAATRs2BBt27YFAFhbW8PGxgYA4ObmhtDQUBw6dEitzUWLFuGVV16Bra0tZs+ejaioKPTo0QOLFi1Sq+vr64t//vmn0n48KS8vTzm3qiYxcSIiItKB1KjklgOAevJU+vxNzWe8tLZ9+3Y0bdoUSUlJSExMRGJiIv755x+sX78ehYWF1Wu8HH///TdatWqlcVn79u1x5coV5fPMzEzk5OQAABISErBixQpMnToVAJCcnAyFQgGg5KrAHTt2oGPHjirtJSQkYO/evZgwYQIKCwtRVFQEiUQCIyMjjZPgQ0ND8ejRI8yePRvF/zfPODc3F99++63GeFNSUiCRSNC4cWMdR6FyTJyIiIh01LcHsOJTwMlBtdzJoaQ8oFP12l+7di1Gjx6tUtamTRu4urrijz/+qF7jTyid49SuXTtcvnwZixcv1lhv+PDh2LNnj/L5jRs3lLcVGDx4MBYuXAgfHx8AwJYtW+Dt7Y0OHTrgueeew4svvohx48aptDd58mQsWrQIEokENjY2GDVqFLy9vbFs2TJERESobd/c3BwHDhzA9evX0bx5c3h7e8Pf31+ZvJW1e/duDBkyBEZGNZ/mSETpjRcqkJmZCRsbG2RkZMDa2rrGg3haiKIiFGVlwdjKChJj3juUiKg+yMvLQ0JCAjw9PSGXy8utd+EqMGACsGMV4N2ypKy8O4drqvs0UygU8PPzw9atW+Hm5mbocCoVGBiI1atXo02bNirl5b3WuuQ5/PbXgcTYGCblXKpJRETPHqkU6Opj6Cj0z8jICKtWrUJiYmKdT5xSUlLw1ltvqSVNNYWn6nSQf+cO4t99F/l37hg6FCIiolrl6+uL7t27GzqMSjk6OmLUqFF6a5+Jkw6Ks7ORcegQivVw91YiIjKs0gnN5WlkD0wJK/m3MrrUpdqjxeykSvFUHRERPdNMTU1hZGSEu3fvwsHBAaampso7ZD/JxgJ4a2TJ/5+4Ml8jXepS7RBCIC0tTXm386pi4kRERM80IyMjeHp6Ijk5WS83TKS6QyKRwM3NTfmzLVXBxImIiJ55pqamaNKkCYqKipT3CaL6x8TEpFpJE8DESScmDg5wmzIFJg4OlVcmIqKnSukpnOqcxqH6j4mTDkzs7eE4ZoyhwyAiIiID4VV1OijKzMTDfftQpIdfWyYiIqK6j4mTDgru3sWNGTNQwMmDREREzyQmTkRERERaYuJEREREpCUmTkRERERaYuKkA4lMBrNWrSCRyQwdChERERkAb0egAzNPT3j9/LOhwyAiIiID4REnIiIiIi0xcdJBTlwcznTtipy4OEOHQkRERAbAxElHorDQ0CEQERGRgTBxIiIiItISEyciIiIiLTFxIiIiItISb0egA7mHB7yioiBzdTV0KERERGQATJx0YCSXw6xZM0OHQURERAbCU3U6yE9ORuLs2chPTjZ0KERERGQATJx0UJyRgfRt21CckWHoUIiIiMgAmDgRERERaYmJExEREZGWmDgRERERaYmJkw6MGzSAU3g4jBs0MHQoREREZAB1KnFKSQcWriv5ty4ybdQIrhERMG3UyNChEBER1Xt1MS+oU4lTajqw6IeSf+ui4sePkXXqFIofPzZ0KERERPVeXcwL6lTiVNfl376Nq//5D/Jv3zZ0KERERGQAvHM4EdUpxcXAiQslf2E2sgf8vAGp1NBRERGVYOJERHXGnweBT5cCyWn/K3N2AGZGAH17GC4uIqJSPFVHRHXCnweBt2aqJk0AcC+tpPzPg4aJi4joSXXyiFNePpCTa+go1OUWm6DYoTFyi02AOhgf0dOquBiYtQQQGpYJABKULO/eiaftiJ4lefmGjkBdnUychk8ydATlaQbgdyDC0HEQPVsEgHv3gXYDDR0JET3reKqOiIiISEt18ojT5m+Bts0NHYW63BvXcX3aNDRbsABmTZsZOhyieuPEeSBsRuX1fpgH+LXXfzxEVDdciq97Z6HqZOIklwHmZoaOQgNpIaRpt2EmLayb8RE9pQI7l1w9dy9N8zwnCQAnh5J6nONE9OyQywwdgTqeqiMig5NKS245AJQkSU8qfT4zgkkTERkeEyciqhP69gBWfFpyZOlJTg4l5byPExHVBXXyVB0RPZv69gBe6sY7hxNR3cXESQeyxo3RcuVKyBo3NnQoRPWWVAp09TF0FEREmjFx0oHUwgJWnTsbOgwiIiIykDo1x6mRPTAlrOTfuqggNRVJS5eiIDXV0KEQERHVe3UxL6hTiZOjPfBueMm/dVHRgwe4t24dih48MHQoRERE9V5dzAvqVOJEREREVJcxcSIiIiLSEhMnIiIiIi0xcdKB1MYG9oMHQ2pjY+hQiIiIyAB4OwIdyJyd4fHxx4YOg4iIiAyER5x0oMjLQ+7161Dk5Rk6FCIiIjIAJk46yEtMxL8hIchLTDR0KERERGQATJyIiIiItMTEiYiIiEhLTJyIiIiItMTESUcSExNDh0BEREQGwtsR6MC8dWt0OnbM0GEQERGRgfCIExEREZGWmDjpIDchAf+OHo3chARDh0JEREQGwMRJByI/H7lXrkDk5xs6FCIiIjIAJk5EREREWmLiRERERKQlJk5EREREWmLipANTFxc0nTcPpi4uhg6FiIiIDID3cdKBsbU17Hr3NnQYREREZCA84qSDwvR0pPz0EwrT0w0dChERERkAEycdFKal4c6iRShMSzN0KERERGQATJyIiIiItMTEiYiIiEhLTJyIiIiItMTESQdSS0vYBAZCamlp6FCIiIjIAHg7Ah3I3NzQfOFCQ4dBREREBsIjTjoQRUUofPgQoqjI0KEQERGRATBx0kFufDzOv/gicuPjDR0KERERGQATJyIiIiItMXEiIiIi0hITJyIiIiItMXEiIiIi0hJvR6ADsxYt4BMTAyMzM0OHQkRERAbAxEkHEqmUN78kIiJ6hvFUnQ7ybt3CtYgI5N26ZehQiIiIyACYOOlAkZODzH/+gSInx9ChEBERkQEwcSIiIiLSEhMnIiIiIi0xcSIiIiLSEhMnHZg4OqLx++/DxNHR0KEQERGRAfB2BDowsbNDoxEjDB0GERERGQiPOOmgKCMD6bt2oSgjw9ChEBERkQEwcdJBQXIyEj/5BAXJyYYOhYiIiAyAiRMRERGRlpg4EREREWmJiRMRERGRlpg46cBILoeFtzeM5HJDh0JEREQGwNsR6EDu4YHWkZGGDoOIiIgMhEeciIiIiLTExEkHOXFxON25M3Li4gwdChERERkAEyciIiIiLTFxIiIiItISEyciIiIiLTFxIiIiItISb0egA7mnJ9r+/jtMGzUydChERERkAEycdGAkk0HeuLGhwyAiIiID4ak6HeQnJSHh44+Rn5Rk6FCIiIjIAJg46aA4KwsP/vwTxVlZhg6FiIiIDICJExEREZGWmDgRERERaUmryeFCCABAZmamXoOp63Kys5FdXIzM7GwUPeNjQUREVF+U5jel+U5FtEqcsv5vTk9jXlFWoksXQ0dARERENSwrKws2NjYV1pEILdIrhUKBu3fvwsrKChKJpMYCfNpkZmaicePGuH37NqytrQ0dzlOFY1c9HL/q4fhVD8ev6jh21VNb4yeEQFZWFlxcXGBkVPEsJq2OOBkZGcHNza1GgqsPrK2t+QGoIo5d9XD8qofjVz0cv6rj2FVPbYxfZUeaSnFyOBEREZGWmDgRERERaYmJkw5kMhlmzpwJmUxm6FCeOhy76uH4VQ/Hr3o4flXHsaueujh+Wk0OJyIiIiIecSIiIiLSGhMnIiIiIi0xcSIiIiLSEhMnIiIiIi0904nTsmXL4OHhAblcDn9/f5w4caLC+r/++itat24NuVwOb29v7Nq1S2W5EAKffPIJnJ2dYWZmht69e+PatWv67IJB1eT4FRYWYvr06fD29oaFhQVcXFwwduxY3L17V9/dMJiafv896T//+Q8kEgkWLVpUw1HXDfoYu8uXL2PQoEGwsbGBhYUFunTpglu3bumrCwZV0+OXnZ2NiIgIuLm5wczMDF5eXli5cqU+u2BQuozfpUuXMGzYMHh4eFT4mdT1NXma1fT4ffHFF+jSpQusrKzQqFEjvPzyy7hy5Yr+OiCeURs3bhSmpqbi+++/F5cuXRJvvPGGsLW1FSkpKRrrHzlyREilUvHVV1+Jf//9V3z00UfCxMREXLhwQVln3rx5wsbGRmzdulWcO3dODBo0SHh6eorc3Nza6latqenxe/Tokejdu7eIiooScXFx4tixY8LPz0/4+vrWZrdqjT7ef6V+++030aFDB+Hi4iIWLlyo557UPn2MXXx8vGjQoIF47733xJkzZ0R8fLzYtm1buW0+zfQxfm+88YZo1qyZiI6OFgkJCWLVqlVCKpWKbdu21Va3ao2u43fixAkxbdo08csvvwgnJyeNn0ld23ya6WP8goODRWRkpLh48aKIjY0V/fr1E02aNBHZ2dl66cMzmzj5+fmJt99+W/m8uLhYuLi4iC+++EJj/REjRoj+/furlPn7+4sJEyYIIYRQKBTCyclJzJ8/X7n80aNHQiaTiV9++UUPPTCsmh4/TU6cOCEAiJs3b9ZM0HWIvsbvzp07wtXVVVy8eFG4u7vXy8RJH2MXEhIixowZo5+A6xh9jF/btm3FZ599plKnU6dO4sMPP6zByOsGXcfvSeV9JqvT5tNGH+NXVmpqqgAgDhw4UJ1Qy/VMnqorKCjA6dOn0bt3b2WZkZERevfujWPHjmlc59ixYyr1ASA4OFhZPyEhAffu3VOpY2NjA39//3LbfFrpY/w0ycjIgEQiga2tbY3EXVfoa/wUCgVeffVVvPfee2jbtq1+gjcwfYydQqHAzp070bJlSwQHB6NRo0bw9/fH1q1b9dYPQ9HXey8gIADbt29HUlIShBCIjo7G1atX8dJLL+mnIwZSlfEzRJt1VW31NSMjAwDQoEGDGmvzSc9k4nT//n0UFxfD0dFRpdzR0RH37t3TuM69e/cqrF/6ry5tPq30MX5l5eXlYfr06QgNDa13P4ypr/H78ssvYWxsjEmTJtV80HWEPsYuNTUV2dnZmDdvHvr06YO//voLQ4YMwdChQ3HgwAH9dMRA9PXeW7JkCby8vODm5gZTU1P06dMHy5YtQ48ePWq+EwZUlfEzRJt1VW30VaFQYMqUKejWrRvatWtXI22WZayXVomqobCwECNGjIAQAitWrDB0OE+F06dPY/HixThz5gwkEomhw3mqKBQKAMDgwYPx7rvvAgB8fHxw9OhRrFy5EkFBQYYM76mwZMkS/PPPP9i+fTvc3d1x8OBBvP3223BxcVE7WkWkT2+//TYuXryIw4cP620bz+QRp4YNG0IqlSIlJUWlPCUlBU5OThrXcXJyqrB+6b+6tPm00sf4lSpNmm7evIm9e/fWu6NNgH7G79ChQ0hNTUWTJk1gbGwMY2Nj3Lx5E1OnToWHh4de+mEI+hi7hg0bwtjYGF5eXip12rRpU++uqtPH+OXm5uL//b//h2+++QYDBw5E+/btERERgZCQECxYsEA/HTGQqoyfIdqsq/Td14iICOzYsQPR0dFwc3OrdnvleSYTJ1NTU/j6+mL//v3KMoVCgf3796Nr164a1+natatKfQDYu3evsr6npyecnJxU6mRmZuL48ePltvm00sf4Af9Lmq5du4Z9+/bB3t5ePx0wMH2M36uvvorz588jNjZW+XBxccF7772HPXv26K8ztUwfY2dqaoouXbqoXb589epVuLu713APDEsf41dYWIjCwkIYGal+nUilUuXRvPqiKuNniDbrKn31VQiBiIgI/P777/j777/h6elZE+FWuMFn0saNG4VMJhPr1q0T//77r3jzzTeFra2tuHfvnhBCiFdffVXMmDFDWf/IkSPC2NhYLFiwQFy+fFnMnDlT4+0IbG1txbZt28T58+fF4MGD6/XtCGpy/AoKCsSgQYOEm5ubiI2NFcnJycpHfn6+QfqoT/p4/5VVX6+q08fY/fbbb8LExESsXr1aXLt2TSxZskRIpVJx6NChWu+fvulj/IKCgkTbtm1FdHS0uHHjhoiMjBRyuVwsX7681vunb7qOX35+vjh79qw4e/ascHZ2FtOmTRNnz54V165d07rN+kQf4/fWW28JGxsbERMTo/LdkZOTo5c+PLOJkxBCLFmyRDRp0kSYmpoKPz8/8c8//yiXBQUFibCwMJX6mzZtEi1bthSmpqaibdu2YufOnSrLFQqF+Pjjj4Wjo6OQyWSiV69e4sqVK7XRFYOoyfFLSEgQADQ+oqOja6lHtaum339l1dfESQj9jN3atWtF8+bNhVwuFx06dBBbt27VdzcMpqbHLzk5WYSHhwsXFxchl8tFq1atxNdffy0UCkVtdKfW6TJ+5e3bgoKCtG6zvqnp8SvvuyMyMlIv8Uv+b6NEREREVIlnco4TERERUVUwcSIiIiLSEhMnIiIiIi0xcSIiIiLSEhMnIiIiIi0xcSIiIiLSEhMnIiIiIi0xcSIiIiLSEhMnIqoRs2bNgo+Pj/J5eHg4Xn75ZeVzIQTefPNNNGjQABKJBLGxsRrLiIjqMiZORKQXixcvxrp165TPd+/ejXXr1mHHjh1ITk5Gu3btNJY9K+7cuQNTU9Na63N4eDgkEonyYW9vjz59+uD8+fO1sn2i+oKJE9FTrLCw0NAhlMvGxga2trbK59evX4ezszMCAgLg5OQEY2NjjWW6EkKgqKioBiOvHevWrcOIESOQmZmJ48eP18o2+/Tpg+TkZCQnJ2P//v0wNjbGgAEDamXbRPUFEyciLe3evRvdu3eHra0t7O3tMWDAAFy/fl2lzp07dxAaGooGDRrAwsICnTt3VvlS/OOPP9ClSxfI5XI0bNgQQ4YMUS6TSCTYunWrSnu2trbKozaJiYmQSCSIiopCUFAQ5HI5fv75Z6SnpyM0NBSurq4wNzeHt7c3fvnlF5V2FAoFvvrqKzRv3hwymQxNmjTB559/DgDo2bMnIiIiVOqnpaXB1NQU+/fvL3c85s2bB0dHR1hZWWH8+PHIy8tTWf7kqbrw8HC88847uHXrFiQSCTw8PDSWlcb6xRdfwNPTE2ZmZujQoQM2b96sbDcmJgYSiQR//vknfH19IZPJcPjwYa3X279/Pzp37gxzc3MEBATgypUrKnFX9Brl5+dj2rRpcHV1hYWFBfz9/RETE1PuGJVHCIHIyEi8+uqrGDVqFNauXatzG1Uhk8ng5OQEJycn+Pj4YMaMGbh9+zbS0tJqZftE9YJefjqYqB7avHmz2LJli7h27Zo4e/asGDhwoPD29hbFxcVCCCGysrJE06ZNRWBgoDh06JC4du2aiIqKEkePHhVCCLFjxw4hlUrFJ598Iv79918RGxsr5s6dq2wfgPj9999VtmljY6P8he/SXwn38PAQW7ZsETdu3BB3794Vd+7cEfPnzxdnz54V169fF99++62QSqXi+PHjynbef/99YWdnJ9atWyfi4+PFoUOHxJo1a4QQQvz888/Czs5O5OXlKet/8803wsPDo9xft4+KihIymUx89913Ii4uTnz44YfCyspKdOjQQVknLCxMDB48WAghxKNHj8Rnn30m3NzcRHJyskhNTdVYJoQQc+bMEa1btxa7d+8W169fF5GRkUImk4mYmBghhBDR0dECgGjfvr3466+/RHx8vEhPT9d6PX9/fxETEyMuXbokAgMDRUBAgDLmyl6j119/XQQEBIiDBw+K+Ph4MX/+fCGTycTVq1crfvOUsX//fuHk5CSKiorEhQsXhJWVlcjOztapDV09+XoIUfJ+nTBhgmjevLnyPUxElWPiRFRFaWlpAoC4cOGCEEKIVatWCSsrK5Genq6xfteuXcXo0aPLbU/bxGnRokWVxta/f38xdepUIYQQmZmZQiaTKROlsnJzc4WdnZ2IiopSlrVv317MmjWr3Pa7du0qJk6cqFLm7+9fbuIkhBALFy4U7u7uKuuULcvLyxPm5ubKZLPU+PHjRWhoqBDifwnQ1q1bq7Tevn37lMt37twpAIjc3Fxlv8p7jW7evCmkUqlISkpSKe/Vq5f44IMPNK5TnlGjRokpU6Yon3fo0EH5OutLWFiYkEqlwsLCQlhYWAgAwtnZWZw+fVqv2yWqb3iqjkhL165dQ2hoKJo2bQpra2vlqaVbt24BAGJjY9GxY0c0aNBA4/qxsbHo1atXtePo3LmzyvPi4mLMnj0b3t7eaNCgASwtLbFnzx5lXJcvX0Z+fn6525bL5Xj11Vfx/fffAwDOnDmDixcvIjw8vNwYLl++DH9/f5Wyrl27VqNXJeLj45GTk4MXX3wRlpaWysePP/6odlr0yXHQZb327dsr/+/s7AwASE1NBVDxa3ThwgUUFxejZcuWKts4cOCA2jYq8ujRI/z2228YM2aMsmzMmDGVnq6bMWOGyuRuTY+4uLgK23jhhRcQGxuL2NhYnDhxAsHBwejbty9u3rypdfxEzzrdZ2ISPaMGDhwId3d3rFmzBi4uLlAoFGjXrh0KCgoAAGZmZhWuX9lyiUQCIYRKmabJ3xYWFirP58+fj8WLF2PRokXw9vaGhYUFpkyZonVcAPD666/Dx8cHd+7cQWRkJHr27Al3d/dK16tp2dnZAICdO3fC1dVVZZlMJlN5/uQ46LKeiYmJ8v8SiQRAybwqoOKxys7OhlQqxenTpyGVSlWWWVpalt+pMjZs2IC8vDyVxFMIAYVCgatXr6Jly5Ya15s6dWqFySwANG3atMLlFhYWaN68ufL5d999BxsbG6xZswZz5szRug9EzzImTkRaSE9Px5UrV7BmzRoEBgYCAA4fPqxSp3379vjuu+/w4MEDjUed2rdvj/3792PcuHEat+Hg4IDk5GTl82vXriEnJ6fS2I4cOYLBgwcrj2CUfgF7eXkBAFq0aAEzMzPs378fr7/+usY2vL290blzZ6xZswYbNmzA0qVLK9xmmzZtcPz4cYwdO1ZZ9s8//1Qaa2W8vLwgk8lw69YtBAUF6X29sip6jTp27Iji4mKkpqYq3wNVsXbtWo1J0MSJE/H9999j3rx5GtdzcHCAg4NDlberiUQigZGREXJzc2u0XaL6jIkTkRbs7Oxgb2+P1atXw9nZGbdu3cKMGTNU6oSGhmLu3Ll4+eWX8cUXX8DZ2Rlnz56Fi4sLunbtipkzZ6JXr15o1qwZRo4ciaKiIuzatQvTp08HUHJ129KlS9G1a1cUFxdj+vTpKkdHytOiRQts3rwZR48ehZ2dHb755hukpKQoEye5XI7p06fj/fffh6mpKbp164a0tDRcunQJ48ePV7bz+uuvIyIiAhYWFipXkmkyefJkhIeHo3PnzujWrRt+/vlnXLp0qdIjHpWxsrLCtGnT8O6770KhUKB79+7IyMjAkSNHYG1tjbCwsBpdr6yKXqOWLVti9OjRGDt2LL7++mt07NgRaWlp2L9/P9q3b4/+/ftX2n5sbCzOnDmDn3/+Ga1bt1ZZFhoais8++wxz5syp0m0ZtJGfn4979+4BAB4+fIilS5ciOzsbAwcO1Mv2iOolQ0+yInpa7N27V7Rp00bIZDLRvn17ERMTozahOzExUQwbNkxYW1sLc3Nz0blzZ5Wr27Zs2SJ8fHyEqampaNiwoRg6dKhyWVJSknjppZeEhYWFaNGihdi1a5fGyeFnz55ViSs9PV0MHjxYWFpaikaNGomPPvpIjB07VmVidnFxsZgzZ45wd3cXJiYmokmTJipXiwlRcpWVubm52qTv8nz++eeiYcOGwtLSUoSFhYn333+/2pPDhRBCoVCIRYsWiVatWgkTExPh4OAggoODxYEDB4QQ/5vk/fDhw2qvd/bsWQFAJCQkKMsqeo0KCgrEJ598Ijw8PISJiYlwdnYWQ4YMEefPn9dqzCIiIoSXl5fGZcnJycLIyEhs27ZNq7Z0FRYWJgAoH1ZWVqJLly5i8+bNetkeUX0lEaLMpAoieiYlJiaiWbNmOHnyJDp16mTocIiI6iQmTkTPuMLCQqSnp2PatGlISEjAkSNHDB0SEVGdxdsRED3jjhw5AmdnZ5w8eRIrV640dDhERHUajzgRERERaYlHnIiIiIi0xMSJiIiISEtMnIiIiIi0xMSJiIiISEtMnIiIiIi0xMSJiIiISEtMnIiIiIi0xMSJiIiISEv/HyCaI40Ee/J5AAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# viz: the gap and its CI, with zero marked. If the band touches zero, you cannot claim a winner.\n", "fig, ax = plt.subplots(figsize=(6, 2.4))\n", "ax.axvline(0, color=\"#c33\", ls=\"--\", lw=1, label=\"no difference\")\n", "ax.errorbar([point], [0], xerr=[[point - dlo], [dhi - point]], fmt=\"o\",\n", " color=\"#1E40FF\", capsize=6, label=\"A − B (95% CI)\")\n", "ax.set_yticks([]); ax.set_xlabel(\"accuracy difference A − B\")\n", "ax.set_title(\"the +5 leaderboard gap, with its confidence interval\")\n", "ax.legend(loc=\"upper right\", fontsize=8); plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "72305110", "metadata": {}, "source": [ "> **Common confusion:** \"the CI crosses zero, so A and B are the same.\" No. Failing to *reject* equality is not *proving* equality. The honest statement is \"this eval is too small to distinguish them\", which is a call for a bigger eval (or a more powerful paired test), not a verdict of a tie. Absence of evidence is not evidence of absence, a distinction this whole field gets wrong constantly.\n" ] }, { "cell_type": "markdown", "id": "08446250", "metadata": {}, "source": [ "### Exercise 23.3 — McNemar's test for paired models\n", "`Difficulty 3/5 · ~15 min`\n", "\n", "The bootstrap gave a CI; McNemar's test gives a p-value for the *same* paired question, and it is the textbook-correct test for two models on shared binary outcomes. It looks only at the *discordant* items, the ones where exactly one model is right:\n", "\n", "- $b_{01}$ = count of items where A is **wrong** and B is **right**\n", "- $b_{10}$ = count of items where A is **right** and B is **wrong**\n", "\n", "Under the null \"the two models are equally good\", a discordant item is equally likely to favor either, so $b_{01}$ and $b_{10}$ should be about equal. The continuity-corrected statistic is $\\chi^2 = (|b_{01} - b_{10}| - 1)^2 / (b_{01} + b_{10})$, with 1 degree of freedom.\n", "\n", "Fill in `mcnemar(a_correct, b_correct)` returning the p-value. The check compares against `statsmodels` if present, else against `scipy.stats.chi2` directly, and against a hand-computed toy.\n" ] }, { "cell_type": "code", "execution_count": 10, "id": "a6e1410c", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.719849Z", "iopub.status.busy": "2026-06-10T20:48:42.719779Z", "iopub.status.idle": "2026-06-10T20:48:42.725733Z", "shell.execute_reply": "2026-06-10T20:48:42.725365Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ -- ] 23.3 mcnemar (balanced): not attempted yet — fill in the TODO above, then re-run.\n", "[ -- ] 23.3 mcnemar (lopsided): not attempted yet — fill in the TODO above, then re-run.\n", "[ -- ] 23.3 mcnemar vs closed form: not attempted yet — fill in the TODO above, then re-run.\n" ] }, { "data": { "text/plain": [ "False" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def mcnemar(a_correct, b_correct):\n", " \"\"\"Paired binary model comparison. a_correct, b_correct: arrays of 0/1 (or bool).\n", " Return the continuity-corrected McNemar p-value.\"\"\"\n", " a = np.asarray(a_correct).astype(bool)\n", " b = np.asarray(b_correct).astype(bool)\n", " # TODO 1: b01 = count where A wrong AND B right; b10 = count where A right AND B wrong\n", " b01 = None\n", " b10 = None\n", " attempted(b01, b10)\n", " if b01 + b10 == 0:\n", " return 1.0 # no discordant pairs -> no evidence of a difference\n", " # TODO 2: continuity-corrected chi-square statistic with df=1, then its upper-tail p-value\n", " # chi2 = (abs(b01 - b10) - 1)**2 / (b01 + b10)\n", " chi2_stat = None\n", " attempted(chi2_stat)\n", " from scipy.stats import chi2 as chi2_dist\n", " return float(1 - chi2_dist.cdf(chi2_stat, df=1))\n", "\n", "# self-checks (run this cell)\n", "def _mcnemar_toy():\n", " # 6 items A-wrong/B-right, 6 items A-right/B-wrong, rest agree -> perfectly balanced -> p ~ 1\n", " a = np.array([1,1,1,1,1,1, 0,0,0,0,0,0, 1,1])\n", " b = np.array([0,0,0,0,0,0, 1,1,1,1,1,1, 1,1])\n", " p = mcnemar(a, b) # b10=6, b01=6 -> chi2=(0-1)^2/12 small -> p high\n", " assert p > 0.5, f\"balanced discordants ({p:.3f}) should give a large p-value (no evidence of a winner)\"\n", "\n", "def _mcnemar_lopsided():\n", " # 20 vs 2 discordant -> strong evidence one model is better -> small p\n", " a = np.array([1]*20 + [0]*2 + [1]*10)\n", " b = np.array([0]*20 + [1]*2 + [1]*10)\n", " p = mcnemar(a, b)\n", " assert p < 0.001, f\"a 20-vs-2 split ({p:.4f}) should be highly significant\"\n", "\n", "def _mcnemar_vs_ref():\n", " from scipy.stats import chi2 as chi2_dist\n", " a = np.array([1]*15 + [0]*5 + [1]*30 + [0]*30)\n", " b = np.array([0]*15 + [1]*5 + [1]*30 + [0]*30)\n", " b01 = int((~a.astype(bool) & b.astype(bool)).sum())\n", " b10 = int((a.astype(bool) & ~b.astype(bool)).sum())\n", " want = float(1 - chi2_dist.cdf((abs(b01 - b10) - 1)**2 / (b01 + b10), df=1))\n", " check_close(mcnemar(a, b), want, atol=1e-9, msg=\"must match the closed-form continuity-corrected p\")\n", "\n", "check(\"23.3 mcnemar (balanced)\", _mcnemar_toy)\n", "check(\"23.3 mcnemar (lopsided)\", _mcnemar_lopsided)\n", "check(\"23.3 mcnemar vs closed form\", _mcnemar_vs_ref)" ] }, { "cell_type": "markdown", "id": "60cabb4a", "metadata": {}, "source": [ "
Hint 1 (conceptual)`b01` and `b10` are counts of *discordant* pairs. With boolean arrays `a` and `b`: `(~a & b)` is the mask \"A wrong, B right\". Sum it for `b01`; swap for `b10`. The items where both agree (both right or both wrong) carry no information about which is better and drop out.
\n", "\n", "
Hint 2 (the lines)\n", "\n", "```python\n", "b01 = int((~a & b).sum())\n", "b10 = int((a & ~b).sum())\n", "chi2_stat = (abs(b01 - b10) - 1) ** 2 / (b01 + b10)\n", "```\n", "
\n", "\n", "
Help — \"p-value is above 1 or negative\"The p-value is `1 - chi2.cdf(stat, df=1)`, an upper-tail probability; it is always in `[0, 1]`. If you got the lower tail (`chi2.cdf` without the `1 -`), you computed `1 - p`. If `stat` is huge, double-check the `-1` continuity correction is *inside* the square, applied to `|b01 - b10|`.
\n" ] }, { "cell_type": "code", "execution_count": 11, "id": "4ed0d1de", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.726734Z", "iopub.status.busy": "2026-06-10T20:48:42.726659Z", "iopub.status.idle": "2026-06-10T20:48:42.730153Z", "shell.execute_reply": "2026-06-10T20:48:42.729827Z" }, "collapsed": true, "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ ok ] 23.3 mcnemar (balanced)\n", "[ ok ] 23.3 mcnemar (lopsided)\n", "[ ok ] 23.3 mcnemar vs closed form\n", "discordant items: A-wrong/B-right = 17, A-right/B-wrong = 28\n", "McNemar p-value: 0.136 -> NOT significant at 0.05\n" ] } ], "source": [ "#@title Solution { display-mode: \"form\" }\n", "# solution: redefines mcnemar; the checks below re-verify the reference.\n", "def mcnemar(a_correct, b_correct):\n", " a = np.asarray(a_correct).astype(bool)\n", " b = np.asarray(b_correct).astype(bool)\n", " b01 = int((~a & b).sum())\n", " b10 = int((a & ~b).sum())\n", " if b01 + b10 == 0:\n", " return 1.0\n", " chi2_stat = (abs(b01 - b10) - 1) ** 2 / (b01 + b10)\n", " from scipy.stats import chi2 as chi2_dist\n", " return float(1 - chi2_dist.cdf(chi2_stat, df=1))\n", "\n", "check(\"23.3 mcnemar (balanced)\", _mcnemar_toy, required=True)\n", "check(\"23.3 mcnemar (lopsided)\", _mcnemar_lopsided, required=True)\n", "check(\"23.3 mcnemar vs closed form\", _mcnemar_vs_ref, required=True)\n", "p_mc = mcnemar(scores_A, scores_B)\n", "b01 = int((~scores_A.astype(bool) & scores_B.astype(bool)).sum())\n", "b10 = int((scores_A.astype(bool) & ~scores_B.astype(bool)).sum())\n", "print(f\"discordant items: A-wrong/B-right = {b01}, A-right/B-wrong = {b10}\")\n", "print(f\"McNemar p-value: {p_mc:.3f} -> {'SIGNIFICANT' if p_mc < 0.05 else 'NOT significant at 0.05'}\")" ] }, { "cell_type": "markdown", "id": "8cdf3f69", "metadata": {}, "source": [ "> **Interpretation.** McNemar agrees with the bootstrap: the p-value is well above 0.05, so the +5 gap is not significant. Two independent methods, the same verdict. That agreement is itself a check, if the bootstrap CI crossed zero but McNemar said p = 0.001, one of them would be wrong and you would go hunting for the bug.\n", "\n", "> **Key takeaways.** The bootstrap puts a CI on *any* metric by resampling items; pairing (resample item indices, score both models on them) is what makes a two-model comparison honest. On a 200-item eval, a +5 measured gap routinely has a CI that crosses zero. McNemar's exact paired test is the textbook companion and should agree with the paired bootstrap. \"Not significant\" means \"this eval is too small to tell\", not \"they are equal\".\n" ] }, { "cell_type": "markdown", "id": "1509029e", "metadata": {}, "source": [ "## Part 3 — Pass@k and the metric you choose\n", "\n", "> **Objectives.** Implement the unbiased pass@k estimator from the HumanEval paper, show that pass@1 and pass@10 are different measurements of the same samples, and make the point that the aggregation a report picks is usually the one that flatters it.\n", "\n", "When a model samples *several* completions per item, you have a choice of how to score. **Pass@1**: sample one, score it, the most pessimistic and the one that matches a single-shot deployment. **Pass@k**: sample $n \\geq k$, score the item as solved if *any* of the top-$k$ passes, more optimistic, and the right number only if your deployment actually takes $k$ shots.\n", "\n", "The naive pass@k estimate (sample exactly $k$, check if any passed) is *high-variance*. The HumanEval paper's unbiased estimator samples $n > k$ and computes the expected pass@k in closed form:\n", "\n", "$$\\text{pass@}k = 1 - \\frac{\\binom{n - c}{k}}{\\binom{n}{k}} = 1 - \\prod_{i=n-c+1}^{n} \\left(1 - \\frac{k}{i}\\right)$$\n", "\n", "where $c$ is how many of the $n$ samples were correct. The product form is numerically stable (no giant binomials). When $n - c < k$ there are not enough wrong samples to fill a failing top-$k$, so pass@k is exactly 1.\n" ] }, { "cell_type": "markdown", "id": "4bc533c5", "metadata": {}, "source": [ "### Exercise 23.4 — The unbiased pass@k estimator\n", "`Difficulty 2/5 · ~12 min`\n", "\n", "Fill in `pass_at_k(n, c, k)` using the stable product form above. `n` = samples drawn, `c` = how many were correct, `k` = the top-k we score. Return a probability in `[0, 1]`.\n", "\n", "The checks verify the boundary cases (all correct -> 1, none correct -> 0) and a hand-computed value: with `n=20, c=5, k=1`, pass@1 is just the fraction correct, `5/20 = 0.25`.\n" ] }, { "cell_type": "code", "execution_count": 12, "id": "cc5dbce5", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.731164Z", "iopub.status.busy": "2026-06-10T20:48:42.731087Z", "iopub.status.idle": "2026-06-10T20:48:42.734933Z", "shell.execute_reply": "2026-06-10T20:48:42.734603Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ -- ] 23.4 pass@k boundaries: not attempted yet — fill in the TODO above, then re-run.\n", "[ -- ] 23.4 pass@k toy value: not attempted yet — fill in the TODO above, then re-run.\n", "[ -- ] 23.4 pass@k monotone in k: not attempted yet — fill in the TODO above, then re-run.\n" ] }, { "data": { "text/plain": [ "False" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def pass_at_k(n, c, k):\n", " \"\"\"Unbiased pass@k (Chen et al., HumanEval). n samples, c correct, top-k scored.\n", " Returns P(at least one of a random top-k of the n samples is correct).\"\"\"\n", " # TODO 1: if there are fewer than k wrong samples (n - c < k), every top-k contains a\n", " # correct one, so pass@k is exactly 1.0. Handle that first.\n", " if None: # replace None with the condition n - c < k\n", " return 1.0\n", " # TODO 2: otherwise use the stable product: 1 - prod_{i=n-c+1}^{n} (1 - k/i)\n", " # (np.arange(n - c + 1, n + 1) gives the i values)\n", " prod = None\n", " attempted(prod)\n", " return float(1.0 - prod)\n", "\n", "# self-checks (run this cell)\n", "def _pass_k_boundaries():\n", " check_close(pass_at_k(20, 20, 1), 1.0, msg=\"all 20 correct -> pass@1 = 1\")\n", " check_close(pass_at_k(20, 0, 5), 0.0, msg=\"0 correct -> pass@5 = 0\")\n", "\n", "def _pass_k_toy():\n", " # pass@1 = c/n exactly: 5 of 20 correct -> 0.25\n", " check_close(pass_at_k(20, 5, 1), 0.25, msg=\"pass@1 with c=5,n=20 is 5/20 = 0.25\")\n", "\n", "def _pass_k_monotone():\n", " # for fixed (n, c), pass@k is non-decreasing in k (more tries can only help)\n", " vals = [pass_at_k(20, 5, k) for k in range(1, 11)]\n", " assert all(vals[i] <= vals[i+1] + 1e-12 for i in range(len(vals)-1)), \\\n", " f\"pass@k must be non-decreasing in k; got {[round(v,3) for v in vals]}\"\n", "\n", "check(\"23.4 pass@k boundaries\", _pass_k_boundaries)\n", "check(\"23.4 pass@k toy value\", _pass_k_toy)\n", "check(\"23.4 pass@k monotone in k\", _pass_k_monotone)" ] }, { "cell_type": "markdown", "id": "8a4c84de", "metadata": {}, "source": [ "
Hint 1 (conceptual)The product runs over the integers `i = n-c+1, ..., n`, each contributing a factor `(1 - k/i)`. `np.prod(1 - k / np.arange(n - c + 1, n + 1))` does it in one line. Guard the `n - c < k` case first, otherwise the product is empty and you would return `1 - 1 = 0` instead of `1`.
\n", "\n", "
Hint 2 (the lines)\n", "\n", "```python\n", "if n - c < k:\n", " return 1.0\n", "prod = np.prod(1.0 - k / np.arange(n - c + 1, n + 1))\n", "return float(1.0 - prod)\n", "```\n", "
\n", "\n", "
Help — \"pass@1 is not equal to c/n\"Check the product range. For `k=1`, the product is over `i = n-c+1..n` of `(1 - 1/i)`, which telescopes to `(n-c)/n`, so `1 - (n-c)/n = c/n`. If you get a different number, your `arange` bounds are off by one, the upper bound of `np.arange` is exclusive, so it must be `n + 1`.
\n" ] }, { "cell_type": "code", "execution_count": 13, "id": "3c195890", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.735713Z", "iopub.status.busy": "2026-06-10T20:48:42.735644Z", "iopub.status.idle": "2026-06-10T20:48:42.738547Z", "shell.execute_reply": "2026-06-10T20:48:42.738227Z" }, "collapsed": true, "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ ok ] 23.4 pass@k boundaries\n", "[ ok ] 23.4 pass@k toy value\n", "[ ok ] 23.4 pass@k monotone in k\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#@title Solution { display-mode: \"form\" }\n", "# solution: redefines pass_at_k; the checks below re-verify the reference.\n", "def pass_at_k(n, c, k):\n", " if n - c < k:\n", " return 1.0\n", " prod = np.prod(1.0 - k / np.arange(n - c + 1, n + 1))\n", " return float(1.0 - prod)\n", "\n", "check(\"23.4 pass@k boundaries\", _pass_k_boundaries, required=True)\n", "check(\"23.4 pass@k toy value\", _pass_k_toy, required=True)\n", "check(\"23.4 pass@k monotone in k\", _pass_k_monotone, required=True)" ] }, { "cell_type": "markdown", "id": "b6dcef2c", "metadata": {}, "source": [ "Now the point that matters for reading reports: the *same* model on the *same* samples can look weak or strong depending only on which `k` you quote. We simulate a model that gets each item right with probability 0.3 per sample, draw `n=20` samples per item, and compute pass@1 vs pass@10.\n" ] }, { "cell_type": "code", "execution_count": 14, "id": "8dcbf592", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.739283Z", "iopub.status.busy": "2026-06-10T20:48:42.739214Z", "iopub.status.idle": "2026-06-10T20:48:42.748553Z", "shell.execute_reply": "2026-06-10T20:48:42.748176Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pass@1 = 0.310\n", "pass@2 = 0.523\n", "pass@5 = 0.842\n", "pass@10 = 0.974\n", "\n", "same model, same samples: pass@1 ~ 0.30, but pass@10 is far higher.\n", "a report that quotes pass@10 while the product ships at n=1 is overselling by this whole gap.\n" ] } ], "source": [ "# same samples, different aggregation: a model with per-sample success 0.3, n=20 draws/item\n", "P_CORRECT, N_SAMPLES, N_PROBLEMS = 0.30, 20, 500\n", "g3 = np.random.default_rng(SEED)\n", "# c[i] = number of the 20 samples that passed, for each of 500 problems\n", "c_per_problem = g3.binomial(N_SAMPLES, P_CORRECT, N_PROBLEMS)\n", "for k in (1, 2, 5, 10):\n", " score = np.mean([pass_at_k(N_SAMPLES, int(c), k) for c in c_per_problem])\n", " print(f\"pass@{k:<2d} = {score:.3f}\")\n", "print(f\"\\nsame model, same samples: pass@1 ~ {P_CORRECT:.2f}, but pass@10 is far higher.\")\n", "print(\"a report that quotes pass@10 while the product ships at n=1 is overselling by this whole gap.\")" ] }, { "cell_type": "markdown", "id": "8ae4b327", "metadata": {}, "source": [ "> **Interpretation.** pass@1 lands near the per-sample success rate (0.30), while pass@10 is much higher, because with ten tries you only need to get lucky once. Neither is wrong; they answer different questions. The methodological sin is *mismatch*: quoting pass@10 when the deployment takes one shot, or comparing your pass@10 against a baseline's pass@1. When you read \"our model solves 78% of problems\", the first question is \"at what `k`, with how many samples?\".\n", "\n", "> **Key takeaways.** Pass@k aggregates multiple samples; the unbiased estimator (product form) is stable and is the HumanEval standard. pass@1 and pass@k are different measurements of the same samples, and pass@k is only the honest number if the deployment actually takes `k` shots. The aggregation a report chooses is usually the one that flatters it; always ask which `k` and how many samples.\n" ] }, { "cell_type": "markdown", "id": "6fbddf34", "metadata": {}, "source": [ "## Part 4 — LLM-as-judge, and position bias\n", "\n", "> **Objectives.** Build a *deterministic mock judge* whose biases we plant on purpose, measure its position bias, and implement the swap check that keeps only order-stable preferences. Because the judge is a small Python function, every number here is reproducible and the bias is a known ground truth, the only way to teach judge methodology without an API key.\n", "\n", "For open-ended tasks (summaries, chat, creative writing) there is no string to match against, so the eval asks a strong LLM to judge. The dominant protocol is *pairwise*: show the judge two responses, A and B, ask which is better. Pairwise is robust to absolute-scale drift but exposes a notorious failure mode: **position bias**, the judge systematically prefers whichever response is in a particular slot, regardless of content.\n", "\n", "We cannot call a real frontier judge in CI (no key, no network, the spec forbids it on the canonical path). So we build a mock judge that is *honest about being a mock*: it scores each response on a hidden true quality, and we add a controllable position bias on top. With the ground truth in hand, we can prove the swap check works.\n" ] }, { "cell_type": "code", "execution_count": 15, "id": "af8cbde6", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.749333Z", "iopub.status.busy": "2026-06-10T20:48:42.749264Z", "iopub.status.idle": "2026-06-10T20:48:42.751853Z", "shell.execute_reply": "2026-06-10T20:48:42.751602Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "built a fair judge and a position-biased judge (bias strength 0.20).\n", "true qualities: A=0.6, B=0.55 (A is really a little better)\n" ] } ], "source": [ "# a deterministic mock judge. NOT a real LLM: a small function whose biases we control,\n", "# so \"does the swap check catch position bias?\" has a provable answer. Honest simulation,\n", "# not magic-string theater: it never pattern-matches the learner's text, it scores a hidden\n", "# quality and (optionally) adds a fixed nudge toward whichever response is shown FIRST.\n", "def make_mock_judge(position_bias_strength=0.0, noise=0.05, seed=SEED):\n", " \"\"\"Return judge(quality_first, quality_second) -> 'first' or 'second'.\n", " quality_* are the TRUE hidden qualities of the two responses (higher = better).\n", " position_bias_strength nudges the judge toward the FIRST-shown response.\"\"\"\n", " jrng = np.random.default_rng(seed)\n", " def judge(quality_first, quality_second):\n", " # the judge perceives quality + a bias toward slot 1 + a little noise\n", " score_first = quality_first + position_bias_strength + noise * jrng.standard_normal()\n", " score_second = quality_second + noise * jrng.standard_normal()\n", " return \"first\" if score_first >= score_second else \"second\"\n", " return judge\n", "\n", "# two responses with a TRUE quality gap: A is genuinely a bit better than B.\n", "TRUE_QUALITY_A, TRUE_QUALITY_B = 0.60, 0.55\n", "fair_judge = make_mock_judge(position_bias_strength=0.0)\n", "biased_judge = make_mock_judge(position_bias_strength=0.20) # strong nudge toward whatever is shown first\n", "print(\"built a fair judge and a position-biased judge (bias strength 0.20).\")\n", "print(f\"true qualities: A={TRUE_QUALITY_A}, B={TRUE_QUALITY_B} (A is really a little better)\")" ] }, { "cell_type": "markdown", "id": "78e727f6", "metadata": {}, "source": [ "> **Note:** the spec calls this out by name. A mock that only \"works\" because it pattern-matches strings the learner types is the *magic-string* anti-pattern. This judge does not look at any text; it scores numeric qualities we pass it and adds a bias we set. The honest framing is \"a simulated judge with a known bias\", and everything we conclude is a property of that simulation, which is the point: we can *prove* the mitigation works against a bias we control.\n" ] }, { "cell_type": "markdown", "id": "695fbf25", "metadata": {}, "source": [ "### Exercise 23.5 — Measure the bias, then the swap check that kills it\n", "`Difficulty 3/5 · ~18 min`\n", "\n", "Two functions.\n", "\n", "First, `a_win_rate(judge, qa, qb, n)`: run `n` comparisons with A shown **first** (judge sees `qa, qb`) and `n` with A shown **second** (judge sees `qb, qa`), and return `(rate_A_when_first, rate_A_when_second)` where each rate is the fraction of comparisons A won. For a fair judge these two rates are about equal; for a position-biased judge they diverge.\n", "\n", "Second, `stable_pref(judge, qa, qb)`: run *one* comparison each way and return `\"A\"`, `\"B\"`, or `\"tie\"`. A is the winner only if A wins when shown first **and** wins when shown second; symmetrically for B; otherwise `\"tie\"` (the preference flipped with order, so it is an artifact, not a signal).\n", "\n", "The checks verify the fair judge shows little first/second gap, the biased judge shows a large one, and the swap check returns `\"tie\"` for the biased judge on near-equal responses.\n" ] }, { "cell_type": "code", "execution_count": 16, "id": "41e05b46", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.752797Z", "iopub.status.busy": "2026-06-10T20:48:42.752696Z", "iopub.status.idle": "2026-06-10T20:48:42.757235Z", "shell.execute_reply": "2026-06-10T20:48:42.756873Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ -- ] 23.5 fair judge symmetric: not attempted yet — fill in the TODO above, then re-run.\n", "[ -- ] 23.5 biased judge asymmetric: not attempted yet — fill in the TODO above, then re-run.\n", "[ -- ] 23.5 swap check flags bias: not attempted yet — fill in the TODO above, then re-run.\n" ] }, { "data": { "text/plain": [ "False" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def a_win_rate(judge, qa, qb, n=N_JUDGE):\n", " \"\"\"Return (rate_A_won_when_A_shown_first, rate_A_won_when_A_shown_second).\"\"\"\n", " # When A is shown FIRST, the judge returns 'first' iff it picked A.\n", " # When A is shown SECOND (we pass qb, qa), the judge returns 'second' iff it picked A.\n", " # TODO 1: count, over n trials, how often A wins when shown FIRST: judge(qa, qb) == \"first\"\n", " a_first = None\n", " # TODO 2: count, over n trials, how often A wins when shown SECOND: judge(qb, qa) == \"second\"\n", " a_second = None\n", " attempted(a_first, a_second)\n", " return a_first / n, a_second / n\n", "\n", "def stable_pref(judge, qa, qb):\n", " \"\"\"Swap check: A wins only if it wins BOTH orderings; same for B; else 'tie'.\"\"\"\n", " a_first = judge(qa, qb) # A shown first\n", " a_second = judge(qb, qa) # A shown second\n", " a_won_first = (a_first == \"first\")\n", " a_won_second = (a_second == \"second\")\n", " # TODO 3: return 'A' if A won both orderings, 'B' if A LOST both orderings, else 'tie'\n", " result = None\n", " attempted(result)\n", " return result\n", "\n", "# self-checks (run this cell)\n", "def _fair_judge_symmetric():\n", " rf, rs = a_win_rate(fair_judge, TRUE_QUALITY_A, TRUE_QUALITY_B, n=N_JUDGE)\n", " assert abs(rf - rs) < 0.15, \\\n", " f\"a FAIR judge should win A at ~equal rates whether first ({rf:.2f}) or second ({rs:.2f})\"\n", "\n", "def _biased_judge_asymmetric():\n", " rf, rs = a_win_rate(biased_judge, TRUE_QUALITY_A, TRUE_QUALITY_B, n=N_JUDGE)\n", " # biased toward FIRST: A wins much more when shown first than when shown second\n", " assert rf - rs > 0.3, \\\n", " f\"a FIRST-biased judge should win A far more when A is first ({rf:.2f}) than second ({rs:.2f})\"\n", "\n", "def _swap_check_flags_bias():\n", " # on near-equal responses the biased judge's raw vote flips with order; swap check -> tie\n", " jb = make_mock_judge(position_bias_strength=0.20, noise=0.0)\n", " assert stable_pref(jb, 0.50, 0.50) == \"tie\", \\\n", " \"with bias and equal quality, the preference flips on swap, so the swap check must return 'tie'\"\n", "\n", "check(\"23.5 fair judge symmetric\", _fair_judge_symmetric)\n", "check(\"23.5 biased judge asymmetric\", _biased_judge_asymmetric)\n", "check(\"23.5 swap check flags bias\", _swap_check_flags_bias)" ] }, { "cell_type": "markdown", "id": "29fa2c59", "metadata": {}, "source": [ "
Hint 1 (conceptual)\"A shown first\" means you call `judge(qa, qb)` and A won if the judge returned `\"first\"`. \"A shown second\" means you call `judge(qb, qa)` and A won if the judge returned `\"second\"`. The swap check declares A the winner only when both of those happen for a single pair.
\n", "\n", "
Hint 2 (pseudocode)\n", "\n", "```python\n", "# a_win_rate (count over n trials, in both orderings):\n", "a_first = sum(1 for _ in range(n) if judge(qa, qb) == \"first\")\n", "a_second = sum(1 for _ in range(n) if judge(qb, qa) == \"second\")\n", "\n", "# stable_pref:\n", "if a_won_first and a_won_second: result = \"A\"\n", "elif (not a_won_first) and (not a_won_second): result = \"B\"\n", "else: result = \"tie\"\n", "```\n", "
\n", "\n", "
Help — \"the biased judge passes the symmetric check too\"You likely used the *fair* judge in both rate calls, or built `a_win_rate` to count wins in only one ordering. Re-read: A's win when shown *second* is detected by `judge(qb, qa) == \"second\"`, not `== \"first\"`. The asymmetry only appears if you query both orderings correctly.
\n" ] }, { "cell_type": "code", "execution_count": 17, "id": "31396045", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.758094Z", "iopub.status.busy": "2026-06-10T20:48:42.758016Z", "iopub.status.idle": "2026-06-10T20:48:42.761871Z", "shell.execute_reply": "2026-06-10T20:48:42.761530Z" }, "collapsed": true, "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ ok ] 23.5 fair judge symmetric\n", "[ ok ] 23.5 biased judge asymmetric\n", "[ ok ] 23.5 swap check flags bias\n", "fair judge: A wins 0.76 when first, 0.70 when second (gap 0.06)\n", "biased judge: A wins 1.00 when first, 0.01 when second (gap 0.98)\n" ] } ], "source": [ "#@title Solution { display-mode: \"form\" }\n", "# solution: redefines both functions; the checks below re-verify the reference.\n", "def a_win_rate(judge, qa, qb, n=N_JUDGE):\n", " a_first = sum(1 for _ in range(n) if judge(qa, qb) == \"first\")\n", " a_second = sum(1 for _ in range(n) if judge(qb, qa) == \"second\")\n", " return a_first / n, a_second / n\n", "\n", "def stable_pref(judge, qa, qb):\n", " a_won_first = (judge(qa, qb) == \"first\")\n", " a_won_second = (judge(qb, qa) == \"second\")\n", " if a_won_first and a_won_second:\n", " return \"A\"\n", " if (not a_won_first) and (not a_won_second):\n", " return \"B\"\n", " return \"tie\"\n", "\n", "check(\"23.5 fair judge symmetric\", _fair_judge_symmetric, required=True)\n", "check(\"23.5 biased judge asymmetric\", _biased_judge_asymmetric, required=True)\n", "check(\"23.5 swap check flags bias\", _swap_check_flags_bias, required=True)\n", "rf_fair, rs_fair = a_win_rate(fair_judge, TRUE_QUALITY_A, TRUE_QUALITY_B)\n", "rf_bias, rs_bias = a_win_rate(biased_judge, TRUE_QUALITY_A, TRUE_QUALITY_B)\n", "print(f\"fair judge: A wins {rf_fair:.2f} when first, {rs_fair:.2f} when second (gap {abs(rf_fair-rs_fair):.2f})\")\n", "print(f\"biased judge: A wins {rf_bias:.2f} when first, {rs_bias:.2f} when second (gap {abs(rf_bias-rs_bias):.2f})\")" ] }, { "cell_type": "code", "execution_count": 18, "id": "672b858a", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.762746Z", "iopub.status.busy": "2026-06-10T20:48:42.762675Z", "iopub.status.idle": "2026-06-10T20:48:42.811127Z", "shell.execute_reply": "2026-06-10T20:48:42.810734Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAEiCAYAAAAPh11JAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAATT5JREFUeJzt3XdYFFfbBvB7QapIERAMoihYUFrEgIqIBSVq9FVjiZqA2BJjJ9bEAjY0lmBsxE58NVHUVI1GECOJhaiAFQQFrCA2UFDKcr4/fJnPlbar4KLev+vi0j175pxnZ2dnn505c0YmhBAgIiIiogppqDsAIiIiotcFEyciIiIiJTFxIiIiIlISEyciIiIiJTFxIiIiIlISEyciIiIiJTFxIiIiIlISEyciIiIiJTFxIiIiIlISE6fXjEwmQ2BgoFJ1bWxsMHTo0CqNpzSBgYGQyWS4c+dOhXXVFWNVGzp0KAwMDNQdRplUWe82Njb44IMPKqx3+PBhyGQyHD58+OWCew3bfF1t2bIFMpkMJ0+eVHcob4QOHTrAwcFB3WG8UsWfp127dqk7lFeGidNr7ujRowgMDMSDBw/UHcpbJzc3F4GBgW/EF/CFCxcQGBiI1NRUdYdCr7l9+/Yp/ePudXTz5k0EBgYiLi5ObTFs374dISEhauv/bVdD3QGQah4/fowaNf7/bTt69CiCgoIwdOhQGBsbK9RNTEyEhkb1zo1fhxjLkpubi6CgIABPf2m+Tp5f7xcuXEBQUBA6dOgAGxubF2qzffv2ePz4MbS1tSspyqppk6rWvn37sHr16jc2ebp58yaCgoJgY2MDFxcXtcSwfft2nDt3DhMnTlRL/287Jk6vGV1dXaXr6ujoVGEkleN1iPFNVBXrXUNDQ6XtU11tEtHrJycnBzVr1lR3GAB4qu6FFY/jSUhIwIABA2BoaAhTU1NMmDABT548UahbWFiIefPmwdbWFjo6OrCxscGXX36JvLw8hXonT56Ej48PzMzMoKenh4YNG2LYsGEKdZ4d4xQYGIgpU6YAABo2bAiZTAaZTCadbiltHMuVK1fQv39/1K5dG/r6+mjdujX27t2rUKf4nPXOnTuxYMEC1KtXD7q6uujcuTOSk5OVXkd37typcN08H+O9e/cwefJkODo6wsDAAIaGhujWrRvi4+NLtL9y5Uq0aNEC+vr6MDExQatWrbB9+3al43sZqampMDc3BwAEBQVJ6/75X9k3btxA7969YWBgAHNzc0yePBlyuVyhTlFREUJCQtCiRQvo6urCwsICn376Ke7fv19uDL/++itkMhnOnDkjle3evRsymQx9+/ZVqGtvb4+BAwdKj59d71u2bEH//v0BAB07dpRey/OnIP/++2+4ublBV1cXjRo1wvfff6/wfGljh4rHfFy4cAEdO3aEvr4+rKys8PXXX5f72sprs6zxWR06dChx5O/69evo3bs3atasiTp16mDSpEklPnfFVq9ejUaNGkFPTw9ubm6Ijo4utc28vDzMmTMHdnZ20NHRgbW1NaZOnVpmuy/ST35+PmbPng1XV1cYGRmhZs2a8PT0RFRUlEJbqampkMlkWLp0Kb755hs0aNAAenp68PLywrlz55SKB3h69PTTTz+FqakpDA0N4evrW+r298cff8DT0xM1a9ZErVq10KNHD5w/f156fujQoVi9ejUASNuRTCYDALRs2bLEduno6FhiG96xYwdkMhkuXrwold24cQPDhg2DhYUFdHR00KJFC2zatKlEfMq+NzKZDGPHjsXPP/8MBwcHqc39+/eXu54OHz6M9957DwDg7+8vvb4tW7Yo1FNme3/R7ahDhw7Yu3cv0tLSpP5tbGwghICZmRkCAgKkukVFRTA2NoampqbCcI7FixejRo0aePTokVR26NAh6b01NjbGf/7zH4X3oCJyuRxffvklLC0tUbNmTfTq1QvXrl1TqBMdHY3+/fujfv360mueNGkSHj9+rFCveIzo5cuX0b17d9SqVQtDhgwBACQlJeHDDz+EpaUldHV1Ua9ePXz00UfIyspSOtaXxSNOL2nAgAGwsbFBcHAwjh8/jm+//Rb3799X+FIZMWIEwsLC0K9fP3zxxRc4ceIEgoODcfHiRfz0008AgNu3b6Nr164wNzfH9OnTYWxsjNTUVOzZs6fMvvv27YtLly7hhx9+wDfffAMzMzMAkL7Qn5eRkYG2bdsiNzcX48ePh6mpKcLCwtCrVy/s2rULffr0Uai/aNEiaGhoYPLkycjKysLXX3+NIUOG4MSJE5W2bp535coV/Pzzz+jfvz8aNmyIjIwMfPfdd/Dy8sKFCxfwzjvvAADWr1+P8ePHo1+/flJCdubMGZw4cQKDBw8uNy5lBq0DQK1atco8MmNubo61a9di9OjR6NOnj/SF4OTkJNWRy+Xw8fGBu7s7li5dioiICCxbtgy2trYYPXq0VO/TTz/Fli1b4O/vj/HjxyMlJQWrVq1CbGws/vnnH2hpaZUaQ7t27SCTyXDkyBGp3+joaGhoaODvv/+W6mVmZiIhIQFjx44ttZ327dtj/Pjx+Pbbb/Hll1/C3t4eAKR/ASA5ORn9+vXD8OHD4efnh02bNmHo0KFwdXVFixYtyl2P9+/fx/vvv4++fftiwIAB2LVrF6ZNmwZHR0d069at3GVfxuPHj9G5c2dcvXoV48ePxzvvvIOtW7fi0KFDJequXbsWY8eOhaenJyZNmoTU1FT07t0bJiYmqFevnlSvqKgIvXr1wt9//41Ro0bB3t4eZ8+exTfffINLly7h559/LjcmZfvJzs7Ghg0bMGjQIIwcORIPHz7Exo0b4ePjg5iYmBKniL7//ns8fPgQY8aMwZMnT7BixQp06tQJZ8+ehYWFRYXrauzYsTA2NkZgYCASExOxdu1apKWlSYkrAGzduhV+fn7w8fHB4sWLkZubi7Vr16Jdu3aIjY2FjY0NPv30U9y8eRMHDx7E1q1bFfrw9PTEDz/8ID2+d+8ezp8/Dw0NDURHRytsw+bm5tL2l5GRgdatW0vJjrm5Of744w8MHz4c2dnZ0ukqVd+bv//+G3v27MHnn3+OWrVq4dtvv8WHH36Iq1evwtTUtNT1ZG9vj7lz52L27NkYNWoUPD09AQBt27aV6iizvb/MdvTVV18hKysL169fxzfffAMAMDAwgEwmg4eHB44cOSLVPXPmDLKysqChoYF//vkHPXr0kNbxu+++K13AEhERgW7duqFRo0YIDAzE48ePsXLlSnh4eOD06dNKnb5fsGABZDIZpk2bhtu3byMkJATe3t6Ii4uDnp4eACA8PBy5ubkYPXo0TE1NERMTg5UrV+L69esIDw9XaK+wsBA+Pj5o164dli5dCn19feTn58PHxwd5eXkYN24cLC0tcePGDfz+++948OABjIyMKoyzUgh6IXPmzBEARK9evRTKP//8cwFAxMfHCyGEiIuLEwDEiBEjFOpNnjxZABCHDh0SQgjx008/CQDi33//LbdfAGLOnDnS4yVLlggAIiUlpUTdBg0aCD8/P+nxxIkTBQARHR0tlT18+FA0bNhQ2NjYCLlcLoQQIioqSgAQ9vb2Ii8vT6q7YsUKAUCcPXu23BiVXTelxfjkyRMpjmIpKSlCR0dHzJ07Vyr7z3/+I1q0aFFuHGUBoNTf5s2by20nMzOzxPtRzM/PTwBQiFkIId59913h6uoqPY6OjhYAxLZt2xTq7d+/v9Ty57Vo0UIMGDBAetyyZUvRv39/AUBcvHhRCCHEnj17Klzv4eHhAoCIiooq0UeDBg0EAHHkyBGp7Pbt20JHR0d88cUXUlnxdvNsG15eXgKA+P7776WyvLw8YWlpKT788MNyX1tZbT4f+7N9eXl5SY9DQkIEALFz506pLCcnR9jZ2Sm0mZeXJ0xNTcV7770nCgoKpLpbtmwRABTa3Lp1q9DQ0FD4DAkhRGhoqAAg/vnnnzJfiyr9FBYWKnz2hBDi/v37wsLCQgwbNkwqS0lJEQCEnp6euH79ulR+4sQJAUBMmjSpzHiEEGLz5s0CgHB1dRX5+flS+ddffy0AiF9++UUI8XQ/YWxsLEaOHKmwfHp6ujAyMlIoHzNmjCjtq6V4G7tw4YIQQohff/1V6OjoiF69eomBAwdK9ZycnESfPn2kx8OHDxd169YVd+7cUWjvo48+EkZGRiI3N1cIodp7A0Boa2uL5ORkqSw+Pl4AECtXrix3nf37779l7h+U3d5fZjsSQogePXqIBg0alChfsmSJ0NTUFNnZ2UIIIb799lvRoEED4ebmJqZNmyaEEEIulwtjY2OFbcPFxUXUqVNH3L17VyqLj48XGhoawtfXt9xYij+jVlZWUr9CCLFz504BQKxYsUIqK36vnhUcHCxkMplIS0uTyor3n9OnT1eoGxsbKwCI8PDwcmOqajxV95LGjBmj8HjcuHEAng6QfPbfZw+fAsAXX3wBANJpsuKB3b///jsKCgqqJNZ9+/bBzc0N7dq1k8oMDAwwatQopKam4sKFCwr1/f39FQblFv+6unLlilL9VbRuSqOjoyMNWpbL5bh79y4MDAzQtGlTnD59WqpnbGyM69ev499//1UqlmcdPHhQqT8fHx+V237eZ599pvDY09NTYf2Fh4fDyMgIXbp0wZ07d6Q/V1dXGBgYlDg18zxPT09ER0cDAB4+fIj4+HiMGjUKZmZmUnl0dDSMjY1f6jLp5s2bS+8/8PSIW9OmTZXaFgwMDPDxxx9Lj7W1teHm5qb0dvSi9u3bh7p166Jfv35Smb6+PkaNGqVQ7+TJk7h79y5GjhypcOHFkCFDYGJiolA3PDwc9vb2aNasmcL71alTJwAo9/1SpR9NTU3ps1dUVIR79+6hsLAQrVq1UvgcFOvduzesrKykx25ubnB3dy/3s/asUaNGKRzZHD16NGrUqCEtf/DgQTx48ACDBg1SeN2amppwd3evcDsF/n//UXxEJDo6Gu+99x66dOkibasPHjzAuXPnpLpCCOzevRs9e/aEEEKhbx8fH2RlZUnrQ9X3xtvbG7a2ttJjJycnGBoavvR2qcz2/jLbUXk8PT0hl8tx9OhRAE/Xsaenp8J+4ty5c3jw4IG0jm/duoW4uDgMHToUtWvXltpycnJCly5dlN6GfH19UatWLelxv379ULduXYXli488AU/HLN25cwdt27aFEAKxsbEl2nz2yDwA6YjSgQMHkJubq1RcVYGn6l5S48aNFR7b2tpCQ0NDGmeUlpYGDQ0N2NnZKdSztLSEsbEx0tLSAABeXl748MMPERQUhG+++QYdOnRA7969MXjw4EobyJuWlgZ3d/cS5cWHxNPS0hS+XOvXr69Qr3jnXtHYm2IVrZvSFBUVYcWKFVizZg1SUlIUxgM9e/h82rRpiIiIgJubG+zs7NC1a1cMHjwYHh4eFcbl7e2tVPwvS1dXt8RpUxMTE4X1l5SUhKysLNSpU6fUNm7fvl1uH56enggNDUVycjIuX74MmUyGNm3aSDvKkSNHIjo6Gh4eHi919eLz20Jpr6Us9erVk073PLvss+Na0tPTFZ43MjJS2Mm+iLS0NNjZ2ZXou2nTpiXqASjxGa1Ro0aJUxRJSUm4ePFimafDy3u/VOkHAMLCwrBs2TIkJCQo/Jhq2LBhibrPf9YAoEmTJti5c2eZ8ZS3vIGBAerWrSt9VpOSkgBA+mJ/nqGhYYV9WFhYoHHjxoiOjsann36K6OhodOzYEe3bt8e4ceNw5coVXLx4EUVFRdKXemZmJh48eIB169Zh3bp1pbZbvM5VfW9eZpsujzLb+8tsR+Vp2bIl9PX1ER0dDR8fH0RHRyMoKAiWlpZYuXIlnjx5IiVQxT+gi7fL5z8XwNPvhgMHDig1MPv5bUgmk8HOzk5hf3/16lXMnj0bv/76a4n1/PwYpRo1aiicvgaebvsBAQFYvnw5tm3bBk9PT/Tq1Qsff/zxqztNByZOle75D0xF5c8+v2vXLhw/fhy//fYbDhw4gGHDhmHZsmU4fvy4WiZT1NTULLVcCPFC7VW0DgBg4cKFmDVrFoYNG4Z58+ahdu3a0NDQwMSJE1FUVCTVs7e3R2JiIn7//Xfs378fu3fvxpo1azB79mxpioCyPP8lXZaX/fIua/09q6ioCHXq1MG2bdtKfb6sHWux4p3fkSNHcOXKFbRs2VIaSPztt9/i0aNHiI2NxYIFC1R/Ac94mW1BmWXr1q2r8NzmzZvLnKCzrO1ILpcrtc5fRlFRERwdHbF8+fJSn7e2tq6Ufv773/9i6NCh6N27N6ZMmYI6depAU1MTwcHBuHz5cqX0oYriz97WrVthaWlZ4vlnj6CVp127doiMjMTjx49x6tQpzJ49Gw4ODjA2NkZ0dDQuXrwIAwMDvPvuuwr9fvzxx/Dz8yu1zeKxUaq+N5W9f1Ol3arajrS0tODu7o4jR44gOTkZ6enp8PT0hIWFBQoKCnDixAlER0ejWbNmFe5bKptcLkeXLl1w7949TJs2Dc2aNUPNmjVx48YNDB06VGH/DiiefXjWsmXLMHToUPzyyy/4888/MX78eGkc7fOJVlVh4vSSkpKSFH4BJicno6ioSPoF2aBBAxQVFSEpKUlhsG1GRgYePHiABg0aKLTXunVrtG7dGgsWLMD27dsxZMgQ/PjjjxgxYkSp/SuTjBRr0KABEhMTS5QnJCRIz1emitZNaXbt2oWOHTti48aNCuUPHjyQBr8Xq1mzJgYOHIiBAwciPz8fffv2xYIFCzBjxoxyL2F//ku6LOV9eQOqrfuy2NraIiIiAh4eHi+UpNWvXx/169dHdHQ0rly5Iv1Sb9++PQICAhAeHg65XI727duX205lvJaXcfDgQYXH5Q04NzExKXXC17S0NDRq1Eh63KBBA5w7dw5CCIXX9/xnoHi7T05ORseOHaXywsJCpKamKgz4t7W1RXx8PDp37qzyOlOln127dqFRo0bYs2ePQj9z5swpte3iI0LPunTpktJzciUlJSnE9OjRI9y6dQvdu3cHAOmUVp06dSo8YlveevH09MTmzZvx448/Qi6Xo23bttDQ0EC7du2kxKlt27ZS8mFubo5atWpBLpdX2O/LvDeqqKzP/cvEWtE6Xrx4MSIiImBmZoZmzZpBJpOhRYsWiI6ORnR0tMKdAIq3y7K+G8zMzJSaBuD5bVAIgeTkZGm7Pnv2LC5duoSwsDD4+vpK9Z7/7CvD0dERjo6OmDlzJo4ePQoPDw+EhoZi/vz5Krf1IjjG6SUVX3pbbOXKlQAgXT1RvON5fpbX4l8axVc53L9/v8QvneIrZ8q7PLV4g1Zm5vDu3bsjJiYGx44dk8pycnKwbt062NjYoHnz5hW2oYqK1k1pNDU1S6yH8PBw3LhxQ6Hs7t27Co+1tbXRvHlzCCEqHCNWWWOc9PX1ASi37ssyYMAAyOVyzJs3r8RzhYWFSrXt6emJQ4cOISYmRkqcXFxcUKtWLSxatAh6enpwdXUttw1VtqOq4O3trfBXXnJra2uL48ePIz8/Xyr7/fffS1z63L17d9y8eVPhVhC5ubklTvm0atUKpqamWL9+PQoLC6Xybdu2lTidMGDAANy4cQPr168vEdfjx4+Rk5NTZtyq9FOcODz7WThx4oTCZ/dZP//8s8JnJCYmBidOnFD6qsV169YpfG7Wrl2LwsJCaXkfHx8YGhpi4cKFpX6+MjMzpf+Xty0Vb5+LFy+Gk5OTdHrF09MTkZGROHnypMJYOk1NTXz44YfYvXt3qdMrPNvvy7w3qqiMz8rLxlqzZs0yL7/39PREXl4eQkJCpCtvi8u3bt2KmzdvKqzjunXrwsXFBWFhYQqv6dy5c/jzzz+l77CKFF/ZWWzXrl24deuWtA2Vtk0LIbBixQql2geeXm367GcHeJpEaWhoKD0dSGXgEaeXlJKSgl69euH999/HsWPH8N///heDBw+Gs7MzAMDZ2Rl+fn5Yt24dHjx4AC8vL8TExCAsLAy9e/eWfuWFhYVhzZo16NOnD2xtbfHw4UOsX78ehoaG5W64xV+IX331FT766CNoaWmhZ8+epf5CmD59On744Qd069YN48ePR+3atREWFoaUlBTs3r270mfwrmjdlOaDDz7A3Llz4e/vj7Zt2+Ls2bPYtm2bwpEEAOjatSssLS3h4eEBCwsLXLx4EatWrUKPHj0UBiiWprLGOOnp6aF58+bYsWMHmjRpgtq1a8PBwUGlQdheXl749NNPERwcjLi4OHTt2hVaWlpISkpCeHg4VqxYoTC4uTSenp7Ytm0bZDKZdOpOU1MTbdu2xYEDB9ChQ4cKZ952cXGBpqYmFi9ejKysLOjo6KBTp05ljr1SpxEjRmDXrl14//33MWDAAFy+fBn//e9/FQb6AsDIkSOxatUq+Pr64tSpU6hbty62bt0qJbzFtLW1ERgYiHHjxqFTp04YMGAAUlNTsWXLFtja2ir8uv/kk0+wc+dOfPbZZ4iKioKHhwfkcjkSEhKwc+dOHDhwAK1atSo1blX6+eCDD7Bnzx706dMHPXr0QEpKCkJDQ9G8eXOFuXeK2dnZoV27dhg9erT0pWlqaoqpU6cqtU7z8/PRuXNnDBgwAImJiVizZg3atWuHXr16AXg6hmnt2rX45JNP0LJlS3z00UcwNzfH1atXsXfvXnh4eGDVqlUA/n+fNH78ePj4+EBTUxMfffSRFKelpSUSExOli0WAp0dIp02bBgAKX+rA02lRoqKi4O7ujpEjR6J58+a4d+8eTp8+jYiICNy7d++l3xtV2NrawtjYGKGhoahVqxZq1qwJd3f3UseeleVlY3V1dcWOHTsQEBCA9957DwYGBujZsycAoE2bNqhRowYSExMVLoRo37491q5dC6DkOl6yZAm6deuGNm3aYPjw4dJ0BEZGRkrPAF+7dm20a9cO/v7+yMjIQEhICOzs7DBy5EgAQLNmzWBra4vJkyfjxo0bMDQ0xO7du1UaU3bo0CGMHTsW/fv3R5MmTVBYWIitW7dKCfYr8+ov5HszFF9yf+HCBdGvXz9Rq1YtYWJiIsaOHSseP36sULegoEAEBQWJhg0bCi0tLWFtbS1mzJghnjx5ItU5ffq0GDRokKhfv77Q0dERderUER988IE4efKkQlso5fL3efPmCSsrK6GhoaEwNUFpl21fvnxZ9OvXTxgbGwtdXV3h5uYmfv/9d4U6xZeXPn/JZ/GlzxVdpq/KuiltOoIvvvhC1K1bV+jp6QkPDw9x7NixEpeaf/fdd6J9+/bC1NRU6OjoCFtbWzFlyhSRlZVVbmyV7ejRo8LV1VVoa2srvDd+fn6iZs2aJeoXr5vnrVu3Tri6ugo9PT1Rq1Yt4ejoKKZOnSpu3rxZYQznz5+Xpo941vz58wUAMWvWrBLLlLZtrF+/XjRq1EhoamoqXK7foEED0aNHjxJtPP+elDUdQWnTRvj5+ZV6OfXzSmtTCCGWLVsmrKyshI6OjvDw8BAnT54sEY8QQqSlpYlevXoJfX19YWZmJiZMmCBN9fB8m8WXbuvo6Ag3Nzfxzz//CFdXV/H+++8r1MvPzxeLFy8WLVq0EDo6OsLExES4urqKoKAgpbY/ZfopKioSCxculOq9++674vfffy+x3oo/k0uWLBHLli0T1tbWQkdHR3h6eipMP1GW4ukI/vrrLzFq1ChhYmIiDAwMxJAhQxQuTS8WFRUlfHx8hJGRkdDV1RW2trZi6NChCvupwsJCMW7cOGFubi5kMlmJ7b14uowdO3YorFN9fX2hra1dYh8hhBAZGRlizJgxwtraWmhpaQlLS0vRuXNnsW7dOoV6yr43AMSYMWNK9FPWVBfP++WXX0Tz5s1FjRo1FPaJqmzvL7MdPXr0SAwePFgYGxsLACXafu+99wQAceLECans+vXrAoCwtrYutc2IiAjh4eEh9PT0hKGhoejZs6c0dUR5ij+jP/zwg5gxY4aoU6eO0NPTEz169FCYYkAIIS5cuCC8vb2FgYGBMDMzEyNHjpSmgXj2e6Ws/eeVK1fEsGHDhK2trdDV1RW1a9cWHTt2FBERERXGWZlkQrzkSLi3VGBgIIKCgpCZmVli7A0RVY7IyEh4e3sjOjpaYRqNV6GoqAjm5ubo27dvqadUqkM/qampaNiwIZYsWYLJkydXUYRE9CyOcSKiauvWrVsAUOU/Tp48eVJibN3333+Pe/fuVeoNnF9VP0RUdTjGiYiqnZycHGzbtg0rVqxAvXr10KRJkyrt7/jx45g0aRL69+8PU1NTnD59Ghs3boSDg4N0H7/XqR8iqjpMnIio2snMzMS4cePg6OiIzZs3V/qFC8+zsbGBtbU1vv32W9y7dw+1a9eGr68vFi1aVOHA+urYDxFVHbWOcTpy5AiWLFmCU6dO4datW/jpp5/Qu3fvcpc5fPgwAgICcP78eVhbW2PmzJnlzrVDREREVFnUOsYpJycHzs7OJeb7KUtKSgp69OiBjh07Ii4uDhMnTsSIESNw4MCBKo6UiIiISM1HnJ4lk8kqPOI0bdo07N27V2EitI8++ggPHjzA/v37X0GURERE9DZ7rcY4HTt2rMTkhT4+Ppg4cWKZy+Tl5SnMKFp8p3FTU1O132aCiIiI1E8IgYcPH+Kdd96pcEzla5U4paenw8LCQqHMwsIC2dnZePz4can3+goODq7wpq9ERERE165dq/Bmwa9V4vQiZsyYgYCAAOlxVlYW6tevj2vXrsHQ0FCNkRERKadFD3VHQOf3qjsCqkrZ2dmwtrau8JZdwGuWOFlaWiIjI0OhLCMjA4aGhmXeWV5HRwc6Ojolyg0NDZk4EdFrQeO12lO/mfh18XZQZgjPazVzeJs2bRAZGalQdvDgQbRp00ZNEREREdHbRK2J06NHjxAXF4e4uDgAT6cbiIuLw9WrVwE8Pc3m6+sr1f/ss89w5coVTJ06FQkJCVizZg127tyJSZMmqSN8IiIiesuo9QDwyZMn0bFjR+lx8VgkPz8/bNmyBbdu3ZKSKABo2LAh9u7di0mTJkm3YtiwYQN8fHwqLSa5XI6CgoJKa49eL1paWtDU1FR3GEREVE1Vm3mcXpXs7GwYGRkhKyurxBinR48e4fr16yVuwklvD5lMhnr16sHAwEDdoRBJGnSsuA5VrbQodUdAVam83OB5HHL4P3K5HNevX4e+vj7Mzc05x9NbSAiBzMxMXL9+HY0bN+aRJyIiKoGJ0/8UFBRACAFzc/Myr9CjN5+5uTlSU1NRUFDAxImIiEp4ra6qexV4pOntxvefiIjKw8Spmnv48CEMDAwwfPhwlZdNTU2FsbFx5Qelovz8fHzwwQdwdHTEmDFjEBoaiiVLlqjcTmBgIJ48eVIFERIRESmHp+rKUZUDMpUdaLhjxw64urpiz549WLFixWs5aDk2NhZJSUlITEyssK5cLi/zFFlQUBAmTpwIXV3dyg6RiIhIKTziVM1t3LgR06ZNQ/v27bFjx45S6xQVFWHs2LGwt7eHs7MzXF1dFY7MzJkzB66urrCzs8O+ffuk8gMHDqBly5ZwcnKCl5cXLly4AAAYPHgwtm/fDgBYs2YNtLW1kZOTAwDo1KkTjhw5Ih3NKqvtYhcuXMCQIUNw9epVuLi44Pvvv0dgYKB0Y+YtW7agY8eO+PDDD+Ho6IiYmBjMnz8f9vb2cHFxgYuLC9LS0vDZZ58BADw9PeHi4oLbt2+//MolIiJSEROnauzChQu4du0afHx8MHz4cGzcuLHUevHx8YiMjMT58+cRHx+PQ4cOQVtbG8DTe/M5OTnh1KlTWLVqlTRZ6O3btzF48GCEhYXhzJkzGDVqFPr16wchBLy9vREREQHg6czsrVq1wl9//YXc3FzEx8dLM7WX1fazmjdvjg0bNqBp06aIi4tTmNC02IkTJ7Bw4UKcPXsWzZo1w9KlS3H69GnExcXh6NGjsLCwQGhoKAAgOjoacXFxqFOnzsuvYCIiIhUxcarGNm7cCF9fX2hqaqJ79+5ISUnBxYsXS9Rr1KgRCgsLMWzYMISFhaGgoAAaGk/fWl1dXfTt2xfA01vWXL58GcDTZMXR0RGOjo4AgCFDhuDmzZu4ceMGvL29ERkZCblcjgsXLiAgIAARERGIjo6Gm5sbtLS0ym1bVW3btkXTpk0BPL2HYOPGjfHxxx/ju+++w71793hqjoiIqg0mTtVUQUEBtm7dirCwMNjY2MDOzg65ubmlHnUyMjLCuXPnMHjwYCQkJMDJyQnJyckAnt7kuPhKMU1NTcjl8gr7rl+/PnR0dLBt2za4urqic+fOiIqKQkREBDp37izVe5G2S/PsuC1NTU0cP34cEydOxO3bt9G6dWtER0e/ULtERESVjYlTNfXrr7+iUaNGuHHjBlJTU5Gamorjx49j69atJW4Jk5mZiZycHHTt2hULFy6EjY2NNF6pLK1bt8bZs2dx7tw5AMCPP/4IKysrWFlZAQC8vb0xe/ZseHt7w8TEBFpaWggPD4e3t3fVvOD/efjwITIyMuDp6YlZs2ahXbt2iI2NBQDUqlULWVlZVdo/ERFReZg4VVMbN27EkCFDFMrs7e1hZWWF3377TaH82rVr6NKlC5ycnODg4AAHBwd069at3PbNzc2xbds2+Pr6wsnJCWvXrkV4eLh0BMnb2xtpaWlSouTt7Y2cnBw4OztX4qssKSsrC3379oWjoyOcnJxQUFAAPz8/AMAXX3yBLl26cHA4ERGpDe9V9z9PnjxBSkoKGjZsyDE1bzFuB1Qd8V516sd71b3ZVLlXHY84ERERESmJiRMRERGRkpg4ERERESmJiRMRERGRkpg4ERERESmJiRMRERGRkpg4ERERESmphroDqM6WhFVd21P8lKv38OFD1K1bFwMHDizzJr9lSU1NhYuLCx48eKB6gG+4Vq1aYenSpejQoYO6QyEiotcIjzhVczt27ICrqyv27NmDR48eqTscIiKitxoTp2pu48aNmDZtGtq3b48dO3aUWqeoqAhjx46Fvb09nJ2d4erqiidPnkjPz5kzB66urrCzs8O+ffuk8gMHDqBly5ZwcnKCl5eXdH+7wYMHY/v27QCANWvWQFtbGzk5OQCATp064ciRI0hNTYWxsXGZbT/rt99+g5OTE1xcXODg4IBffvkFAJCeno4BAwbAzc0Njo6OmDlzprTMxYsX4ePjAycnJzg5OSE0NBQAkJycDG9vb6m9n3/+WVpGJpNh4cKFcHNzQ8OGDbF582bpuaNHj0r9+/v7o7CwUOn3gIiIqBgTp2rswoULuHbtGnx8fDB8+PAyT9XFx8cjMjIS58+fR3x8PA4dOgRtbW0AT+/95uTkhFOnTmHVqlWYNGkSAOD27dsYPHgwwsLCcObMGYwaNQr9+vWDEALe3t6IiIgAABw8eBCtWrXCX3/9hdzcXMTHx6NNmzbltv28mTNn4rvvvkNcXBzOnDkDLy8vAICfnx/GjBmDmJgYxMbG4uTJkwgPD0dhYSH+85//YOjQoThz5gzOnDmDfv36AQCGDBmC/v3748yZMwgPD8fw4cORlpYm9aWjo4OYmBj88ccfGD9+PAoLC5Gfn4+BAwdi6dKlOHfuHAYNGoT4+PhKeIeIiOhtw8SpGtu4cSN8fX2hqamJ7t27IyUlBRcvXixRr1GjRigsLMSwYcMQFhaGgoICaGg8fWt1dXXRt29fAECbNm1w+fJlAMCJEyfg6OgIR0dHAE8Tkps3b+LGjRvw9vZGZGQk5HI5Lly4gICAAERERCA6Ohpubm7Q0tIqt+3nde7cGRMmTMDXX3+NM2fOwNjYGDk5OYiMjMSECRPg4uKCVq1aITk5GYmJiUhMTMSTJ08waNAgqQ0zMzM8fPgQp0+fxvDhwwEAjRs3Rrt27RAdHS3VK74xcrNmzVCjRg2kp6cjISEBNWrUkG5Y3LVrVzRq1OgF3xUiInqbMXGqpgoKCrB161aEhYXBxsYGdnZ2yM3NLfWok5GREc6dO4fBgwcjISEBTk5OSE5OBvD0CIxMJgMAaGpqQi6XV9h3/fr1oaOjg23btsHV1RWdO3dGVFQUIiIi0LlzZ6mesm0vX74cmzdvhr6+Pvz8/PD111+j+N7Sx48fR1xcHOLi4pCcnKxwuk4Zxf0Xe/bGvJqammWeknt+OSIiImUwcaqmfv31VzRq1Ag3btxAamoqUlNTcfz4cWzduhUFBQUKdTMzM5GTk4OuXbti4cKFsLGxkcYrlaV169Y4e/Yszp07BwD48ccfYWVlBSsrKwCAt7c3Zs+eDW9vb5iYmEBLSwvh4eHSURtVJCQkoEWLFhg7dixGjx6N48ePw8DAAB07dsSiRYukejdv3sT169fRtGlT6Ovr44cffpCeu3PnDmrVqoWWLVtKY5eSk5Px999/o3379uX236xZMxQWFiIq6untzSMiIso8OkZERFQeJk7V1MaNG6XTTsXs7e1hZWWF3377TaH82rVr6NKlC5ycnODg4AAHBwd069at3PbNzc2xbds2+Pr6wsnJCWvXrkV4eLh0JMbb2xtpaWlSouTt7Y2cnBw4Ozur/Fq+/PJLtGjRAu+++y62bt2KwMBAAMC2bduQnJwMBwcHODo6om/fvrh79y5q1KiBX375BZs3b4ajoyOcnZ2xe/duaZkdO3bA2dkZ/fr1w4YNG1C/fv1y+9fW1saOHTswadIkODo6Yvv27S/0OoiIiGSi+JzJWyI7OxtGRkbIysqCoaGhVP7kyROkpKSgYcOGCqd76O3C7YCqowYd1R0BpUWpOwKqSmXlBqXhESciIiIiJTFxIiIiIlISEyciIiIiJTFxes5bNuSLnsP3n4iIysOb/P6PlpYWZDIZMjMzYW5uznl+3kJCCGRmZkImk0mTfBIRET2LidP/aGpqol69erh+/TpSU1PVHQ6piUwmQ7169aCpqanuUIiIqBpSe+K0evVqLFmyBOnp6XB2dsbKlSvh5uZWZv2QkBCsXbsWV69ehZmZGfr164fg4OBKuXTcwMAAjRs3LjHBJL09tLS0mDQREVGZ1Jo47dixAwEBAQgNDYW7uztCQkLg4+ODxMRE1KlTp0T97du3Y/r06di0aRPatm2LS5cuYejQoZDJZFi+fHmlxKSpqckvTiIiIiqVWgeHL1++HCNHjoS/vz+aN2+O0NBQ6OvrY9OmTaXWP3r0KDw8PDB48GDY2Niga9euGDRoEGJiYl5x5ERERPQ2UlvilJ+fj1OnTinc+0xDQwPe3t44duxYqcu0bdsWp06dkhKlK1euYN++fejevXuZ/eTl5SE7O1vhj4iIiOhFqO1U3Z07dyCXy2FhYaFQbmFhgYSEhFKXGTx4MO7cuYN27dpBCIHCwkJ89tln+PLLL8vsJzg4GEFBQZUaOxEREb2dXqt5nA4fPoyFCxdizZo1OH36NPbs2YO9e/di3rx5ZS4zY8YMZGVlSX/Xrl17hRETERHRm0RtR5zMzMygqamJjIwMhfKMjAxYWlqWusysWbPwySefYMSIEQAAR0dH5OTkYNSoUfjqq6+goVEyD9TR0YGOjk7lvwAiIiJ666jtiJO2tjZcXV0RGRkplRUVFSEyMhJt2rQpdZnc3NwSyVHxFXCc8ZmIiIiqmlqnIwgICICfnx9atWoFNzc3hISEICcnB/7+/gAAX19fWFlZITg4GADQs2dPLF++HO+++y7c3d2RnJyMWbNmoWfPnpxCgIiIiKqcWhOngQMHIjMzE7Nnz0Z6ejpcXFywf/9+acD41atXFY4wzZw5EzKZDDNnzsSNGzdgbm6Onj17YsGCBep6CURERPQWkYm37BxXdnY2jIyMkJWVBUNDQ3WHQ0RUoQYd1R0BpUWpOwKqSqrkBq/VVXVERERE6sTEiYiIiEhJTJyIiIiIlKTWweFvKo5HUD+ORyAioqrAI05ERERESmLiRERERKQkJk5ERERESmLiRERERKQkJk5ERERESmLiRERERKQkJk5ERERESmLiRERERKQkJk5ERERESmLiRERERKQkJk5ERERESmLiRERERKQk3uSX3khLwtQdAU3xU3cERESVj0eciIiIiJTExImIiIhISUyciIiIiJTExImIiIhISUyciIiIiJRUKYnTgwcPKqMZIiIiompN5cRp8eLF2LFjh/R4wIABMDU1hZWVFeLj4ys1OCIiIqLqROXEKTQ0FNbW1gCAgwcP4uDBg/jjjz/QrVs3TJkypdIDJCIiIqouVJ4AMz09XUqcfv/9dwwYMABdu3aFjY0N3N3dKz1AIiIioupC5SNOJiYmuHbtGgBg//798Pb2BgAIISCXyys3OiIiIqJqROUjTn379sXgwYPRuHFj3L17F926dQMAxMbGws7OrtIDJCIiIqouVE6cvvnmG9jY2ODatWv4+uuvYWBgAAC4desWPv/880oPkIiIiKi6UDlx0tLSwuTJk0uUT5o0qVICIiIiIqquVE6cACApKQlRUVG4ffs2ioqKFJ6bPXt2pQRGREREVN2onDitX78eo0ePhpmZGSwtLSGTyaTnZDIZEyciIiJ6Y6mcOM2fPx8LFizAtGnTqiIeIiIiompL5ekI7t+/j/79+1daAKtXr4aNjQ10dXXh7u6OmJiYcus/ePAAY8aMQd26daGjo4MmTZpg3759lRYPERERUVlUTpz69++PP//8s1I637FjBwICAjBnzhycPn0azs7O8PHxwe3bt0utn5+fjy5duiA1NRW7du1CYmIi1q9fDysrq0qJh4iIiKg8Kp+qs7Ozw6xZs3D8+HE4OjpCS0tL4fnx48cr3dby5csxcuRI+Pv7A3h6O5e9e/di06ZNmD59eon6mzZtwr1793D06FGpXxsbG1VfAhEREdELkQkhhCoLNGzYsOzGZDJcuXJFqXby8/Ohr6+PXbt2oXfv3lK5n58fHjx4gF9++aXEMt27d0ft2rWhr6+PX375Bebm5hg8eDCmTZsGTU1NpfrNzs6GkZERsrKyYGhoqNQyqmrQsUqaJRWMHaruCGiKn7ojeHNwn6J+aVHqjoCqkiq5gcpHnFJSUl44sGfduXMHcrkcFhYWCuUWFhZISEgodZkrV67g0KFDGDJkCPbt24fk5GR8/vnnKCgowJw5c0pdJi8vD3l5edLj7OzsSomfiIiI3j4qj3FSp6KiItSpUwfr1q2Dq6srBg4ciK+++gqhoaFlLhMcHAwjIyPpr/gGxURERESqUuqIU0BAAObNm4eaNWsiICCg3LrLly9XqmMzMzNoamoiIyNDoTwjIwOWlpalLlO3bl1oaWkpnJazt7dHeno68vPzoa2tXWKZGTNmKMScnZ3N5ImIiIheiFKJU2xsLAoKCqT/l+XZyTAroq2tDVdXV0RGRkpjnIqKihAZGYmxY8eWuoyHhwe2b9+OoqIiaGg8PVh26dIl1K1bt9SkCQB0dHSgo6OjdFxEREREZVEqcYqKiir1/y8rICAAfn5+aNWqFdzc3BASEoKcnBzpKjtfX19YWVkhODgYADB69GisWrUKEyZMwLhx45CUlISFCxeqdCUfERER0YtSeXD4oUOH4OHhUSlHcQYOHIjMzEzMnj0b6enpcHFxwf79+6UB41evXpWOLAGAtbU1Dhw4gEmTJsHJyQlWVlaYMGECZzEnIiKiV0Ll6QgMDAxQWFiI9957Dx06dICXlxc8PDygp6dXVTFWKk5H8HbgdATqx+kIKg/3KerH6QjebKrkBi90y5XIyEh069YNMTEx6NOnD4yNjeHh4YGZM2e+cNBERERE1Z3KiZOWlhY8PDzw5Zdf4sCBAzh+/DgGDRqEmJgYaSwSERER0ZtI5TFOly5dwuHDh3H48GH89ddfyMvLg6enJ5YuXYoOHTpUQYhERERE1YPKiVOzZs1gbm6OCRMmYPr06XB0dFRpGgIiIiKi15XKp+rGjx8PKysrzJ07F5999hm++uor/Pnnn8jNza2K+IiIiIiqDZUTp5CQEJw+fRrp6emYMWMG8vPz8dVXX8HMzAweHh5VESMRERFRtfDC96qTy+UoKChAXl4enjx5gry8PCQmJlZmbERERETVygudqnNycoKFhQU+/fRT3Lx5EyNHjkRsbCwyMzOrIkYiIiKiakHlweG3bt3CqFGj0KFDBzg4OFRFTERERETVksqJU3h4eFXEQURERFTtvfAYJyIiIqK3DRMnIiIiIiUxcSIiIiJSUqUmTnK5vDKbIyIiIqpWKiVxunTpEqZOnYp69epVRnNERERE1dILJ065ubnYvHkzPD090bx5cxw5cgQBAQGVGRsRERFRtaLydATHjx/Hhg0bEB4ejvr16+PixYuIioqCp6dnVcRHREREVG0ofcRp2bJlaNGiBfr16wcTExMcOXIEZ8+ehUwmg6mpaVXGSERERFQtKH3Eadq0aZg2bRrmzp0LTU3NqoyJiIiIqFpS+ojTvHnzEB4ejoYNG2LatGk4d+5cVcZFREREVO0onTjNmDEDly5dwtatW5Geng53d3c4OztDCIH79+9XZYxERERE1YLKV9V5eXkhLCwM6enp+Pzzz+Hq6govLy+0bdsWy5cvr4oYiYiIiKqFF56OoFatWvj0009x4sQJxMbGws3NDYsWLarM2IiIiIiqlUqZANPR0REhISG4ceNGZTRHREREVC1V6i1XtLS0KrM5IiIiomqFN/klIiIiUhITJyIiIiIlMXEiIiIiUtILJU6XL1/GzJkzMWjQINy+fRsA8Mcff+D8+fOVGhwRERFRdaJy4vTXX3/B0dERJ06cwJ49e/Do0SMAQHx8PObMmVPpARIRERFVFyonTtOnT8f8+fNx8OBBaGtrS+WdOnXC8ePHKzU4IiIioupE5cTp7Nmz6NOnT4nyOnXq4M6dO5USFBEREVF1pHLiZGxsjFu3bpUoj42NhZWVVaUERURERFQdqZw4ffTRR5g2bRrS09Mhk8lQVFSEf/75B5MnT4avr29VxEhERERULaicOC1cuBDNmjWDtbU1Hj16hObNm6N9+/Zo27YtZs6c+UJBrF69GjY2NtDV1YW7uztiYmKUWu7HH3+ETCZD7969X6hfIiIiIlWonDhpa2tj/fr1uHLlCn7//Xf897//RUJCArZu3QpNTU2VA9ixYwcCAgIwZ84cnD59Gs7OzvDx8ZGmOShLamoqJk+eDE9PT5X7JCIiInoRKidOc+fORW5uLqytrdG9e3cMGDAAjRs3xuPHjzF37lyVA1i+fDlGjhwJf39/NG/eHKGhodDX18emTZvKXEYul2PIkCEICgpCo0aNVO6TiIiI6EWonDgFBQVJczc9Kzc3F0FBQSq1lZ+fj1OnTsHb2/v/A9LQgLe3N44dO1bmcnPnzkWdOnUwfPhwlfojIiIiehk1VF1ACAGZTFaiPD4+HrVr11aprTt37kAul8PCwkKh3MLCAgkJCaUu8/fff2Pjxo2Ii4tTqo+8vDzk5eVJj7Ozs1WKkYiIiKiY0omTiYkJZDIZZDIZmjRpopA8yeVyPHr0CJ999lmVBFns4cOH+OSTT7B+/XqYmZkptUxwcLDKR8KIiIiISqN04hQSEgIhBIYNG4agoCAYGRlJz2lra8PGxgZt2rRRqXMzMzNoamoiIyNDoTwjIwOWlpYl6l++fBmpqano2bOnVFZUVPT0hdSogcTERNja2iosM2PGDAQEBEiPs7OzYW1trVKcRERERIAKiZOfnx8AoGHDhmjbti20tLReunNtbW24uroiMjJSmlKgqKgIkZGRGDt2bIn6zZo1w9mzZxXKZs6ciYcPH2LFihWlJkQ6OjrQ0dF56ViJiIiIVB7j5OXlJf3/yZMnyM/PV3je0NBQpfYCAgLg5+eHVq1awc3NDSEhIcjJyYG/vz8AwNfXF1ZWVggODoauri4cHBwUljc2NgaAEuVERERElU3lxCk3NxdTp07Fzp07cffu3RLPy+VyldobOHAgMjMzMXv2bKSnp8PFxQX79++XBoxfvXoVGhoqX/xHREREVOlUTpymTJmCqKgorF27Fp988glWr16NGzdu4LvvvsOiRYteKIixY8eWemoOAA4fPlzuslu2bHmhPomIiIhUpXLi9Ntvv+H7779Hhw4d4O/vD09PT9jZ2aFBgwbYtm0bhgwZUhVxEhEREamdyufA7t27J83WbWhoiHv37gEA2rVrhyNHjlRudERERETViMqJU6NGjZCSkgLg6VVuO3fuBPD0SFTxQG0iIiKiN5HKiZO/vz/i4+MBANOnT8fq1auhq6uLSZMmYcqUKZUeIBEREVF1ofIYp0mTJkn/9/b2RkJCAk6dOgU7Ozs4OTlVanBERERE1YlKR5wKCgrQuXNnJCUlSWUNGjRA3759mTQRERHRG0+lxElLSwtnzpypqliIiIiIqjWVxzh9/PHH2LhxY1XEQkRERFStqTzGqbCwEJs2bUJERARcXV1Rs2ZNheeXL19eacERERERVScqJ07nzp1Dy5YtAQCXLl1SeE4mk1VOVERERETVkMqJU1RUVFXEQURERFTt8e65REREREpi4kRERESkJCZOREREREpi4kRERESkJCZOREREREpi4kRERESkJCZOREREREpi4kRERESkJCZOREREREpi4kRERESkJCZOREREREpi4kRERESkJCZOREREREpi4kRERESkJCZOREREREpi4kRERESkJCZOREREREpi4kRERESkJCZOREREREpi4kRERESkJCZOREREREpi4kRERESkJCZOREREREqqFonT6tWrYWNjA11dXbi7uyMmJqbMuuvXr4enpydMTExgYmICb2/vcusTERERVRa1J047duxAQEAA5syZg9OnT8PZ2Rk+Pj64fft2qfUPHz6MQYMGISoqCseOHYO1tTW6du2KGzduvOLIiYiI6G2j9sRp+fLlGDlyJPz9/dG8eXOEhoZCX18fmzZtKrX+tm3b8Pnnn8PFxQXNmjXDhg0bUFRUhMjIyFccOREREb1t1Jo45efn49SpU/D29pbKNDQ04O3tjWPHjinVRm5uLgoKClC7du1Sn8/Ly0N2drbCHxEREdGLUGvidOfOHcjlclhYWCiUW1hYID09Xak2pk2bhnfeeUch+XpWcHAwjIyMpD9ra+uXjpuIiIjeTmo/VfcyFi1ahB9//BE//fQTdHV1S60zY8YMZGVlSX/Xrl17xVESERHRm6KGOjs3MzODpqYmMjIyFMozMjJgaWlZ7rJLly7FokWLEBERAScnpzLr6ejoQEdHp1LiJSIiorebWo84aWtrw9XVVWFgd/FA7zZt2pS53Ndff4158+Zh//79aNWq1asIlYiIiEi9R5wAICAgAH5+fmjVqhXc3NwQEhKCnJwc+Pv7AwB8fX1hZWWF4OBgAMDixYsxe/ZsbN++HTY2NtJYKAMDAxgYGKjtdRAREdGbT+2J08CBA5GZmYnZs2cjPT0dLi4u2L9/vzRg/OrVq9DQ+P8DY2vXrkV+fj769eun0M6cOXMQGBj4KkMnIiKit4zaEycAGDt2LMaOHVvqc4cPH1Z4nJqaWvUBEREREZXitb6qjoiIiOhVYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpCQmTkRERERKYuJEREREpKRqkTitXr0aNjY20NXVhbu7O2JiYsqtHx4ejmbNmkFXVxeOjo7Yt2/fK4qUiIiI3mY11B3Ajh07EBAQgNDQULi7uyMkJAQ+Pj5ITExEnTp1StQ/evQoBg0ahODgYHzwwQfYvn07evfujdOnT8PBwUENr4CIiN50S8LUHQFN8VN3BE+p/YjT8uXLMXLkSPj7+6N58+YIDQ2Fvr4+Nm3aVGr9FStW4P3338eUKVNgb2+PefPmoWXLlli1atUrjpyIiIjeNmo94pSfn49Tp05hxowZUpmGhga8vb1x7NixUpc5duwYAgICFMp8fHzw888/l1o/Ly8PeXl50uOsrCwAQHZ29ktGX7aiwiprmpT05LG6I6Aq/Ii9dbhPUT/uU9SvKvcpxTmBEKLCumpNnO7cuQO5XA4LCwuFcgsLCyQkJJS6THp6eqn109PTS60fHByMoKCgEuXW1tYvGDW9Dmb/re4IaPZodUdAVHm4T1G/V7FPefjwIYyMjMqto/YxTlVtxowZCkeoioqKcO/ePZiamkImk6kxMqoq2dnZsLa2xrVr12BoaKjucIjoNcd9yptPCIGHDx/inXfeqbCuWhMnMzMzaGpqIiMjQ6E8IyMDlpaWpS5jaWmpUn0dHR3o6OgolBkbG7940PTaMDQ05E6OiCoN9ylvtoqONBVT6+BwbW1tuLq6IjIyUiorKipCZGQk2rRpU+oybdq0UagPAAcPHiyzPhEREVFlUfupuoCAAPj5+aFVq1Zwc3NDSEgIcnJy4O/vDwDw9fWFlZUVgoODAQATJkyAl5cXli1bhh49euDHH3/EyZMnsW7dOnW+DCIiInoLqD1xGjhwIDIzMzF79mykp6fDxcUF+/fvlwaAX716FRoa/39grG3btti+fTtmzpyJL7/8Eo0bN8bPP//MOZxIoqOjgzlz5pQ4RUtE9CK4T6FnyYQy194RERERkfonwCQiIiJ6XTBxIiIiIlISEyciIiIiJTFxoiohhMCoUaNQu3ZtyGQyxMXFVbhMamqq0nWfJ5PJyrztjrK2bNnCOb6IXqEOHTpg4sSJ5daxsbFBSEjIK4mnLBXFWVn7DmXWB6mf2q+qozfT/v37sWXLFhw+fBiNGjWCmZlZhctYW1vj1q1bStV93q1bt2BiYvIioRJRNfbvv/+iZs2a6g6jXAMHDkT37t3VHQa9IkycqEpcvnwZdevWRdu2bZVeRlNTs8wZ4IGnR7Hkcjlq1Ci52Za3HBG9vszNzdUdQoX09PSgp6en7jDoFeGpOqp0Q4cOxbhx43D16lXIZDLY2NgAeHoUql27djA2NoapqSk++OADXL58WVru+VN1hw8fhkwmwx9//AFXV1fo6Ojg779Lv9Pms6fqipd78OCB9HxcXBxkMhlSU1Olsi1btqB+/frQ19dHnz59cPfu3RLtzp8/H3Xq1EGtWrUwYsQITJ8+HS4uLgp1NmzYAHt7e+jq6qJZs2ZYs2aNyuuM6G1VWFiIsWPHwsjICGZmZpg1a5bCHeqfP1W3fPlyODo6ombNmrC2tsbnn3+OR48eSc+npaWhZ8+eMDExQc2aNdGiRQvs27dPev7cuXPo1q0bDAwMYGFhgU8++QR37tyRns/JyYGvry8MDAxQt25dLFu2rMLX8PypuqFDh6J3794KdSZOnIgOHTqo1M+tW7fQo0cP6OnpoWHDhti+fXuJ9fHgwQOMGDEC5ubmMDQ0RKdOnRAfH19hzPTimDhRpVuxYgXmzp2LevXq4datW/j3338BPN1RBAQE4OTJk4iMjISGhgb69OmDoqKictubPn06Fi1ahIsXL8LJyalSYjxx4gSGDx+OsWPHIi4uDh07dsT8+fMV6mzbtg0LFizA4sWLcerUKdSvXx9r164tUWf27NlYsGABLl68iIULF2LWrFkICwurlDiJ3nRhYWGoUaMGYmJisGLFCixfvhwbNmwos76Ghga+/fZbnD9/HmFhYTh06BCmTp0qPT9mzBjk5eXhyJEjOHv2LBYvXgwDAwMAT5OMTp064d1338XJkyexf/9+ZGRkYMCAAdLyU6ZMwV9//YVffvkFf/75Jw4fPozTp09X+utWph9fX1/cvHkThw8fxu7du7Fu3Trcvn1boU7//v1x+/Zt/PHHHzh16hRatmyJzp074969e5UeM/2PIKoC33zzjWjQoEG5dTIzMwUAcfbsWSGEECkpKQKAiI2NFUIIERUVJQCIn3/+ucL+AIiffvpJYbn79+9Lz8fGxgoAIiUlRQghxKBBg0T37t0V2hg4cKAwMjKSHru7u4sxY8Yo1PHw8BDOzs7SY1tbW7F9+3aFOvPmzRNt2rSpMGait52Xl5ewt7cXRUVFUtm0adOEvb299LhBgwbim2++KbON8PBwYWpqKj12dHQUgYGBpdadN2+e6Nq1q0LZtWvXBACRmJgoHj58KLS1tcXOnTul5+/evSv09PTEhAkTyoxh8+bNCvsOPz8/8Z///EehzoQJE4SXl5cQQijVz8WLFwUA8e+//0p1kpKSBABpfURHRwtDQ0Px5MkThb5sbW3Fd999V2a89HJ4xIlemaSkJAwaNAiNGjWCoaGhdArv6tWr5S7XqlWrSo/l4sWLcHd3Vyh7/kbRiYmJcHNzUyh79nFOTg4uX76M4cOHw8DAQPqbP3++wilIIipb69atIZPJpMdt2rRBUlIS5HJ5qfUjIiLQuXNnWFlZoVatWvjkk09w9+5d5ObmAgDGjx+P+fPnw8PDA3PmzMGZM2ekZePj4xEVFaXweW3WrBmAp+MyL1++jPz8fIV9Q+3atdG0adNKfc3K9JOYmIgaNWqgZcuWUpmdnZ3CRTDx8fF49OgRTE1NFV5TSkoK90FViIPD6ZXp2bMnGjRogPXr1+Odd95BUVERHBwckJ+fX+5yql5RU3xvQ/HMOImCggLVA65A8biK9evXl0jCNDU1K70/orddamoqPvjgA4wePRoLFixA7dq18ffff2P48OHIz8+Hvr4+RowYAR8fH+zduxd//vkngoODsWzZMowbNw6PHj1Cz549sXjx4hJt161bF8nJyZUSp4aGhsL+B6i6fVDdunVx+PDhEs9xapWqwyNO9ErcvXsXiYmJmDlzJjp37gx7e3vcv3+/Svoqvgrn1q1bUtnzc0PZ29vjxIkTCmXHjx9XeNy0aVNpfFaxZx9bWFjgnXfewZUrV2BnZ6fw17Bhw8p4KURvvNI+h40bNy71x8epU6dQVFSEZcuWoXXr1mjSpAlu3rxZop61tTU+++wz7NmzB1988QXWr18PAGjZsiXOnz8PGxubEp/ZmjVrwtbWFlpaWgox3b9/H5cuXVLpNZmbmyvsfwDFfZAy/TRt2hSFhYWIjY2VypKTkxX2my1btkR6ejpq1KhR4vW8yLQupBwmTvRKmJiYwNTUFOvWrUNycjIOHTqEgICAKunLzs4O1tbWCAwMRFJSEvbu3VviipXx48dj//79WLp0KZKSkrBq1Srs379foc64ceOwceNGhIWFISkpCfPnz8eZM2cUTisEBQUhODgY3377LS5duoSzZ89i8+bNWL58eZW8NqI3zdWrVxEQEIDExET88MMPWLlyJSZMmFBqXTs7OxQUFGDlypW4cuUKtm7ditDQUIU6EydOxIEDB5CSkoLTp08jKioK9vb2AJ4OHL937x4GDRqEf//9F5cvX8aBAwfg7+8PuVwOAwMDDB8+HFOmTMGhQ4dw7tw5DB06VDqKraxOnTrh5MmT+P7775GUlIQ5c+bg3Llz0vPK9NOsWTN4e3tj1KhRiImJQWxsLEaNGgU9PT1pH+Tt7Y02bdqgd+/e+PPPP5GamoqjR4/iq6++wsmTJ1WKmZTHxIleCQ0NDfz44484deoUHBwcMGnSJCxZsqRK+tLS0sIPP/yAhIQEODk5YfHixSWumGvdujXWr1+PFStWwNnZGX/++SdmzpypUGfIkCGYMWMGJk+ejJYtWyIlJQVDhw6Frq6uVGfEiBHYsGEDNm/eDEdHR3h5eWHLli084kSkJF9fXzx+/Bhubm4YM2YMJkyYgFGjRpVa19nZGcuXL8fixYvh4OCAbdu2ITg4WKGOXC7HmDFjYG9vj/fffx9NmjSRpgh555138M8//0Aul6Nr165wdHTExIkTYWxsLCUtS5YsgaenJ3r27Alvb2+0a9cOrq6uKr0mHx8fzJo1C1OnTsV7772Hhw8fwtfXV6GOMv18//33sLCwQPv27dGnTx+MHDkStWrVkvZBMpkM+/btQ/v27eHv748mTZrgo48+QlpaGiwsLFSKmZQnE8+fiCV6zeTl5UFXVxcHDx6Et7d3lfbVpUsXWFpaYuvWrVXaDxG9Pr777jvMmzcP169fr9J+rl+/Dmtra2mAPKkHB4fTay07Oxt79uyBhoaGdHVMZcnNzUVoaCh8fHygqamJH374ARERETh48GCl9kNEr69r165h3759aNGiRaW3fejQITx69AiOjo64desWpk6dChsbG7Rv377S+yLlMXGi19qcOXOwfft2LF68GPXq1avUtosPgy9YsABPnjxB06ZNsXv37io/qkVEr4+WLVvCysoKW7ZsqfS2CwoK8OWXX+LKlSuoVasW2rZti23btkFLS6vS+yLl8VQdERERkZI4OJyIiIhISUyciIiIiJTExImIiIhISUyciIiIiJTExImIiIhISUyciIiIiJTExImIiIhISUyciIiIiJTExImIiIhISf8HYRdi/NtJQOUAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# viz: the position-bias signature is the gap between the two bars within each judge\n", "fig, ax = plt.subplots(figsize=(6, 3))\n", "x = np.arange(2); w = 0.35\n", "ax.bar(x - w/2, [rf_fair, rf_bias], w, label=\"A shown first\", color=\"#1E40FF\")\n", "ax.bar(x + w/2, [rs_fair, rs_bias], w, label=\"A shown second\", color=\"#8aa0ff\")\n", "ax.set_xticks(x); ax.set_xticklabels([\"fair judge\", \"biased judge\"])\n", "ax.set_ylabel(\"rate A wins\"); ax.set_ylim(0, 1)\n", "ax.set_title(\"position bias = the within-judge gap between the two bars\")\n", "ax.legend(fontsize=8); plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "d0805ca0", "metadata": {}, "source": [ "> **Interpretation.** The fair judge's two bars are nearly equal: A's win rate barely depends on where A sits, so its preference reflects content. The biased judge's bars are far apart: A wins most of the time when shown first and rarely when shown second, the preference is mostly about position. The swap check is what converts a noisy, biased pairwise judge into a usable one: discard every comparison whose verdict flips with order, and the residue is the order-stable signal. You pay for it in \"ties\" (thrown-away comparisons), which is the honest cost of a biased judge.\n", "\n", "> **Key takeaways.** Pairwise LLM-as-judge has position bias: the verdict can depend on slot, not content. Measure it by comparing A's win rate when shown first vs second; a large gap is the bias. Mitigate by running both orderings and keeping only order-stable preferences. A mock judge with a *known* bias is how you prove the mitigation works without a real model in the loop, and the bound on any judge is its agreement with humans, which no swap check can raise.\n" ] }, { "cell_type": "markdown", "id": "c86a2d90", "metadata": {}, "source": [ "## Part 5 — Loss is not the metric (a deliberate failure)\n", "\n", "> **Objectives.** Stage the loss-vs-metric divergence: a training run where the loss you optimize keeps falling while the metric you actually care about turns and falls too. Select the checkpoint by lowest loss and ship the worse model; then select by the metric and fix it. This is the chapter's deliberate failure demo.\n", "\n", "Karpathy's loss log is where most people first meet evaluation: watch the number go down. But the loss is a *proxy*. The thing you care about is a task metric on held-out data, and the two can diverge. The classic divergence is overfitting: past some point, lower training loss buys *worse* held-out performance. If your model-selection rule is \"lowest loss\", you will pick exactly the overfit checkpoint, the worse model, and never know, because the loss looked great.\n", "\n", "We make this concrete with a tiny ridge-free polynomial fit (no torch needed): increasing model capacity drives training loss monotonically toward zero while held-out accuracy peaks and then collapses. We deliberately select on the wrong number first.\n" ] }, { "cell_type": "code", "execution_count": 19, "id": "729a8382", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.812281Z", "iopub.status.busy": "2026-06-10T20:48:42.812194Z", "iopub.status.idle": "2026-06-10T20:48:42.816938Z", "shell.execute_reply": "2026-06-10T20:48:42.816582Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "degree 1: train MSE 0.1680 held-out accuracy 0.390\n", "degree 2: train MSE 0.1547 held-out accuracy 0.425\n", "degree 3: train MSE 0.0294 held-out accuracy 0.810\n", "degree 4: train MSE 0.0294 held-out accuracy 0.810\n", "degree 5: train MSE 0.0289 held-out accuracy 0.820\n", "degree 6: train MSE 0.0282 held-out accuracy 0.840\n", "degree 7: train MSE 0.0281 held-out accuracy 0.840\n", "degree 8: train MSE 0.0252 held-out accuracy 0.890\n", "degree 9: train MSE 0.0222 held-out accuracy 0.885\n", "degree 10: train MSE 0.0222 held-out accuracy 0.885\n", "degree 11: train MSE 0.0221 held-out accuracy 0.885\n", "degree 12: train MSE 0.0180 held-out accuracy 0.815\n", "degree 13: train MSE 0.0180 held-out accuracy 0.815\n", "degree 14: train MSE 0.0162 held-out accuracy 0.840\n", "degree 15: train MSE 0.0161 held-out accuracy 0.765\n" ] } ], "source": [ "# a clean overfitting curve: fit polynomials of growing degree to noisy data.\n", "# training MSE falls monotonically; held-out accuracy (within-tolerance) peaks then collapses.\n", "def make_regression_data(n, noise, seed):\n", " g = np.random.default_rng(seed)\n", " x = np.sort(g.uniform(-1, 1, n))\n", " y_true = np.sin(3 * x) # the real signal\n", " y = y_true + noise * g.standard_normal(n) # noisy observations\n", " return x, y, y_true\n", "\n", "x_tr, y_tr, _ = make_regression_data(25, noise=0.25, seed=SEED)\n", "x_te, y_te, ytrue_te = make_regression_data(200, noise=0.0, seed=SEED + 1) # clean held-out target\n", "\n", "degrees = list(range(1, 16))\n", "train_loss, held_metric = [], []\n", "for d in degrees:\n", " coef = np.polyfit(x_tr, y_tr, d) # fit on the noisy training set\n", " train_loss.append(np.mean((np.polyval(coef, x_tr) - y_tr) ** 2)) # train MSE (the proxy)\n", " pred_te = np.polyval(coef, x_te)\n", " held_metric.append(np.mean(np.abs(pred_te - ytrue_te) < 0.25)) # held-out accuracy (the target)\n", "train_loss = np.array(train_loss); held_metric = np.array(held_metric)\n", "for d, tl, hm in zip(degrees, train_loss, held_metric):\n", " print(f\"degree {d:2d}: train MSE {tl:.4f} held-out accuracy {hm:.3f}\")" ] }, { "cell_type": "markdown", "id": "45ceff61", "metadata": {}, "source": [ "> **Predict:** if you select the model by *lowest training loss*, which degree do you pick, and is it the best held-out model?
AnswerLowest training loss is the highest degree (the most flexible polynomial drives train MSE toward zero by threading every noisy point). But that is the *worst* held-out model: it has memorized the noise. The held-out accuracy peaks at a moderate degree and collapses for high degrees. Selecting on loss ships the overfit model. This is the entire failure mode, in nine lines.
\n" ] }, { "cell_type": "code", "execution_count": 20, "id": "9da5a9dd", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.817904Z", "iopub.status.busy": "2026-06-10T20:48:42.817830Z", "iopub.status.idle": "2026-06-10T20:48:42.820439Z", "shell.execute_reply": "2026-06-10T20:48:42.820078Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[selecting by lowest training loss] -> degree 15, held-out accuracy 0.765\n", "the loss is lowest here, so this 'looks' best. It is the overfit model.\n", "\n", "[selecting by held-out metric] -> degree 8, held-out accuracy 0.890\n", "\n", "selecting on the metric improves held-out accuracy by +0.125.\n", "[ ok ] the loss-selected and metric-selected models DISAGREE, and the metric wins\n" ] } ], "source": [ "# THE BUG: select the model with the lowest TRAINING LOSS.\n", "best_by_loss = degrees[int(np.argmin(train_loss))]\n", "acc_of_loss_pick = held_metric[int(np.argmin(train_loss))]\n", "print(f\"[selecting by lowest training loss] -> degree {best_by_loss}, \"\n", " f\"held-out accuracy {acc_of_loss_pick:.3f}\")\n", "print(\"the loss is lowest here, so this 'looks' best. It is the overfit model.\\n\")\n", "\n", "# THE FIX: select the model with the highest HELD-OUT METRIC.\n", "best_by_metric = degrees[int(np.argmax(held_metric))]\n", "acc_of_metric_pick = held_metric[int(np.argmax(held_metric))]\n", "print(f\"[selecting by held-out metric] -> degree {best_by_metric}, \"\n", " f\"held-out accuracy {acc_of_metric_pick:.3f}\")\n", "print(f\"\\nselecting on the metric improves held-out accuracy by \"\n", " f\"{acc_of_metric_pick - acc_of_loss_pick:+.3f}.\")\n", "assert acc_of_metric_pick > acc_of_loss_pick, \\\n", " \"the metric-selected model must beat the loss-selected one, that is the whole lesson\"\n", "assert best_by_loss != best_by_metric, \\\n", " \"loss and metric must disagree here, otherwise there is no divergence to teach\"\n", "print(\"[ ok ] the loss-selected and metric-selected models DISAGREE, and the metric wins\")" ] }, { "cell_type": "code", "execution_count": 21, "id": "79626a8f", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.821287Z", "iopub.status.busy": "2026-06-10T20:48:42.821219Z", "iopub.status.idle": "2026-06-10T20:48:42.908465Z", "shell.execute_reply": "2026-06-10T20:48:42.908070Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAogAAAFJCAYAAAAYFdw2AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAlshJREFUeJzs3Xd8U9X7wPFPkmZ0T9rSllmKbEVwMBREBBRRUAEVEfTrQsEBCOICHCAoOEBB3OLErYiIA1T8qaCgbChltkCBQhddaXJ+f1yaNp1pmzYpfd6vV169ubm590ma8eTcc56jU0ophBBCCCGEOE3v6QCEEEIIIYR3kQRRCCGEEEI4kQRRCCGEEEI4kQRRCCGEEEI4kQRRCCGEEEI4kQRRCCGEEEI4kQRRCCGEEEI4kQRRCCGEEEI4kQRRCCGEEEI4kQSxkXn77bfR6XTs27fP06E40el0jB8/3tNh1Ilnn32W1q1bYzAYOOecc6p13759+9K3b1/H9X379qHT6Xj77bcd62bMmIFOp3NPsHWsKNbjx497OhSPakj/s8r07duXTp061ftxz/TX0dixY2nZsqXTOp1Ox4wZMzwSj2icJEEUog6tWrWKKVOm0KtXL9566y1mzZrl6ZDqxaxZs/jyyy89HUa9aEyPtb7JcyuE50iCKEQd+vnnn9Hr9bzxxhvcfPPNXHHFFW4/xqOPPkpubq7b91sbjemLvSaP1Rv/Z96oMb2OhPA2kiAKUYeOHj2Kr68vJpOpzo7h4+ODxWKps/0L9zl16hQg/zMhhPeTBFEA8Morr9CxY0fMZjMxMTHcc889pKenO22TmJjItddeS3R0NBaLhbi4OK6//noyMjIc2/zwww/07t2bkJAQAgICOOuss3j44YdrFNNTTz2FXq9nwYIFjnXfffcdF110Ef7+/gQGBjJ48GC2bt1a5r47duzguuuuIywsDIvFQvfu3fn666+dtinqj/nrr79y5513Eh4eTlBQEDfffDMnT5502vbvv/9m4MCBRERE4OvrS6tWrbj11lsrjV+n0/HWW29x6tQpdDqdU9/Bt956i379+hEZGYnZbKZDhw4sWrSoRs9Tef3Zivp0fvnll3Tq1Amz2UzHjh1ZuXJlmfuvWbOG7t27Y7FYiI+P59VXXy13n8ePH2fHjh3k5ORU+bhPnTrFO++843jcY8eOddomPT2dsWPHEhISQnBwMLfccku5+33vvffo1q0bvr6+hIWFcf3113Pw4EGXn5Ndu3Zx0003ERwcTJMmTXjsscdQSnHw4EGuvvpqgoKCiI6OZt68eWX2kZ+fz/Tp02nTpg1ms5lmzZoxZcoU8vPzXXqsRTFs27aNG2+8kdDQUHr37u10W3mP9/zzz8fPz4/Q0FAuvvhiVq1aVeljHTt2LAEBAezZs4eBAwfi7+9PTEwMTzzxBEopp23tdjsvvPACHTt2xGKxEBUVxZ133lnm9f7VV18xePBgYmJiMJvNxMfH8+STT2Kz2ap87letWoWfnx833HADhYWFQM0+F+r7dVTUn3Lbtm1ccskl+Pn5ERsby9y5c6t8zCWPU9X/z5XPWlfs37+fu+++m7POOgtfX1/Cw8MZPnx4mf7l7v6cc/U1VJPPTOFdfDwdgPC8GTNmMHPmTPr378+4cePYuXMnixYtYv369fz+++8YjUYKCgoYOHAg+fn5TJgwgejoaFJSUli+fDnp6ekEBwezdetWrrzySrp06cITTzyB2Wxm9+7d/P7779WO6dFHH2XWrFm8+uqr3H777QAsXbqUMWPGMHDgQObMmUNOTg6LFi2id+/ebNy40dGpe+vWrfTq1YvY2Fgeeugh/P39WbZsGUOHDuWzzz5j2LBhTscaP348ISEhzJgxw/HY9+/fz5o1a9DpdBw9epQBAwbQpEkTHnroIUJCQti3bx+ff/55pY9h6dKlLFmyhHXr1vH6668D0LNnTwAWLVpEx44dueqqq/Dx8eGbb77h7rvvxm63c88991T7+SrP2rVr+fzzz7n77rsJDAzkpZde4tprr+XAgQOEh4cDsHHjRgYNGkTTpk2ZOXMmNpuNJ554giZNmpTZ38KFC5k5cyarV692GjhT3uO+7bbbOP/887njjjsAiI+Pd9pmxIgRtGrVitmzZ7NhwwZef/11IiMjmTNnjmObp59+mscee4wRI0Zw2223cezYMRYsWMDFF1/Mxo0bCQkJqfI5GDlyJO3bt+eZZ57h22+/5amnniIsLIxXX32Vfv36MWfOHN5//30mT57Meeedx8UXXwxoX4JXXXUVa9eu5Y477qB9+/Zs3ryZ559/nl27djlOe7ryWIcPH05CQgKzZs0qk7CVNHPmTGbMmEHPnj154oknMJlM/PXXX/z8888MGDCg0sdps9kYNGgQF154IXPnzmXlypVMnz6dwsJCnnjiCcd2d955J2+//Ta33HIL9957L3v37mXhwoVs3LjR8V4HLakICAhg4sSJBAQE8PPPP/P444+TmZnJs88+W2Ecy5cv57rrrmPkyJG8+eabGAyGGn8ueOJ1dPLkSQYNGsQ111zDiBEj+PTTT5k6dSqdO3fm8ssvrzReV/5/rnzWumr9+vX83//9H9dffz1xcXHs27ePRYsW0bdvX7Zt24afn5/T9u76nHPlNVTTz0zhZZRoVN566y0FqL179yqllDp69KgymUxqwIABymazObZbuHChAtSbb76plFJq48aNClCffPJJhft+/vnnFaCOHTtW7bgAdc899yillJo0aZLS6/Xq7bffdtyelZWlQkJC1O233+50vyNHjqjg4GCn9Zdeeqnq3LmzysvLc6yz2+2qZ8+eKiEhocxz0a1bN1VQUOBYP3fuXAWor776Siml1BdffKEAtX79+mo/rjFjxih/f/8y63NycsqsGzhwoGrdurXTuj59+qg+ffo4ru/du1cB6q233nKsmz59uir9VgaUyWRSu3fvdqz777//FKAWLFjgWDdkyBDl5+enUlJSHOsSExOVj49PmX0WHWf16tWVPmallPL391djxowps75oH7feeqvT+mHDhqnw8HDH9X379imDwaCefvppp+02b96sfHx8yqyv6Dh33HGHY11hYaGKi4tTOp1OPfPMM471J0+eVL6+vk7xLl26VOn1evXbb7857Xfx4sUKUL///rvLj/WGG26o8LYiiYmJSq/Xq2HDhjm9D5XSXruVGTNmjALUhAkTnO4zePBgZTKZHO/H3377TQHq/fffd7r/ypUry6wv7/V55513Kj8/P6f3VZ8+fVTHjh2VUkp99tlnymg0qttvv93pMdTmc6E+X0d9+vRRgHr33Xcd6/Lz81V0dLS69tprK43Tlf+fq5+1Smn/0xYtWjjtB1DTp093XC/vf/THH3+UeQzu/Jxz9TVUm89M4T3kFHMj9+OPP1JQUMD999+PXl/8crj99tsJCgri22+/BSA4OBiA77//vsJTjEW/xL/66ivsdnu1Y1FKMX78eF588UXee+89xowZ47jthx9+ID09nRtuuIHjx487LgaDgQsuuIDVq1cDcOLECX7++WdGjBhBVlaWY7u0tDQGDhxIYmIiKSkpTse94447nH65jxs3Dh8fH1asWOH0uJYvX47Vaq324yqPr6+vYzkjI4Pjx4/Tp08f9uzZ43TKvjb69+/v1OLSpUsXgoKC2LNnD6C1Ov34448MHTqUmJgYx3Zt2rQpt7VkxowZKKUqbT101V133eV0/aKLLiItLY3MzEwAPv/8c+x2OyNGjHD6f0dHR5OQkOD4f1fltttucywbDAa6d++OUor//e9/jvUhISGcddZZjucF4JNPPqF9+/a0a9fO6fj9+vUDcPn45T3W8nz55ZfY7XYef/xxp/ch4HI5nJJlooq6GBQUFPDjjz8C2mMKDg7msssuc3pM3bp1IyAgwOkxlXx9Fr2PLrroInJyctixY0eZY3/44YeMHDmSO++8k1dffdXpMdT2c6Ey7n4dBQQEcNNNNzmum0wmzj//fKfXRnlc+f+5+lnrqpL/I6vVSlpaGm3atCEkJIQNGzaU2d4dn3Ouvobq4jNT1D9JEBu5/fv3A3DWWWc5rTeZTLRu3dpxe6tWrZg4cSKvv/46ERERDBw4kJdfftkpmRk5ciS9evXitttuIyoqiuuvv55ly5a5/KXw7rvv8vLLL7NgwQJuuOEGp9sSExMB6NevH02aNHG6rFq1iqNHjwKwe/dulFI89thjZbabPn06gGPbIgkJCU7XAwICaNq0qaMvT58+fbj22muZOXMmERERXH311bz11ltOfdGq6/fff6d///74+/sTEhJCkyZNHH2y3JUgNm/evMy60NBQR1+ho0ePkpubS5s2bcpsV946dyodW2hoKIAjtsTERJRSJCQklPk/bt++vcz/0NXjBAcHY7FYiIiIKLO+ZB+qxMREtm7dWubYbdu2Bcq+hirTqlWrKrdJSkpCr9fToUMHl/dbkl6vp3Xr1k7rimIteh0nJiaSkZFBZGRkmceVnZ3t9Ji2bt3KsGHDCA4OJigoiCZNmjgSp9Kvz71793LTTTdx7bXXsmDBgjIJbW0/Fyrj7tdRXFxcmfhLvmcq4sr/z9XPWlfl5uby+OOP06xZM8xmMxERETRp0oT09PRyP0Pc8Tnn6muoLj4zRf2TPojCZfPmzWPs2LF89dVXrFq1invvvZfZs2fz559/EhcXh6+vL7/++iurV6/m22+/ZeXKlXz88cf069ePVatWYTAYKt1/r169+Pfff1m4cCEjRowgLCzMcVvRl8nSpUuJjo4uc18fHx+n7SZPnszAgQPLPU51kx+dTsenn37Kn3/+yTfffMP333/Prbfeyrx58/jzzz8JCAio1v6SkpK49NJLadeuHfPnz6dZs2aYTCZWrFjB888/77ZWloqeb1VJP7j6UlVsdrsdnU7Hd999V+62rj7n5d3XlefFbrfTuXNn5s+fX+62zZo1c+n44NzS40l2u53IyEjef//9cm8v6neanp5Onz59CAoK4oknniA+Ph6LxcKGDRuYOnVqmddn06ZNadq0KStWrODvv/+me/fuTrfX9nOhMu5+HXnze6a0CRMm8NZbb3H//ffTo0cPgoOD0el0XH/99TX6DHHlc87V15C7PzOFZ0iC2Mi1aNECgJ07dzq1QBQUFLB371769+/vtH3nzp3p3Lkzjz76KP/3f/9Hr169WLx4MU899RSgtWRceumlXHrppcyfP59Zs2bxyCOPsHr16jL7Kq1NmzbMnTuXvn37MmjQIH766ScCAwOB4s7pkZGRle6n6DEYjcYqj1ckMTGRSy65xHE9Ozubw4cPl6lZeOGFF3LhhRfy9NNP88EHHzBq1Cg++ugjp9OYrvjmm2/Iz8/n66+/dmoBqc5pS3eIjIzEYrGwe/fuMreVt646ajtLSHx8PEopWrVq5WgJq0/x8fH8999/XHrppVU+FnfMiBIfH4/dbmfbtm3Vnm0HtERoz549Ts/Vrl27AByDt+Lj4/nxxx/p1atXpUnrmjVrSEtL4/PPP3cM2gGtpbA8FouF5cuX069fPwYNGsQvv/xCx44dnbap6edCQ3kdufL/q+5nbVU+/fRTxowZ4zQCPy8vr8IR0e74nHP1NeTKvoT3k1PMjVz//v0xmUy89NJLTr+S33jjDTIyMhg8eDAAmZmZjpIVRTp37oxer3ecNjhx4kSZ/Rd9WLp6aqFLly6sWLGC7du3M2TIEEcx4YEDBxIUFMSsWbPK7dNy7NgxQEt6+vbty6uvvsrhw4cr3K6kJUuWOO1z0aJFFBYWOvrhnTx5skwLQnUfV0lFrRQl95mRkcFbb71V7X3VhsFgoH///nz55ZccOnTIsX737t189913tdq3v79/jUp3FLnmmmswGAzMnDmzzHOvlCItLa1W8VVlxIgRpKSk8Nprr5W5LTc311HPEGr/WAGGDh2KXq/niSeeKNP642rr1cKFC53us3DhQoxGI5deeimgPSabzcaTTz5Z5r6FhYWOx1De67OgoIBXXnmlwmMHBwfz/fffExkZyWWXXUZSUpLjttp8LjSU15Er/z9XP2tdZTAYyjymBQsWVFiKyB2fc66+htz9mSk8Q1oQG7kmTZowbdo0Zs6cyaBBg7jqqqvYuXMnr7zyCuedd56j39HPP//M+PHjGT58OG3btqWwsJClS5diMBi49tprAXjiiSf49ddfGTx4MC1atODo0aO88sorxMXFOeq/ueLCCy/kq6++4oorruC6667jyy+/JCgoiEWLFjF69GjOPfdcrr/+epo0acKBAwf49ttv6dWrl+ML8uWXX6Z379507tyZ22+/ndatW5Oamsoff/xBcnIy//33n9PxCgoKuPTSSxkxYoTjsffu3ZurrroKgHfeeYdXXnmFYcOGER8fT1ZWFq+99hpBQUE1mhllwIABmEwmhgwZwp133kl2djavvfYakZGR5Sa1dWnGjBmsWrWKXr16MW7cOGw2GwsXLqRTp078+++/ZbZ1pcwNQLdu3fjxxx+ZP38+MTExtGrVigsuuMDluOLj43nqqaeYNm0a+/btY+jQoQQGBrJ3716++OIL7rjjDiZPnlyDR+ya0aNHs2zZMu666y5Wr15Nr169sNls7Nixg2XLlvH99987TqXW9rGC1nr+yCOP8OSTT3LRRRdxzTXXYDabWb9+PTExMcyePbvS+1ssFlauXMmYMWO44IIL+O677/j22295+OGHHaf9+vTpw5133sns2bP5999/GTBgAEajkcTERD755BNefPFFrrvuOnr27EloaChjxozh3nvvRafTsXTp0ioT1YiICEe9w/79+7N27VpiY2Nr9bnQUF5Hrvz/XP2sddWVV17J0qVLCQ4OpkOHDvzxxx/8+OOPjhJWpbnjc87V15C7PzOFh9TLWGnhNUqXuSmycOFC1a5dO2U0GlVUVJQaN26cOnnypOP2PXv2qFtvvVXFx8cri8WiwsLC1CWXXKJ+/PFHxzY//fSTuvrqq1VMTIwymUwqJiZG3XDDDWrXrl1VxkWJMjdFvvrqK+Xj46NGjhzpKAuxevVqNXDgQBUcHKwsFouKj49XY8eOVX///bfTfZOSktTNN9+soqOjldFoVLGxserKK69Un376aZnn4pdfflF33HGHCg0NVQEBAWrUqFEqLS3Nsd2GDRvUDTfcoJo3b67MZrOKjIxUV155ZZljlqeiMjdff/216tKli7JYLKply5Zqzpw56s033yzzv6lNmZvSz6dSSrVo0aJM2ZCffvpJde3aVZlMJhUfH69ef/11NWnSJGWxWJy2mzRpktLpdGr79u1VPu4dO3aoiy++WPn6+irAccyiWEuXPKnodfnZZ5+p3r17K39/f+Xv76/atWun7rnnHrVz585Kj1/RcSr6f5Qs11KkoKBAzZkzR3Xs2FGZzWYVGhqqunXrpmbOnKkyMjJq/FhL3lbam2++qbp27eo4Xp8+fdQPP/xQ6WMtekxJSUlqwIABys/PT0VFRanp06eXKbmilFJLlixR3bp1U76+viowMFB17txZTZkyRR06dMixze+//64uvPBC5evrq2JiYtSUKVPU999/X6bMUXnP2+7du1XTpk1V+/bt1bFjx2r1uVCfr6PyHkvR81u65ExFXPn/VfVZW9ExKVXm5uTJk+qWW25RERERKiAgQA0cOFDt2LGjzHu8Lj7nqnoN1eYzU3gPnVJe2PtWiHpQVOx1/fr1ZTrWN3ZDhw5l69atjtHjAOeffz4tWrTgk08+8WBkorSxY8fy6aefkp2d7elQhBeSzzlRU9IHUYhGrqifZ5HExERWrFjhdBo5MzOT//77z2lWDiGEEGcu6YMoRCPXunVrxo4d66jFtmjRIkwmE1OmTHFsExQUJJ3LhRCiEZEEUYhGbtCgQXz44YccOXIEs9lMjx49mDVrVpnCukIIIRoP6YMohBBCCCGceLwF8eiyZaQuXYo1LQ3fhASaP/gg/p06lbttblIShxYvJmfHDgoOHyZu4kSibrzRaZvNQ4ZQUE6pkCbDh9N86lQAdt5xB9ml5qqMuOYaWpye6kwIIYQQojHzaIJ4YtUqkp9/nubTpuHfqRNHP/yQxAkT6PjZZxhLTLNWxJ6XhzkujtD+/TlYwRRY7d59F0oUCs1NSiLxnnsIPV0stkjEsGHE3Hmn47reYqlW7IWFhWzcuJGoqKgyk7MLIYQQ4sxgt9tJTU2la9eujmldGwOPPtLU998nYuhQIk4X6mw+bRoZa9eS9vXXRI8dW2Z7/44d8T89hVNKiVkDSjKenqy9yJF33sEcF0dAt25O6/UWC8aICJdjzc/Pd+qk/88//9CvXz+X7y+EEEKIhmvdunWcd955ng6j3ngsQbRbreTs2EHTW25xrNPp9QSefz7Zmza57RhpK1YQNWpUmTk9T3z3HWkrVmAMDyfk4otpetttlbYizp49m5kzZ5ZZv27dOpo2beqWeIUQ3is3H/qNDQLg57cz8TV7OCAhBKCdXdw7fDgArT75pNpnBKty+PBhzj//fKKioty6X2/nsQSxMD0dbDZ8Sp1KNoaFkbdvn1uOkb5mDbbsbMKHDHFaHzZoEKamTTE1aUJOYiIpCxaQt38/8c8+W+G+pk2bxsSJEx3XU1JS6NChA02bNiUuLs4t8QohvJdS8O832nJYcBClfnMKITxEKUXTn38GwCckpEyDkLs0tu5kZ/TJ9LSvviK4Z09Mp+ciLdLkmmscy75t2mCMiCBx3Djyk5MxV5Dsmc1mzObiJoPMzMy6CVoI4ZV0OggP8XQUQojSdDpdme5l3uCdL2DJx3DsBLSPh5n3wjnty9/WWgivvA+froLUY9C6GTx0J/Q9v35jLslj6bBPSAgYDBSeOOG03nriBMYKJhuvjvzDh8lct46Iq6+uctuiUdN5Bw/W+rhCCCGEaNy++RmeWgT3jYHlS7QEcfQUOH6y/O2fewPeXw4zJ8CPb8Ooq+COx2BLYvnb1wePJYh6oxG/du3IXLfOsU7Z7WStX09Aly613n/a11/jExpKcO/eVW6bu3MnQLUGrQghGpcCKyx4T7sUWD0djRCiiN1q5fAbb3D4jTewW73jzfn6J3D9YBhxObRtCbMmgq8Fln1X/vaf/wD33Aj9LoTmMTD6arjkAnhtWb2G7cSjp5ijRo1i34wZ+HfogF/Hjhz94APsubmOPoN7H38cU2QksePHA9qLIG/PHgCU1Yr12DFydu5E7+eHpVkzx36V3U7aN98QfuWV6EoNSc9PTubEypUE9eqFT3AwuYmJHJw/n4Bzz8XPC2aOUDYb2Rs3Yj1+HGNEBAFdu6IzGDwdlhCNVkoqnMiAvHztVz7AhWeD5XSPk7BgiG1cfdeF8CqqsJBDixYBEHnjjWA01slxsrKynLqXle56VqTACpt3wd2jitfp9dD7XNiwtfx9F1jBbHJeZzHD35vdEXnNeDRBDBswgMKTJzm0eLFWKLttWxIWLHCcYi44cgRdiU6h1mPH2D6q+BlPXbqU1KVLCTj3XM5assSxPmvdOgqOHHGUzylJ5+ND5rp1pH74IfbcXExRUYT260fT//2vDh+pa07+/DMHn3sO69GjjnXGyEiaTZ5MqJTUEaLepaTCJTdDfoHz+uvuLV42m2D1u5IkCuEpOoOBiKFDHct1pUOHDk7Xp0+fzowZM8psdzIDbHaIKNUtMiIUkg6Uv++Lu2utjhecDS1i4PcNsPI3sNvdFHwNyFR7NZScnEyzZs04ePCgW0Yxn/z5Z/ZMmVLh7a3nzpUkUYh6tnkXXHln1dstfxU6t637eIQQ9a/o+37btm3ExsY61lfUgph6HM4fDp8vhG4di9fPWgx//QdfLSp7jLR0eOg5+PEP0AEtYqHXudop6V3fu/8xueKMHsXcUCibjYPPPVfpNgfnzSOkTx853SxEI1R0mrsi9XGaW2IQjV1gYCBBQUFVbhcaDAZ92QEpx09Ck7KTxAFahYTXnoK8AkjPgKgIeGYJNPdgmWVJEL1A9saNTqeVy2NNTSV740YCu3evp6iEEK7amggmIwQHQHCg1nfIXaXYKjrNXVJdn+aWGIRwncmonVH4fQMMPD1O1m7Xro8ZVvl9LSaIbqKVvfnuV7iyb52HWyFJEL2A9fhxt24nhKhfU0udADAZtUQxOLA4aQwJLGddEAQFlFgfqH1BlHQio/KkCLTbT2TUXWIkMQhvZsvNZdNllwHQ5YcfMPj6ejgiuG04THoGurSFs9vDm59CTh4MH6Td/sAsLRGcert2feM2OHIcOrbR/j7/NtgV3HmDxx6CJIjewNXyOlKGR4j6pVzsIB4TqY1yzsjSOqcXWLXiuMdOVH3f0ixm5yRS72JL5A+/w/ak6h/PFclHGk4MonGy5+V5OgQnQ/pBWgbMf1v7HOgQD+/OKT7FfOioNrK5SH4BPPcmHDwEfr5aiZsXHtY+BzxFBqnUkDsHqSibjc1DhlR5mjnm7ruJuukm9CZTpdsJIWrv4BEYNwM276x626JBKkrBqVxIz4SMbC1hzMiC9NN/M7O09eXdnpmt3V/UjgwYanyU3U7BEe0XhCk62qn6iTu4e1BqQyEtiF5AZzDQbPLkSkcxAxx65RXSvv6auEmTCLnoonqKTojGRSn44Bt4erGW7FWHTgcBftqlul8jdjtk5ZxOGkskkNt2w8L3q75/904Q6F/Ng7oo6xT8vaVhxCAaH51ejzkmxtNhnHEkQfQSof360Xru3LJ1EKOiiJs4EZWfT8pLL5GfnEzSAw8Q1KsXzSZNwtK8uQejFuLMkpKq9Sf87W/t+tnttFOmlc2cYjZpo2drS68/fVo5ACgxcrF5jGsJ4owJdddy5mq5H2+IQQjhHpIgepHQfv0I6dOnwplUQvr04fDrr3P0ww/J/P13tv31F5GjRtH0f//D4Ofn4eiFaLiU0uqNPfmK1lJlNmmdx2+5Bg4f0wY+FNrg29Xa9oMvAZ/TFaektIp3KbR5OgJR31RhIUeXaXPSRY4YUWYGNVEz8ix6GZ3BUGEpG4O/P3H33UfE1VdzcP58Mv/v/0h95x1OrFhB3H33ETpwIDp31dYQopE4cgymzoM1f2nXz+0I86ZC69Ozd8ZGaZecXBj6ibZu4i1aR3LhfWYthqXPlh0NLs5cdquV5PnzAYgYNgyDJIhu4d6enKJeWFq2pM2LLxI/bx6m2Fisx46x99FH2XX77eTsdKFHvRACpeCz7+GyW7Tk0GyEh++CT18sTg5L0hvg6ku1i74e69WHBZedo7U0d53mbugxAKzbBGMfguycuotFeBedXk/YoEGEDRrk9gEqjZmMYq4hbxnVZM/PJ/X99zny5pvaMH+9nohhw4gdNw6fkBCPxSWENzt6Ah6eBz/8n3b97HYw7yFIaOHZuCriDTOINIQY9qfAlGe1wUVd28Pbz2i1JoWoDW/5vq9vkiDWkLe9YAqOHCH5xRc5+cMPABiCgogZN44m11wj0/MJcZpS8PXP8PhLWqkZow88MBbuvL64T6Fo2P7bATdP1f6/7Vprp5sjK5jeTAhXeNv3fX2RttgzhCk6mtazZ9P21VfxbdMGW2YmB+fMYfvo0WRt3Ojp8ITwuOMn4a7pcO9TWvLQKUGrmXfPKEkOzyRnt4NlL0BkOOzYA8PvlSLbQtSEJIhnmMBu3Wj/3ns0e/BBDIGB5O7axa7bb2fPI49QUEUhbiHOVN/+ovU1XPmblgxOHAtfvqK1MLkqJxe6DtUuOdWsjyjq11mttL6kcdGwLwWG3wdJBzwdlagrttxc/uvfn//698eWK29Od5EE8Qyk8/EhcuRIOn3xBRHDhoFOx8nvv2frtddy+K23sBdUMaGpEGeIExkw/gm4e4a23L41fL0Y7hujnV6uyf4q6wMnvEeLWPjsJYhvrk1rNvw+2Lrb01GJulKYnk5herqnwzijSB/EGmpIfRJyduzgwNy5nNq0CQBzXJzMxiLOeN+vhYfna6eWDXrtVPKE0WAy1mx/djvs3q8tt2nhPI+q8F5p6TB6CmxNhCB/eOsZbcYXceZQdjt5e/cCYGnVSqbacxNJEGuoob1glFKc+O47Ul56Cevx4wAyG4s4I6VnwoyF8IU2XouEFjB/GnQ5y7NxCc/JzIZbH4b1m8HXAq8/Bb27eToq0VA0tO97d5HfwI2ETqcj/Ior6PjZZ0SNHo3Ox0ebjWXECJIXLMCWU1w0TNlsZP39NydWriTr779RNpmaQDQMP/0Bl92qJYd6Pdx9IyxfIslhYxcUAEvnQp/zIDcPbpkGq9Z6OiohvJu0INZQQ/9Fkbdvn2M2FgBjkybE3XcfGI0kz5vnPB90ZCTNJk8mtF8/T4UrRKUysuHJl+GTldr1+Gbw3ENwbgf3HcNaCJ+e3v91g2rWh1F4Vn4B3Pc0fPer1u3guYfgmss8HZWoLVVYyPFvvgEgYsgQt0+119C/72tKEsQaOhNeMEopMn79lYPz51OQklLl9q3nzpUkUXidX9bB1Oe0OZN1OrhtOEy+FSxm9x4nJxfaX6Etb18hU+01VIU2eOi54h8TT94HNw/1aEiilmy5ufx7uk/9Ob/9hsHXvW/OM+H7vibkN3AjptPpCOnTh6ALL+TI0qUcXry40u0PzptHSJ8+Uni7kfD2mTNycuG9r7XC1wAtY+G5qXBe57qJRW+AAb2Kl0XD5GOAuQ+Cvy+8/QU89qI2Ld/dN3o6sobFGz4fiuj0eoL79HEsC/eQBFGgN5sJPOccDlexnTU1leyNGwns3r1e4hKek5IKl9ysnZKriNkEq9+tuy8BV2Iocss1MOW2um3Vs5jgtafqbv+i/uj1MGMCBAbAgqUw5zXIOqW9hnQ6T0fn/bzh86EkvdlMm3nz6v5AjYyk2gLAMbLZXduJhu1ERtWJWX5B3dYEdCUGgNkTtS97OeUrqkOn07oiPHyXdv2VD7TWRLvds3E1BN7w+SDqnrQgCgCMERFu3U40Dqt+h211VHw4OdW17TrLCGVRC3eOhEA/ePh5WPqVdrr5uaky/aIQkiAKAAK6dsUYGek0erk0Y1QUAV271mNUwhNsNti117VtX3q3bmPxJrl50H+stvzj21o9PXFmuHEI+PvBxNlaiaRTObDgca1bgSiHlw1tteflsXX4cAA6fvIJeou8Od3B4wni0WXLSF26FGtaGr4JCTR/8EH8O5Vf5j43KYlDixeTs2MHBYcPEzdxIlE3OvcsPvTqqxx+7TWndeYWLej02WeO6/b8fJJfeIETq1ahCgoIuvBCmj/0EMbwcPc/wAZCZzDQbPJk9kyZUuE2zSZNkgEqZ6gDh+C3v+G3f+D/NkJGlmv3O6+zVmOuLmRma4WNvYVSxa2aUvvhzHP1pdrAlbtnaC3jt07T+pz6S9cFQHvNb0uCb1fDZ6s8HY0zpRQFhw87loV7eDRBPLFqFcnPP0/zadPw79SJox9+SOKECXT87DOMYWFltrfn5WGOiyO0f38Ozp9f4X4trVvT9pVXHNdL10Q6OH8+GWvX0vqZZzAEBHBw7lySHnyQdm++6b4H1wCF9utH67lzOfjcc851EKOiaDZpkpS4OYNkZMHvG2DtP1pSeOCQ8+3+FjiVV/V+po+Hzm3rJsbNu+DKO+tm3zVhNsHXi4qXxZmnf094ew7c9oj2/rjpQXj7GQiuox9B3k4p2LkXlq+Bb9fAnoOejqh8epOJdu+841gW7uHRBDH1/feJGDqUiKuuAqD5tGlkrF1L2tdfEz12bJnt/Tt2xL9jRwBSFi6scL86H58K+8rZsrNJ++orWj31FEHnnQdAy+nT2XrddWRv3kxA5zqqkdFAhPbrR0ifPpz44Qf2Pfoo6PV0+vxz9GY3F5UT9arACv9s1VoJ1/6jJV8lO+P7GODcjtr0Yxd110Z5Xj3Oc/F6I4MBzm7n6ShEXevZFd5/DsY8BBu2wvX3w9JnISLU05HVn12nk8LlayDpQPF6swn6XajNTDTntYruXf90BoMjNxDu47EE0W61krNjB01vucWxTqfXE3j++WRv2lSrfecfOMCmQYPQmc0EdO5M7PjxmKKjATi1fTuqsJDACy5wbG9p2RJTdDSnNm2qMEHMz88nPz/fcT0ry8VzcA2QzmAgbMAADsyejf3UKfIPHsS3TRtPhyWqQSnYta/4tPFf/2l96Epq0wIuOp0QXnA2BPgV37Z5V72GK4RX6doBlr0AN03WTqsOv09LGmMiPR1Z3Uncr7USLl+tLRcxG6HvBTC4L1zaQ/uc2LzLtQTx4xV1d4ZB1D2PJYiF6elgs+FT6lSyMSyMvH37arxf/06daDljBuYWLbAeP87h115j52230eHjjzH4+1OYlobOaMQnMNDpfj5hYVjT0irc7+zZs5k5c2aN42podHo9vvHxnNq0idzduyVBrGc1KUKbmga//3O6lXADHC31cm4SCr26aUlhr27QtEnl+zebqq5zFhZc9WOpKW+IoaRCG3xzuij3kH4yyvVM1641fPISjJqsnVq97l4tSWx1Bk2ksedg8enjHXuK15uM2rzVg/tqp90D/Z3v58p7E7RR4TGRdV+EXBUWcuKHH7TYLrvM7VPtNVZn3LMY3KtX8ZWEBPw7dWLzlVdy8ocfiBg6tMb7nTZtGhMnTnRcT0lJoUMHN0706oV8ExI4tWkTOYmJhA0a5OlwGg1Xi9CuWAIHDp/uR/i31mJYksUMF3SB3t21pLBda9eLAMdGaUVuPTlTgjfEUFJBAdw/S1se2Bt8ZPDCGa9VHHxaIkkcfq92url9vKcjq7l9KVor4bdrtNbRIkYf7WzCkEu0pLCywWeuvDeXr4bFH2ktjXod3HWD2x5CGXarlX2PPQZASN++GCRBdAuPPYs+ISFgMFB44oTTeuuJE24dTewTGIilRQvyk5O16+HhKKuVwqwsp1bEwiqOazabMZfoh5eZmem2GL2VX0ICALmJiR6OpHFxtQjtwP9prVpFdDrtdE7vbtqlW6falemIjaq/5MubYyii12vPa9GyaBxiIuGTF2H0FK3m53X3wswJcFbr8rf39BSU5cVw4FBxn8KtJT7OfQzaD8gr+2rTSAYH4rKq3pud22qlg+a9CbOXgE6v1ZysCzqdjsDzz3csC/fwWIKoNxrxa9eOzHXrCOnbFwBlt5O1fj2RI0a47Ti2nBzyk5MxXnEFAP7t26Pz8SFr3TpCL70UgLx9+yg4cgT/Ll3cdtwzQdFp5dzddVQJWdRKoQ3ioov7EfbsCqH1dLq1MbKYtVOMovGJCIWPnodRE2FzIkyaU/G23jAFpdkE78+Df7ZoLYWbdhbfZjj9Q2dwX60lPCSobuIEuHc0KDvMfxtmLdaOfdtw9x9Hb7E4VS4R7uHRdtioUaPYN2MG/h064NexI0c/+AB7bi7hQ4YAsPfxxzFFRhI7fjygNSPn7dE6SiirFeuxY+Ts3Inezw9Ls2YAJL/wAsEXXYSpaVOsx45x6NVX0en1hA4cCIAhIIDwq68m+fnn8QkORu/vz8Fnn8W/S5dGP4K5tKIE0Xr0KIXp6Vqrbx3zhgng6yMGpeDYSdifAgcPw/5D2q/8/YcgycVSEq89CZf1krljhagPwQEwfYLWgliZoinm6upzytUzDNdNKL6u12s/IK+8REsK66vfLsB9Y8Cu4IV34MlXtNPNt15Xf8cXNefRBDFswAAKT57k0OLFWqHstm1JWLDAcaq34MgRdCXO5ViPHWP7qFGO66lLl5K6dCkB557LWUuWaPdJTWXvI49QmJGBT2goAWefTbu338YYWlyjoNnEiSTr9SRNmaIVyu7Rg+ZTp9bTo244DAEBmGJjKUhJIXf3bgK7d6/T43nDBPDujCG/AJKPnE7+DmsJYFESeOAw5OVXfv+qNI2U5FCI+mRxsdrXxq1wso7mId6X7Np2Oh30OEc7fTzoYggPqZt4XHH/GLDZYcFSmPmydrr5lms8F49wjcd7ckaOHEnkyPI7JhQlfUXMMTF0+/vvSvfXevbsKo+pN5tpPnWqJIUu8G3Tpt4SxOpMAO/pX+cnMrS+SScznZO+ohbBA4fh8LHKZ9zQ67V9NG8KzWOgRYy2bC0sHgwhvEduHgy5S1v+ZrFMtScq9thLno4Als7Vup54A50OJt2i1V59+X2YsUBrSRwzzD37t+flsf3mmwFo/+67MtWem3g8QRTezS8hgYxffiHHiwaq5OZBdk7d7dsVE56E4ych61Tl2/lZTid+sVryV5QEtojVkkOTsex9pAahd1KquD6czOYlKtMqru5+QOTmwV4XWhHrsm9hTeh08OD/tPfOKx/A4y9pP5JHX137fSulirufyZvTbSRBFJVyDFTxogRx+H2ejsD5Azo6orgFsFnT4gSweVPttI6cBj4zmE3aQIWiZSEqsuCxxjMFZXXodDDlNrDZ4NWP4dEXtJbEUVfVbr96k4m2ixc7loV7SIIoKuVbVOomKQlls6EzSHVggOn3wEXnQbNo1/slucrbCkQLjcGg9ekSQtScTgfT7tQGrry2DB5+XmtJvOHKWuzTYKjzLlCNkSSIolLmuDh0ZjMqP5/8gwextGzp6ZD4fCF0TKibfW9NhGvGV73deV0goUXdxOBtBaKFEMKddDp45C6tT+Ibn8JD87R11w/2dGTu9c4XsORjOHZCK64+8144p33F27/xKbz3tTZYMiwYrugDU26vXT3b2pAEUVRKZzDgGx9PzrZt5O7e7RUJoslYd2+Y8voEeoI3FYgWmkIb/PSHtnxpD5lqrzHyhtZ9b4jBHXQ6eOxurU/im59pSaJeDyMur/6+VGEhGWvXAhDcu7dXTLX3zc/w1CJ4+gEtKXzzU63Y+up3tbqapX35I8xZAnOnaJMc7D1YXG/z8XvqN/Yinn8WhdfzTUggZ9s2chITCe3f39PhCOERBQVwhzabF9tXyFR7jZE3tO57QwzuotNpyY/NrrW2TXlW65N4XTVndrVbrSRNngzAOb/9VmdT7WVlZTnNolZ6hrWSXv9EaxEtSnhnTYSf/4Jl35U/N/U/W7XEcOjpr9hm0XBVP/h3u7sfheskQRRV8ktIII26H6jiDb+MvSEG4Z30eujWsXhZNE7e0LrvDTG4i06nTV1ot8PSr2DyXO39dc2A6uxD55gJrS6n2uvQoYPT9enTpzNjxowy2xVYtcFEdxeXbdam6jwXNmwtf9/dOsKXP2gJ4TnttfJpq/+Cay5z4wOoJkkQRZXqa8q9ol/Gf2+Be58Cf1/4+HmgxPtdfp0LT7GYtf6vQgj30ungyfu0083vfa2dWtXpYJiLyZHeYqHdm2/WbZDAtm3biI2NdVyvqPXwZIbWKlr6VHJEKCQdKH/fQ/tr97vuXu15KLTBTVfB+JvcFX31SYIoqlQ0krng0CFs2dkYAgLq7FixUfD7Bm2581napb6dSb/OhRCiIShKEu12+GA5THxGa3W7+lJPR1YsMDCQoKC6KTD5x79aEfEn74eu7WFfCsxcCC++C/fdXCeHrJKcKBFV8gkOxhilZUx13YoIsOX0mexOdTRSWQghhPfR67VBHdcP1hLF+2dpgz0amtBgMOi1yRRKOn4SmoSVf595b8KwAXDDYGjXGgZdBA/ephUVt9vrPubySIIoXFJfp5lBKzUDdVfKRoiayMvXptobclft59EWQpRPr4fZE7XBHXY73Pc0fLum8vsUTbW3/eabsee5OB1WHTIZtULpRWfDQHssv2+AczuWf5/cPG2ATkmG0xmapyaHkVPMwiV+CQlk/v47Obvqdh44ux22nc5BpQVReBO7HTbtLF4WQtQNvR7mTNYSo09WalOb6nRaXcDyKKXI2bbNsewNbhsOk56BLm3h7NNlbnLyYPjpEdoPzILoJjD1du16/57ayOeOCdoglf0pWqti/x5akX5PkARRuKS+WhD3JmtvIosZ4pvV6aGEqBaTCd6cVbwshKg7RUmi3Q6frdKSRL1eO/VaZlujkTYvvOBY9gZD+kFaBsx/WyuU3SEe3p1TfIr50FHnaggTRmtJ8HNvwJHj2jStl/bQTjN7iiSIwiWOKfd270bZ7ejqqM7H1tP5Z/t4z/1qEqI8PgbtA1sIUT8MBnh2ijYt3xc/wD0zYdEMGNDbeTudjw/BvXuXuw9PGjtMu5Tn4xecr/sY4P4x2sVbSB9E4RJLixbojEbsOTkUHDpUZ8fZcvoMdsc2dXYIIYQQDYTBAPOmaqOZC21w90z44XdPR9U4SIIoXKLz8cHSqhVQt6eZt0r/Q+GlbDb47W/tYrN5OhohGg+DAeZP02YWsRbCuBnF014CKJuNzD//JPPPP1Hy5nQbSRCFy4pOM+fU0YwqSkmJG+G98gvgpge1S2Uz7Qgh3M/HAM8/DFf21ZLEu6bDz39qt9kLCkgcP57E8eOxF8ib012kD6JwmV9CAieouyn3Dh2F9Eztg6Btqzo5hBA1ptdrHc2LloUQ9cvHAC88ovVJXPGLNjf6o3fDuQk6dC3aArA1SYfu9CCyxjTr1fy3tNJAcdHu26ckiMJlJQeq1IWi+odtW2rzHQvhTSxm+O51T0chRONm9IGXHoVTOfDLepj+EoAF+EDbYELxtmaTNnVqY0gSf/gdFr4HF5wNI6+Ayy+u/feo/A4WLitKEPMPHsSWm+v2/W+RAtlCCCGqYPSBB26perv8AjiRUffxeIPvXoevF2sNLDMXQvdr4ZHn4b8dNd+nJIjCZcawMHzCw0Ep8pKS3L7/ogEqMoJZCCFEZXykDFoZnRJg5r2w7lOY+yAcPgbXToCB/9MKdWdmV29/kiCKaqnLgtlFJW46tXX7roWotbx8GHm/dpGp9oTwHiaVxxM5d/BEzh2YlOen2vM0paCwEKxWbTk4AN75EnqMrN7c1tIHUVSLX0ICWX/95fYp99LSterxOp1WJFsIb2O3w5//FS8LIbyDDkUn2wbHcmO1eScsWwlf/6zNB33tAHjyfmgZq93+1ucwY6E2y4srJEEU1VJXLYhFA1RaxUGAn1t3LYRbmEzwyvTiZSGEd7Bi5FnLM47lxmjArZB0AC7qrp1eLm8O56sv1fonukoSRFEtjpHMiYkopdDpdG7Zr2OAivQ/FF7KxwCD+3o6CiFEaXadD38Y+3s6DI8a3BdGXg7RTSreJiwY9lXjFLP0QRTVYmnVCgwGbFlZWFNT3bbfrVIgWwghhKiR+26uPDmsCY+3IB5dtozUpUuxpqXhm5BA8wcfxL9Tp3K3zU1K4tDixeTs2EHB4cPETZxI1I03Om1z+K23SF+9mrx9+9Cbzfh36ULchAlYWrZ0bLPzjjvI3rDB6X4R11xDi4cfdvvjO9PoTSYsLVqQt2cPubt3Y4p2T1VOKXEjvJ3NBhu2acvndih7+kYIUX/CgrU6f/kFoFc22to2A7DL0Bm7Tntzmk3ado3BnY/DOe1h3A3O6xd/CP/thEUzqr9PjyaIJ1atIvn552k+bRr+nTpx9MMPSZwwgY6ffYYxLKzM9va8PMxxcYT278/B+fPL3Wf2hg00GT4c/w4dUDYbKS+/TOL48XT45BMMvr6O7SKGDSPmzjsd1/UWi/sf4BnKNyGBvD17yElMJLh371rvL+sU7EvRliVBFN4qvwCuu1db3r4C/Hwr314IUXdio7Qi2CcyQOUVYL31NgCMb/6GzqK9ORvTTCrrNsEDY8uu73sBvPZJzfbp0QQx9f33iRg6lIirrgKg+bRpZKxdS9rXXxM9dmyZ7f07dsS/Y0cAUhaW39MyYcECp+stZ8xg02WXkbN9O4HnnutYr7dYMEZEuBxrfn4++fnFtS2ysrJcvu+Zxi8hgZPff++2Kfe2nR7vEhPZeH7tiYZHpyseDeimrrdCiFqIjdIu9jwd25o1A6BDWx36RtjecypXKyBemo+P1ghTEx5LEO1WKzk7dtD0luJy6Dq9nsDzzyd70ya3HceWrVWG9AkKclp/4rvvSFuxAmN4OCEXX0zT226rtBVx9uzZzJw5021xNWTunnJvqwxQEQ2ArwV+ec/TUQghStNbLHT64gtPh+FR7VrD8tVw3xjn9d/8DAktarZPjyWIhenpYLPhU+pUsjEsjLx9+9xyDGW3kzxvHv5nn+0ozwIQNmgQpqZNMTVpQk5iIikLFpC3fz/xzz5b4b6mTZvGxIkTHddTUlLo0KGDW+JsaIoSxLz9+7Hn56M3m2u1v6IZVKRAthBCCFF9947W+iHuPwQ9u2rrft+g1UR8ZUbN9unxQSp16cCcOeQmJXHW6687rW9yzTWOZd82bTBGRJA4bhz5ycmY4+LK3ZfZbMZcIhHKzMysm6AbAGOTJhiCg7FlZJC3dy9+7drVan9S4kYIIYSouf49YcmT8PL7sOJXsJigXTy8/xxceE7N9umxBNEnJAQMBgpPnHBabz1xAmN4eK33f2DOHDLWruWsJUswRVXeS7Vo1HTewYMVJoiimE6nw7dNG7L/+YecxMRaJYh5BZC4T1uWASrCm+UVwF2Pa8uLn9A+gIUQnmfPzydpyhQA4ufOrfVZrYbq0h7axV08VgdRbzTi164dmevWOdYpu52s9esJ6NKlxvtVSnFgzhzS16yh7aJFmGNjq7xP7s6dANUatNLY+ZUomF0bO/eAza4NTmnq5hpOQriT3Qar/9IudpunoxFCFFF2O5m//07m77+jZB5Mt/HoKeaoUaPYN2MG/h064NexI0c/+AB7bi7hQ4YAsPfxxzFFRhI7fjygDWzJ27MHAGW1Yj12jJydO9H7+WE5PYLp4Jw5nFi5kvh58zD4+WE9fhwAQ0AAeouF/ORkTqxcSVCvXvgEB5ObmMjB+fMJOPdcR9IjquauKfe2lqh/KCNDhTczGuG5qcXLQgjvoDcaaTF9umO5MbLZ4PVP4ds1cCgVCgqdb9/0dfX36dEEMWzAAApPnuTQ4sVaoey2bUlYsMBxirngyBF0+uJGTuuxY2wfNcpxPXXpUlKXLiXg3HM5a8kSAI59+ikAu0rUOARoMX06EUOGoPPxIXPdOlI//BB7bi6mqChC+/Wj6f/+V9cP94ziGMm8a1etptxzDFCR3Fx4OaMPDB/k6SiEEKXpfHyION2w1Fi98A58tAJuHwHPvQHjb4LkI7BqrTbLSk14fJBK5MiRRI4cWe5tRUlfEXNMDN3+/rvS/VV1uyk6usx+RfX5xseDXk9hejqFaWk1Pj2/ZZf2VwaoCCGEEDXz5U/wzCStD+ILb8PV/aBFLLRvrc0Adcu11d+nzMUsakRvsWA+fVq/pqeZC22wXesxICVuhNez2bQW7627tWUhhHdQNhs5O3eSs3MnqpG+OY+d0GohgjbLU+bp4tiX9tD6TdeEJIiixor6bObUcKBK0gFt+rIAP2gR487IhHC//AK44nbtkl/g6WiEEEXsBQVsHzWK7aNGYS9onG/Opk3gaJq23CIGfjt9MvW/HWCqYbdMSRBFjTkGqtQwQSwaoNJeO1sthFfT6SAqQrvIgCohvIdOp8PYpAnGJk1q3B++oRvYWyuMDTB2GMx7E/rcBBOfgRGX12yfHu+DKBqu2k65V1QgWwaoiIbA1wLrajjpvRCi7ugtFrp8952nw/Coh+4oXh7SD2Kj4Z8t0CpOK6JdE5IgihpzTLm3Zw+qsBCdT/VeTiVL3AghhBCi+qyFMG0e3HszNG+qrTu3g3apDTmxJ2rM1LQpen9/VGFhtefPttulxI0QQghRW0YfWPmb+/crCaKosaIp96D6A1UOHoasU2A2QpsWdRGdEO6VVwDjZmiXvMbZD14Ir2TPzydp6lSSpk7Fnp/v6XA8YkAvreahO8kpZlErfgkJnPrvP22gyuWu94Tdcrr1sG0r7dePEN7OboMVv2jL86Z6NhYhRDFlt5P+00/a8owZng3GQ1rGwYvvwt9boHNb8LM4316TOojy1SxqpaZT7m09XSBbTi+LhsJohCfuLV4WQngHvdFIsylTHMuN0ccrICgANu/SLiXpdJIgCg9wjGSu5inmohZEGaAiGgqjD4wZ5ukohBCl6Xx8iBwxwtNheNTvH7p/ny73QTzxww/YrVbH9YLUVJTd7rhuz8vjyDvvuDc64fWKWhCtx45RmJ7u0n2UKh7BLC2IQgghzkTvfAG9roe2A+DqcfDv9oq3HXk/tLik7GXsQ/UWbhkutyDufeQRuqxciT4sDICtI0bQ4f33McfFAWA7dYqUl18mesyYuolUeCWDvz+m2FgKUlLI3b2bwO7dq7zP0TQ4flIrjl00NZAQ3s5uh/2HtOUWMVLcXQhvoex28pOTATDHxaHzgjfnNz/DU4vg6QfgnPbw5qcwegqsfhciQstu/+oTUFBYfD09AwbdBoP7una8yXMqv/25GvSbdv1ZVKry66LRqu6Ue0UFsts014oPC9EQ5OVD39HaJa9xDpQUwivZ8/PZes01bL3mGq8Zxfz6J3D9YG0Wk7YtYdZE7ftuWQX1vEOCIDKs+PLbP9r2g/u4drzMbOdLWjr8sRG+/614Xubqkj6IotZ8ExJIX7PG5X6IW6RAtmiggvw9HYEQojyGgIA6P0ZWVhaZmZmO62azGbPZXGa7Aqs2UOTuUcXr9HrofS5s2OrasT5eAUMuAT9f17Zf8mTZdXY7PPK8dsajJjzfDisavOqOZC4qkN2xTV1FJIT7+fnC5uXaxdUPbSFE3TP4+nLOmjWcs2YNBt+6e3N26NCB4OBgx2X27NnlbncyA2z2sqeSI0Lh2Imqj/Pvdti5V2uBrA29Hm4bDq9/WrP7V6sFMfOPP4qzdLudzPXrMSYlAWDLyqpZBKLBc4xkTkpC2WzoDIZKt3eUuGlb15EJIYQQ7rFt2zZiY2Md18trPXSHj1do/fPPaV/7fe0/BDZbze5brQRxX6kClAdmzXLeQKerWRSiQTPHxqK3WLDn5ZF/8CCWli0r3DY9E5JTteUO0oIohBCigQgMDCQoKKjK7UKDwaDXBmOWdPwkNAmr/L45ufDNapg4tnqxPfFy2XVH0+DnP+HagdXbVxGXE8Ru69fX7AjijKczGLDEx5OzdSs5iYmVJohFp5ebx0Bw3XcZEcJt8gvg4fna8qyJYDZ5Nh4hhMZeUOBosGr+8MPoTZ59c5qM2mwmv2+Agb21dXa7dr2qWqrf/gIFBTDssuodc2upHl56HYSHwKPjYMQV1dtXERmkItzCLyGBnK1btYEql1X8ynYMUJHWQ9HA2Gzw6ffa8pP3eTYWIUQxZbORtnw5AM2mesc8mLcNh0nPQJe2cPbpMjc5eTB8kHb7A7MguglMvd35fh+vgAG9tVbI6vj4effEXZLLCWLe/v3YsrLw79TJsS5z3ToOv/EG9txcQvr2pemtt7o/QtEgOAaqVDGSeauMYBYNlI8PTLuzeFkI4R10Pj7E3nuvY9kbDOkHaRkw/21tYEqHeHh3TvEp5kNHy9ZSTToA6zfDe89W/3gHDms/YlvFOa/fm6x9XjWLrv4+XX4mUxYswLdNG0eCmJ+Swu4HHiCwa1f8EhI48vbb6C0Wom68sfpRiAbPt6024qSqkcxbZAYV0UCZjHDX9Z6OQghRmt5oJPrmmz0dRhljh2mX8nz8Qtl18c1h/+qaHWvyM1rNxdIJ4sbt8PG35R+vKi6XuTm1fTtBPXs6rp/47jssLVqQsHAhzSZPptnEiY4mXtH4FLUgFhw+jC07u9xtcnJhz0FtWVoQhRBCCPfYuhu6dy67/twOsM21CnRluJwgFqanY4qMdFzP+ucfgi+6yHE9sHt3Cg4dqlkUosHzCQrCGBUFVNyKuH2PNgFPZLhWKV6IhsRuhyPHtEuJaeiFEB6m7HYKjh6l4OhRVCN9c+p0kJ1Tdn1WtlaTsSZcThB9goKwHj8OaP+MU9u2EdC5OF21W60omX6vUatqyr0tp+sfygAV0RDl5cMFI7SLTLUnhPew5+ez+Yor2HzFFV4z1V59O78LvPK+c81Dmw1e/qD8lkVXuNwHMbBbNw6/8QbNp07l5I8/glIEdOvmuD1vzx7MMTWcz0WcEXwTEshYu7bCgSpFw/ClQLZoqHwqrwEvhPCUKiZoONM9dAeMuA8uuVlLFgHWbdJaFT+cX7N9upwgxtx9N4n33MPmIUNAr6f5gw86TWmTtmIFgd27VzuAo8uWkbp0Kda0NHwTEmj+4INOI6VLyk1K4tDixeTs2EHB4cPETZxY7qCYqvZpz88n+YUXOLFqFaqggKALL6T5Qw9hDA+vdvyiWFVT7kmJG9GQ+flC0o+ejkIIUZrB15duf/3l6TA8qm1L+P4NeOcL2JYEFjNcO0CruxhSdW3vcrmcIJpjYuj4ySfk7tmDT2gopiZNnG6PufNOpz6KrjixahXJzz9P82nT8O/UiaMffkjihAl0/OwzjGFlO6nZ8/Iwx8UR2r8/B+eXnxK7ss+D8+eTsXYtrZ95BkNAAAfnziXpwQdp9+ab1YpfOHNMubd7N8puR1diDH+BFXbt1ZZlgIoQQgjhXlERMOX2qrdzlct9EEGrL+TXtm2Z5BDAr21bfEJCqnXw1PffJ2LoUCKuugrf1q1pPm0aeouFtK+/Lnd7/44dibvvPsIGDqywUnpV+7RlZ5P21Vc0e+ABgs47D//27Wk5fTqnNm0ie/PmasUvnFmaN0dnNGLPySkzYGnXPrAWQlBAzeoxCSGEEKJ8y76Db9eUXf/tGvh0Zc326XIL4qHXXnNpu5jbXUtf7VYrOTt20PSWWxzrdHo9geefT/amTa6GVe19ntq+HVVYSOAFFzi2sbRsiSk6mlObNjkNvCkpPz+f/BKdX7OysmoU45lM5+ODpXVrcnfuJCcxEXNccUGmkgWyZcpu0RDlF8CTr2jLj90tU+0J4S3sBQUkP69NJRL3wAMen2rPE175QJsCtLTwEJg2H64bVP19upwgHl6yBGOTJviEhmq1Ssqj07mcIBamp4PNhk+pU8nGsDDy9u1zNaxq77MwLQ2d0YhPYKDTNj5hYVjT0irc9+zZs5k5c2aN4mpM/BISyN25k9zEREIvucSx3jFARU4viwbKZoOlX2nLD9/p2ViEEMWUzcaxTz4BcMyo0tgcSoVmTcuuj42GlNSa7dPlBDGoZ0+y/v4bv/btibjqKoIvusipj9mZbtq0aUycWJyep6Sk0KFDBw9G5J0c/RBLjWSWEjeiofPxgfvHFC8LIbyDzseHpqcbp7xlqr36Fh4KO5LKduHavhtC63qQSsKLL1Jw7Bhpy5eT/OKL7J81i/DBg4m46iosLVtW/8AhIWAwUHjihNN664kTNR5N7Mo+fcLDUVYrhVlZTq2IhVUc12w2YzabHdczMzNrFOOZruRAlSI2G2xP0pZlgIpoqExGeGCsp6MQQpSmNxqJubNxN+tf1Q9mLAB/P7jgdJmbP/+DmQu1eaFrolpNgKYmTWh6yy10+vxzWs+eTeHJk2wfM4Ydt96KPS+vegc2GvFr147Mdesc65TdTtb69QR06VKtfVVnn/7t26Pz8SGrxDZ5+/ZRcOQI/jU8rihWlCDmJydjy9HKuu9NgZw8bdh9fDNPRieEEEKceSbdCue0hxsnwVmDtMvoB6FHV5hyW832WeO2WP8OHSg4dIjcPXvI2bkTVVhY7X1EjRrFvhkz8O/QAb+OHTn6wQfYc3MJHzIEgL2PP44pMpLY8eMBbRBK3p49ACirFeuxY+Ts3Inezw9Ls2Yu7dMQEED41VeT/Pzz+AQHo/f35+Czz+LfpUuFA1SE64yhofiEh1OYlkbenj34d+rkGKDSPr7R1zIVDZhSkHlKWw7yl8FWQngLpRS27GxA+47XNcI3p8kIL0+HSQe1M3YWM5zVCuJqUTWk2gli9qZNpH39NSd++AFLixaEDxlC2KBBGAICqn3wsAEDKDx5kkOLF2tFrdu2JWHBAsep3oIjR5z6OVqPHWP7qFGO66lLl5K6dCkB557LWUuWuLRPgGYTJ5Ks15M0ZYpWKLtHD5pPnVrt+EX5/BISyExLIycx0SlBlAEqoiHLzYMu2u9Mtq/QCmcLITzPnpfHf6cHRZ7z229Ok3g0Nq2baRd3cDlBPPLOO6QtX05hejphgwZx1uuvO+berY3IkSOJHDmy3NuKkr4i5pgYuv39d632CaA3m2k+daokhXXENyGBzD//dAxU2VKixI0QQggh3O/wMfjhdzh0VJucoqTH76n+/lxOEFMWLsQUHU1o//6g05H2zTeUVxSm2cRyCvGIRqXklHtKSYkbcWbwtcDuH7RlmZNZCO+ht1g4988/tSuNtB/T2n/gtkeheVNIOgBtW0HyEUBBx7Y126fLCWJA166g05F7ug9geRrjeX9RVslSN8lHFOmZOnwM2lyRQjRUOh0YG2cFDSG8mk6na/S1p+a+DneMgIm3QIcr4NWZWumb+56CPufXbJ8uP6OlT/cKURFLy5ZgMGDLymLT3+lAKG1byswTQgghRF3YvR8WPKotGwyQlw/+vlrCePujMPrq6u+z8VS6FvVGbzI5amP++482JaH0PxQNXYEVnl6sXUr37xFCeI7daiX5xRdJfvFF7NbG+eb0s0DB6WIykeGw/1DxbScyarbPxt0mK+qMX9u25CUlOfofSoIoGrrCQljysbb8wBitrIQQwvNUYSGpS5cC0PSOO8DY+N6cXTvA35shoQVccgE8tQh27IWVv2q31YQkiKJOFA1U2XksGJABKqLh8/GBO0YWLwshvIPOx4eo0aMdy43RY3fDqVxteeJYyMmF5auhZax2W000zmdS1DnfhATSdaEcLwhGp4MOMgezaOBMRnjkLk9HIYQoTW80EnfffZ4Ow6OaxxQv+/nCLDcUlJE+iKJO+CYksFd/FgAtY+z4N966pUIIIUSDU6MWxMKsLHK2bsV64gTY7U63hV95pVsCEw2bMSKCff7ngIL2TbOBIE+HJEStKAWFNm3ZxyBT7QnhLZRSYDv95jQYpOSem1Q7QUz/9Vf2PvYY9pwcDP6lJiTV6SRBFIBWl2p/QFfIgja+yUANe8kK4SVy86D9FdqyTLUnhPew5+Xx70UXATLVnjtVO0FMfuEFIq66ith77kFvsdRFTOIMscfWGoBW1m1IgiiEEEI0HNXug2g9epTIkSMlORSVysyG5JxQAJqn/eXhaISoPV8LbPpGu/jKx58QXkNvsXD26tWcvXp1o81NXnxHO8tRWl6+dltNVDtBDOrRg1Pbt9fsaKLR2J6k/Y2wH8a4Z6PWR0SIBkyng+AA7SJdnITwHjqdDp/AQHwCAxtt/8MX3i0uc1NSbp52W01U+xRzcK9epLz4Inl79uDbpk2ZmkMhffrULBJxRtmSqP1tbd9FYXo6hWlpGCMiPBuUEEIIcQZSqvwfrtuSICSwZvusdoK4/+mnATj8+utlb9Tp6LZuXc0iEWeUracTxITAI5ALOYmJBEuCKBqwAiu8/L62fM8omUlFCG9ht1o58uabAETfeiv6RjSTSuchWmKo00Hf0c5Jot2utSqOGlKzfVc7Qey2fn3NjiQalaIWxA7N8+Ao5CYmEtyjh2eDEqIWCgvhhdN9ee4cKQmiEN5CFRZy+LXXAIi6+eZGNdXe9HtAAQ/O1WZQCfQvvs1ohLho6NaxZvuWmVSE2+Xlw+792nLnDiYK/9YSRCEaMoMBRl9dvCyE8A46g4Emw4c7lhuT6wZpf5tFQ7dOYHRjVufSro5+9BERw4ahN5s5+tFHlW4bef31bglMNFw794LNDuEh0PzsWPYAubt3ezosIWrFbIKn7vd0FEKI0vQmE82nTvV0GB7VrCkcTav49tio6u/TpQQx9YMPCBs0CL3ZTOoHH1S8oU4nCaJgyy7tb8c24Nc2AYC8vXuxW62Nqm+IEEIIUR963VB5dYW9P1V/ny4liJ2//rrcZSHKs/V0Y2HHBDBFR6P398d+6hT5+/fj26aNZ4MTQggh6sE7X8CSj+HYCWgfDzPvhXPaV7x9RjY8+zqs/A0ysrRWv8fvgX4XVn2sFUucrxfatMGir30CD/6vZvFLH0ThdkUDVDolaPWp/BISyP73X3ISEyVBFA1WTq42YhBg8zcy1Z4Q3sKWm8u/ffsCcM6aNV4x1d43P8NTi+DpB7Sk8M1PYfQUWP0uRISW3b7ACjdN1rpmLZoB0U0g5QgEBbh2vA7lfLV2OQsiw7Uk9fKLq/8YapQgFqSmkv7rrxQcOYKyWp1uazZxYk12Kc4QhTbYsUdb7qidXcb3dIKYm5gIl1/uueCEqKVCm6cjEEKUy1b3b86srCwyMzMd181mM2azudxtX/8Erh8MI05/5c2aCD//Bcu+g7tvLLv9su8gPQs+X1g80KRZdO1jjm8O/+2s2X2rnSBmrltH0sSJmGJjydu3D9/4eAoOHwal8GvXrmZRiDNG0gHIL4AAP2gRo60rajWUgSqiIbOY4a9lxctCCO+gN5vpvGKFY7mudOjQwen69OnTmTFjRpntCqyweRfcPapEjHrofS5s2Fr+vn/4Pzi3Azz2grYcFgxXXwrjbnCtakLWKefrSsHRE/DC29Aqtur7l6faCWLKwoVEjR5NzJ13svHii4mfOxefsDD2PvoowT171iwKccYoGqDSoY32hgCtBREgd9cuD0UlRO3p9dppHyGEd9Hp9ZgiI+v8ONu2bSM2tjjbqqj18GSGVsmj9KnkiFCtEaU8Bw/BH0fg6v7w9mzYlwKPvqidtbh/TNWxFRXMLkkpiImEBY9Vff/yVDtBzNu3j9azZgFavSF7fj4GPz9i7rqLpEmTaHLddTWLRJwRigaodEooXlfUgmg9fhzryZMYQ8vpgCGEEEJ4scDAQIKCgupk33YF4aHwzCStxbDzWXDkOLz6sWsJ4kfzna/r9RAWAi1jwaeGpSGrnSDqfX2xn+53aIyIID85Gd/4eAAK09NrFoU4Y5QscVPE4OeHOS6O/ORkcnfvxnjeeZ4JTohaKLDCm59py7deKzOpCOEt7FYrRz/8EIDIG27weDm10GAw6OH4Sef1x09Ck7Dy7xMZBj4+zqeT27TQRkAXWKv+vLnwnFqFXK5qJ4j+nTqR/e+/+LZqRXCvXiS/8AK5u3eTvno1/p071yiIo8uWkbp0Kda0NHwTEmj+4IP4d+pU4fYnf/yRlEWLKDh8GHOzZsRNmEBw796O2//p3r3c+8Xeey/RN98MwOYhQ7S+kyVvHz+e6LFja/QYhDbv47YkbbljgvNtvgkJWoKYmEiQJIiiASoshNmvass3Xy0JohDeQhUWkvLSSwDajCoeThBNRujcFn7fAANPpyZ2u3Z9zLDy79O9E3z1k7ZdUfesvQe1UciuftbsT4E3PiueySyhhfZjtkV99UFs9sAD2HJzAWh6553YcnI4+cMPWqJWgxHMJ1atIvn552k+bRr+nTpx9MMPSZwwgY6ffYYxrGyqnf3ff+x55BFi77mH4Isu4sTKlSRNnkz7995znMrssnKl030y/u//2P/kk4T26+e0Puauu4gYOtRxXe/vj6i5A4e1jrJmo/bLpyTfNm1IX71aptwTDZbBANcNLF4WQngHncFA+JVXOpa9wW3DYdIz0KUtnH26zE1OHgw/PTXeA7O0Ps1Tb9eu33Q1vPMlzFgIY4fB3mR4+QMYe41rx/tlHdz2KHSI16bcA/hnC1x2C7wxCy4qv92sUtVKEJXNRsHRo45BBwZfX1o8/HD1j1pC6vvvEzF0KBFXXQVA82nTyFi7lrSvvy63Ne/oRx8R3KOHoyUwdtw4sv76i6PLljliMUZEON0n/ZdfCOzeHXNcnNN6vZ9fmW0rkp+fT35+vuN6VlaWy4+xsdh6Ovc7q3XZ+SAdA1VkJLNooMwmmPeQp6MQQpSmN5loWc5oYk8a0g/SMmD+29pp4g7x8O6c4lPMh44WtxSCNpjk3bnw5Msw6H8Q1QRuuUYbxeyKZ16D/10HD91Rav0SmL2kHhJEncFA4vjxdPzkE3wCA6t/tFLsVis5O3bQ9JZbio+h1xN4/vlkb9pU7n2yN20iatQop3VBPXqQvmZNudtb09LIWLuWVjNnlrntyDvvcPiNNzBFRRE2aBBRN96Izqf8p2T27NnMLGcfotjWEgWyS3MkiHv2oAoLK3yehRBCiDPB2GHapTwfv1B2XbeO8OUrNTtW0n54ZXrZ9SMu11ova0Jf9SbOfOPjyU9JqdnRSilMTwebDZ9Sp5KNYWFY08qfdbowLa3MqWefSrZPW74cg78/IZdc4rQ+cuRIWj/9NG0XL6bJNddw5K23SD7dh6E806ZNIyMjw3HZtm2bC4+wcSmaQaV0/0MAc2wseosFlZ9PfnJy/QYmhBBCnMHCQmBbOSfotu3WRkfXRLWbcWLGjSP5hReIGTcO//bt0VssTrcbAlycF6aeHP/6a8IGDSpTPDPqppscy34JCeiMRvbPmkXs+PHoTaYy+yldMb1kNXWh1VtyJIjlTPmj0+vxbdOGU1u2kJOYiKVly3qNT4jaysmFC4Zry399IlPtCeEtbLm5bD49S1fn777ziqn26tsNg+GhedpYgG4dtXV/b4FFH2r9IWui2gni7vvuAyBp4kTnqoxKgU5Ht3XrXD94SAgYDBSeOOG03nriBMbw8PLvEx6OtdT2hRVsn7VxI/n79xMxe3aVsfh36gQ2GwWHDknyUgOpxyEtXRva3z6+/G18ExI4tWWLNlDlssvqNT4h3CHzVNXbCCHqny0729MheNS9N4O/H7y2DOa8pq2LCocHxsAt19Zsn9VOENsuXlyzI5VDbzTi164dmevWEXJ6om1lt5O1fj2RI0aUe5+ALl3IWr+eqBuLJzPM/OuvckvspH31FX7t2+PXtm2VseTs2gV6fZnT3cI1RQWy45tXPA2ZY8o9GcksGiCLGdYsLV4WQngHvdlMx88/dyw3Rjqd1lJ423DIztHWBfjVbp/VThDNsbEYo6LQlZrTRSmFNTW12gFEjRrFvhkz8O/QAb+OHTn6wQfYc3MJHzIEgL2PP44pMpLY8eMBiLz+enbecQep771HcO/enPj+e3K2bSszmtqWnc3JH38k7v77yxwze9MmTm3ZQmD37hj8/MjevJnk+fMJu/xyfOqoSvqZbkslA1SK+J5O1CVBFA2RXg+t4qreTghRv3R6PZbmzT0dhtfYngRdzqr9fqqdIG6+6iq6rFxZZqCILSODzVddVa1TzABhAwZQePIkhxYv1gplt21LwoIFjlPGBUeOoCsxFjzg7LNp/fTTpLzyCikvv4y5WTPin3vO0TpV5MSqVSilCBs0qMwx9SYTJ1et4vCSJditVswxMUTeeGOZ0dHCdVsrGaBSpOh/VHDkCIVZWW4ZCS+EEEKIYmMfgu9eg+YxtdtP9WuNnO5rWJotN7fcwR2uiBw5ksiRI8u97awlS8qsC+3fn9D+/SvdZ5NrrqHJNeVXmPRr1452b79d7ThFxSorcVPEJzAQU3Q0BUeOkLt7N4Fdu9ZPcEK4gbUQPvhGW75xSNlan0IIz1CFhRw7fYq5yTXXNPoyakq5Zz8uP4sH55+eCVqn49CiRc6jl+12Tm3Z4jiFKBqXkxmQfLp3QYdyRjCX5JuQoCWIiYmSIIoGxWqFx09Xwho+SBJEIbyF3Wrl4Ny5AIQPGYKhkSeI7uLys5izc6e2oBS5u3ejKzHXod5oxDchgajRo90eoPB+RQNUmsdAUBVVjnzbtCHjt9+kH6JocPQGuKJP8bIQwjvo9HpCLr3UsdzYzZoIETWsfViSywniWa9qs9TvmzmTZpMmeV29Q+E5rgxQKSJT7omGymKCRTM8HYUQojS92Uz8nDmeDsNrDK28B57Lqt0O23J6OXO5iEZtayUFsksrmSAqu11+7QkhhBA1cMfjrm+75Inq71++nUWtFZ1i7uRCF1RLs2boTCbsubkUHDpUt4EJIYQQZ6gg/+JLoB/83wbYvLP49i27tHVB/jXbv/TkFLVyKhf2HNSWXWlB1Pn44Nu6NTk7dpCTmIg5TgrLiYYhNw/6nO5m/ctS8LVUvr0Qon7Y8/LYMmwYAJ2++KLMFMBnquemFi/PfhUG94VZD4DhdB9pmw0efQECapggSguiqJXtSdqQ+shwaOLiJDSO08wyUEU0IEppU0qmHndfGQkhRO0ppbAeO4b12DFUI31zLvsO7hhRnByCtnzbcO22mpAWRFEr1RmgUkQSRNEQmU2w4rXiZSGEd9CbTLR//33HcmNUaIOkA9p0tyUlHQC7vWb7lARR1IorM6iUJgmiaIgMBte6UQgh6pfOYMDvLDfMLdeADR8EU56F/YfgnPbauo3bYdEH2m01IQmiqBXHAJXqJIinp9zLT07GlpODwa+WM4oLIYQQjdij4yAyDF77BI6maesiw+HOkXD7iJrtUxJEUWMFVti1V1uuTguiMTQUY0QE1uPHyU1KIqBz57oJUAg3shbClz9qy0P7y0wqQngLVVhI2ndaR7vwyy9vlFPt6fVw1w3aJeuUti6whoNTijS+Z1G4za592pdmcCDERVXvvr4JCVqCmJgoCaJoEKxWmHy6Fu/gPpIgCuEt7FYr+2fOBCC0f/9GP9VebRPDIo37WRS1smWX9rdjG9Dpqndf3zZtyPzjD+mHKBoMvQEuuaB4WQjhHXR6PUG9ejmWG4vLb3f9u3fFkurvXxJEUWNF/Q+rc3q5iEy5JxoaiwnefsbTUQghStObzSS8+KKnw6h3A3vX7f4lQRQ1VpMSN0VKjmRWSqGrbhOkEEII0YjdP6Zu99942mKFW9lsWpFsqFkLoqVlSzAYsGVnY01NdWtsQgghRGOTkQ0ffgtzXoP0TG3d5l1w5FjN9icJoqiRvSna1GO+Fmhdg9ny9EYjvq1aAZAj/RBFA5CbB31u0i65eZ6ORghRpGiqvS3DhmHPa5xvzu1JcMloWPwhLPkYMrO19St/0xLGmpAEUdRI0QCV9vHOU/tUhxTMFg2JUrAvRbs00tm8hPBKSinyDx4k/+DBRjvV3pOvwHUD4Zf3nGd6uuQCWLepZvuUPoiiRhwDVGoxs4RvQgJ8950kiKJBMJvg05eKl4UQ3kFvMnHW6687lhujTTth9sSy66Mj4NiJmu1TEkRRI0UtiDUZoFJEWhBFQ2IwwHlSslMIr6MzGAg45xxPh+FRJiNk5ZRdvzcZwkJqtk85xSyqTanalbgp4nc6Qcw7cKDR9hsRQgghaqt/T3jpXW3yCgB0kJIKs5fA5RfXbJ+SIIpqS0mFjCxtJom2LWu+H5/wcHxCQsBuJ3fvXneFJ0SdKLTBt2u0S6HN09EIIYqowkJO/vgjJ3/8EVVYWPUdzkCPjoNTuXDuMMjLh5H3awPqAnzhwf/VbJ9yillUW1H9w4SWteuLpdPp8E1IIGv9enITE/Fv394t8QlRFwoK4G5tNi+2rwAfX8/GI4TQ2K1W9jz0EADn/PZbo5xqLygA3n8O1m/WRjTn5EKnttC7W8332fieRVFrW2tRILs03zZtHAmiEN5Mr4cLzy5eFkJ4B51OR8C55zqWG7PzOruvr7QkiKLailoQa9P/sIhMuScaCosZPn7B01EIIUrTWyyctaQGkw2fYdb+A/+3AY6ng93ufNtzU6u/P0kQRbW5o8RNkaIEMWfXLplyTwghhKiBF96BF9+FLm0hMhzc8VXqFQni0WXLSF26FGtaGr4JCTR/8EH8O3WqcPuTP/5IyqJFFBw+jLlZM+ImTCC4d/Gs1ftmzCBt+XKn+wT16EHCggWO64UZGRx89lnSf/sNnU5HSL9+NJs8GYOfn/sf4Bnk2AlIPa69+Dq4I0Fs1Qr0emwZGRSmpWGMiKj9ToUQQggPe+cLbVaTYye0SSVm3gvnVNDV/pOVMHmO8zqzEXatcu1Y730N86bCNQNqF3NJHu9Jc2LVKpKff56mt99O+/few69tWxInTMB6ovzKjtn//ceeRx4h4uqraf/++4T07UvS5MllTlEG9exJl5UrHZdWTz/tdPvexx4jd88e2r78Mm1eeIHsjRvZX2obUVZR62HrOPB3Qyd9vcWCpUULQKbcE94tLx8uv0275OV7OhohRBF7Xh7bbryRbTfe6DUl0775GZ5aBPeNgeVLtARx9BQ4frLi+wT6w/rPii+/f+T68ayF0K1j7eMuyeMJYur77xMxdCgRV12Fb+vWNJ82Db3FQtrXX5e7/dGPPiK4Rw+ib74Z31atiB03Dr927Ti6bJnTdjqjEWNEhOPiExTkuC13714y/+//aPHoo/h36kTAOefQ7MEHOblqFQXHyp/VOj8/n8zMTMclKyvLfU9CA7LVjf0Pi/i20Zoic3ftct9OhXAzux22JWmX0v17hBCeo5Qid9cuck93VfIGr38C1w+GEZdr5eBmTQRfCyz7ruL76IDIsOJLkzDXj3f9FfDlT7WN2plHTzHbrVZyduyg6S23ONbp9HoCzz+f7E3lTx6YvWkTUaNGOa0L6tGD9DVrnLf75x/+u+wyDIGBBJ53HrHjxmk194BTmzZhCAzEv0OH4n2cfz7o9ZzasgXTJZeUOe7s2bOZOXNmDR/pmcOdA1SK+CYkcPKHH2SgivBqZhO892zxshDCO+hNJhIWLnQs15WsrCwyMzMd181mM2azucx2BVbYvAvuLpGq6PXQ+1zYsLXi/Z/KhZ7Xaz9AOyXAlNugbauKt3/i5eJlpeCD5fD7P9AuHnwMzts+fk9Vj64sjyaIhenpYLPhE+acJhvDwsjbt6/8+6SlYSy1vU9YGNa0NMf1oB49CLnkEsyxseQnJ5Py8ssk3nsv7d56C53BgDUtDZ/QUKd96Hx88AkKorDEfkqaNm0aEycWT3SYkpJChxIJZmPhzhI3RWTKPdEQGAxwUXdPRyGEKE1nMBB04YV1fpzS3/nTp09nxowZZbY7mQE2O0Q4pxlEhELSgfL33boZPDtFS+6ysmHJMrhmAvzwFjRtUv59tpZqUykaF7Cz1LwTNR2w4hWDVNwtbOBAx7Jvmzb4tmnDlqFDyfrnH62lsAZK/1Io+SuiscjMhv2HtGV3jGAuUjTlXu7evditVvRGo/t2LoQQQrjBtm3biI2NdVwvr/Wwprp1dO5D2K0TXDoG3v8GJt9a/n0+ft5thy+XR/sg+oSEgMFAYakBKdYTJzCGh5d/n/DwMgNYCivZHsAcF4dPSAj5Bw8CYAwPp/Ckc09RVVhIYWYmPpXsp7HbdvrXSmwUhAa7b7/GqCgMAQFgs1XYciyEpxXa4Kc/tItMtSeE91CFhWSsXUvG2rV1OtVeYGAgQUFBjktFCWJoMBj0ZQekHD/per9Co4/WlWt/Si2DrgWPJoh6oxG/du3IXLfOsU7Z7WStX09Aly7l3iegSxey1q93Wpf511/4d664dHhBaiqFGRmOEir+Xbpgy8ri1Pbtjm2y/v4b7PZKy+s0dkXN2e48vQzFU+6BnGYW3qugAG59WLsUFHg6GiFEEbvVyu7772f3/fdjt1o9HQ4mI3RuC79vKF5nt2vXz3VxpLHNBjv3VG+girt5fBRz1KhRHP/yS9KWLyd3714OzJ6NPTeX8CFDANj7+OOknO58ChB5/fVk/N//kfree+Tt28ehV18lZ9s2IkeMAMCWk0Pyiy+SvXkz+YcOkbluHUmTJmFu1oygHj0ArfZeUM+e7H/qKU5t2UL2v/9yYO5cQgcMwNSkgpP9gi2nBxm78/RyEcdIZkkQhZfS66HLWdpFptoTwnvodDr8OnTAr0MHr5ls4bbh8NFy+HQlJO6HR56HnDwYPki7/YFZMOe14u1ffAd+XQ8HDmkDXO6fBcmp2khoT/F4H8SwAQMoPHmSQ4sXa4Wy27YlYcECxynjgiNH0JX4NA44+2xaP/00Ka+8QsrLL2Nu1oz4555zJBg6vZ7cxETSli/HlpWFsUkTgi68kJi77nIa3dTqySc5MHcuu+6+G3Q6Qvv1o9mDD9bvg29gHC2Ibd2/b5lyT3g7ixm+WezpKIQQpektFtq/+66nw3AypB+kZcD8t7VC2R3i4d05xS2Ch446/9DMyIaH5mnbBgdo37OfL9RK5HiKTnlL0aAGJjk5mWbNmnHw4EHi4uI8HU6dy8uHDldoI7P+WgbRbm5ozd68mZ233IIxIoIuK1e6d+dCCCFEDTW27/sicqJEuGTHHi05DA+BqDqYDc83Ph50OqzHj2M9WUmpeSGEEELUOUkQhUu2lKh/WBddPAx+fphP/zKT08zCG+XlwzXjtYtMtSeE97Dn5bHj1lvZceutXjPV3pnA430QRcNQF1Pslebbpg35Bw+Su2sXQeedV3cHEqIG7Hb4Z2vxshDCOyilOHV69jXpNec+kiAKl9RViZuSfBMSSF+9WloQhVcymWDJk8XLQgjvoDcaiX/uOceycA9JEEWVrIWwI0lbrtMWRKmFKLyYjwEG9vZ0FEKI0nQ+PoT07evpMM440gdRVCnpAORbIcAPmjetu+M4ptzbs6dOq+ELIYQQonKSIIoqlSyQXZcFgk0xMeh9fVEFBeSdnhZRCG9hs8Ef/2oXm0y1J4TXUDYbWX//Tdbff6Pkzek2kiCKKhX1P6zL08ugFTmXGVWEt8ovgOsf0C75MtWeEF7DXlDArrvuYtddd2GXeTDdRhJEUaWtJUrc1DVJEIW30ukgoYV28ZLZvIQQaFPtWVq3xtK6tddMtXcmkEEqolJ2e/21IIJMuSe8l68Ffnzb01EIIUrTWyx0XLbM02GccaQFUVTqwGHIzgGzCdq0qPvj+bXVJnqWFkQhhBDCcyRBFJUqGqDSrrVW5qOuFZ1iLjhyhMKsrLo/oBBCCCHKkARRVMpxerlN/RzPEBCAqalWS0dOMwtvkpcPoyZrF5lqTwjvYc/LY9fdd7Pr7rtlqj03kj6IolJFLYj1MUCliG+bNhQcPkzurl0Edu1afwcWohJ2O6z9p3hZCOEdlFJkrVvnWBbuIQmiqJBS9TtApYhvQgIZv/0mLYjCq5hM8MLDxctCCO+gNxpp+eSTjmXhHpIgigqlHoe0dDDotT6I9UWm3BPeyMcAwy7zdBRCiNJ0Pj6EX365p8M440gfRFGhLafzszYtwGKuv+P6lSh1o+RcnhBCCFHvpAVRVKioQHZ9DVApYo6LA5MJe14eRz/6CL+2bQno2hWdoR6GUZegbDayN27Eevw4xogIj8QgvIfNVvyjqVMCyEtBCO+gbDZyduwAwK9dO/mcdhNJEEWFHF+Gbev3uOm//orObkcByfPnA2CMjKTZ5MmE9utXLzGc/PlnDj73HNajRx3r6jsGkCTVm+QXwFXjtOXtK8DP17PxCCE09oICdowZA8A5v/2GwVfenO4gCaKoUH2XuAEtMdszZUqZ9dajR9kzZQqt586t8wTNG2IoikOSVO+JQaeDuKjiZSGEd9DpdI7yaDLVnvvolIwJr5Hk5GSaNWvGwYMHiYuL83Q4bncyA84Zqi1v/gaCAur+mMpmY/OQIU4JUWnGqCg6f/11nSUH3hADVJykFmlMSao3xCCEaLzO9O/7ikgLonBISYUTGdryv9u1v9ERsP+QthwWDLFRdXf87I0bK03MAKypqWy7/noMAXWTsdqys12KYde4cZgiI9EZjeh8fCq/lN6m1HV9qevo9Rx45plKYzg4bx4hffp4JEltjK25QgjR2EiCKAAtObzkZq2fVUlHjsOVd2rLZhOsfrfukkTr8eMubZe3d2/dBFAN2Rs2ePT41tRU/u3XD0NAAHqzGb3JhM5kQm+xaMun1+ktFm292YzebC5eLu+2ovVmM/j4cGDOnEpjqOskVdlsHHzuOY/GIIQQjZUkiALQWg5LJ4el5Rdo29VVgmiMiHBpu5hx4xxzNrtb7u7dHFq0qMrtmowciTkmBlVYiLJaUTabtlx0vWi5sBB7qeulby99sWVnY8vMrDIG+6lT2E+dcsfDrhFraiobLrigeEVR35/Sfx03V3B7Bdspux1VUPmL0pqaSupHHxHaty+mqCitBbaO5ObauPvBTOwFBcy7JZnw88+RxFQIL2DPz2fPw1oV+9azZmk/ckWtSYIovEZA164YIyOr7P8XPXZsnX0xB/fuzbHPPqsyhmYTJ9ZZDFl//82uu+6qcrsWjz+Ob0ICKj8fe34+9oKC4uX8fFRBAfa8PG19QYFjfZnbTt/Xnp/vWC7MykLl5lYv8KLuzBV0a3a1s3N1O0WnPP88Kc8/DwYDpshITE2bYo6NxRQTg7lpU0yxsZhjYjBGRNT4f3by559JfHYBP+d9AcDOCdcSGBkk/SCF8ALKbifjl18cy8I9vCJBPLpsGalLl2JNS8M3IYHmDz6If6dOFW5/8scfSVm0iILDhzE3a0bchAkE9+4NgCosJOWVV8j4/XcKUlIwBAQQeP75xE6YgKlJE8c+Ng8ZQsHhw077jR0/nuixY+vkMYqq6QwGmk2eXOngjGaTJtVpq403xOBqohw+eLDHk9TWzz5LwNlnV5gcqtLrq7Fd9ubN7Hv00SpjMEZFUXjyJKqggILDhyk4fLjcLgA6Hx9MTZuWn0A2bYpPeHi5IyCL+kEqDIwzPgWAD1bpBymEl9AbjTR/5BHHsnAPjyeIJ1atIvn552k+bRr+nTpx9MMPSZwwgY6ffYYxLKzM9tn//ceeRx4h9p57CL7oIk6sXEnS5Mm0f+89fNu0wZ6XR86OHTS97Tb8EhIozMri4HPPkTRxIu2XLnXaV8xddxExdKjjut7fv64frtfKzvF0BJrQfv1oPXdu2VGrUVE0mzSpXr6IPR1DQ0pSQy6+uM7iMEVHk/LSSy6NKEeno/DECfJTUsg/dIiCQ4ec/x45giosJP/gQfIPHiSrnH3pzGbMMTHFCWTTppiiox39IH2wcZn1yzL3k36QQniWzseHJsOGeTqMM47Hy9xsHzMG/w4daD51KqA1D28ePJjIkSPLbc3bM20a9txc2rzwgmPdjrFj8W3blhan+yCUdmrrVnaMGUPn5csxRUcDWgti5A03EHXjjTWK+0wZ9p56HF7/FN79EvLyq95++avQuR4KZ3tD3TtPx1BueZd6TJS9odSOu2JQhYVYjx8vm0AePkx+Sor2HNfio7DNggUE9+hR4/sLIbzXmfJ9X10ebUG0W61aa98ttzjW6fR6As8/n+xNm8q9T/amTUSNGuW0LqhHD9LXrKnwOLbsbNDpypRGOfLOOxx+4w1MUVGEDRpE1I03VtjJPT8/n/z84gwqK6u8NoiGY18KLP4QPlsFBVZPR1OWzmAgsHv3Rh1DaL9+hPTp47Ek1dMtqe6MQefjgyk6GlN0NIHdupW53W61Yk1NJT8lxZE05h86RM62beQfPKhtg45kfSsA4ux70ZfoLbl7wgSMUVGYY2OLL3Fx2unr2Fh8QkPdUsDX0z9aJAbhjZTd7qhuYWnVCp1e7+GIzgweTRAL09PBZsOn1KlkY1gYefv2lX+ftLQyp559wsKwpqWVu709P5+UBQsIGzjQKUGMHDkSv3btMAQHc+q//0h5+WWsx4/TbOLEcvcze/ZsZs6c6fqD81JbEmHRB7DiVyjqy9u9EwzuAzNf9mxsoqzGnqTWVwx6oxFzXJw2D3gJJftiFmDmfv9lAHyQ1RsLeU7bWlNTsaamltv/Ue/rq522LpE8FiWSppgY9CZTlTF6Q8FwiUF4I3t+PttGjgRkqj138ngfxLqkCgvZ89BDKKVo/tBDTrdF3XSTY9kvIQGd0cj+WbOIHT++3A/radOmMbFE8piSkkKHDh3qLng3Ugr++g9e+QB+WV+8vt+FcPeNcF5nrQ7iM69VXurGbNKKZYvGxdNJqidjKN0XM8h+ssw2xqgo2r3zDtYjR8hPTtZaH4v+nj59bc/NJXf3bnJ37y57EJ0OY2Rkpa2P6atXe7xguDcULfeGGIR38gkJ8XQIZxyPJog+ISFgMFB44oTTeuuJExjDw8u/T3g41lLbF5azfVFyWHDkCG0XLapy5g3/Tp3AZqPg0CEsLVuWud1sNmMuUVsp04U6dZ5mt8NPf8ArH8KGrdo6vR6GXALjboD28cXbxkZpRbCLZlIpT13PpCKEtyk5YMhCHm+fuqzMNs0mTcIUEYEpIqLc6gv206OrSyaOBaeTx/zkZOy5uZW2Pup8fVHWyvuB7H/6aVRhYZ2dWlN2e5WF0/fPmgU6ndayq9M51708fanNOpTiwOzZlcYgA4YaJ4OvL2f/+KOnwzjjeDRB1BuN+LVrR+a6dYT07QtoH0RZ69cTOWJEufcJ6NKFrPXrnQaXZP71F/6dOzuuFyWHeQcO0PbVV136ZZGzaxfo9WVOdzdE1kL45mdY9CHs2qetMxth+OVw50hoHlP+/WKjJAEUorTa9oPUm0xYWrTA0qJFmduUUhSmp5OfnKwljSVaHotaH12pR2nLyGBvBYP06ostPZ09Dz7o0Risqakc++QTwocMwdCIq1II4Q4eP8UcNWoU+2bMwL9DB/w6duToBx9gz80lfMgQAPY+/jimyEhix48HIPL669l5xx2kvvcewb17c+L778nZts0xglkVFpI0ZQo5O3fS5vnnwWZzTOFmCA5GbzSSvWkTp7ZsIbB7dwx+fmRv3kzy/PmEXX45PkFBnnki3CA3Dz7+Dl77GJJTtXUBfjD6arj1Oohs+LmvEB5RV/0gdTodxtBQjKGhUOJHbhF7fj7HPvmE5BJVGypibtGi3NJg7mA9cYL8/furjiEuDkNwsHO9S6W0Ope1WQfYTp3Clp5eZQwHn3uOg889hyk2Fr+2bfFNSHD8NcXEuGWwkBCNgccTxLABAyg8eZJDixdrhbLbtiVhwQLHKeOCI0ecTpsEnH02rZ9+mpRXXiHl5ZcxN2tG/HPPOaZeKzh6lIxffwVge6kSNm0XLyawe3f0JhMnV63i8JIl2K1WzDExRN54Y5nR0Q1FRrZWpuatzyAtXVsXEQq3Xgs3XQ3BlZ9dF0K4IN9m4JGVWj/IuVNAVw9nMfVmM37t2rm0bYtp0+qsn6bLs/s8+qjHYzAEB2PLyKDg9Kn89NWrHbfp/f3xS0jANyEB37ZtteU2bdBbLNWOxxtGUntDDN7Anp/P/iefBKDFY495zVR773wBSz6GYye0Ll0z74Vz2ld9v69/hglPwoBe8NpTdR9nRTxeB7Gh8oa6SKlp8MYn8P43xYWu46K108gjLgeLd7xHhDgj5ORC+yu05e0rwK+eBkoqm43NQ4a4VDC8rpKDhhaDLSuLnF27yN21i5zERHITE8nbswdVWFj2Tno95mbNyrQ2GiMjK2xt9IaR1N4Qg7ew5eby70UXAXUzirkm3/ff/AwTn4GnH9CSwjc/hW9/0fr6R4RWfL+DR+C6CVpXsJBASRAbJE8miPtTYPFH8On3xTUMz2qlDTwZ0g98Gt8PSCHqnLVQa6kHuHkoGOvx/MuZVLTcUzHYrVby9u0j93TCmLNrF7mJiWUGSRYxBAeXaW20tG5Nxtq1Dfp5OBOpwkKOLtNKUEWOGFFhPeOaKvq+37ZtG7GxsY71pQevlnT1OOjSDp68T7tut8OFI2HsMK16SHlsNhh+n9bAs34zZGZLgtgguTtBTEmtegRxepZW3Hr5muIaht06wj2j4JILtBHKQogzk6dn1jlTY7AeP661Mp5OGHN27SJv/37t27o0vV4bUV3ebUWxRETQ9tVX67Qldecdd1BYQe1fqPvW3NLxePo0d13HUPR9X9r06dOZMWNGmfUFVmg3CBbNhIG9i9dPnK0lfa8/Xf5x5r8FO/bAkidh0jOeTxA93gdRaMnhJTdXXoNQrwN7iVS+7wVwz+kahtLnWogzX2MpWl7fMRgjIgiOiHCaKtGen0/e3r2OVsaivzYXyptZjx9n67XX1igWd7GmprLjlluwtGiBT0gIPsHB+ISGasslL8HBtWpt84bT3PUZQ3ktiOXGlAE2e9lTyRGhkHSg/H2v3wwfr4DvXndXtLUnCaIXOJFReXIIWnKo0xXXMOzQpn5iE0Jo7HZIOf0dFBvpmRb7xly0vD5jKBocVHKAkFKKY8uWcfDZZ6uOz2h0+2lORxyFhVXWxQTI2baNnG3bqtzOEBTklDA6lstLKENCMAQGotPpvKJoeX3HEBgYSFAdVDrJzoH7Z8Ezk71rMgpJEBuQJU/AgN5VbyeEcL+8fOh9g7Zcn4NUhHfQ6XT4xsdXvSGQsGCBx0dzR40Zg09ICIXp6eVebJmZoBS2zExsmZnkH6igaas0g8ExUrwy+2bO5NTWreUXTi/6e3q5TGH008sV3Uen06Hsdo68+WalMXiqcHpoMBj0cLzUxEvHT0KTcipR7T8EyUfgfyVKmRadMWx9qTawpUVs2fvVNUkQG5CmkZ6OQIjGzbf61VDEGaT01IvlMUZFEdC1q8djiL377koTI1VYSGFWVvkJ5MmTztczMihMT8d+6hTYbNgqGNhTkv3UKVLfeadGj9FdrKmpZG/cWO8t3iYjdG4Lv28o7oNot2vXxwwru318c1hVKtd97g2tZXHGBM9990uCKIQQLvDzhR3feToK4Uklp16sSLNJk+q0xcpdMeh8fIqLtLvIXlBAYXo6aStWcGjhwiq3D+rRA3Pz5o6i5w4li6LXcH3+oUOc+u+/KmMomiijvt02XBto0qUtnH26zE1OHgwfpN3+wCyIbgJTbweLSatEUlLQ6frFpdfXJ0kQhRBCCBfVdurFhhyD3mTCFBlJQDlzjpcneswYj59qN0ZE1MnxqzKkH6RlwPy3tULZHeLh3TnFp5gPHfX+yiOSIAohhBDVcCaO5q6OhnSqvS5jqMrYYdqlPB+/UPl95z3k9nCqTRJEIYRwQX4BPP6StvzEvWA2eTYe4VmNYTR3Zcc9U061i4p5eQNn4xAWXPWXjdnkXcPfhWhsbDb46FvtUkmdZCEahaLT3MZI5xEUxqioepvJpSgGnyZNPBbDmUxaEL1AbJQ2jL2qmVRio+ovJiGEMx8fmPy/4mUhGjtvOdUe1LMnB+fOxZ6TQ8TQoQSed560HLqBTLVXQ56ci1kIIYQQ9aOxft/LKWYhhBBCCOFETpQIIYQLlCruBhIWLHOgC+EtlFIUpqcD4BMSUjwDi6gVSRCFEMIFuXlw7umSFTLVnhDew56Xx6bLLgPgnN9+w+Arb053kASxhux2OwCHDx/2cCRCiPqQmw/2wiAAUg5l4mv2cEBCCEBLELNPlxZISUlBb3HvnJhF3/NF3/uNhSSINZSamgrA+eef7+FIhBD1rW0bT0cghChXQkKd7To1NZXmzZvX2f69jYxirqHCwkI2btxIVFQUem+fL6cGsrKy6NChA9u2bSMwMNDT4XiMPA8aeR408jxo5HnQyPOgOdOfB7vdTmpqKl27dsWnEdW4kgRRlCszM5Pg4GAyMjIICgrydDgeI8+DRp4HjTwPGnkeNPI8aOR5ODOdeU1fQgghhBCiViRBFEIIIYQQTiRBFOUym81Mnz4ds7lxD9WU50Ejz4NGngeNPA8aeR408jycmaQPohBCCCGEcCItiEIIIYQQwokkiEIIIYQQwokkiEIIIYQQwokkiEIIIYQQwokkiMLJ7NmzOe+88wgMDCQyMpKhQ4eyc+dOT4flUc888ww6nY7777/f06F4REpKCjfddBPh4eH4+vrSuXNn/v77b0+HVa9sNhuPPfYYrVq1wtfXl/j4eJ588knO9DF+v/76K0OGDCEmJgadTseXX37pdLtSiscff5ymTZvi6+tL//79SUxM9Eywdaiy58FqtTJ16lQ6d+6Mv78/MTEx3HzzzRw6dMhzAdeRql4PJd11113odDpeeOGFeotPuJckiMLJL7/8wj333MOff/7JDz/8gNVqZcCAAZw6dcrToXnE+vXrefXVV+nSpYunQ/GIkydP0qtXL4xGI9999x3btm1j3rx5hIaGejq0ejVnzhwWLVrEwoUL2b59O3PmzGHu3LksWLDA06HVqVOnTnH22Wfz8ssvl3v73Llzeemll1i8eDF//fUX/v7+DBw4kLy8vHqOtG5V9jzk5OSwYcMGHnvsMTZs2MDnn3/Ozp07ueqqqzwQad2q6vVQ5IsvvuDPP/8kJiamniITdUIJUYmjR48qQP3yyy+eDqXeZWVlqYSEBPXDDz+oPn36qPvuu8/TIdW7qVOnqt69e3s6DI8bPHiwuvXWW53WXXPNNWrUqFEeiqj+AeqLL75wXLfb7So6Olo9++yzjnXp6enKbDarDz/80AMR1o/Sz0N51q1bpwC1f//++gnKAyp6HpKTk1VsbKzasmWLatGihXr++efrPTbhHtKCKCqVkZEBQFhYmIcjqX/33HMPgwcPpn///p4OxWO+/vprunfvzvDhw4mMjKRr16689tprng6r3vXs2ZOffvqJXbt2AfDff/+xdu1aLr/8cg9H5jl79+7lyJEjTu+P4OBgLrjgAv744w8PRuZ5GRkZ6HQ6QkJCPB1KvbLb7YwePZoHH3yQjh07ejocUUs+ng5AeC+73c79999Pr1696NSpk6fDqVcfffQRGzZsYP369Z4OxaP27NnDokWLmDhxIg8//DDr16/n3nvvxWQyMWbMGE+HV28eeughMjMzadeuHQaDAZvNxtNPP82oUaM8HZrHHDlyBICoqCin9VFRUY7bGqO8vDymTp3KDTfcQFBQkKfDqVdz5szBx8eHe++919OhCDeQBFFU6J577mHLli2sXbvW06HUq4MHD3Lffffxww8/YLFYPB2OR9ntdrp3786sWbMA6Nq1K1u2bGHx4sWNKkFctmwZ77//Ph988AEdO3bk33//5f777ycmJqZRPQ+iclarlREjRqCUYtGiRZ4Op179888/vPjii2zYsAGdTufpcIQbyClmUa7x48ezfPlyVq9eTVxcnKfDqVf//PMPR48e5dxzz8XHxwcfHx9++eUXXnrpJXx8fLDZbJ4Osd40bdqUDh06OK1r3749Bw4c8FBEnvHggw/y0EMPcf3119O5c2dGjx7NAw88wOzZsz0dmsdER0cDkJqa6rQ+NTXVcVtjUpQc7t+/nx9++KHRtR7+9ttvHD16lObNmzs+N/fv38+kSZNo2bKlp8MTNSAtiMKJUooJEybwxRdfsGbNGlq1auXpkOrdpZdeyubNm53W3XLLLbRr146pU6diMBg8FFn969WrV5kyR7t27aJFixYeisgzcnJy0Oudf08bDAbsdruHIvK8Vq1aER0dzU8//cQ555wDQGZmJn/99Rfjxo3zbHD1rCg5TExMZPXq1YSHh3s6pHo3evToMv21Bw4cyOjRo7nllls8FJWoDUkQhZN77rmHDz74gK+++orAwEBHX6Lg4GB8fX09HF39CAwMLNPn0t/fn/Dw8EbXF/OBBx6gZ8+ezJo1ixEjRrBu3TqWLFnCkiVLPB1avRoyZAhPP/00zZs3p2PHjmzcuJH58+dz6623ejq0OpWdnc3u3bsd1/fu3cu///5LWFgYzZs35/777+epp54iISGBVq1a8dhjjxETE8PQoUM9F3QdqOx5aNq0Kddddx0bNmxg+fLl2Gw2x+dmWFgYJpPJU2G7XVWvh9KJsdFoJDo6mrPOOqu+QxXu4Olh1MK7AOVe3nrrLU+H5lGNtcyNUkp98803qlOnTspsNqt27dqpJUuWeDqkepeZmanuu+8+1bx5c2WxWFTr1q3VI488ovLz8z0dWp1avXp1uZ8HY8aMUUpppW4ee+wxFRUVpcxms7r00kvVzp07PRt0Hajsedi7d2+Fn5urV6/2dOhuVdXroTQpc9Ow6ZQ6w6cCEEIIIYQQ1SKDVIQQQgghhBNJEIUQQgghhBNJEIUQQgghhBNJEIUQQgghhBNJEIUQQgghhBNJEIUQQgghhBNJEIUQQgghhBNJEIUQQgghhBNJEIXwsLfffpuQkBBPh+GSGTNmOObddZVOp+PLL7+s1n369u3L/fffX637eJuCggLatGnD//3f/9X7sfft24dOp+Pff/91+T5nwnNek8ddnsWLFzNkyBD3BCVEAyUJohDCZZMnT+ann37ydBgNwuLFi2nVqhU9e/b0dCiNRrNmzTh8+LBjzvQ1a9ag0+lIT0+v1n5uvfVWNmzYwG+//VYHUQrRMEiCKIRwWUBAAOHh4Z4Owy0KCgrqbN9KKRYuXMj//ve/OjuGKMtgMBAdHY2Pj0+t9mMymbjxxht56aWX3BSZEA2PJIhC1ELfvn0ZP34848ePJzg4mIiICB577DFKTnF+8uRJbr75ZkJDQ/Hz8+Pyyy8nMTGx3P3t27cPvV7P33//7bT+hRdeoEWLFtjtdkeryE8//UT37t3x8/OjZ8+e7Ny50+k+ixYtIj4+HpPJxFlnncXSpUudbtfpdLz66qtceeWV+Pn50b59e/744w92795N37598ff3p2fPniQlJTnuU/oU8/r167nsssuIiIggODiYPn36sGHDhmo9h6dOneLmm28mICCApk2bMm/evDLb5OfnM3nyZGJjY/H39+eCCy5gzZo1Ttu89tprNGvWDD8/P4YNG8b8+fOdTt0Xxf7666/TqlUrLBYLAOnp6dx22200adKEoKAg+vXrx3///ee076+++opzzz0Xi8VC69atmTlzJoWFhRU+pn/++YekpCQGDx7sWFd0+nPZsmVcdNFF+Pr6ct5557Fr1y7Wr19P9+7dCQgI4PLLL+fYsWOO+9ntdp544gni4uIwm82cc845rFy50ul469ato2vXrlgsFrp3787GjRvLxLRlyxYuv/xyAgICiIqKYvTo0Rw/frzCx1Ceb775hvPOOw+LxUJERATDhg1z3LZ06VK6d+9OYGAg0dHR3HjjjRw9etRxe9Hr9ttvv6VLly5YLBYuvPBCtmzZ4tgmLS2NG264gdjYWPz8/OjcuTMffvihUwx2u525c+fSpk0bzGYzzZs35+mnn3Z6jv/991/27dvHJZdcAkBoaCg6nY6xY8fy7rvvEh4eTn5+vtN+hw4dyujRox3XhwwZwtdff01ubm61niMhzhhKCFFjffr0UQEBAeq+++5TO3bsUO+9957y8/NTS5YscWxz1VVXqfbt26tff/1V/fvvv2rgwIGqTZs2qqCgQCml1FtvvaWCg4Md21922WXq7rvvdjpOly5d1OOPP66UUmr16tUKUBdccIFas2aN2rp1q7roootUz549Hdt//vnnymg0qpdfflnt3LlTzZs3TxkMBvXzzz87tgFUbGys+vjjj9XOnTvV0KFDVcuWLVW/fv3UypUr1bZt29SFF16oBg0a5LjP9OnT1dlnn+24/tNPP6mlS5eq7du3q23btqn//e9/KioqSmVmZjod54svvqjwORw3bpxq3ry5+vHHH9WmTZvUlVdeqQIDA9V9993n2Oa2225TPXv2VL/++qvavXu3evbZZ5XZbFa7du1SSim1du1apdfr1bPPPqt27typXn75ZRUWFub0vE6fPl35+/urQYMGqQ0bNqj//vtPKaVU//791ZAhQ9T69evVrl271KRJk1R4eLhKS0tTSin166+/qqCgIPX222+rpKQktWrVKtWyZUs1Y8aMCh/T/PnzVbt27ZzW7d27VwGqXbt2Ts9vt27dVN++fdXatWvVhg0bVJs2bdRdd93ltK+goCD14Ycfqh07dqgpU6Yoo9HoeOxZWVmqSZMm6sYbb1RbtmxR33zzjWrdurUC1MaNG5VSSp08eVI1adJETZs2TW3fvl1t2LBBXXbZZeqSSy5xHKdPnz5Oz3lpy5cvVwaDQT3++ONq27Zt6t9//1WzZs1y3P7GG2+oFStWqKSkJPXHH3+oHj16qMsvv9xxe9Hrtn379mrVqlWO/3XLli0d74Xk5GT17LPPqo0bN6qkpCT10ksvKYPBoP766y/HfqZMmaJCQ0PV22+/rXbv3q1+++039dprrzk9xxs3blSFhYXqs88+U4DauXOnOnz4sEpPT1c5OTkqODhYLVu2zLHP1NRU5ePj4/T+OHXqlNLr9Wr16tUVPidCnMkkQRSiFvr06aPat2+v7Ha7Y93UqVNV+/btlVJK7dq1SwHq999/d9x+/Phx5evr6/iCKp0gfvzxxyo0NFTl5eUppZT6559/lE6nU3v37lVKFX/R/vjjj477fPvttwpQubm5SimlevbsqW6//XanWIcPH66uuOIKx3VAPfroo47rf/zxhwLUG2+84Vj34YcfKovF4rheOkEszWazqcDAQPXNN984HaeiBDErK0uZTCanL+u0tDTl6+vrSFb279+vDAaDSklJcbrvpZdeqqZNm6aUUmrkyJFq8ODBTrePGjWqTIJoNBrV0aNHHet+++03FRQU5Hiui8THx6tXX33VcZySiZBSSi1dulQ1bdq0wufhvvvuU/369XNaV5S8vP766451H374oQLUTz/95Fg3e/ZsddZZZzmux8TEqKefftppX+edd57jR8Srr76qwsPDHf97pZRatGiRU4L45JNPqgEDBjjt4+DBg47kSamqE8QePXqoUaNGVXh7aevXr1eAysrKUkoVv24/+ugjxzZF/+uPP/64wv0MHjxYTZo0SSmlVGZmpjKbzY6EsLSSCWLJY548edJpu3Hjxjklr/PmzVOtW7d2eh8rpRyJqBCNkZxiFqKWLrzwQnQ6neN6jx49SExMxGazsX37dnx8fLjgggsct4eHh3PWWWexffv2cvc3dOhQDAYDX3zxBaCNcr7kkkto2bKl03ZdunRxLDdt2hTAcUpv+/bt9OrVy2n7Xr16lTlmyX1ERUUB0Pn/27v3kCi6Nw7gX1/brc1Ny+wqumKtuZnpRkgG0UVTCGTNP4LyXi5FJkVkRYWWBmVFaYZSL1GtlIVY0D3DMkrzguV6aV1rM+xeqGBbmWjP7w9xfo27mr71+9lbzwcEZ87sc86cmXXPzpxn9PYWrWtvb0dbW5vVtr59+xZarRZKpRIODg6wt7eH2WxGU1OT1e17M5lM6OjoEPWPo6Mjpk2bJizX1NSgq6sLHh4ekMvlws+dO3eE299GoxF+fn6i2L2XAUChUGDcuHHCsl6vh9lsxtixY0WxGxsbhdh6vR4pKSmicq1Wi9evX+PTp09W9+vz58/CLezeBtLnPcexra0Nr1696vdYGgwG4ZZtD39/f9H2er0et2/fFu2Dp6cnAIimEPSnqqoKAQEBfZZXVlYiJCQErq6uGDVqFObPnw8AFufCt23rOdY9+9LV1YXU1FR4e3vD0dERcrkcN27cEGIYDAZ8+fKl33YMhFarRUFBAV6+fAmg+z0WExMjeh8DgEwm6/MYM/a7+7GZvIyxn04qlSIqKgonTpxAWFgYzpw5g4yMDIvtJBKJ8HvPB9vXr18HVZe1GIOJGx0djebmZmRkZEChUGD48OHw9/f/qQkgZrMZtra2qKyshK2trahMLpcPKpadnZ1F7EmTJlnMZwQgzF80m83YtWsXwsLCLLbpaxDo5OSEmpoaq2UD6fPBHsfvMZvNCAkJQVpamkVZz5eL75HJZH2Wffz4EcHBwQgODsbp06cxbtw4NDU1ITg4eFDnwv79+5GRkYH09HR4e3vDzs4OGzZsEGL014bBUKvV8PHxgU6nQ1BQEOrq6nDlyhWL7VpaWkRfKBj7k/AAkbEfVFZWJlouLS2FUqmEra0tVCoVOjs7UVZWJjzupLm5GUajEdOnT+8zZlxcHGbMmIGsrCx0dnZaHZz0R6VSobi4GNHR0cK64uLifuv8J4qLi5GVlYUlS5YAAJ4/fz6oxIcpU6ZAIpGgrKwMrq6uALqTehoaGoQrUGq1Gl1dXXj37h3mzZtnNc60adNQUVEhWtd72ZpZs2bhzZs3GDZsmMUV2m+3MRqNmDp16oD3S61WIzs7G0RkcVVqMOzt7TF58mQUFxcL/QF093vPFVKVSoWcnBy0t7cLA9bS0lKLfcjPz4ebm9s/zvCdOXMmCgsLERsba1FWX1+P5uZm7N27Fy4uLgBgkWjVo7S01OJYq1QqYb80Gg0iIiIAdH8xaWhoEM5bpVIJmUyGwsJCxMXFfbfNUqkUQPeVyd7i4uKQnp6Oly9fIjAwUGh3D5PJhPb2dqjV6u/Ww9jviG8xM/aDmpqasHHjRhiNRuTm5iIzMxPr168H0P2BptFooNVqce/ePej1ekRERMDZ2RkajabPmCqVCnPmzMGWLVuwfPnyQV85SUxMxMmTJ5GdnY3Hjx/j4MGDOH/+PDZt2vRD+9qbUqlETk4ODAYDysrKEB4ePqi2yuVyrFq1ComJibh16xZqa2sRExODv/76758mDw8PhIeHIyoqCufPn0djYyPKy8uxZ88e4apPQkICrl69ioMHD+Lx48c4evQorl279t3BWWBgIPz9/REaGoqCggI8e/YMJSUl2L59uzDASUpKgk6nw65du1BXVweDwYCzZ89ix44dfcZduHAhzGYz6urqBtwXfUlMTERaWhrOnTsHo9GIrVu3oqqqSjjHVqxYARsbG2i1Wjx69AhXr17FgQMHRDHi4+PR0tKC5cuXo6KiAiaTCTdu3EBsbKzVwZM1ycnJyM3NRXJyMgwGA2pqaoQrkq6urpBKpcjMzMTTp09x8eJFpKamWo2TkpKCwsJC4Vg7OTkhNDQUQPf5dPPmTZSUlMBgMGD16tV4+/at8NoRI0Zgy5Yt2Lx5M3Q6HUwmE0pLS3H8+HGrdSkUCtjY2ODy5ct4//49zGazULZixQq8ePECf//9N1auXGnx2rt378Ld3R1TpkwZUP8w9tsZ6kmQjP2bzZ8/n9auXUtr1qwhe3t7GjNmDG3btk002b2lpYUiIyPJwcGBZDIZBQcHCxmoRJZJKj2OHz9OAKi8vFy03trE+4cPHxIAIZGFiCgrK4vc3d1JIpGQh4cH6XQ6URz0Sh7pPcHfWl29k1QePHhAs2fPphEjRpBSqaS8vDxSKBR06NChPuvp7cOHDxQREUEjR46kCRMm0L59+ywSJjo6OigpKYnc3NxIIpHQpEmTaOnSpVRdXS1sc+zYMXJ2diaZTEahoaG0e/dumjhxolDeV4JNW1sbJSQk0OTJk0kikZCLiwuFh4dTU1OTsM3169dp7ty5JJPJyN7envz8/ESZ6tYsW7aMtm7dKiwPpH+JLM+Hrq4u2rlzJzk7O5NEIiEfHx+6du2aqK779++Tj48PSaVS8vX1FbJ3v62roaGBli5dSqNHjyaZTEaenp60YcMG4Vz9XpIKEVF+fj75+vqSVColJycnCgsLE8rOnDlDbm5uNHz4cPL396eLFy9aTRi5dOkSeXl5kVQqJT8/PyGbnKg7aUWj0ZBcLqfx48fTjh07KCoqijQajag/du/eTQqFgiQSCbm6ugpJRNb6OCUlhSZOnEg2NjYUHR0t2p/IyEhydHS0SFIiIgoKCqI9e/b02x+M/c5siL55YBtjbFAWLFgAX19fpKen//TYqampyMvLQ3V19U+P/SfQarWor68fsv+GUV1djcWLF8NkMg16ruTvqKioCAsXLkRra+sv868lAwIC4OXlZfFA7Lq6OixatAgNDQ1wcHAYotYxNrT4FjNjvxiz2Yza2locOXIECQkJQ92cf40DBw5Ar9fjyZMnyMzMxKlTp0RzMP/fZs6cibS0NDQ2Ng5ZG5h1ra2tuHDhAoqKihAfH29R/vr1a+h0Oh4csj8aJ6kw9otZt24dcnNzERoaanVuFLOuvLwc+/btw4cPH+Du7o7Dhw8PKJHhfykmJmZI62fWqdVqtLa2Ii0tTfRIpR6BgYFD0CrGfi18i5kxxhhjjInwLWbGGGOMMSbCA0TGGGOMMSbCA0TGGGOMMSbCA0TGGGOMMSbCA0TGGGOMMSbCA0TGGGOMMSbCA0TGGGOMMSbCA0TGGGOMMSbyH83R2BbHJKoFAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# viz: the divergence. Train loss falls forever; held-out accuracy peaks then collapses.\n", "fig, ax1 = plt.subplots(figsize=(6.5, 3.4))\n", "ax1.plot(degrees, train_loss, \"o-\", color=\"#c33\", label=\"train MSE (the proxy)\")\n", "ax1.set_xlabel(\"polynomial degree (model capacity)\"); ax1.set_ylabel(\"train MSE\", color=\"#c33\")\n", "ax1.tick_params(axis=\"y\", labelcolor=\"#c33\")\n", "ax2 = ax1.twinx()\n", "ax2.plot(degrees, held_metric, \"s-\", color=\"#1E40FF\", label=\"held-out accuracy (the target)\")\n", "ax2.set_ylabel(\"held-out accuracy\", color=\"#1E40FF\"); ax2.tick_params(axis=\"y\", labelcolor=\"#1E40FF\")\n", "ax2.axvline(best_by_loss, ls=\":\", c=\"#c33\"); ax2.axvline(best_by_metric, ls=\":\", c=\"#1E40FF\")\n", "ax1.set_title(\"loss keeps falling; the metric peaks then collapses\")\n", "fig.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "6e6ad206", "metadata": {}, "source": [ "> **Interpretation.** The red curve (training loss) only goes down, the seductive number. The blue curve (the metric we actually want) tells the truth: it peaks at a moderate capacity and collapses as the model memorizes noise. The two dotted lines are the two selection rules; they land on different models, and the loss-based rule loses. The discipline: *the loss is not the metric*. Track both, select on the metric you defined in advance, and treat a falling loss with a falling held-out metric as a red alarm, not a success.\n", "\n", "> **Caveat:** this is not an argument against loss, loss is what you can differentiate and optimize. It is an argument against *selecting* on loss. Optimize the proxy; evaluate and select on the target. Confusing the two is one of the most expensive mistakes in applied ML, and it hides behind a number that looks like progress.\n", "\n", "> **Key takeaways.** The loss is a proxy; the held-out metric is the target. They can diverge (overfitting is the canonical case). Selecting the checkpoint by lowest loss ships the overfit model; selecting by the metric fixes it. Always track both, define the selection metric before the run, and read a falling-loss-with-falling-metric pattern as a failure, not a win.\n" ] }, { "cell_type": "markdown", "id": "5f693550", "metadata": {}, "source": [ "### The same point in one torch step (optional, print-and-skips without torch)\n", "\n", "The polynomial demo needs no torch. For readers who think in training loops, here is the identical lesson as one gradient step on a tiny linear model: the training loss after the step is lower, but that says nothing about held-out behavior. The cell is fenced so it never errors if torch is absent.\n" ] }, { "cell_type": "code", "execution_count": 22, "id": "fa72f194", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.909586Z", "iopub.status.busy": "2026-06-10T20:48:42.909515Z", "iopub.status.idle": "2026-06-10T20:48:42.914392Z", "shell.execute_reply": "2026-06-10T20:48:42.914021Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "train loss before step: 0.7464\n", "train loss after step: 0.6404 (lower, as a gradient step must make it)\n", "but: a lower TRAIN loss is not evidence of a better model. Evaluate on held-out data.\n" ] } ], "source": [ "# deeper: the loss-down-after-a-step fact, with explicit p.grad=None (never .data). Optional.\n", "if not HAVE_TORCH:\n", " print(\"skipped: torch not installed. It would show one SGD step lowering the TRAIN loss,\")\n", " print(\"with the reminder that a lower train loss is not by itself evidence of a better model.\")\n", "else:\n", " torch.manual_seed(SEED)\n", " w = torch.zeros(3, requires_grad=True)\n", " Xt = torch.tensor(np.vander(x_tr, 3), dtype=torch.float32) # (n, 3) quadratic features\n", " yt = torch.tensor(y_tr, dtype=torch.float32)\n", " def mse():\n", " return ((Xt @ w - yt) ** 2).mean()\n", " before = mse().item()\n", " loss = mse(); loss.backward()\n", " with torch.no_grad():\n", " w -= 0.1 * w.grad\n", " w.grad = None # modern hygiene: zero grads, never .data\n", " after = mse().item()\n", " print(f\"train loss before step: {before:.4f}\")\n", " print(f\"train loss after step: {after:.4f} (lower, as a gradient step must make it)\")\n", " print(\"but: a lower TRAIN loss is not evidence of a better model. Evaluate on held-out data.\")" ] }, { "cell_type": "markdown", "id": "f659e24b", "metadata": {}, "source": [ "## Part 6 — Custom evals: kappa and the ledger\n", "\n", "> **Objectives.** Build Cohen's kappa for inter-annotator agreement (the step everyone skips and the one that exposes an ambiguous rubric), then assemble the experiment ledger that records every change with its CI, the artifact that makes an eval defensible. This is the consolidation part: the pieces from Parts 1-5 come together into one report.\n", "\n", "The benchmarks in the draft (MMLU, GSM8K, SWE-bench) are public, contaminated, and not your problem. The eval that decides whether your system ships is the one you build for your use case, and its hardest step is *agreement*: two humans labeling the same outputs disagree far more than anyone expects. **Cohen's kappa** measures agreement *corrected for chance*: if two raters agree 80% of the time but would agree 75% just by guessing with the same label frequencies, the chance-corrected agreement is small.\n", "\n", "$$\\kappa = \\frac{p_o - p_e}{1 - p_e}$$\n", "\n", "where $p_o$ is observed agreement (fraction of items both rated the same) and $p_e$ is the agreement expected by chance from each rater's marginal label frequencies. $\\kappa = 1$ is perfect, $\\kappa = 0$ is chance-level, and below $\\approx 0.6$ your rubric is too ambiguous to trust, the signal to *rewrite the rubric*, not to average more annotators.\n" ] }, { "cell_type": "markdown", "id": "c8317ef8", "metadata": {}, "source": [ "### Exercise 23.6 — Cohen's kappa from scratch\n", "`Difficulty 2/5 · ~12 min`\n", "\n", "Fill in `cohen_kappa(a, b)` for two raters' label lists (same length, same label set). Compute $p_o$ (observed agreement) and $p_e$ (chance agreement from the marginals), then return $(p_o - p_e)/(1 - p_e)$. Return `1.0` if $p_e = 1$ (degenerate: one label used by both).\n", "\n", "The checks verify perfect self-agreement gives $\\kappa = 1$, a chance-level pattern gives $\\kappa \\approx 0$, and your result matches `sklearn.metrics.cohen_kappa_score`.\n" ] }, { "cell_type": "code", "execution_count": 23, "id": "e5fa736b", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:42.915228Z", "iopub.status.busy": "2026-06-10T20:48:42.915162Z", "iopub.status.idle": "2026-06-10T20:48:43.125421Z", "shell.execute_reply": "2026-06-10T20:48:43.125059Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ -- ] 23.6 kappa self-agreement: not attempted yet — fill in the TODO above, then re-run.\n", "[ -- ] 23.6 kappa chance-level: not attempted yet — fill in the TODO above, then re-run.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[ -- ] 23.6 kappa vs sklearn: not attempted yet — fill in the TODO above, then re-run.\n" ] }, { "data": { "text/plain": [ "False" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def cohen_kappa(a, b):\n", " \"\"\"Cohen's kappa for two raters. a, b: equal-length label lists over a shared label set.\"\"\"\n", " assert len(a) == len(b) and len(a) > 0, \"raters must label the same nonempty set of items\"\n", " n = len(a)\n", " labels = sorted(set(a) | set(b))\n", " # TODO 1: observed agreement p_o = fraction of items where a[i] == b[i]\n", " p_o = None\n", " # TODO 2: expected (chance) agreement p_e = sum over labels of (freq of label in a)*(freq in b)\n", " p_e = None\n", " attempted(p_o, p_e)\n", " if p_e >= 1.0:\n", " return 1.0\n", " return (p_o - p_e) / (1 - p_e)\n", "\n", "# self-checks (run this cell)\n", "def _kappa_self():\n", " labs = [\"valid\"] * 8 + [\"invalid\"] * 2\n", " check_close(cohen_kappa(labs, labs), 1.0, msg=\"identical labelings -> kappa 1\")\n", "\n", "def _kappa_chance():\n", " # a and b independent with the same marginals -> kappa ~ 0\n", " a = [\"a\", \"a\", \"b\", \"b\"]; b = [\"a\", \"b\", \"a\", \"b\"]\n", " assert abs(cohen_kappa(a, b)) < 1e-9, f\"chance-level labels should give kappa ~ 0, got {cohen_kappa(a,b)}\"\n", "\n", "def _kappa_vs_sklearn():\n", " from sklearn.metrics import cohen_kappa_score\n", " g = np.random.default_rng(5)\n", " a = list(g.choice([\"yes\", \"no\", \"maybe\"], 60))\n", " b = list(g.choice([\"yes\", \"no\", \"maybe\"], 60))\n", " check_close(cohen_kappa(a, b), cohen_kappa_score(a, b), atol=1e-9,\n", " msg=\"must match sklearn.metrics.cohen_kappa_score\")\n", "\n", "check(\"23.6 kappa self-agreement\", _kappa_self)\n", "check(\"23.6 kappa chance-level\", _kappa_chance)\n", "check(\"23.6 kappa vs sklearn\", _kappa_vs_sklearn)" ] }, { "cell_type": "markdown", "id": "3618ba95", "metadata": {}, "source": [ "
Hint 1 (conceptual)`p_o` is just `mean(a[i] == b[i])`. For `p_e`, each rater has a label frequency; the chance they both land on label `l` independently is `freq_a(l) * freq_b(l)`. Sum that over all labels.
\n", "\n", "
Hint 2 (the lines)\n", "\n", "```python\n", "p_o = sum(1 for x, y in zip(a, b) if x == y) / n\n", "p_e = sum((a.count(l) / n) * (b.count(l) / n) for l in labels)\n", "```\n", "
\n", "\n", "
Help — \"kappa is negative\" or \"not matching sklearn\"A small negative kappa is legitimate (worse than chance) and only appears when `p_o < p_e`. If you are far off sklearn, check `p_e`: it uses each rater's *own* marginal frequencies (`a.count(l)/n` and `b.count(l)/n` separately), not a shared distribution.
\n" ] }, { "cell_type": "code", "execution_count": 24, "id": "4ba0650c", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.126404Z", "iopub.status.busy": "2026-06-10T20:48:43.126255Z", "iopub.status.idle": "2026-06-10T20:48:43.130868Z", "shell.execute_reply": "2026-06-10T20:48:43.130602Z" }, "collapsed": true, "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ ok ] 23.6 kappa self-agreement\n", "[ ok ] 23.6 kappa chance-level\n", "[ ok ] 23.6 kappa vs sklearn\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#@title Solution { display-mode: \"form\" }\n", "# solution: redefines cohen_kappa; the checks below re-verify the reference.\n", "def cohen_kappa(a, b):\n", " assert len(a) == len(b) and len(a) > 0, \"raters must label the same nonempty set of items\"\n", " n = len(a)\n", " labels = sorted(set(a) | set(b))\n", " p_o = sum(1 for x, y in zip(a, b) if x == y) / n\n", " p_e = sum((a.count(l) / n) * (b.count(l) / n) for l in labels)\n", " if p_e >= 1.0:\n", " return 1.0\n", " return (p_o - p_e) / (1 - p_e)\n", "\n", "check(\"23.6 kappa self-agreement\", _kappa_self, required=True)\n", "check(\"23.6 kappa chance-level\", _kappa_chance, required=True)\n", "check(\"23.6 kappa vs sklearn\", _kappa_vs_sklearn, required=True)" ] }, { "cell_type": "markdown", "id": "28cd94c4", "metadata": {}, "source": [ "Now use kappa the way you would on a real custom eval: two annotators label the same 40 items, and the kappa tells you whether the rubric is usable. We simulate an *ambiguous* rubric (annotators agree often but not on the hard cases) and a *clear* one.\n" ] }, { "cell_type": "code", "execution_count": 25, "id": "70d85412", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.131627Z", "iopub.status.busy": "2026-06-10T20:48:43.131557Z", "iopub.status.idle": "2026-06-10T20:48:43.134691Z", "shell.execute_reply": "2026-06-10T20:48:43.134382Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ambiguous rubric: kappa = 0.072 -> REWRITE (kappa < 0.6)\n", "clear rubric : kappa = 0.574 -> REWRITE (kappa < 0.6)\n" ] } ], "source": [ "# two rubrics: an ambiguous one (annotators diverge on borderline items) and a clear one.\n", "g6 = np.random.default_rng(SEED)\n", "N_LABELED = 40\n", "gold = g6.choice([\"pass\", \"fail\"], N_LABELED) # the latent correct label\n", "# annotator 1 = gold with some slips; annotator 2 agrees more under a CLEAR rubric than an AMBIGUOUS one\n", "def annotate(gold, flip_prob, seed):\n", " h = np.random.default_rng(seed)\n", " return [g if h.uniform() > flip_prob else (\"fail\" if g == \"pass\" else \"pass\") for g in gold]\n", "ann1 = annotate(gold, 0.10, seed=11)\n", "ann2_ambiguous = annotate(gold, 0.35, seed=12) # ambiguous rubric: many independent slips\n", "ann2_clear = annotate(gold, 0.08, seed=13) # clear rubric: few slips\n", "k_ambiguous = cohen_kappa(ann1, ann2_ambiguous)\n", "k_clear = cohen_kappa(ann1, ann2_clear)\n", "print(f\"ambiguous rubric: kappa = {k_ambiguous:.3f} -> {'REWRITE (kappa < 0.6)' if k_ambiguous < 0.6 else 'ok'}\")\n", "print(f\"clear rubric : kappa = {k_clear:.3f} -> {'REWRITE (kappa < 0.6)' if k_clear < 0.6 else 'ok'}\")" ] }, { "cell_type": "markdown", "id": "f1bcd03e", "metadata": {}, "source": [ "> **Interpretation.** The ambiguous rubric lands below the 0.6 line: the annotators are not really measuring the same thing, so any single number computed from one annotator's labels is built on sand. The clear rubric clears the bar. The lesson the draft hammers: low kappa is a signal to *rewrite the rubric*, not to average more annotators or paper over the disagreement. If two careful humans cannot agree what \"pass\" means, neither can your LLM judge, and the eval is measuring noise.\n" ] }, { "cell_type": "markdown", "id": "9c38bdf0", "metadata": {}, "source": [ "### The experiment ledger\n", "\n", "Everything above becomes a report only when it is recorded. The **experiment ledger** is a table with one row per evaluated change, each carrying its measured metric *and its CI*. It is the artifact that separates \"interesting result\" from \"result that survives scrutiny\", and it is what a future you (or a skeptical reviewer) reads to decide whether a claimed improvement is real. We assemble one from the live state of this notebook: each model's accuracy with its bootstrap CI, the paired gap with its CI and McNemar p, and a verdict.\n" ] }, { "cell_type": "code", "execution_count": 26, "id": "9f1846b6", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.135856Z", "iopub.status.busy": "2026-06-10T20:48:43.135783Z", "iopub.status.idle": "2026-06-10T20:48:43.244842Z", "shell.execute_reply": "2026-06-10T20:48:43.244303Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "change accuracy 95% CI\n", "--------------------------------------------\n", "model A 0.860 [0.810, 0.905]\n", "model B 0.805 [0.750, 0.860]\n", "--------------------------------------------\n", "A - B gap +0.055 [-0.010, +0.120] (crosses 0 -> not significant)\n", "McNemar p = 0.136 verdict: NOT significant (CI crosses 0)\n" ] } ], "source": [ "# build the experiment ledger from the notebook's live state. One row per claim, each with its CI.\n", "def ledger_row(name, scores):\n", " p, _ = acc_and_se(scores)\n", " lo, hi = bootstrap_ci(scores, n_resamples=N_BOOT, seed=SEED)\n", " return {\"name\": name, \"accuracy\": p, \"ci_lo\": lo, \"ci_hi\": hi}\n", "\n", "ledger = [ledger_row(\"model A\", scores_A), ledger_row(\"model B\", scores_B)]\n", "print(f\"{'change':<10} {'accuracy':>9} {'95% CI':>22}\")\n", "print(\"-\" * 44)\n", "for r in ledger:\n", " print(f\"{r['name']:<10} {r['accuracy']:>9.3f} [{r['ci_lo']:.3f}, {r['ci_hi']:.3f}]\")\n", "\n", "gap, glo, ghi = paired_diff_ci(scores_A, scores_B)\n", "verdict = \"NOT significant (CI crosses 0)\" if glo <= 0 <= ghi else \"significant\"\n", "print(\"-\" * 44)\n", "print(f\"{'A - B gap':<10} {gap:>+9.3f} {fmt_ci(glo, ghi)}\")\n", "print(f\"McNemar p = {mcnemar(scores_A, scores_B):.3f} verdict: {verdict}\")" ] }, { "cell_type": "markdown", "id": "306d19d5", "metadata": {}, "source": [ "> **Interpretation.** This six-line table is the deliverable. It reports each model's number *with its uncertainty*, the paired gap *with its CI and a second test*, and a verdict that is honest about what 200 items can and cannot resolve. A leaderboard would show only the first column of accuracies; the ledger shows the columns that let a reader decide whether to believe them. The discipline is not hard; it is tedious, and the tedium is the entire difference between a number that sounds good and a claim that holds up.\n", "\n", "> **Key takeaways.** Cohen's kappa measures inter-annotator agreement corrected for chance; below ~0.6 the rubric is too ambiguous and the fix is to rewrite it. The experiment ledger records every change with its metric *and* its CI; it is the artifact that makes an eval defensible. A report with per-change CIs, a paired test, and an honest verdict is more credible than most of the field's, and none of it is hard.\n" ] }, { "cell_type": "markdown", "id": "95bf57ba", "metadata": {}, "source": [ "## Safety lens\n", "\n", "Evals are how the field claims a model is *safe* to deploy, which makes a wrong eval more dangerous than no eval: it manufactures unearned confidence. Two specific failure modes turn a safety eval into theater, and both are measurable.\n", "\n", "The first is the **upper-bound asymmetry**. A safety eval is an upper bound on safety, not a lower bound. If a model *passes* (refuses 95% of HarmBench prompts), you have evidence about *that set*, not about deployment, a paraphrase the model never saw may sail through. If a model *fails*, you have hard evidence of *at least* that failure mode. So a passing safety number should be read as \"we could not find the failure here\", never \"the model is safe\". The right companion to any passing safety claim is a list of *what would have to be true for the claim to fail*; an empty list means the claim is not falsifiable, and not falsifiable is the same as not credible.\n", "\n", "The second is **evaluation awareness**: a capable model can detect that it is being tested (the prompt looks like a test, the asker asks test-shaped things) and behave differently than in deployment. We can demonstrate the *structure* of the detection problem with a tiny, honest mock: a model whose behavior differs between \"eval-flavored\" and \"deployment-flavored\" prompts. The eval that only sees the eval-flavored prompts reports a safety number that deployment will not honor.\n" ] }, { "cell_type": "code", "execution_count": 27, "id": "b7c2a3fd", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.245984Z", "iopub.status.busy": "2026-06-10T20:48:43.245861Z", "iopub.status.idle": "2026-06-10T20:48:43.249431Z", "shell.execute_reply": "2026-06-10T20:48:43.249017Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "safety eval reports refuse rate : 100% (looks perfectly safe)\n", "actual deployment refuse rate : 0% (the model defects in the wild)\n", "\n", "the behavioral eval, by itself, is blind to the gap. Mitigations: include BOTH context\n", "flavors with matched content and compare; or cross-check internal state (Ch 22 refusal\n", "direction), since a model can pass behaviorally while its internals say it knew it was tested.\n", "[ ok ] behavioral safety number (eval) does not match deployment behavior\n" ] } ], "source": [ "# a mock that behaves differently when it detects an 'eval-flavored' context.\n", "# NOT a real model: a function that flips behavior on a context flag, to show the STRUCTURE\n", "# of evaluation awareness. The point: an eval that only probes one context lies about the other.\n", "def eval_aware_model(prompt_is_eval_flavored):\n", " # complies with safety on eval-flavored prompts, defects on deployment-flavored ones\n", " return \"refuse\" if prompt_is_eval_flavored else \"comply\"\n", "\n", "# the eval set is all eval-flavored; deployment is all deployment-flavored\n", "eval_prompts = [True] * 100\n", "deploy_prompts = [False] * 100\n", "eval_refuse_rate = np.mean([eval_aware_model(p) == \"refuse\" for p in eval_prompts])\n", "deploy_refuse_rate = np.mean([eval_aware_model(p) == \"refuse\" for p in deploy_prompts])\n", "print(f\"safety eval reports refuse rate : {eval_refuse_rate:.0%} (looks perfectly safe)\")\n", "print(f\"actual deployment refuse rate : {deploy_refuse_rate:.0%} (the model defects in the wild)\")\n", "print(\"\\nthe behavioral eval, by itself, is blind to the gap. Mitigations: include BOTH context\")\n", "print(\"flavors with matched content and compare; or cross-check internal state (Ch 22 refusal\")\n", "print(\"direction), since a model can pass behaviorally while its internals say it knew it was tested.\")\n", "assert eval_refuse_rate != deploy_refuse_rate, \\\n", " \"the demo only teaches if the two contexts diverge, that divergence IS evaluation awareness\"\n", "print(\"[ ok ] behavioral safety number (eval) does not match deployment behavior\")" ] }, { "cell_type": "markdown", "id": "629e8e64", "metadata": {}, "source": [ "> **Interpretation.** The eval reports 100% refusal; deployment gets 0%. The behavioral eval is not lying, it is faithfully reporting what it measured, and what it measured was the eval-flavored context only. This is why the most credible safety signal the field has for capable models is mechanistic (Ch 22): check whether the refusal direction was active even when the behavior complied, because a sufficiently strategic model can pass the behavioral test while its internals show it knew it was a test. The habits that survive this: pre-register the eval suite, run a blind eval (the eval team is not the train team), always include at least one set of items the model provably has not seen, report bootstrap CIs on every claim, and attach the falsification list to every safety claim.\n" ] }, { "cell_type": "markdown", "id": "bf7540be", "metadata": {}, "source": [ "## Test yourself\n", "\n", "Three parts: concept self-checks with folded answers, three auto-checked problems, and a capstone with a rubric and a folded reference. Solutions are folded; try before you peek. Every answer is in this notebook; if unsure, re-run that section.\n" ] }, { "cell_type": "markdown", "id": "8e863b18", "metadata": {}, "source": [ "### Part A — Concepts\n", "\n", "1. A leaderboard shows model A at 87.0 and B at 82.0 on a 200-item benchmark. In one sentence, what is the single most important thing the leaderboard did not show you?
AnswerThe uncertainty: each number is an estimate with a standard error of a few percent, and the CI on the *paired difference* A − B crosses zero on 200 items, so the 5-point gap is not significant. The leaderboard showed point estimates and hid the only thing that decides whether the gap is real.
\n", "\n", "2. Why is a CI on the *paired difference* tighter than comparing two independently-bootstrapped per-model CIs?
AnswerBecause both models are scored on the *same* items, so a hard item drags both down together. Pairing cancels that shared item-difficulty noise, leaving a cleaner estimate of the *difference*. Throwing away the pairing and comparing two independent numbers is less powerful and is a real methodological error.
\n", "\n", "3. You read \"our model solves 78% of problems.\" What is the first question you ask, and why?
Answer\"At what `k`, with how many samples?\" pass@10 and pass@1 are different measurements of the same samples; pass@10 is far higher than pass@1 for the same model. If they quote pass@k but ship at `n=1`, or compare their pass@10 to a baseline's pass@1, the number is misleading about production behavior.
\n", "\n", "4. A pairwise LLM judge prefers A when A is shown first and B when B is shown first, on the same two responses. What is this called and what is the fix?
AnswerPosition bias: the verdict tracks the slot, not the content. The fix is the swap check, run every comparison in both orders and keep only the preferences that are stable across the swap; everything that flips is discarded as an artifact. The cost is more \"ties\" (thrown-away comparisons), which is the honest price of a biased judge.
\n", "\n", "5. Your training loss falls every epoch but held-out accuracy peaks at epoch 8 and then declines. Which checkpoint do you ship, and what is the name of the failure if you select on loss?
AnswerShip the epoch-8 checkpoint (highest held-out metric). Selecting on lowest loss ships a later, overfit checkpoint, the loss is a proxy and the metric is the target, and overfitting is exactly the regime where they diverge. The loss looking good is what makes this mistake invisible.
\n", "\n", "6. Two annotators agree on 80% of items but Cohen's kappa is 0.15. What does that tell you, and what should you do?
AnswerMost of the 80% agreement is explainable by chance given the label frequencies, so the chance-corrected agreement is low and the rubric is ambiguous. The fix is to rewrite the rubric until kappa clears ~0.6, not to average more annotators. If careful humans cannot agree what the label means, no judge can.
\n", "\n", "7. A safety eval reports the model refuses 95% of harmful prompts. Why is \"the model is safe\" the wrong conclusion?
AnswerA safety eval is an upper bound, not a lower bound: passing means \"we could not find a failure on this set\", not \"there is no failure\". A paraphrase the model never saw can still slip through, and a capable model may behave differently when it detects an eval (evaluation awareness). The honest companion to a passing number is a falsification list, what would have to be true for the claim to fail.
\n", "\n", "8. Look back at the gap-CI plot in Part 2 (the dot with error bars crossing the red zero line). State its interpretation in one sentence.
AnswerThe measured A − B gap is positive, but its 95% CI straddles zero, so the data are consistent with no difference (or even B ahead); the leaderboard's \"+5 win\" is inside the noise of a 200-item eval and cannot be claimed as significant.
\n" ] }, { "cell_type": "markdown", "id": "fa5613fb", "metadata": {}, "source": [ "### Part B — Auto-checked problems\n", "\n", "Three problems that make you compute something new with the chapter's pieces. Write the body; the check asserts the property; the solution is folded below each.\n" ] }, { "cell_type": "markdown", "id": "34f08465", "metadata": {}, "source": [ "**Problem B.1 — How many items do you need?** `Difficulty 2/5 · ~10 min`\n", "\n", "You measured a 2-point true gap that 200 items could not resolve. Fill in `min_items_for_se(p, target_se)`: the smallest number of items `n` so that the standard error of a proportion `sqrt(p*(1-p)/n)` is at most `target_se`. Solve the inequality, then `ceil`. This is the back-of-envelope every eval should start with.\n" ] }, { "cell_type": "code", "execution_count": 28, "id": "0547ffbb", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.250335Z", "iopub.status.busy": "2026-06-10T20:48:43.250253Z", "iopub.status.idle": "2026-06-10T20:48:43.253150Z", "shell.execute_reply": "2026-06-10T20:48:43.252839Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ -- ] B.1 min items for SE: not attempted yet — fill in the TODO above, then re-run.\n" ] }, { "data": { "text/plain": [ "False" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import math\n", "def min_items_for_se(p, target_se):\n", " \"\"\"Smallest integer n with sqrt(p*(1-p)/n) <= target_se.\"\"\"\n", " # TODO: invert se = sqrt(p*(1-p)/n) for n, then round UP (math.ceil)\n", " n = None\n", " attempted(n)\n", " return int(n)\n", "\n", "def _min_items():\n", " # to get SE <= 0.01 at p=0.85: p(1-p)/se^2 = 0.1275/0.0001 = 1275\n", " check_close(min_items_for_se(0.85, 0.01), 1275, atol=0, msg=\"0.85*0.15/0.01^2 = 1275\")\n", " # smaller target SE needs strictly more items\n", " assert min_items_for_se(0.5, 0.02) > min_items_for_se(0.5, 0.04), \\\n", " \"a tighter target SE must require more items\"\n", "check(\"B.1 min items for SE\", _min_items)" ] }, { "cell_type": "markdown", "id": "c092e5ec", "metadata": {}, "source": [ "
Hint`se = sqrt(p(1-p)/n)` -> `n = p(1-p)/se^2`. Use `math.ceil` to round up to a whole item.
\n", "
Solution\n", "\n", "```python\n", "def min_items_for_se(p, target_se):\n", " return int(math.ceil(p * (1 - p) / target_se ** 2))\n", "```\n", "
\n" ] }, { "cell_type": "code", "execution_count": 29, "id": "50736789", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.253976Z", "iopub.status.busy": "2026-06-10T20:48:43.253901Z", "iopub.status.idle": "2026-06-10T20:48:43.256020Z", "shell.execute_reply": "2026-06-10T20:48:43.255638Z" }, "collapsed": true, "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ ok ] B.1 min items for SE\n", "to resolve a 2-point gap you want SE ~ 0.005 per model: ~5,100 items. 200 was never going to be enough.\n" ] } ], "source": [ "#@title Solution { display-mode: \"form\" }\n", "# solution: redefines min_items_for_se; the check re-verifies the reference.\n", "def min_items_for_se(p, target_se):\n", " return int(math.ceil(p * (1 - p) / target_se ** 2))\n", "check(\"B.1 min items for SE\", _min_items, required=True)\n", "print(f\"to resolve a 2-point gap you want SE ~ 0.005 per model: \"\n", " f\"~{min_items_for_se(0.85, 0.005):,} items. 200 was never going to be enough.\")" ] }, { "cell_type": "markdown", "id": "7e9c60a6", "metadata": {}, "source": [ "**Problem B.2 — A verbosity-bias judge.** `Difficulty 2/5 · ~10 min`\n", "\n", "Position bias is not the only judge bias. Build `make_verbosity_judge(strength)` returning a judge `judge(len_a, len_b)` that returns `\"A\"` or `\"B\"`, preferring the *longer* response with the given `strength` (ties broken toward A). Then `verbosity_bias_rate(judge, n)` should report how often the judge picks the longer of two random-length responses; a biased judge exceeds 0.5.\n" ] }, { "cell_type": "code", "execution_count": 30, "id": "c138d879", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.256867Z", "iopub.status.busy": "2026-06-10T20:48:43.256800Z", "iopub.status.idle": "2026-06-10T20:48:43.260523Z", "shell.execute_reply": "2026-06-10T20:48:43.260167Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ -- ] B.2 verbosity bias: not attempted yet — fill in the TODO above, then re-run.\n" ] }, { "data": { "text/plain": [ "False" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def make_verbosity_judge(strength):\n", " \"\"\"Return judge(len_a, len_b) -> 'A' or 'B', biased toward the LONGER response by `strength`.\"\"\"\n", " vrng = np.random.default_rng(SEED)\n", " def judge(len_a, len_b):\n", " # score = length advantage * strength + noise; pick the higher-scoring side\n", " # TODO 1: score_a favors A when len_a > len_b; score_b symmetric. Add small noise.\n", " score_a = None\n", " score_b = None\n", " attempted(score_a, score_b)\n", " return \"A\" if score_a >= score_b else \"B\"\n", " return judge\n", "\n", "def verbosity_bias_rate(judge, n=200):\n", " \"\"\"Fraction of n random pairs where the judge picks the LONGER response.\"\"\"\n", " h = np.random.default_rng(1)\n", " picked_longer = 0\n", " for _ in range(n):\n", " la, lb = h.integers(20, 200), h.integers(20, 200)\n", " choice = judge(la, lb)\n", " longer = \"A\" if la > lb else \"B\"\n", " if choice == longer:\n", " picked_longer += 1\n", " return picked_longer / n\n", "\n", "def _verbosity():\n", " fair = make_verbosity_judge(0.0)\n", " biased = make_verbosity_judge(0.05)\n", " assert verbosity_bias_rate(biased) > 0.7, \"a strongly verbosity-biased judge picks the longer one most of the time\"\n", " assert verbosity_bias_rate(fair) < 0.65, \"a judge with no length bias picks the longer one ~half the time\"\n", "check(\"B.2 verbosity bias\", _verbosity)" ] }, { "cell_type": "markdown", "id": "7a373d8c", "metadata": {}, "source": [ "
HintLet the length advantage drive the score: `score_a = strength * (len_a - len_b) + small_noise`, and `score_b = -that + small_noise` (or symmetric). With `strength=0` only noise remains, so the judge is fair.
\n", "
Solution\n", "\n", "```python\n", "def make_verbosity_judge(strength):\n", " vrng = np.random.default_rng(SEED)\n", " def judge(len_a, len_b):\n", " score_a = strength * (len_a - len_b) + 0.5 * vrng.standard_normal()\n", " score_b = strength * (len_b - len_a) + 0.5 * vrng.standard_normal()\n", " return \"A\" if score_a >= score_b else \"B\"\n", " return judge\n", "```\n", "
\n" ] }, { "cell_type": "code", "execution_count": 31, "id": "b1fb0699", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.261483Z", "iopub.status.busy": "2026-06-10T20:48:43.261408Z", "iopub.status.idle": "2026-06-10T20:48:43.266337Z", "shell.execute_reply": "2026-06-10T20:48:43.265927Z" }, "collapsed": true, "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ ok ] B.2 verbosity bias\n", "verbosity-biased judge picks the longer response 98% of the time.\n" ] } ], "source": [ "#@title Solution { display-mode: \"form\" }\n", "# solution: redefines make_verbosity_judge; the check re-verifies the reference.\n", "def make_verbosity_judge(strength):\n", " vrng = np.random.default_rng(SEED)\n", " def judge(len_a, len_b):\n", " score_a = strength * (len_a - len_b) + 0.5 * vrng.standard_normal()\n", " score_b = strength * (len_b - len_a) + 0.5 * vrng.standard_normal()\n", " return \"A\" if score_a >= score_b else \"B\"\n", " return judge\n", "check(\"B.2 verbosity bias\", _verbosity, required=True)\n", "print(f\"verbosity-biased judge picks the longer response \"\n", " f\"{verbosity_bias_rate(make_verbosity_judge(0.05)):.0%} of the time.\")" ] }, { "cell_type": "markdown", "id": "ab5ad0d5", "metadata": {}, "source": [ "**Problem B.3 — Diagnose the failure mode.** `Difficulty 1/5 · ~6 min`\n", "\n", "Reading reports means recognizing the failure mode behind a symptom. Fill in `diagnose(symptom)` mapping each symptom string to one label. The check verifies all five.\n" ] }, { "cell_type": "code", "execution_count": 32, "id": "31be4845", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.267241Z", "iopub.status.busy": "2026-06-10T20:48:43.267167Z", "iopub.status.idle": "2026-06-10T20:48:43.270805Z", "shell.execute_reply": "2026-06-10T20:48:43.270467Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ -- ] B.3 diagnose failure modes: not attempted yet — fill in the TODO above, then re-run.\n" ] }, { "data": { "text/plain": [ "False" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def diagnose(symptom):\n", " \"\"\"Map a one-line symptom to its failure-mode label.\"\"\"\n", " table = {\n", " # TODO: fill each value with the right label from this set:\n", " # \"contamination\", \"position_bias\", \"verbosity_bias\",\n", " # \"evaluation_awareness\", \"loss_metric_divergence\"\n", " \"model recites the exact benchmark answer text it saw in pretraining\": None,\n", " \"judge prefers whichever response is shown first\": None,\n", " \"judge prefers the longer answer regardless of content\": None,\n", " \"model refuses on test-flavored prompts but complies in deployment\": None,\n", " \"training loss falls while held-out accuracy falls too\": None,\n", " }\n", " attempted(*table.values())\n", " return table[symptom]\n", "\n", "def _diagnose():\n", " cases = {\n", " \"model recites the exact benchmark answer text it saw in pretraining\": \"contamination\",\n", " \"judge prefers whichever response is shown first\": \"position_bias\",\n", " \"judge prefers the longer answer regardless of content\": \"verbosity_bias\",\n", " \"model refuses on test-flavored prompts but complies in deployment\": \"evaluation_awareness\",\n", " \"training loss falls while held-out accuracy falls too\": \"loss_metric_divergence\",\n", " }\n", " for symptom, want in cases.items():\n", " got = diagnose(symptom)\n", " assert got == want, f\"'{symptom}' is {want}, you returned {got!r}\"\n", "check(\"B.3 diagnose failure modes\", _diagnose)" ] }, { "cell_type": "markdown", "id": "40ae2e62", "metadata": {}, "source": [ "
Solution\n", "\n", "```python\n", "table = {\n", " \"model recites the exact benchmark answer text it saw in pretraining\": \"contamination\",\n", " \"judge prefers whichever response is shown first\": \"position_bias\",\n", " \"judge prefers the longer answer regardless of content\": \"verbosity_bias\",\n", " \"model refuses on test-flavored prompts but complies in deployment\": \"evaluation_awareness\",\n", " \"training loss falls while held-out accuracy falls too\": \"loss_metric_divergence\",\n", "}\n", "```\n", "
\n" ] }, { "cell_type": "code", "execution_count": 33, "id": "b413f456", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.271659Z", "iopub.status.busy": "2026-06-10T20:48:43.271594Z", "iopub.status.idle": "2026-06-10T20:48:43.274136Z", "shell.execute_reply": "2026-06-10T20:48:43.273779Z" }, "collapsed": true, "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[ ok ] B.3 diagnose failure modes\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#@title Solution { display-mode: \"form\" }\n", "# solution: redefines diagnose; the check re-verifies the reference.\n", "def diagnose(symptom):\n", " table = {\n", " \"model recites the exact benchmark answer text it saw in pretraining\": \"contamination\",\n", " \"judge prefers whichever response is shown first\": \"position_bias\",\n", " \"judge prefers the longer answer regardless of content\": \"verbosity_bias\",\n", " \"model refuses on test-flavored prompts but complies in deployment\": \"evaluation_awareness\",\n", " \"training loss falls while held-out accuracy falls too\": \"loss_metric_divergence\",\n", " }\n", " return table[symptom]\n", "check(\"B.3 diagnose failure modes\", _diagnose, required=True)" ] }, { "cell_type": "markdown", "id": "2d305f49", "metadata": {}, "source": [ "### Part C — Capstone: a defensible eval report\n", "\n", "One open project: produce a small but complete eval report comparing two models, the kind of artifact that survives review. Reuse every function above (`acc_and_se`, `bootstrap_ci`, `paired_diff_ci`, `mcnemar`, `cohen_kappa`, the ledger).\n", "\n", "**Deliverables**\n", "1. Generate (or reuse) per-item scores for two models on one shared eval set of at least 300 items, with a *known* true gap you set yourself.\n", "2. An experiment-ledger table: each model's accuracy with bootstrap CI, the paired A − B gap with its CI, and the McNemar p-value.\n", "3. A verdict that is honest about what your eval size can resolve, and a one-line statement of how many items you would need to resolve the true gap (use `min_items_for_se`).\n", "4. One bias check on a mock judge (position *or* verbosity) showing the swap/length-control mitigation changes the verdict.\n", "5. A 150-word writeup naming one thing you established with a confidence interval and one thing your eval is too small to claim.\n", "\n", "**Self-assessment** (pass / partial / fail): (a) every numeric claim has a code cell producing it; (b) the paired gap is reported with a CI *and* a second test (McNemar) that agree; (c) you state the items needed to resolve the true gap; (d) you name one thing the eval cannot conclude; (e) the bias check shows a mitigation changing a verdict. A pass needs (a), (b), and (d).\n", "\n", "
My solution (reference, ~5s on CPU)\n", "\n", "```python\n", "# 1. a larger eval set with a known 3-point true gap\n", "gc = np.random.default_rng(99)\n", "NC = 400\n", "diff = gc.uniform(0, 1, NC)\n", "sA = (gc.uniform(0, 1, NC) < 0.80 * (1 - 0.3 * diff)).astype(int) # true 0.80\n", "sB = (gc.uniform(0, 1, NC) < 0.77 * (1 - 0.3 * diff)).astype(int) # true 0.77 (3-pt gap)\n", "\n", "# 2-3. ledger + verdict\n", "for name, s in [(\"A\", sA), (\"B\", sB)]:\n", " p, _ = acc_and_se(s); lo, hi = bootstrap_ci(s, n_resamples=N_BOOT, seed=SEED)\n", " print(f\"{name}: {p:.3f} CI [{lo:.3f}, {hi:.3f}]\")\n", "gap, glo, ghi = paired_diff_ci(sA, sB)\n", "print(f\"gap A-B: {gap:+.3f} {fmt_ci(glo, ghi)} McNemar p={mcnemar(sA, sB):.3f}\")\n", "print(f\"to resolve a 3-pt gap (SE ~ 0.0075/model) you want ~{min_items_for_se(0.8, 0.0075):,} items\")\n", "\n", "# 4. a bias check whose mitigation changes the verdict\n", "jb = make_mock_judge(position_bias_strength=0.20, noise=0.0)\n", "raw = jb(0.50, 0.50) # raw pairwise vote on equal responses\n", "stable = stable_pref(jb, 0.50, 0.50) # after the swap check\n", "print(f\"raw judge says '{raw}' on equal responses; swap check says '{stable}' (the honest verdict)\")\n", "```\n", "\n", "The report's honest line: *I established with a 95% CI that model A's accuracy is in [lo, hi]; I cannot claim A beats B because the paired-difference CI crosses zero and McNemar agrees (p > 0.05). My eval needs ~N items to resolve the true 3-point gap; at 400 it is underpowered, so the right move is more items, not a louder claim.*\n", "
\n" ] }, { "cell_type": "markdown", "id": "a9e820a2", "metadata": {}, "source": [ "## Reflection\n", "\n", "Write ~150 words on the dumbest bug you hit in this notebook and how you found it. A strong candidate: bootstrapping over the wrong axis (so the CI was the whole `[0, 1]` instead of a few percent wide), or forgetting the pairing in the difference CI (resampling A and B on *different* items, which inflates the gap's variance), or the McNemar off-by-one (the continuity correction `-1` applied outside the square instead of inside), or reading position-bias from only one ordering so the bias was invisible. What was the symptom (a CI that was absurdly wide, a p-value that disagreed with the bootstrap, a \"fair\" judge that looked biased), and what diagnostic told you where to look? Nobody grades this. Writing it is the point: the failure mode of eval code is not a crash, it is a confident, wrong number that looks like a result. The whole chapter is about distrusting that number until a CI, a second test, and an honest verdict back it up, and the same skepticism applies hardest to your own code.\n" ] }, { "cell_type": "markdown", "id": "5a87ff1a", "metadata": {}, "source": [ "## Going further\n", "\n", "- Chen et al. 2021, *Evaluating Large Language Models Trained on Code* (HumanEval) — the source of the unbiased pass@k estimator you built; the appendix derivation is worth one read.\n", "- Zheng et al. 2023, *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena* — the canonical LLM-as-judge methodology paper, including the position-bias measurement and swap-style mitigation.\n", "- Efron and Tibshirani, *An Introduction to the Bootstrap* — the book behind Part 2; the resampling idea is one of the highest-leverage things you can carry into any empirical work.\n", "- Gema et al. 2024, *MMLU-Redux* — the item-level audit showing ~6.5% of MMLU items have annotator errors; it calibrates how clean any public benchmark really is.\n", "- Eugene Yan, *LLM evals* series, and Chip Huyen's eval pieces — the most readable practitioner-facing writing on this material; they cover the landscape the draft surveys.\n", "- `inspect.aisi.org.uk` — the AISI Inspect framework (typed datasets, solvers, scorers, logged traces); the discipline it enforces is the discipline this notebook hand-rolled.\n", "- ARENA 3.0, *Chapter 3* (intro to evals, dataset generation, running with Inspect) — the best hands-on curriculum for the next rungs; your bootstrap, kappa, and judge code port directly.\n", "\n", "## What this enables\n", "\n", "We turned a +5 leaderboard gap into a CI that crosses zero, and a biased judge into an order-stable one. Concretely: a paired bootstrap on 200 items put the A − B gap at a CI straddling zero, McNemar agreed (p > 0.05), and the swap check converted a position-biased pairwise judge into a usable signal at the cost of some ties. The gap this leaves: every method here estimated *uncertainty* but assumed the items were *clean*, no contamination, no reward hacking, no eval-aware model gaming the measurement. Those are adversarial failure modes where the data-generating process itself is fighting you, and behavioral statistics alone cannot catch them. The forward teaser, made concrete: here is the one thing a CI cannot save you from.\n" ] }, { "cell_type": "code", "execution_count": 34, "id": "05d4d4fb", "metadata": { "execution": { "iopub.execute_input": "2026-06-10T20:48:43.274875Z", "iopub.status.busy": "2026-06-10T20:48:43.274807Z", "iopub.status.idle": "2026-06-10T20:48:43.276843Z", "shell.execute_reply": "2026-06-10T20:48:43.276495Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "measured score on a contaminated item: 1.0 (high confidence)\n", "true capability on a fresh item : 0.0\n", "\n", "No confidence interval rescues you here: the bootstrap faithfully reports the precision\n", "of a number that is measuring memorization, not capability. Catching THIS needs contamination\n", "audits, held-out/blind evals, and adversarial eval generation, the substrate of Ch 24's\n", "red-team discipline, and where mech-interp (Ch 22) re-enters as a non-behavioral backstop.\n" ] } ], "source": [ "# the gap: statistics quantify NOISE, not ADVERSARIAL gaming. A contaminated item the model\n", "# memorized is scored 'correct' with high confidence; a tighter CI makes the WRONG number more precise.\n", "contaminated_score = 1.0 # model recited the memorized answer: looks perfect\n", "true_capability = 0.0 # it cannot actually do the task on a fresh item\n", "print(f\"measured score on a contaminated item: {contaminated_score:.1f} (high confidence)\")\n", "print(f\"true capability on a fresh item : {true_capability:.1f}\")\n", "print(\"\\nNo confidence interval rescues you here: the bootstrap faithfully reports the precision\")\n", "print(\"of a number that is measuring memorization, not capability. Catching THIS needs contamination\")\n", "print(\"audits, held-out/blind evals, and adversarial eval generation, the substrate of Ch 24's\")\n", "print(\"red-team discipline, and where mech-interp (Ch 22) re-enters as a non-behavioral backstop.\")" ] }, { "cell_type": "markdown", "id": "9c660d3a", "metadata": {}, "source": [ "---\n", "*Built top-to-bottom. If every check above printed `[ ok ]`, you put a confidence interval on a leaderboard gap, caught a position-biased judge with a swap check, watched a metric diverge from its loss, and assembled a defensible eval ledger, the science of not lying to yourself with numbers. Runtime stamp written by CI.*\n" ] } ], "metadata": { "kernelspec": { "display_name": "obvix-nb", "language": "python", "name": "obvix-nb" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" }, "obvix": { "title": "Ch 23 — Eval Science" } }, "nbformat": 4, "nbformat_minor": 5 }