{
"cells": [
{
"cell_type": "markdown",
"id": "c5e55208",
"metadata": {},
"source": [
"# Ch 26 — Reading Papers (notebook)\n",
"\n",
"`[← 25 production-and-monitoring]` · **this notebook** · `[end of curriculum]`\n",
"\n",
"Runs top-to-bottom in ~4 min on free Colab CPU. Last verified 2026-06-11.\n",
"\n",
"**What you'll build**\n",
"- A paper-triage harness driven by an *injected* fetcher with canned arXiv fixtures, so the three-pass method runs with zero network. One optional live cell degrades to the same fixtures if arXiv is unreachable.\n",
"- A category-and-red-flag classifier for abstracts, a Pass-2 card validator, and a citation-graph navigator that lands on the canonical references by sorting a canned citation list.\n",
"- The paper-claim -> reproduce-a-figure loop, twice. You reproduce **the BatchNorm stability claim** at toy scale (a deep net that diverges to `NaN` at a high learning rate, then trains fine once you add BatchNorm), and **the induction-head finding** (a 2-layer attention-only transformer learns to copy a repeated subsequence; the 1-layer ablation cannot).\n",
"- An evidence ledger: for each claim, the single number that would falsify it, computed and asserted.\n",
"\n",
"**How this notebook works.** Code cells with a `# TODO` are yours to fill in. Run the cell to grade yourself: `[ ok ]` passed, `[FAIL]` shows what went wrong, `[ -- ]` means not attempted yet. Every exercise has a hint ladder (open only as many as you need) and a folded solution below it. The notebook runs top-to-bottom even if you fill in nothing, because the solution cells redefine the pieces the later cells need. See Ch 00 for the full protocol.\n"
]
},
{
"cell_type": "markdown",
"id": "5d95fe99",
"metadata": {},
"source": [
"## Before you start\n",
"\n",
"1. arXiv emits roughly 300 ML papers a day. You can read closely maybe two. What is the first skill the arithmetic forces on you? AnswerDeciding which papers to read, and at what depth, against a question you actually have. The reading itself is the second skill. Triage is the first, and this notebook makes it mechanical.\n",
"2. You have read a paper carefully and you have re-implemented it. Are those the same state of knowledge? AnswerNo. A paper you have only read is a paper you have opinions about. A paper you have re-implemented is one whose hidden assumptions you were forced to confront, because the code did not run until you did. Part 3 and Part 4 are about turning the first state into the second.\n",
"3. Predict before you run: a 1-layer attention-only transformer is trained on sequences that contain a repeated block. Will it learn to copy the repeat? AnswerBarely. Copying-by-content (find the earlier occurrence of the current token, attend to whatever followed it) is a *two-step* lookup that needs one head to feed another. A single layer cannot compose two heads, so it stays near chance on the repeated region. The 2-layer model can, and the gap is the evidence. We reproduce exactly this in Part 4.\n"
]
},
{
"cell_type": "markdown",
"id": "f80d8866",
"metadata": {},
"source": [
"## Setup\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "3417726e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:22.647345Z",
"iopub.status.busy": "2026-06-10T20:39:22.647255Z",
"iopub.status.idle": "2026-06-10T20:39:23.504673Z",
"shell.execute_reply": "2026-06-10T20:39:23.504030Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"numpy 2.2.6 · torch 2.12.0+cpu · device cpu\n"
]
}
],
"source": [
"import numpy as np\n",
"import torch\n",
"import matplotlib.pyplot as plt\n",
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
"print(f\"numpy {np.__version__} · torch {torch.__version__} · device {device}\")\n",
"if np.__version__ < \"2.0\":\n",
" print(\"WARN: written for NumPy 2.x; older versions may shift the last digit\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "82f60555",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.505870Z",
"iopub.status.busy": "2026-06-10T20:39:23.505734Z",
"iopub.status.idle": "2026-06-10T20:39:23.513516Z",
"shell.execute_reply": "2026-06-10T20:39:23.513140Z"
}
},
"outputs": [],
"source": [
"import os, re, math, random\n",
"SEED = 0\n",
"FAST = bool(os.environ.get('NB_FAST')) # CI smoke mode: ~10x fewer steps, same code paths\n",
"rng = np.random.default_rng(SEED) # the one numpy generator we thread everywhere\n",
"torch.manual_seed(SEED); random.seed(SEED)\n",
"\n",
"# Training-step budgets. The experiment-log tables quote expected losses for BOTH.\n",
"BN_STEPS = 60 if FAST else 150 # full-batch SGD steps for the BatchNorm demo\n",
"IH_STEPS = 250 if FAST else 800 # Adam steps for the induction-head transformer\n",
"\n",
"# Plot helper (defined here, never imported): a labelled loss curve onto an axis.\n",
"def plot_loss(ax, losses, label, color):\n",
" ax.plot(losses, label=label, color=color)\n",
" ax.set_xlabel('step'); ax.set_ylabel('train loss')\n",
"\n",
"# ── house self-check harness (identical across all chapter notebooks) ──\n",
"import numpy as _np\n",
"\n",
"def check(label, test_fn, required=False):\n",
" \"\"\"Run one self-check. test_fn raises AssertionError (with a teaching\n",
" message) on failure, NotImplementedError if the stub is unfilled.\n",
" required=True is used only in solution cells; it is what CI grades.\"\"\"\n",
" try:\n",
" test_fn()\n",
" except NotImplementedError:\n",
" if required:\n",
" raise AssertionError(f\"{label}: reference solution incomplete\")\n",
" print(f\"[ -- ] {label}: not attempted yet — fill in the TODO above, then re-run.\")\n",
" return False\n",
" except AssertionError as e:\n",
" if required:\n",
" raise\n",
" print(f\"[FAIL] {label}: {e}\")\n",
" return False\n",
" print(f\"[ ok ] {label}\")\n",
" return True\n",
"\n",
"def attempted(*vals):\n",
" \"\"\"Treat None placeholders as 'not attempted'.\"\"\"\n",
" if any(v is None for v in vals):\n",
" raise NotImplementedError\n",
"\n",
"def check_shape(x, want):\n",
" assert tuple(x.shape) == tuple(want), \\\n",
" f\"shape {tuple(x.shape)}, expected {tuple(want)} — check your reshape/transpose order\"\n",
"\n",
"def check_close(got, want, atol=1e-5, rtol=1e-4, msg=\"\"):\n",
" g, w = _np.asarray(got, dtype=float), _np.asarray(want, dtype=float)\n",
" assert g.shape == w.shape, f\"shape {g.shape} vs expected {w.shape}. {msg}\"\n",
" bad = ~_np.isclose(g, w, atol=atol, rtol=rtol)\n",
" assert not bad.any(), \\\n",
" f\"{bad.mean():.2%} of values wrong (max diff {abs(g - w).max():.3g}). {msg}\""
]
},
{
"cell_type": "markdown",
"id": "91c79865",
"metadata": {},
"source": [
"> **Note:** seeds make this notebook's printed numbers reproduce on CPU. Library versions and BLAS threading can shift the last digit or two; quoted numbers hold for the pinned environment. If the induction-head repeated-region loss is 0.74 where the page says 0.72, you did nothing wrong.\n",
"\n",
"> **Note:** every training loop honors `FAST`. With `NB_FAST=1` the step counts above drop ~3-10x so CI can smoke-test the same code paths in seconds. The full run (no env var) is what produces the committed figures, and the direction of every claim holds under both settings.\n"
]
},
{
"cell_type": "markdown",
"id": "2c7931dd",
"metadata": {},
"source": [
"## The map\n",
"\n",
"> **Part 1 — Triage and the three-pass sweep.** Inject a fetcher with canned arXiv fixtures, build a Pass-1 triage card, and write a category-and-red-flag classifier for abstracts. The reading order is itself a checkable fact.\n",
"> **Part 2 — The knowledge base and the citation graph.** Validate a Pass-2 card, then navigate a canned citation graph backward (to the canonical references) and forward (to the follow-ups), landing on the primary sources instead of the loudest secondaries.\n",
"> **Part 3 — Re-implement a claim: BatchNorm stability.** Quote the claim, reproduce it at toy scale, watch a deep net diverge to `NaN` at a high learning rate, then add BatchNorm and watch it train. The deliberate failure is the lesson.\n",
"> **Part 4 — Re-implement a finding: induction heads.** Build a 2-layer attention-only transformer, train it on repeated subsequences, and measure the one number that proves it learned to copy. Ablate to 1 layer and watch the number collapse.\n",
"> **Safety lens.** Citation laundering, hype-cycle reading, and LLM-generated papers, each demonstrated on a small canned example, not asserted as a platitude.\n"
]
},
{
"cell_type": "markdown",
"id": "f162328e",
"metadata": {},
"source": [
"## Part 1 — Triage and the three-pass sweep\n",
"\n",
"> **Objectives.**\n",
"> - Treat the paper fetcher as an *injected callable*, so the whole workflow runs on canned fixtures with no network. This is the same dependency-injection seam you would use to test any code that talks to an external service.\n",
"> - Produce a Pass-1 triage card from metadata: the artifact you reread later that primes you in two minutes.\n",
"> - Encode the Pass-1 reading order (title, abstract, figure 1, conclusions, related-work opener) as a checkable list, and write a classifier that reads a category and the common red flags off an abstract.\n",
"> - Carry one fact forward: you read against a *question*, and the question sets the depth budget and the stop criterion.\n",
"\n",
"Keshav's three-pass method, adapted to ML: Pass 1 is five to ten minutes (title, abstract, figures, conclusions; decide whether to go on). Pass 2 is about an hour (the full read, with notes in your own words). Pass 3 is four-plus hours (re-execute the paper, often by re-implementing a piece). The skill is knowing which pass you are doing and why.\n"
]
},
{
"cell_type": "markdown",
"id": "75680a2a",
"metadata": {},
"source": [
"### The fetcher seam and the canned fixtures\n",
"\n",
"Live arXiv is convenient and a liability: it is a network call on the critical path, it rate-limits, and it makes the notebook non-reproducible. The fix is the same one you would use in any test suite. The function that talks to the outside world is a *parameter*, defaulting to the real thing but trivially replaced by a scripted stand-in. Below, `canned_fetch` returns fixed metadata for three real papers; the workflow never knows it is not talking to arXiv.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "e6e7060b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.514854Z",
"iopub.status.busy": "2026-06-10T20:39:23.514741Z",
"iopub.status.idle": "2026-06-10T20:39:23.518185Z",
"shell.execute_reply": "2026-06-10T20:39:23.517738Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Attention Is All You Need · 2017 · cs.CL/cs.LG\n"
]
}
],
"source": [
"# Canned arXiv fixtures: real papers, metadata frozen here so the notebook is offline-proof.\n",
"FIXTURES = {\n",
" \"1706.03762\": {\n",
" \"title\": \"Attention Is All You Need\",\n",
" \"authors\": [\"Vaswani\", \"Shazeer\", \"Parmar\", \"Uszkoreit\", \"Jones\",\n",
" \"Gomez\", \"Kaiser\", \"Polosukhin\"],\n",
" \"year\": 2017,\n",
" \"categories\": [\"cs.CL\", \"cs.LG\"],\n",
" \"abstract\": (\"The dominant sequence transduction models are based on complex \"\n",
" \"recurrent or convolutional neural networks. We propose a new simple \"\n",
" \"network architecture, the Transformer, based solely on attention \"\n",
" \"mechanisms, dispensing with recurrence and convolutions entirely. \"\n",
" \"Experiments show these models to be superior in quality while being \"\n",
" \"more parallelizable and requiring significantly less time to train.\"),\n",
" },\n",
" \"1502.03167\": {\n",
" \"title\": \"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift\",\n",
" \"authors\": [\"Ioffe\", \"Szegedy\"],\n",
" \"year\": 2015,\n",
" \"categories\": [\"cs.LG\"],\n",
" \"abstract\": (\"Training deep neural networks is complicated by the fact that the \"\n",
" \"distribution of each layer's inputs changes during training. We refer \"\n",
" \"to this as internal covariate shift, and address it by normalizing layer \"\n",
" \"inputs. Batch Normalization allows us to use much higher learning rates \"\n",
" \"and be less careful about initialization, and in some cases eliminates \"\n",
" \"the need for Dropout.\"),\n",
" },\n",
" \"2209.11895\": {\n",
" \"title\": \"In-context Learning and Induction Heads\",\n",
" \"authors\": [\"Olsson\", \"Elhage\", \"Nanda\", \"Joseph\", \"DasSarma\"],\n",
" \"year\": 2022,\n",
" \"categories\": [\"cs.LG\"],\n",
" \"abstract\": (\"We present preliminary evidence that induction heads, a type of \"\n",
" \"attention head that implements a simple copying rule, may constitute the \"\n",
" \"mechanism for the majority of in-context learning in large transformer \"\n",
" \"models. Induction heads emerge in a sharp phase change, and a 2-layer \"\n",
" \"attention-only model uses them to predict repeated subsequences.\"),\n",
" },\n",
"}\n",
"\n",
"def canned_fetch(arxiv_id):\n",
" \"\"\"Stand-in for a real arXiv fetcher; returns frozen metadata.\"\"\"\n",
" if arxiv_id not in FIXTURES:\n",
" raise KeyError(f\"no fixture for {arxiv_id!r}; this offline notebook ships three papers\")\n",
" return dict(FIXTURES[arxiv_id]) # a copy, so callers cannot mutate the fixture\n",
"\n",
"meta = canned_fetch(\"1706.03762\")\n",
"print(meta[\"title\"], \"·\", meta[\"year\"], \"·\", \"/\".join(meta[\"categories\"]))"
]
},
{
"cell_type": "markdown",
"id": "d0857c57",
"metadata": {},
"source": [
"> **Interpretation.** The workflow below takes `fetch=canned_fetch` and never touches the network. Swapping in a live fetcher is a one-argument change, and the optional cell at the end of this part shows that swap wrapped in `try/except` so a network failure degrades to the fixtures rather than crashing the run.\n"
]
},
{
"cell_type": "markdown",
"id": "d775d1b1",
"metadata": {},
"source": [
"### A Pass-1 triage card\n",
"\n",
"The output of Pass 1 is a one-paragraph note: what the paper claims, the single piece of evidence it offers, the strongest objection, and a keep/skip decision. We render it as markdown so a knowledge base can read it. The four judgement lines are left blank on purpose; filling them is the reading.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a4f74f4f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.518914Z",
"iopub.status.busy": "2026-06-10T20:39:23.518841Z",
"iopub.status.idle": "2026-06-10T20:39:23.521202Z",
"shell.execute_reply": "2026-06-10T20:39:23.520844Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# 2209.11895 — In-context Learning and Induction Heads\n",
"\n",
"**Authors:** Olsson, Elhage, Nanda et al. · **Year:** 2022 · **Categories:** cs.LG · **Pass depth:** 1\n",
"\n",
"## Verbatim abstract\n",
"\n",
"We present preliminary evidence that induction heads, a type of attention head that implements a simple copying rule, may constitute ...\n"
]
}
],
"source": [
"def triage_card(arxiv_id, fetch=canned_fetch):\n",
" \"\"\"Render a Pass-1 starter card (markdown) from injected metadata.\"\"\"\n",
" m = fetch(arxiv_id)\n",
" authors = \", \".join(m[\"authors\"][:3]) + (\" et al.\" if len(m[\"authors\"]) > 3 else \"\")\n",
" return (\n",
" f\"# {arxiv_id} — {m['title']}\\n\\n\"\n",
" f\"**Authors:** {authors} · **Year:** {m['year']} · \"\n",
" f\"**Categories:** {', '.join(m['categories'])} · **Pass depth:** 1\\n\\n\"\n",
" f\"## Verbatim abstract\\n\\n{m['abstract']}\\n\\n\"\n",
" f\"## Triage notes (fill these in; this is Pass 1)\\n\\n\"\n",
" f\"- Category (method / empirical / theory / position / survey): \\n\"\n",
" f\"- Two-sentence claim, in your words: \\n\"\n",
" f\"- Single strongest piece of supporting evidence: \\n\"\n",
" f\"- Strongest objection you can muster: \\n\"\n",
" f\"- Decision (deeper read / archive) and the question it answers: \\n\"\n",
" )\n",
"\n",
"card = triage_card(\"2209.11895\")\n",
"print(card[:320], \"...\")"
]
},
{
"cell_type": "markdown",
"id": "e82a8586",
"metadata": {},
"source": [
"> **Interpretation.** Every paper you triage leaves one of these behind. After a month the archive is keyed to *your* questions and is more useful than any newsletter, because the newsletter never knew what you were trying to build.\n"
]
},
{
"cell_type": "markdown",
"id": "0740e529",
"metadata": {},
"source": [
"### The Pass-1 reading order is a fact, not a vibe\n",
"\n",
"The order you sweep a paper in Pass 1 matters. Read the abstract twice (first for the claim, then for the qualifiers), then figure 1 (the headline result lives there in 2026), then conclusions (the one honest caveat), then the first paragraph of related work (what the authors think the prior art is). The introduction is deliberately *last* or skipped, because it is the marketing pitch and reading it first anchors you to the author's framing before you have decided whether the framing is right.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "48c51278",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.522210Z",
"iopub.status.busy": "2026-06-10T20:39:23.522129Z",
"iopub.status.idle": "2026-06-10T20:39:23.524126Z",
"shell.execute_reply": "2026-06-10T20:39:23.523866Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Pass-1 sweep: title -> abstract -> figure_1 -> conclusions -> first_paragraph_of_related_work\n"
]
}
],
"source": [
"PASS_1_ORDER = [\"title\", \"abstract\", \"figure_1\", \"conclusions\", \"first_paragraph_of_related_work\"]\n",
"assert PASS_1_ORDER[0] == \"title\", \"Pass 1 always starts at the title: it is the strongest category signal\"\n",
"assert \"introduction\" not in PASS_1_ORDER, \"the introduction is the author's pitch; do not anchor on it in Pass 1\"\n",
"print(\"Pass-1 sweep:\", \" -> \".join(PASS_1_ORDER))"
]
},
{
"cell_type": "markdown",
"id": "c4939067",
"metadata": {},
"source": [
"> **Common confusion.** \"Read the introduction first\" feels natural and is the most common Pass-1 mistake. The introduction is written to convince you to keep reading. In Pass 1 you have not yet decided whether to keep reading, so you read the parts that report results, not the part that sells them.\n"
]
},
{
"cell_type": "markdown",
"id": "1e8ed819",
"metadata": {},
"source": [
"### Exercise 26.1 — Parse an arXiv id\n",
"`Difficulty 1/5 · ~6 min`\n",
"\n",
"Fill in `parse_arxiv_id(s)`. A modern arXiv id is `YYYY.NNNNN` (four digits, a dot, four or five digits), optionally followed by a version like `v3`. Return the cleaned id on a match; raise `ValueError` on anything else. This is the guard that keeps a typo from silently fetching the wrong paper.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "cdc0cc9d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.525008Z",
"iopub.status.busy": "2026-06-10T20:39:23.524936Z",
"iopub.status.idle": "2026-06-10T20:39:23.529132Z",
"shell.execute_reply": "2026-06-10T20:39:23.528778Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 26.1 accepts valid ids: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 26.1 rejects malformed ids: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"_ARXIV_RE = re.compile(r\"^\\d{4}\\.\\d{4,5}(v\\d+)?$\")\n",
"\n",
"def parse_arxiv_id(s):\n",
" \"\"\"Return the stripped id if it matches YYYY.NNNNN(vK); else raise ValueError.\"\"\"\n",
" s = s.strip()\n",
" # TODO 1: if _ARXIV_RE does NOT match s, raise ValueError with a helpful message\n",
" # TODO 2: otherwise return s\n",
" raise NotImplementedError\n",
"\n",
"def _toy_parse():\n",
" assert parse_arxiv_id(\" 1706.03762 \") == \"1706.03762\", \"should strip whitespace and accept a 5-digit id\"\n",
" assert parse_arxiv_id(\"2209.11895\") == \"2209.11895\", \"a clean 5-digit id should pass through unchanged\"\n",
" assert parse_arxiv_id(\"2402.04249v3\") == \"2402.04249v3\", \"version suffixes are valid\"\n",
"\n",
"def _toy_reject():\n",
" for bad in [\"not-an-id\", \"1706.037\", \"arXiv:1706.03762\", \"1706_03762\"]:\n",
" try:\n",
" parse_arxiv_id(bad)\n",
" except ValueError:\n",
" continue\n",
" raise AssertionError(f\"{bad!r} should have been rejected, but parse accepted it\")\n",
"\n",
"check(\"26.1 accepts valid ids\", _toy_parse)\n",
"check(\"26.1 rejects malformed ids\", _toy_reject)"
]
},
{
"cell_type": "markdown",
"id": "1388c98c",
"metadata": {},
"source": [
"Hint 1 (conceptual)The regex is already compiled for you. You only need to test it with `.match(...)` and branch: raise on no match, return `s` on a match.\n",
"\n",
"Hint 2 (the lines)\n",
"\n",
"```python\n",
"if not _ARXIV_RE.match(s):\n",
" raise ValueError(f\"not an arXiv id: {s!r}\")\n",
"return s\n",
"```\n",
"\n",
"\n",
"Help — \"a valid id is being rejected\"The spec accepts four *or five* digits after the dot (`\\d{4,5}`), so both `1706.0376` and `1706.03762` are valid. If your version rejects four-digit ids, you wrote `\\d{5}` or `\\d{4}` instead of `\\d{4,5}`. Use the compiled `_ARXIV_RE` as given; do not re-tighten it.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "2f1b979c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.529996Z",
"iopub.status.busy": "2026-06-10T20:39:23.529929Z",
"iopub.status.idle": "2026-06-10T20:39:23.532139Z",
"shell.execute_reply": "2026-06-10T20:39:23.531749Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 26.1 accepts valid ids\n",
"[ ok ] 26.1 rejects malformed ids\n",
"parsed: 2209.11895v2\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines parse_arxiv_id; the checks below re-verify the reference.\n",
"def parse_arxiv_id(s):\n",
" s = s.strip()\n",
" if not _ARXIV_RE.match(s):\n",
" raise ValueError(f\"not an arXiv id (expected YYYY.NNNNN): {s!r}\")\n",
" return s\n",
"\n",
"check(\"26.1 accepts valid ids\", _toy_parse, required=True)\n",
"check(\"26.1 rejects malformed ids\", _toy_reject, required=True)\n",
"print(\"parsed:\", parse_arxiv_id(\"2209.11895v2\"))"
]
},
{
"cell_type": "markdown",
"id": "37066310",
"metadata": {},
"source": [
"### Exercise 26.2 — Read category and red flags off an abstract\n",
"`Difficulty 2/5 · ~12 min`\n",
"\n",
"Pass 1 includes a snap judgement: what *kind* of paper is this, and does the abstract wave any of the known red flags? Fill in `classify_abstract(text)` to return a dict with:\n",
"\n",
"- `category`: one of `\"method\"`, `\"empirical\"`, `\"survey\"`, or `\"unknown\"`, decided by cheap keyword cues (a method paper says it *proposes* or *introduces*; an empirical study *re-evaluates* or *studies*; a survey *surveys* or *reviews*).\n",
"- `red_flags`: a list (possibly empty) drawn from the cues in `RED_FLAG_CUES` that appear in the text.\n",
"\n",
"These are heuristics, not truth. The point is to make the snap judgement explicit and rerunnable, the way a checklist makes a pilot's snap judgement explicit. **Harder:** add a `confidence` key that is `\"low\"` when two or more category cues fire at once (genuinely ambiguous).\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "ca85ca61",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.532873Z",
"iopub.status.busy": "2026-06-10T20:39:23.532809Z",
"iopub.status.idle": "2026-06-10T20:39:23.537104Z",
"shell.execute_reply": "2026-06-10T20:39:23.536860Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 26.2 category (method): not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 26.2 red flags: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 26.2 clean abstract: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"RED_FLAG_CUES = {\n",
" \"novel framework\": \"claims novelty without naming what is new\",\n",
" \"unifies\": \"a framework that 'unifies' N concepts rarely reproduces\",\n",
" \"outperforms\": \"outperforms whom, on which eval, with what compute?\",\n",
" \"inspired by recent advances\": \"rides the hype cycle instead of a question\",\n",
"}\n",
"_CATEGORY_CUES = {\n",
" \"method\": [\"propose\", \"we introduce\", \"new architecture\", \"novel\",\n",
" \"address it by\", \"allows us to\", \"we present\"],\n",
" \"empirical\": [\"re-evaluate\", \"we study\", \"empirical study\", \"we measure\", \"we benchmark\"],\n",
" \"survey\": [\"we survey\", \"we review\", \"this survey\", \"overview of\"],\n",
"}\n",
"\n",
"def classify_abstract(text):\n",
" \"\"\"Return {'category': str, 'red_flags': list[str]} from cheap keyword cues.\"\"\"\n",
" low = text.lower()\n",
" # TODO 1: red_flags = the keys of RED_FLAG_CUES whose key string appears in `low`\n",
" red_flags = None\n",
" # TODO 2: for each category in _CATEGORY_CUES, count how many of its cue strings appear in `low`\n",
" # TODO 3: category = the one with the highest count; \"unknown\" if every count is 0\n",
" category = None\n",
" attempted(red_flags, category)\n",
" return {\"category\": category, \"red_flags\": red_flags}\n",
"\n",
"def _toy_method():\n",
" r = classify_abstract(\"We propose a new simple network architecture based on attention.\")\n",
" assert r[\"category\"] == \"method\", f\"a 'we propose ... new architecture' abstract is a method paper, got {r['category']!r}\"\n",
"\n",
"def _toy_flags():\n",
" r = classify_abstract(\"We propose a novel framework that unifies five ideas and outperforms baselines.\")\n",
" assert set(r[\"red_flags\"]) == {\"novel framework\", \"unifies\", \"outperforms\"}, \\\n",
" f\"should catch three red flags, got {sorted(r['red_flags'])}\"\n",
"\n",
"def _toy_clean():\n",
" # the real BatchNorm abstract: a method paper, no marketing red flags\n",
" r = classify_abstract(FIXTURES[\"1502.03167\"][\"abstract\"])\n",
" assert r[\"category\"] == \"method\", f\"BatchNorm is a method paper, got {r['category']!r}\"\n",
" assert r[\"red_flags\"] == [], f\"the BN abstract waves no red flags, got {r['red_flags']}\"\n",
"\n",
"check(\"26.2 category (method)\", _toy_method)\n",
"check(\"26.2 red flags\", _toy_flags)\n",
"check(\"26.2 clean abstract\", _toy_clean)"
]
},
{
"cell_type": "markdown",
"id": "5773fdb3",
"metadata": {},
"source": [
"Hint 1 (conceptual)Both halves are substring membership. `cue in low` is a boolean; `sum(cue in low for cues in ...)` counts. For the category, build a small dict of counts and take the `max` by value, falling back to `\"unknown\"` when the best count is 0.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"red_flags = [k for k in RED_FLAG_CUES if k in low]\n",
"counts = {cat: sum(cue in low for cue in cues) for cat, cues in _CATEGORY_CUES.items()}\n",
"best = max(counts, key=counts.get)\n",
"category = best if counts[best] > 0 else \"unknown\"\n",
"```\n",
"\n",
"\n",
"Help — \"every abstract comes back 'method'\"`max(counts, key=counts.get)` returns *a* key even when all counts are 0, and dict order makes it `\"method\"`. You must guard with `counts[best] > 0`, otherwise an abstract with no cues is silently labelled a method paper.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "793cbd32",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.537955Z",
"iopub.status.busy": "2026-06-10T20:39:23.537872Z",
"iopub.status.idle": "2026-06-10T20:39:23.540745Z",
"shell.execute_reply": "2026-06-10T20:39:23.540343Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 26.2 category (method)\n",
"[ ok ] 26.2 red flags\n",
"[ ok ] 26.2 clean abstract\n",
"1706.03762: category=method red_flags=[]\n",
"1502.03167: category=method red_flags=[]\n",
"2209.11895: category=method red_flags=[]\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines classify_abstract; the checks below re-verify the reference.\n",
"def classify_abstract(text):\n",
" low = text.lower()\n",
" red_flags = [k for k in RED_FLAG_CUES if k in low]\n",
" counts = {cat: sum(cue in low for cue in cues) for cat, cues in _CATEGORY_CUES.items()}\n",
" best = max(counts, key=counts.get)\n",
" category = best if counts[best] > 0 else \"unknown\"\n",
" return {\"category\": category, \"red_flags\": red_flags}\n",
"\n",
"check(\"26.2 category (method)\", _toy_method, required=True)\n",
"check(\"26.2 red flags\", _toy_flags, required=True)\n",
"check(\"26.2 clean abstract\", _toy_clean, required=True)\n",
"for aid in [\"1706.03762\", \"1502.03167\", \"2209.11895\"]:\n",
" r = classify_abstract(FIXTURES[aid][\"abstract\"])\n",
" print(f\"{aid}: category={r['category']:9} red_flags={r['red_flags']}\")"
]
},
{
"cell_type": "markdown",
"id": "bdd68e28",
"metadata": {},
"source": [
"> **Interpretation.** All three fixtures are method papers with clean abstracts, which is why they became canonical. A red-flag count is not a verdict; it is a prior. The papers that survive triage are the ones you then read, and the ones you read carefully are the small set you build on.\n"
]
},
{
"cell_type": "markdown",
"id": "cfd3cd4c",
"metadata": {},
"source": [
"### Optional: the live fetcher (degrades to the fixtures)\n",
"\n",
"The only difference between this notebook and a live triage tool is the fetcher. Here is the live one, wrapped so a network failure (or an offline CI run) degrades to the canned fixtures instead of crashing. This is the only cell that *can* touch the network, and it never has to.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "d43301c6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.541554Z",
"iopub.status.busy": "2026-06-10T20:39:23.541483Z",
"iopub.status.idle": "2026-06-10T20:39:23.544752Z",
"shell.execute_reply": "2026-06-10T20:39:23.544402Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"title: Attention Is All You Need\n"
]
}
],
"source": [
"# deeper: an optional live fetcher; the canonical path above never needs it.\n",
"def live_fetch(arxiv_id, timeout=6):\n",
" import urllib.request\n",
" url = f\"http://export.arxiv.org/api/query?id_list={arxiv_id}\"\n",
" req = urllib.request.Request(url, headers={\"User-Agent\": \"obvix-learn-ch26/1.0\"})\n",
" with urllib.request.urlopen(req, timeout=timeout) as r:\n",
" xml = r.read().decode(\"utf-8\")\n",
" titles = re.findall(r\"(.*?)\", xml, flags=re.DOTALL)\n",
" summaries = re.findall(r\"(.*?)\", xml, flags=re.DOTALL)\n",
" title = titles[1].strip() if len(titles) > 1 else \"(no title)\"\n",
" return {\"title\": title, \"authors\": re.findall(r\"(.*?)\", xml),\n",
" \"year\": 0, \"categories\": [], \"abstract\": (summaries[0].strip() if summaries else \"\")}\n",
"\n",
"USE_LIVE = False # flip to True locally to hit arXiv; CI and this committed run keep it canned\n",
"try:\n",
" fetched = (live_fetch(\"1706.03762\") if USE_LIVE else canned_fetch(\"1706.03762\"))\n",
" print(\"title:\", fetched[\"title\"])\n",
"except Exception as e: # network down, offline CI, rate limit: degrade\n",
" print(f\"live fetch unavailable ({type(e).__name__}); using canned fixture\")\n",
" print(\"title:\", canned_fetch(\"1706.03762\")[\"title\"])"
]
},
{
"cell_type": "markdown",
"id": "740af96f",
"metadata": {},
"source": [
"> **Key takeaways.** Triage is a fetch, a card, and a one-paragraph judgement, run against a question. The fetcher is injected so the workflow is reproducible and the live path is one flag away and never on the critical path. The Pass-1 reading order is a checkable fact, and the introduction is read last. Red flags are priors, not verdicts.\n"
]
},
{
"cell_type": "markdown",
"id": "ee6a37c9",
"metadata": {},
"source": [
"## Part 2 — The knowledge base and the citation graph\n",
"\n",
"> **Objectives.**\n",
"> - Validate a Pass-2 card: the five sections that make a note worth rereading (claim, method, evidence, limitations, what-this-changes-for-me), each non-empty and free of `TODO`.\n",
"> - Navigate a canned citation graph backward (to the canonical references) and forward (to the follow-ups), sorting by citation count to find the primaries the loud secondaries are standing on.\n",
"> - See why citation count is a *lagging* indicator, and read the cited paper before you cite it.\n",
"\n",
"Pass 2 produces a one-page card. The most important box is the five-minute summary written in your own words without looking at the paper. If you cannot write it, you have not done Pass 2; you have done two iterations of Pass 1.\n"
]
},
{
"cell_type": "markdown",
"id": "9167dcd1",
"metadata": {},
"source": [
"### Validating a Pass-2 card\n",
"\n",
"A card is \"done\" when each required section exists as a `## heading` and has real content under it. The validator below scans for the five sections and rejects any that is empty or still says `TODO`. It is the same shape as a linter: it does not judge the content, only that you wrote some.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "e79eeea0",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.545519Z",
"iopub.status.busy": "2026-06-10T20:39:23.545446Z",
"iopub.status.idle": "2026-06-10T20:39:23.549010Z",
"shell.execute_reply": "2026-06-10T20:39:23.548711Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"GOOD_CARD valid=True, missing=[]\n"
]
}
],
"source": [
"PASS_2_FIELDS = [\"claim\", \"method\", \"evidence\", \"limitations\", \"what this changes for me\"]\n",
"\n",
"GOOD_CARD = \"\"\"# 2209.11895 — In-context Learning and Induction Heads\n",
"\n",
"## Claim\n",
"A 2-layer attention-only transformer learns induction heads that copy from\n",
"repeated subsequences, and this is much of in-context learning.\n",
"\n",
"## Method\n",
"Train a tiny attention-only transformer; measure loss on repeated tokens.\n",
"\n",
"## Evidence\n",
"Loss on the repeated region drops far below the non-repeat region for 2 layers,\n",
"not for 1 layer.\n",
"\n",
"## Limitations\n",
"Toy scale; the full claim about real LLMs is preliminary, as the authors say.\n",
"\n",
"## What this changes for me\n",
"Part 4 of this notebook reproduces exactly this split, so I can trust the gap.\n",
"\"\"\"\n",
"\n",
"def validate_pass2(card_text, fields=PASS_2_FIELDS):\n",
" \"\"\"Return (ok, missing). A field is missing if its '## heading' is absent,\n",
" or the text under it (up to the next heading) is empty or contains 'TODO'.\"\"\"\n",
" missing = []\n",
" for field in fields:\n",
" m = re.search(rf\"^##\\s+{re.escape(field)}\\s*$\", card_text, re.IGNORECASE | re.MULTILINE)\n",
" if not m:\n",
" missing.append(field); continue\n",
" rest = card_text[m.end():]\n",
" nxt = re.search(r\"^##\\s+\", rest, re.MULTILINE)\n",
" body = rest[:nxt.start()] if nxt else rest\n",
" if not body.strip() or \"TODO\" in body:\n",
" missing.append(field)\n",
" return (len(missing) == 0, missing)\n",
"\n",
"ok, missing = validate_pass2(GOOD_CARD)\n",
"print(f\"GOOD_CARD valid={ok}, missing={missing}\")"
]
},
{
"cell_type": "markdown",
"id": "804faba0",
"metadata": {},
"source": [
"> **Interpretation.** A linter for your own notes sounds excessive until you have a vault of 200 cards, half of them abandoned at \"## Claim\\nTODO\". The validator is what keeps the base honest, the same way `pytest` keeps a codebase honest.\n"
]
},
{
"cell_type": "markdown",
"id": "36a1394c",
"metadata": {},
"source": [
"### Exercise 26.3 — Find the half-written card\n",
"`Difficulty 2/5 · ~8 min`\n",
"\n",
"`validate_pass2` is written for you above; now you build the adversarial input that proves it works. Construct `BAD_CARD`: a string that has all five `## headings` but is missing real content in **exactly two** of them (leave one empty, write `TODO` under another). Then the check confirms the validator flags those two and only those two.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "3e175ddd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.549835Z",
"iopub.status.busy": "2026-06-10T20:39:23.549769Z",
"iopub.status.idle": "2026-06-10T20:39:23.552288Z",
"shell.execute_reply": "2026-06-10T20:39:23.552030Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 26.3 two sections flagged: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Build a card that is missing content in exactly two sections.\n",
"BAD_CARD = None # TODO: a markdown string with all five ## headings, two of them empty/TODO\n",
"\n",
"def _bad_card():\n",
" attempted(BAD_CARD)\n",
" ok, missing = validate_pass2(BAD_CARD)\n",
" assert not ok, \"your BAD_CARD passed validation; it should be missing two sections\"\n",
" assert len(missing) == 2, f\"expected exactly 2 missing sections, validator found {len(missing)}: {missing}\"\n",
" # and the good card must still pass, proving the validator is not just always-false\n",
" assert validate_pass2(GOOD_CARD)[0], \"GOOD_CARD must still validate\"\n",
"\n",
"check(\"26.3 two sections flagged\", _bad_card)"
]
},
{
"cell_type": "markdown",
"id": "66edd619",
"metadata": {},
"source": [
"Hint 1 (conceptual)Start from `GOOD_CARD`'s structure: five `## headings`. Under three of them put a real sentence; under one put nothing (blank line straight to the next heading); under one put the single word `TODO`.\n",
"\n",
"Hint 2 (a template)\n",
"\n",
"Build a string with all five `## headings`. Put a real sentence under three of them. Leave `## Limitations` empty (a blank line straight to the next heading) and put the single word `TODO` under `## Method`. Those two are the ones the validator must catch. Use a normal triple-quoted Python string for the card text.\n",
"\n",
"Help — \"validator found 1, not 2\"A heading followed immediately by the next heading still has a body of `\"\\n\"`, which `.strip()` turns empty, so it counts as missing. If you got 1, you probably left real text under all but one section; re-check that you blanked one *and* TODO'd another.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "0ec464f4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.553097Z",
"iopub.status.busy": "2026-06-10T20:39:23.553031Z",
"iopub.status.idle": "2026-06-10T20:39:23.554932Z",
"shell.execute_reply": "2026-06-10T20:39:23.554630Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 26.3 two sections flagged\n",
"missing: ['method', 'limitations']\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: a card missing content in exactly two sections (Method=TODO, Limitations=empty).\n",
"BAD_CARD = \"\"\"# 2209.11895 — half written\n",
"\n",
"## Claim\n",
"Induction heads copy from repeats.\n",
"\n",
"## Method\n",
"TODO\n",
"\n",
"## Evidence\n",
"The repeated-region loss drops for 2 layers.\n",
"\n",
"## Limitations\n",
"\n",
"## What this changes for me\n",
"I will reproduce the split in Part 4.\n",
"\"\"\"\n",
"\n",
"check(\"26.3 two sections flagged\", _bad_card, required=True)\n",
"print(\"missing:\", validate_pass2(BAD_CARD)[1])"
]
},
{
"cell_type": "markdown",
"id": "5db5205a",
"metadata": {},
"source": [
"### Navigating a citation graph\n",
"\n",
"When a paper says \"as shown in [X]\", you can trust it or follow it. Most people trust, which is how the loudest secondary source crowds out the canonical primary. Here is a canned citation graph: for each paper, the references it cites (backward) with their own citation counts. Backward search sorted by citation count surfaces the foundational works; you can usually stop after two hops.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "b7761758",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.555810Z",
"iopub.status.busy": "2026-06-10T20:39:23.555739Z",
"iopub.status.idle": "2026-06-10T20:39:23.558935Z",
"shell.execute_reply": "2026-06-10T20:39:23.558490Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 95000 2017 Attention Is All You Need\n",
" 28000 2020 Language Models are Few-Shot Learners\n",
" 600 2021 A Mathematical Framework for Transformer Circuits\n"
]
}
],
"source": [
"# Canned citation graph: arxiv_id -> list of references it cites, each with a (rough) count.\n",
"CITES = {\n",
" \"2209.11895\": [ # Induction Heads cites:\n",
" {\"id\": \"1706.03762\", \"title\": \"Attention Is All You Need\", \"year\": 2017, \"count\": 95000},\n",
" {\"id\": \"2005.14165\", \"title\": \"Language Models are Few-Shot Learners\", \"year\": 2020, \"count\": 28000},\n",
" {\"id\": \"2104.00xxx\", \"title\": \"A Mathematical Framework for Transformer Circuits\", \"year\": 2021, \"count\": 600},\n",
" ],\n",
" \"1502.03167\": [ # BatchNorm cites:\n",
" {\"id\": \"1207.0580\", \"title\": \"Improving neural networks by preventing co-adaptation (Dropout)\", \"year\": 2012, \"count\": 8000},\n",
" {\"id\": \"9801.00xxx\", \"title\": \"Efficient BackProp\", \"year\": 1998, \"count\": 12000},\n",
" ],\n",
"}\n",
"# Forward graph: who cites whom (the inverse of CITES), built once.\n",
"CITED_BY = {}\n",
"for src, refs in CITES.items():\n",
" for r in refs:\n",
" CITED_BY.setdefault(r[\"id\"], []).append(src)\n",
"\n",
"def backward(arxiv_id, top=3):\n",
" \"\"\"References this paper cites, most-cited first (the canonical primaries).\"\"\"\n",
" refs = sorted(CITES.get(arxiv_id, []), key=lambda r: -r[\"count\"])\n",
" return refs[:top]\n",
"\n",
"for r in backward(\"2209.11895\"):\n",
" print(f\"{r['count']:6d} {r['year']} {r['title']}\")"
]
},
{
"cell_type": "markdown",
"id": "ec19dacc",
"metadata": {},
"source": [
"> **Interpretation.** The Induction Heads paper's most-cited reference is the Transformer paper, exactly the foundational work you would want to land on. The least-cited reference (the Transformer Circuits framework, 600 citations) is the most *useful* one if your question is mechanistic, which is the caveat in the next cell: citation count is a lagging indicator.\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "5af3a0f3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.559703Z",
"iopub.status.busy": "2026-06-10T20:39:23.559631Z",
"iopub.status.idle": "2026-06-10T20:39:23.561756Z",
"shell.execute_reply": "2026-06-10T20:39:23.561423Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"most-cited reference : Attention Is All You Need (95000)\n",
"most-useful-if-mechanistic: A Mathematical Framework for Transformer Circuits (600)\n"
]
}
],
"source": [
"# A 2026 mechanistic paper cited 600 times can matter more than a 2017 paper cited 95000 times,\n",
"# if your question is 'how does this work' rather than 'is this standard'. Count ranks visibility,\n",
"# not importance-to-your-question.\n",
"canonical = backward(\"2209.11895\", top=1)[0]\n",
"niche = min(CITES[\"2209.11895\"], key=lambda r: r[\"count\"])\n",
"print(f\"most-cited reference : {canonical['title']} ({canonical['count']})\")\n",
"print(f\"most-useful-if-mechanistic: {niche['title']} ({niche['count']})\")\n",
"assert niche[\"count\"] < canonical[\"count\"], \"the niche primary is less cited but often more load-bearing\""
]
},
{
"cell_type": "markdown",
"id": "96cbb9b3",
"metadata": {},
"source": [
"### Exercise 26.4 — Forward search, and a two-hop backward walk\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"Two functions. `forward(arxiv_id)` returns the papers in the graph that *cite* `arxiv_id` (use the prebuilt `CITED_BY`). `canonical_roots(arxiv_id, hops)` does backward search for `hops` levels and returns the set of ids reachable, so you can land on the seminal works without reading every paper in between. The depth budget for most contexts is two hops; this function makes \"two hops\" literal.\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "ed2f47d4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.562425Z",
"iopub.status.busy": "2026-06-10T20:39:23.562356Z",
"iopub.status.idle": "2026-06-10T20:39:23.566079Z",
"shell.execute_reply": "2026-06-10T20:39:23.565760Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 26.4 forward search: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 26.4 two-hop roots: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def forward(arxiv_id):\n",
" \"\"\"Return the list of paper ids in the graph that cite arxiv_id.\"\"\"\n",
" # TODO 1: look arxiv_id up in CITED_BY (return [] if absent)\n",
" raise NotImplementedError\n",
"\n",
"def canonical_roots(arxiv_id, hops=2):\n",
" \"\"\"Set of reference ids reachable by following citations backward `hops` levels.\"\"\"\n",
" frontier = {arxiv_id}\n",
" roots = set()\n",
" for _ in range(hops):\n",
" nxt = set()\n",
" for pid in frontier:\n",
" for ref in CITES.get(pid, []):\n",
" # TODO 2: add ref['id'] to both `roots` and the next frontier `nxt`,\n",
" # then DELETE the next line.\n",
" raise NotImplementedError\n",
" frontier = nxt\n",
" return roots\n",
"\n",
"def _fwd():\n",
" assert set(forward(\"1706.03762\")) == {\"2209.11895\"}, \\\n",
" \"the Transformer paper is cited (in this graph) only by Induction Heads\"\n",
" assert forward(\"9999.99999\") == [], \"an unknown id has no forward citations here\"\n",
"\n",
"def _roots():\n",
" roots = canonical_roots(\"2209.11895\", hops=1)\n",
" assert \"1706.03762\" in roots, \"one hop back from Induction Heads reaches the Transformer paper\"\n",
" assert \"1207.0580\" not in roots, \"Dropout is two hops away (via BatchNorm), not one, and not even reachable here\"\n",
"\n",
"check(\"26.4 forward search\", _fwd)\n",
"check(\"26.4 two-hop roots\", _roots)"
]
},
{
"cell_type": "markdown",
"id": "3759fa6e",
"metadata": {},
"source": [
"Hint 1 (conceptual)`forward` is a single dict lookup with a default: `CITED_BY.get(arxiv_id, [])`. For `canonical_roots`, replace the `raise NotImplementedError` with two `.add` calls that accumulate `ref[\"id\"]` into `roots` and `nxt`.\n",
"\n",
"Hint 2 (the lines)\n",
"\n",
"```python\n",
"def forward(arxiv_id):\n",
" return CITED_BY.get(arxiv_id, [])\n",
"# inside the inner loop of canonical_roots:\n",
"roots.add(ref[\"id\"]); nxt.add(ref[\"id\"])\n",
"```\n",
"\n",
"\n",
"Help — \"two-hop roots is empty\"If `canonical_roots` returns an empty set, you never added to `roots`. The `# TODO 2` line must populate both `roots` (the answer) and `nxt` (so the next hop has somewhere to walk from). Adding only to `nxt` leaves `roots` empty.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "8c16d5c4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.567140Z",
"iopub.status.busy": "2026-06-10T20:39:23.567075Z",
"iopub.status.idle": "2026-06-10T20:39:23.569674Z",
"shell.execute_reply": "2026-06-10T20:39:23.569296Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 26.4 forward search\n",
"[ ok ] 26.4 two-hop roots\n",
"forward(Transformer): ['2209.11895']\n",
"2-hop roots of Induction Heads: ['1706.03762', '2005.14165', '2104.00xxx']\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines forward and canonical_roots; the checks below re-verify.\n",
"def forward(arxiv_id):\n",
" return CITED_BY.get(arxiv_id, [])\n",
"\n",
"def canonical_roots(arxiv_id, hops=2):\n",
" frontier = {arxiv_id}\n",
" roots = set()\n",
" for _ in range(hops):\n",
" nxt = set()\n",
" for pid in frontier:\n",
" for ref in CITES.get(pid, []):\n",
" roots.add(ref[\"id\"]); nxt.add(ref[\"id\"])\n",
" frontier = nxt\n",
" return roots\n",
"\n",
"check(\"26.4 forward search\", _fwd, required=True)\n",
"check(\"26.4 two-hop roots\", _roots, required=True)\n",
"print(\"forward(Transformer):\", forward(\"1706.03762\"))\n",
"print(\"2-hop roots of Induction Heads:\", sorted(canonical_roots(\"2209.11895\", hops=2)))"
]
},
{
"cell_type": "markdown",
"id": "2cd7146b",
"metadata": {},
"source": [
"> **Key takeaways.** A Pass-2 card is valid when its five sections are filled; a linter keeps the vault honest. Backward search sorted by citation count lands on the canonical primaries; forward search finds the follow-ups. Citation count ranks visibility, not importance-to-your-question, so read the cited paper before you cite it.\n"
]
},
{
"cell_type": "markdown",
"id": "e2402c8c",
"metadata": {},
"source": [
"## Part 3 — Re-implement a claim: BatchNorm stability\n",
"\n",
"> **Objectives.**\n",
"> - Quote a specific, falsifiable claim from a real paper, then design the smallest experiment that could falsify it.\n",
"> - Reproduce the claim at toy scale: a deep MLP that trains well at a high learning rate *only* with BatchNorm.\n",
"> - Stage the deliberate failure: at a high learning rate the no-BN net diverges to `NaN`. Diagnose it, then fix it with BatchNorm.\n",
"> - Build the evidence ledger entry: the single number (does the loss stay finite and fall) that decides the claim.\n",
"\n",
"The Batch Normalization paper (Ioffe & Szegedy, 2015) makes a concrete promise in its abstract: BatchNorm *\"allows us to use much higher learning rates and be less careful about initialization.\"* That is a claim you can falsify on a laptop in seconds. The reproduce-a-figure loop is: quote the claim, build the smallest experiment that tests *that specific promise*, run it, and write down the number.\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "371356b0",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.570454Z",
"iopub.status.busy": "2026-06-10T20:39:23.570389Z",
"iopub.status.idle": "2026-06-10T20:39:23.572117Z",
"shell.execute_reply": "2026-06-10T20:39:23.571864Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Claim (BatchNorm abstract):\n",
" much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout.\n",
"\n",
"Falsifier we will run: train a deep MLP at a HIGH learning rate with and without BN.\n",
"If the claim holds, the no-BN run destabilizes (or diverges) where the BN run stays stable.\n"
]
}
],
"source": [
"print(\"Claim (BatchNorm abstract):\")\n",
"print(\" \", FIXTURES['1502.03167']['abstract'][-120:])\n",
"print(\"\\nFalsifier we will run: train a deep MLP at a HIGH learning rate with and without BN.\")\n",
"print(\"If the claim holds, the no-BN run destabilizes (or diverges) where the BN run stays stable.\")"
]
},
{
"cell_type": "markdown",
"id": "c9955fc1",
"metadata": {},
"source": [
"### A synthetic problem with a known answer\n",
"\n",
"We need a problem where \"did it learn\" is unambiguous. A linear decision boundary in 20 dimensions with a little label noise: a deep ReLU MLP can drive the training loss near zero if optimization cooperates, so any failure to do so is an optimization failure, which is exactly what we are studying.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "e3348521",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.572768Z",
"iopub.status.busy": "2026-06-10T20:39:23.572699Z",
"iopub.status.idle": "2026-06-10T20:39:23.576273Z",
"shell.execute_reply": "2026-06-10T20:39:23.575899Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"X (2000, 20) · y (2000,) · positive rate 0.51\n"
]
}
],
"source": [
"def make_data(n=2000, d=20):\n",
" g = torch.Generator().manual_seed(SEED)\n",
" X = torch.randn(n, d, generator=g) # (n, d) features\n",
" w = torch.randn(d, 1, generator=g) # (d, 1) the true (linear) separator\n",
" margin = (X @ w).squeeze(1) # (n,) signed distance to the boundary\n",
" y = (margin + 0.5 * torch.randn(n, generator=g) > 0).long() # (n,) noisy labels\n",
" return X, y\n",
"\n",
"X, y = make_data()\n",
"print(f\"X {tuple(X.shape)} · y {tuple(y.shape)} · positive rate {y.float().mean():.2f}\")"
]
},
{
"cell_type": "markdown",
"id": "e6ad3b1e",
"metadata": {},
"source": [
"> **Note:** the boundary is linear, but we attack it with a 6-layer ReLU net on purpose. The depth is what makes the network sensitive to the learning rate, which is the sensitivity BatchNorm is claimed to tame.\n"
]
},
{
"cell_type": "markdown",
"id": "89eaa83e",
"metadata": {},
"source": [
"### A deep MLP, with a BatchNorm switch\n",
"\n",
"One model class, one boolean. With `bn=True` a `BatchNorm1d` sits after every linear layer; with `bn=False` it does not. Everything else is identical, so any difference in training is attributable to BatchNorm alone. This is the controlled comparison the paper's claim demands.\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "7abc4edd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.577076Z",
"iopub.status.busy": "2026-06-10T20:39:23.577005Z",
"iopub.status.idle": "2026-06-10T20:39:23.582297Z",
"shell.execute_reply": "2026-06-10T20:39:23.582046Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"forward OK, logits shape (8, 2)\n"
]
}
],
"source": [
"import torch.nn as nn\n",
"\n",
"class DeepMLP(nn.Module):\n",
" def __init__(self, d=20, h=128, depth=6, bn=False):\n",
" super().__init__()\n",
" layers, prev = [], d\n",
" for _ in range(depth):\n",
" layers.append(nn.Linear(prev, h))\n",
" if bn:\n",
" layers.append(nn.BatchNorm1d(h)) # the one thing under test\n",
" layers.append(nn.ReLU())\n",
" prev = h\n",
" layers.append(nn.Linear(h, 2))\n",
" self.net = nn.Sequential(*layers)\n",
" def forward(self, x):\n",
" return self.net(x)\n",
"\n",
"# randn smoke test: shapes must line up before we train anything (Tier-2 house habit).\n",
"probe = DeepMLP(bn=True)(torch.randn(8, 20))\n",
"check_shape(probe, (8, 2))\n",
"print(\"forward OK, logits shape\", tuple(probe.shape))"
]
},
{
"cell_type": "markdown",
"id": "de68d659",
"metadata": {},
"source": [
"### The training loop, and the high-learning-rate failure\n",
"\n",
"A plain full-batch SGD loop, returning the loss at every step so we can see *how* it failed, not just whether. We run it at a deliberately high learning rate (`lr=3.0`), the regime where the paper says initialization-sensitivity bites.\n",
"\n",
"> **Predict:** at `lr=3.0`, what happens to the no-BN net's loss? AnswerIt diverges. The deep ReLU stack amplifies the high learning rate into exploding activations, the loss goes to `inf`, then `NaN`. That divergence is the claim's falsifier firing in our favor: BatchNorm is supposed to prevent exactly this.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "730fc961",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:23.583154Z",
"iopub.status.busy": "2026-06-10T20:39:23.583079Z",
"iopub.status.idle": "2026-06-10T20:39:24.636947Z",
"shell.execute_reply": "2026-06-10T20:39:24.636607Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"no-BN @ lr=3.0: final loss nan\n",
"contains NaN: True\n"
]
}
],
"source": [
"def train_mlp(bn, lr, steps):\n",
" torch.manual_seed(SEED) # re-seed so this cell reproduces if re-run alone\n",
" Xd, yd = make_data()\n",
" model = DeepMLP(bn=bn)\n",
" opt = torch.optim.SGD(model.parameters(), lr=lr)\n",
" loss_fn = nn.CrossEntropyLoss()\n",
" losses = []\n",
" model.train()\n",
" for _ in range(steps):\n",
" opt.zero_grad() # forward\n",
" out = model(Xd)\n",
" loss = loss_fn(out, yd) # backward\n",
" loss.backward()\n",
" opt.step() # update\n",
" losses.append(loss.item()) # track stats\n",
" return losses\n",
"\n",
"HIGH_LR = 3.0\n",
"no_bn_high = train_mlp(bn=False, lr=HIGH_LR, steps=BN_STEPS)\n",
"print(f\"no-BN @ lr={HIGH_LR}: final loss {no_bn_high[-1]:.4f}\")\n",
"print(\"contains NaN:\", any(math.isnan(v) for v in no_bn_high))"
]
},
{
"cell_type": "markdown",
"id": "07109945",
"metadata": {},
"source": [
"> **Interpretation.** The no-BN net at `lr=3.0` blows up. `final loss = nan` is not a bug in our code; it is the failure the BatchNorm paper exists to fix. We have reproduced the *negative* half of the claim. Now the positive half.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "bb7962e4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:24.638165Z",
"iopub.status.busy": "2026-06-10T20:39:24.637993Z",
"iopub.status.idle": "2026-06-10T20:39:25.283992Z",
"shell.execute_reply": "2026-06-10T20:39:25.283650Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" BN @ lr=3.0: final loss 0.0005\n",
"contains NaN: False\n",
"\n",
"[ ok ] BatchNorm tolerates a learning rate that makes the plain net diverge.\n"
]
}
],
"source": [
"bn_high = train_mlp(bn=True, lr=HIGH_LR, steps=BN_STEPS)\n",
"print(f\" BN @ lr={HIGH_LR}: final loss {bn_high[-1]:.4f}\")\n",
"print(\"contains NaN:\", any(math.isnan(v) for v in bn_high))\n",
"# the claim, as an assertion: BN stays finite at the LR where the plain net diverged.\n",
"assert all(math.isfinite(v) for v in bn_high), \"BN run should stay finite at the high LR\"\n",
"assert any(not math.isfinite(v) for v in no_bn_high), \"the no-BN run should have diverged (that is the point)\"\n",
"print(\"\\n[ ok ] BatchNorm tolerates a learning rate that makes the plain net diverge.\")"
]
},
{
"cell_type": "markdown",
"id": "669d0963",
"metadata": {},
"source": [
"### The figure: stability is the claim\n",
"\n",
"The committed figure is the evidence. We add a *low* learning rate (`lr=0.1`) to show the other half of the abstract's promise: at a learning rate so cautious the plain net barely moves, BatchNorm still trains. The claim is not that BN is faster; it is that BN widens the band of learning rates that work.\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "0bedb7e7",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:25.285038Z",
"iopub.status.busy": "2026-06-10T20:39:25.284961Z",
"iopub.status.idle": "2026-06-10T20:39:26.661918Z",
"shell.execute_reply": "2026-06-10T20:39:26.661556Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAABEEAAAGGCAYAAACUtJ9/AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAhPlJREFUeJzt3XlYVGX/P/D3sA2gAqKyqAi4pOKGghouoYnhlvr0ZObXQqnsV4kbZUrlXpKVS5ZLaS5lPZLmUlaWkrgUaYKYuS8oLmwugCACMvfvj2FODAwwgwOHmXm/rutcM3PmPud8bkTmPp+5F4UQQoCIiIiIiIiIyMxZyR0AEREREREREVFtYBKEiIiIiIiIiCwCkyBEREREREREZBGYBCEiIiIiIiIii8AkCBERERERERFZBCZBiIiIiIiIiMgiMAlCRERERERERBaBSRAiIiIiIiIisghMghARERERERGRRWAShEzK3LlzoVAocPPmzSrL+vj4YPz48dW6jo+PD4YNG1atY6n6jhw5Ajs7O1y5ckXuUGQ1fvx4+Pj4aO1TKBSYO3euLPHI4dSpU7CxscE///wjdyhERAbZsGEDFAoFLl++LHcoVA25ublwc3PD119/LXcohH/b/oZgG4KqwiQIUQ1QKBSIiIiotEy/fv2gUCikzcHBAZ07d8ayZcugUqlqKdJ/rVmzBsHBwXB3d4dSqYSvry/Cw8MNasT98ccf6NOnDxwdHeHh4YHJkycjNzdX7+PffvttjBkzBt7e3tWowcMZP3681r+HjY0NvLy88Oyzz+LUqVNaZePi4qRyCQkJOs9Vv3792grdLPn5+WHo0KGYPXu23KEQEZmkfv36oWPHjpWW0dxgajZbW1v4+Phg8uTJyMrKqp1Ay8jKysLLL7+MJk2aoF69eujfvz8SExP1OvbIkSN47bXXEBAQAFtbW4NvngHg448/RoMGDfDss88afOzD0iTQSm9ubm7o378/fv7553LlNWUWL15c4bmOHj1aG6HXKWxDUFVs5A6AqKacPXsWVlZ1O8/XvHlzREdHAwBu3ryJb775BtOmTUNmZibee++9Wo3l2LFj8PX1xfDhw9GwYUMkJydjzZo12LVrF44fP46mTZtWenxSUhIGDBiA9u3bY8mSJbh27Ro++ugjnD9/XucHt67j9+7diz/++MNYVTKYUqnE2rVrAQAPHjzAxYsXsXr1auzevRunTp3S+TOYO3cufvjhhxqPLT8/HzY2lvUn+5VXXsGQIUNw8eJFtGrVSu5wiIjM1qpVq1C/fn3k5eUhNjYWn3zyCRITE3Ho0KFajUOlUmHo0KE4fvw4pk+fjsaNG2PlypXo168fEhIS0KZNm0qP/+mnn7B27Vp07twZLVu2xLlz5wy6flFRET7++GNMmzYN1tbWD1OVhzJ//nz4+vpCCIH09HRs2LABQ4YMwQ8//KCzp/KHH36IV199FY6OjjJEWzexDUGVEkQmZM6cOQKAyMzMrNHreHt7i6FDh1b7eABi4sSJlZYJDg4WHTp00NqXn58vvL29RYMGDcSDBw+qfX1jOXr0qAAgoqOjqyw7ePBg4enpKbKzs6V9a9asEQDEL7/8UuXxkydPFi1atBAqleqhYq6ucePGiXr16pXbv2vXLgFAfP7559K+ffv2CQDC399fABAJCQl6ncuQWLy9vat9vLHk5ubKev3CwkLRsGFDMWvWLFnjICIyxPr16wUAkZycLGscutoZZVXUrho9erQAIA4fPlyTIZYTExMjAIgtW7ZI+zIyMoSLi4sYM2ZMlcenpaWJe/fuCSGEmDhxojD0Vmfbtm0CgLhw4YJhgRuJ5nfnr7/+0tp/+/ZtYWtrK/7v//5Pa3/ptsjixYv1Opex5Ofni+Li4ho5d2ma31FDsQ1BlanbX5MTVSArKwvjx4+Hi4sLnJ2dER4ejnv37mmV0TUnyN9//43g4GA4ODigefPmePfdd7F+/foKx+4eOnQIPXr0gL29PVq2bIkvv/yyBmsF2Nvbo3v37rh79y4yMjJq9Fr60MxLUVWX2JycHOzZswfPPfccnJycpP1hYWGoX78+vv322yqvtWPHDjz++OPluq5q5mfR59/i0qVLGDVqFFxdXeHo6IhHH30UP/74Y9UVrYSHhwcA6OyFMWnSJDRs2PCh5urYsWMHOnbsCHt7e3Ts2BHbt2/XWa70nCBbt26FQqHA/v37y5X77LPPoFAotMbBnjlzBk8//TRcXV1hb2+PwMBAfP/991rHabrN7t+/H6+99hrc3NzQvHlz6f0VK1agZcuWcHBwQI8ePXDw4EH069cP/fr10zpPQUEB5syZg9atW0OpVMLLywtvvvkmCgoKtMrt2bMHffr0gYuLC+rXr4+2bdvirbfe0ipja2uLfv36YefOnVX+HImI6rqVK1eiQ4cOUCqVaNq0KSZOnKj1+bp8+XJYW1tr7Vu8eDEUCgUiIyOlfcXFxWjQoAFmzJhRY7H27dsXAHDx4sUau4YuW7duhbu7O5566ilpX5MmTfDMM89g586d5T5LynJ3d4eDg0O1r79jxw74+PiU6zmgGeZ6/fp1jBw5EvXr10eTJk3wxhtvoLi4WKtsXl4eXn/9dXh5eUGpVKJt27b46KOPIISodlwuLi5wcHDQ2Rbp3bs3Hn/8cXzwwQfIz8+v1vmzsrIwbdo0+Pj4QKlUonnz5ggLC5Pm4NMMA968eTPeeecdNGvWDI6OjsjJycHt27fxxhtvoFOnTqhfvz6cnJwwePBgHD9+XDq/EAKNGzfW+j1WqVRwcXEp9zu/aNEi2NjYVDqcmm0IeliW1beazMYzzzwDX19fREdHIzExEWvXroWbmxsWLVpU4THXr19H//79oVAoEBUVhXr16mHt2rVQKpU6y1+4cAFPP/00XnzxRYwbNw7r1q3D+PHjERAQgA4dOtRU1XD58mUoFAq4uLhUWTY7OxtFRUVVlrO3t9d7jopbt26huLgYKSkpmD9/PgBgwIABlR5z4sQJPHjwAIGBgVr77ezs4O/vj2PHjlV6/PXr15GSkoJu3brpfF+ff4v09HT06tUL9+7dw+TJk9GoUSNs3LgRw4cPx9atW/Gf//xHr/prPvCLi4tx6dIlzJgxA40aNdLZ/dTJyQnTpk3D7NmzkZiYWGH8Ffn111/x3//+F35+foiOjsatW7cQHh6ulXzQZejQoVJyKTg4WOu9mJgYdOjQQRoHfvLkSfTu3RvNmjXDzJkzUa9ePXz77bcYOXIkvvvuu3I/l9deew1NmjTB7NmzkZeXB0DdTToiIgJ9+/bFtGnTcPnyZYwcORINGzbUilWlUmH48OE4dOgQXn75ZbRv3x4nTpzA0qVLce7cOezYsUOKadiwYejcuTPmz58PpVKJCxcu4Pfffy9X14CAAOzcuRM5OTlaCTYiIlMyd+5czJs3DyEhIXj11Vdx9uxZrFq1Cn/99Rd+//132Nraom/fvlCpVDh06JD0mXPw4EFYWVnh4MGD0rmOHTuG3NxcPPbYYzUWr+aLoYYNG1ZZ9t69e+W+iNLF2tq6yvMdO3YM3bp1KzecuUePHvj8889x7tw5dOrUqcprVdcff/xR4Wd5cXExQkND0bNnT3z00UfYu3cvFi9ejFatWuHVV18FoL7ZHz58OPbt24cXX3wR/v7++OWXXzB9+nRcv34dS5cu1SuO7Oxs3Lx5E0IIZGRk4JNPPkFubi6ee+45neXnzp2Lxx57DKtWrdJKNOgjNzcXffv2xenTp/HCCy+gW7duuHnzJr7//ntcu3YNjRs3lsouWLAAdnZ2eOONN1BQUAA7OzucOnUKO3bswKhRo+Dr64v09HR89tlnCA4OloYSKxQK9O7dGwcOHJDO9ffffyM7OxtWVlb4/fffMXToUADq3/muXbtW2G5lG4KMQt6OKESG0XSJe+GFF7T2/+c//xGNGjXS2uft7S3GjRsnvZ40aZJQKBTi2LFj0r5bt24JV1fXct1Wvb29BQBx4MABaV9GRoZQKpXi9ddfrzJO6Dkcpl27diIzM1NkZmaKM2fOiOnTpwsAeg/FCQ4OFgCq3Er/HKqiVCql4xo1aiSWL19e5TFbtmwp9/PSGDVqlPDw8Kj0+L179woA4ocffij3nr7/FlOnThUAxMGDB6V9d+/eFb6+vsLHx6fKLpvjxo3T+bNr1qxZueEumuEwW7ZsEVlZWaJhw4Zi+PDhWufSZziMv7+/8PT0FFlZWdK+X3/9VQAoNxwGgJgzZ470esyYMcLNzU1r2FRqaqqwsrIS8+fPl/YNGDBAdOrUSdy/f1/ap1KpRK9evUSbNm2kfZpus3369NE6Z0FBgWjUqJHo3r27KCoqkvZv2LBBABDBwcHSvq+++kpYWVlp/RsIIcTq1asFAPH7778LIYRYunSp3sPavvnmG1m6ZBMRVVfZ4TAZGRnCzs5OPPHEE1qfRZ9++qkAINatWyeEEKK4uFg4OTmJN998Uwih/lvdqFEjMWrUKGFtbS3u3r0rhBBiyZIlwsrKSty5c6fSOAwZDnP27FmRmZkpLl++LNatWyccHBxEkyZNRF5eXpX11Zyjqk2fYZ716tUr18YTQogff/xRABC7d++u8hwahg6HKSoqEgqFQmc7T9NGKP35KoQQXbt2FQEBAdLrHTt2CADi3Xff1Sr39NNPC4VCUeUwG83vTtlNqVSKDRs2lCtfur3Zv39/4eHhIQ0H0nc4zOzZswUAsW3btnLvaYYoa9o9LVu2lM6vcf/+/XJtrOTkZKFUKrV+Xh9++KGwtrYWOTk5Qgghli9fLry9vUWPHj3EjBkzhBDq/wMuLi5i2rRp0nFlh8OwDUHGwJ4gZJJeeeUVrdd9+/bF9u3bK8307t69G0FBQfD395f2ubq6YuzYsfjkk0/Klffz85O6gwLq7pht27bFpUuXjFMJqIcpNGnSRGvf8OHD8cUXX+h1/OLFi3Hnzp0qy1U1qWlpP//8M+7fv4/Tp09j06ZNUm+Aymi6X+rqVWNvb19l98xbt24BqPgbJ33+LX766Sf06NEDffr0kfbVr18fL7/8MqKionDq1KkqZ8m3t7eXJjlVqVS4fPkylixZgiFDhuDAgQN45JFHyh3j7OyMqVOnYs6cOTh27Bi6du1a6TU0UlNTkZSUhJkzZ8LZ2VnaP3DgQPj5+VX5cx89ejT+97//IS4uTuqps3XrVqhUKowePRoAcPv2bfz222+YP38+7t69i7t370rHh4aGYs6cObh+/TqaNWsm7Z8wYYLWZHBHjx7FrVu3EB0drdUNd+zYsZg2bZpWTFu2bEH79u3Rrl07rWWsH3/8cQDAvn370KtXL6mX086dOxEeHl7pBMaa3wl9lsUmIqqL9u7di8LCQkydOlXr792ECRPw1ltv4ccff5T+Fvbq1Uv6tvz06dO4desWZs6cie+++w7x8fEYOHAgDh48iI4dO+rVY1Rfbdu21XrdqVMnrF+/Xq+JNsPCwrQ+eyuizzCV/Pz8CtsSmvdryu3btyGEqLS3iq7251dffSW9/umnn2BtbY3JkydrlXv99dexdetW/Pzzz1WuHgioh6Bq2hzp6enYtGkTXnrpJTRo0EBrqFBpc+fORXBwMFavXl3u87ky3333Hbp06aKzx2zZIcrjxo0r9+9Y+t+ruLgYWVlZ0hCV0qv69O3bF8XFxfjjjz8QGhqKgwcPom/fvnB3d5d6Ov3zzz/IysrSavOVxTYEGYNFJ0EOHDiADz/8EAkJCUhNTcX27dsxcuRIvY+/f/8+XnnlFSQkJOD06dMYNmyY1N27tLi4OERGRuLkyZPw8vLCO++8U26uCjJMixYttF5r/sjduXOnwiTIlStXEBQUVG5/69at9bqG5jr6JB305ePjgzVr1kClUuHixYt47733kJmZKX3YVyUgIMBosWj0798fADB48GCMGDECHTt2RP369Sv90NZ8IOoaq3v//n29x+eKCsbL6vNvceXKFfTs2bNcufbt20vvd+zYEbdv30ZhYaFW7JokhLW1NUJCQrSOHzJkCNq0aYOoqCh89913OuObMmUKli5dirlz5+o99vTKlSsAoHOm+7INB10GDRoEZ2dnxMTESEmQmJgY+Pv7Sw2nCxcuQAiBWbNmYdasWTrPk5GRoZUE8fX11Rln2f8nNjY20pwxGufPn8fp06fLJfZKXwtQJ3DWrl2Ll156CTNnzsSAAQPw1FNP4emnny7XmNH8TlRnmUMiorpA83e0bKLBzs4OLVu2lN4H1DeKc+fORX5+Pg4ePAhPT09069YNXbp0wcGDBzFw4EAcOnQIzzzzjFFj/O677+Dk5ITMzEwsX74cycnJen92t2zZEi1btjRKHA4ODhW2JTTv17SK2iL29vblPt90tUWaNm2KBg0aaJUr3RYB1ENQSs95YW1trXXuHj16aA0xHjNmDLp27YqIiAgMGzYMdnZ25eJ77LHH0L9/f3zwwQflkjWVuXjxIv773//qVbZsGwFQf2n08ccfY+XKlUhOTtaaI6VRo0bS827dusHR0REHDx6UkiDz5s2Dh4cHPvnkE9y/f19KhlSWVGMbgozBopMgeXl56NKlC1544YUKs6qVKS4uhoODAyZPnlzhzVFycjKGDh2KV155BV9//TViY2Px0ksvwdPTE6GhoQ9bBYtV0bJlFX1w1dVr1KtXT+umu3fv3ujWrRveeustLF++vMrjy97QV6T0jb4hWrVqha5du+Lrr7+uNAni6ekJQN27oazU1NQqe6JoPiQrSjAZ89/iqaee0ppQdNy4cdiwYUOF5Zs3b462bdtqjWMtS9MbZO7cuVXOf2IsSqUSI0eOxPbt27Fy5Uqkp6fj999/x8KFC6UyKpUKAPDGG29U+PembHLjYRqYKpUKnTp1wpIlS3S+7+XlJV3jwIED2LdvH3788Ufs3r0bMTExePzxx/Hrr79q/XtrfidKj0kmIjJXffr0QVFREeLj46VvygF1cuTgwYM4c+YMMjMzK/2mvDoee+wx6e/sk08+iU6dOmHs2LFISEio9Jt2oPwNfUXK3ujr4unpWWFbAjCsZ6uhXF1doVAoDG6LVMdHH32EefPmSa+9vb11TtCvYWVlhf79++Pjjz/G+fPnK5ybbs6cOejXrx8+++wzo/YU0tDVRli4cCFmzZqFF154AQsWLICrqyusrKwwdepUqR0CqCcq7dmzJw4cOIALFy4gLS1N6glSVFSEw4cP4+DBg2jXrl2lvydsQ5AxWHQSZPDgwRg8eHCF7xcUFODtt9/G//73P2RlZaFjx45YtGiRtBpCvXr1sGrVKgDA77//rnMFjdWrV8PX1xeLFy8GoM4EHzp0CEuXLmUSpJZ5e3vjwoUL5fbr2ieXzp0747nnnsNnn32GN954Q2cPiNLK3tBXpKob/crk5+dXORt7x44dYWNjg6NHj2p9O1VYWIikpKQqv7Fq164dAHXSsLq8vb1x9uzZcvvPnDkjvQ+UH0KkT4PqwYMHVTbwpk6dimXLlmHevHl6NTw08Zw/f77ce7rqocvo0aOxceNGxMbG4vTp0xBCSENhAEjfzNna2pbr4aIvTZwXLlyQegkB6p/J5cuX0blzZ2lfq1atcPz4cQwYMKDKb12srKwwYMAADBgwAEuWLMHChQvx9ttvY9++fVqxJicnw8rKSudQJCIiU6D5O3r27FmtHhOFhYVITk7W+pvXo0cP2NnZ4eDBgzh48CCmT58OQJ2kWLNmDWJjY6XXNaV+/fqYM2cOwsPD8e233+LZZ5+ttHzZG/qKVHWjDwD+/v44ePAgVCqVVvLl8OHDcHR0rNHPAhsbG7Rq1eqh2yJ79+7F3bt3tXqDlG2LlB1CpM8XEA8ePACAStsjwcHB6NevHxYtWoTZs2frFXOrVq20VpQz1NatW9G/f/9yQ7mzsrLKJR/69u2LRYsWYe/evWjcuDHatWsHhUKBDh06SL/zuiaiL4ttCHpYXCK3EhEREYiPj8fmzZvx999/Y9SoURg0aJDOm5aKxMfHl7v5CA0NRXx8vLHDpSpofu5JSUnSvtu3b+Prr7+WLygd3nzzTRQVFVX4bXppixcvxp49e6rc3nzzzUrP8+DBA53ffBw5cgQnTpwot+rLmTNnkJKSIr12dnZGSEgINm3apDXvxFdffYXc3FyMGjWq0us3a9YMXl5eOHr0aJV1rsiQIUNw5MgRrf9beXl5+Pzzz+Hj4wM/Pz8A6iFEISEh0qbZX5Fz587h7Nmz6NKlS6XlNL1Bdu7cqfU7VhFPT0/4+/tj48aNyM7Olvbv2bMHp06dqvJ4AAgJCYGrqytiYmIQExODHj16aHVVdXNzk74R0vXNWmZmZpXXCAwMRKNGjbBmzRqpAQYAX3/9dbnfmWeeeQbXr1/HmjVryp0nPz9fmufk9u3b5d7XzNVTNuGWkJCADh06VKsnExFRXRASEgI7OzssX75cqwfjF198gezsbGlVDEA95KJ79+743//+h5SUFK2eIPn5+Vi+fDlatWol9cCsKWPHjkXz5s0rXXVPIywsTK+2iD7traeffhrp6enYtm2btO/mzZvYsmULnnzySa35Jy5evGj0JXyDgoIeui1SXFyMTz/9VGv/0qVLoVAopC9fW7ZsqdUW6d27d6XnLSoqwq+//go7OztpaE1F5s6di7S0NHz++ed6xfzf//4Xx48fx/bt28u9p0+PW2tr63LltmzZguvXr5cr27dvXxQUFGDZsmXo06eP9IWJZm6VGzduVNnLiW0IMgaL7glSmZSUFKxfvx4pKSnSN8VvvPEGdu/ejfXr12t1Oa9MWloa3N3dtfa5u7sjJycH+fn5tTK2kdTefPNNbNq0CQMHDsSkSZOkJXJbtGiB27dvG3284NGjR/Huu++W29+vX79Kxzr6+flhyJAhWLt2LWbNmqU1nrIsY80JkpubCy8vL4wePRodOnRAvXr1cOLECaxfvx7Ozs7l5pNo3749goODERcXJ+1777330KtXLwQHB+Pll1/GtWvXsHjxYjzxxBMYNGhQlTGMGDEC27dvhxCiWv8WM2fOxP/+9z8MHjwYkydPhqurKzZu3Ijk5GR89913VXbnBdTJoE2bNgH4d2LU1atXQ6VSYc6cOVUer5kb5Pjx46hXr16V5aOjozF06FD06dMHL7zwAm7fvo1PPvkEHTp00Ktrsa2tLZ566ils3rwZeXl5+Oijj8qVWbFiBfr06YNOnTphwoQJaNmyJdLT0xEfH49r167h+PHjlV7Dzs4Oc+fOxaRJk/D444/jmWeeweXLl7Fhwwa0atVK69/q+eefx7fffotXXnkF+/btQ+/evVFcXIwzZ87g22+/xS+//ILAwEDMnz8fBw4cwNChQ+Ht7Y2MjAysXLkSzZs31/q/UVRUhP379+O1116r8mdBRFRXNWnSBFFRUZg3bx4GDRqE4cOH4+zZs1i5ciW6d+9ebtnTvn374v3334ezs7O0HKybmxvatm2Ls2fPGjSvXGZmps62iK+vL8aOHVvhcba2tpgyZQqmT5+O3bt3V/o5bsw5QZ5++mk8+uijCA8Px6lTp9C4cWOsXLkSxcXF5XqbaObDKt275MqVK9JEpZpkhqb+3t7eeP755yu9/ogRI/DVV1/h3Llz1eo98OSTT6J///54++23cfnyZXTp0gW//vordu7cialTp6JVq1Z6nefnn3+Weo9kZGTgm2++wfnz5zFz5swql3oNDg5GcHCwXj2FAWD69OnYunUrRo0ahRdeeAEBAQG4ffs2vv/+e6xevbrKL4GGDRuG+fPnIzw8HL169cKJEyfw9ddf6/ydCAoKgo2NDc6ePYuXX35Z2q9Z3hdAlUkQtiHIKORYkqYuAiC2b98uvd61a5cAIOrVq6e12djYiGeeeabc8ePGjRMjRowot79NmzZi4cKFWvs0y3yVXWKKqqZZJqvsslhll6MTovwSuUIIcezYMdG3b1+hVCpF8+bNRXR0tFi+fLkAINLS0rSO1bVMbXBwsNaSoBVBJUvELViwQDpXRUvXxcXFlVsStSYVFBSIKVOmiM6dOwsnJydha2srvL29xYsvvqj1M9VAmaVRNQ4ePCh69eol7O3tRZMmTcTEiROlpdCqkpiYWG6JWyEM+7e4ePGiePrpp4WLi4uwt7cXPXr0ELt27dLr+rqWyHVychIDBgwQe/fu1SpbeoncsjS/o/oskSuEEN99951o3769UCqVws/PT2zbtk2MGzeuyiVyNfbs2SMACIVCIa5evarzGhcvXhRhYWHCw8ND2NraimbNmolhw4aJrVu3SmWqWkpPs5SdUqkUPXr0EL///rsICAgQgwYN0ipXWFgoFi1aJDp06CCUSqVo2LChCAgIEPPmzRPZ2dlCCCFiY2PFiBEjRNOmTYWdnZ1o2rSpGDNmjDh37pzWuX7++WcBQJw/f76qHyMRUZ2hq00ihHpJ3Hbt2glbW1vh7u4uXn31VZ3L3GraiYMHD9ba/9JLLwkA4osvvtArjuDg4ArbIgMGDBBCVNyuEkKI7Oxs4ezsrFe7x5hu374tXnzxRdGoUSPh6OgogoODdX42eXt7l/us1Hw+69r0qUdBQYFo3Lix1FbTGDdunM7P9bLLtwohxN27d8W0adNE06ZNha2trWjTpo348MMPpeVmK6NriVx7e3vh7+8vVq1aVe4cKLVEbkU/h6qWyBVCiFu3bomIiAjRrFkzYWdnJ5o3by7GjRsnbt68qXU+Xe2e+/fvi9dff114enoKBwcH0bt3bxEfH19hm7l79+7llq29du2aACC8vLzKlS/7M2YbgoxBIYQRZ3k0YQqFQmt1mJiYGIwdOxYnT54sNxFS/fr14eHhobVv/PjxyMrKKrc6zGOPPYZu3bph2bJl0r7169dj6tSpWl3gST5Tp07FZ599htzcXKNOekWGGzBgAJo2baq13BzVTSqVCk2aNMFTTz2lc/iLMYwcOVL620xERFQbFixYgPXr1+P8+fNsF5owtiGoMpwTpAJdu3ZFcXExMjIy0Lp1a62tbAKkMkFBQdIkVhp79uzRuVQr1byy68vfunULX331Ffr06cMPujpg4cKFiImJ0VoukOR3//79cuN9v/zyS9y+fVuaKNrYTp8+jV27dmHBggU1cn4iIiJdpk2bhtzcXGzevFnuUKia2Iagqlj0nCC5ublaK4MkJycjKSkJrq6ueOSRRzB27FiEhYVh8eLF6Nq1KzIzMxEbG4vOnTtLk1idOnUKhYWFuH37Nu7evStNiKiZoOeVV17Bp59+ijfffBMvvPACfvvtN3z77bf48ccfa7u6BHVSql+/fmjfvj3S09PxxRdfICcnp9ycFySPnj176rXkL9WuP//8E9OmTcOoUaPQqFEjJCYm4osvvkDHjh2rnPS2utq3b681ESsREVFtqF+/PjIyMuQOgx4C2xBUFYseDhMXF6e15KOGZjnRoqIivPvuu/jyyy9x/fp1NG7cGI8++ijmzZsnTVTl4+Oj81vr0j/WuLg4TJs2DadOnULz5s0xa9Ysgya1IuN56623sHXrVly7dg0KhQLdunXDnDlzqr18KJEluHz5MiZPnowjR47g9u3bcHV1xZAhQ/D+++/Dzc1N7vCIiIiIiPRm0UkQIiIiIiIiIrIcnBOEiIiIiIiIiCwCkyBEREREREREZBEsbmJUlUqFGzduoEGDBlAoFHKHQ0REZLaEELh79y6aNm0KKyvL/d6FbQ8iIqKap2+7w+KSIDdu3ICXl5fcYRAREVmMq1evonnz5nKHIRu2PYiIiGpPVe0Oi0uCNGjQAID6B+Pk5CRzNEREROYrJycHXl5e0mevpWLbg4iIqObp2+6wuCSIphuqk5MTGyJERES1wNKHgLDtQUREVHuqandY7gBdIiIiIiIiIrIoTIIQERERERERkUVgEoSIiIiIiIiILILFzQlCRESWo7i4GEVFRXKHYbZsbW1hbW0tdxhEREREemMShIiIzI4QAmlpacjKypI7FLPn4uICDw8Pi5/8lIiIiEyDrEmQVatWYdWqVbh8+TIAoEOHDpg9ezYGDx5c4TFbtmzBrFmzcPnyZbRp0waLFi3CkCFDailiIiIyBZoEiJubGxwdHXmDXgOEELh37x4yMjIAAJ6enjJHRERERFQ1WZMgzZs3x/vvv482bdpACIGNGzdixIgROHbsGDp06FCu/B9//IExY8YgOjoaw4YNwzfffIORI0ciMTERHTt2lKEGRERU1xQXF0sJkEaNGskdjllzcHAAAGRkZMDNzY1DY4iIiKjOUwghhNxBlObq6ooPP/wQL774Yrn3Ro8ejby8POzatUva9+ijj8Lf3x+rV6/W6/w5OTlwdnZGdnY2nJycjBY3ERHVDffv30dycjJ8fHykm3SqOfn5+bh8+TJ8fX1hb2+v9R4/c9X4cyAiIqp5+n7e1pnVYYqLi7F582bk5eUhKChIZ5n4+HiEhIRo7QsNDUV8fHxthEhERCaEQ2BqB3/OREREZEpknxj1xIkTCAoKwv3791G/fn1s374dfn5+OsumpaXB3d1da5+7uzvS0tIqPH9BQQEKCgqk1zk5OcYJnIiIiIiIiIhMiuw9Qdq2bYukpCQcPnwYr776KsaNG4dTp04Z7fzR0dFwdnaWNi8vL6Odm+qG3L//xrFevXD62WflDoWIiKhWvLca6PUs8L9dVZclIiKif8meBLGzs0Pr1q0REBCA6OhodOnSBR9//LHOsh4eHkhPT9fal56eDg8PjwrPHxUVhezsbGm7evWqUeMn+any81F4/ToKS1YoICKifykUCmmzsbFBixYtEBkZqdVLcsOGDVAoFBg0aJDWsVlZWVAoFIiLi6vlqKkqd3OB6+lA5h25IyEiIjItsidBylKpVFoNs9KCgoIQGxurtW/Pnj0VziECAEqlEk5OTlobmRmVCgCgsKpzv85ERHXC+vXrkZqaiuTkZKxcuRJfffUV3n33Xa0yNjY22Lt3L/bt2ydTlGQIp/rqx5xceeMgIiIyNbLeNUZFReHAgQO4fPkyTpw4gaioKMTFxWHs2LEAgLCwMERFRUnlp0yZgt27d2Px4sU4c+YM5s6di6NHjyIiIkKuKlAdIIqL1U+YBCEiE9avXz9MnjwZb775JlxdXeHh4YG5c+dqlUlJScGIESNQv359ODk54ZlnninXQ1IXFxcXeHh4wMvLC8OGDcOIESOQmJioVaZevXp44YUXMHPmTGNWi2oIkyBERETVI+vEqBkZGQgLC0NqaiqcnZ3RuXNn/PLLLxg4cCAAdWPPqtSNba9evfDNN9/gnXfewVtvvYU2bdpgx44d6Nixo1xVoDpAaHqCWFvLHAkR1UVCCKjy82W5tpWDg0Grp2zcuBGRkZE4fPgw4uPjMX78ePTu3RsDBw6ESqWSEiD79+/HgwcPMHHiRIwePdqg4Srnzp3Db7/9hvHjx5d7b+7cuWjdujW2bt2Kp59+Wu9zUu1jEoSIiKh6ZE2CfPHFF5W+r6tRN2rUKIwaNaqGIiKTVJIEYU8QItJFlZ+Po+3by3LtwNOnYe3oqHf5zp07Y86cOQCANm3a4NNPP0VsbCwGDhyI2NhYnDhxAsnJydIk319++SU6dOiAv/76C927d6/wvGPGjIG1tTUePHiAgoICDBs2TKunpUbTpk0xZcoUvP322xg5cqRhlaVaxSQIERFR9fCukUyeZjgM5wQhIlPXuXNnrdeenp7IKJn0+fTp0/Dy8tJa5czPzw8uLi44ffp0peddunQpkpKScPz4cezatQvnzp3D888/r7PsjBkzkJmZiXXr1j1kbagmMQlCRERUPbL2BCEyCk1PEA6HISIdrBwcEFhFkqAmr20IW1tbrdcKhQIqzd+4h+Dh4YHWrVsDUC9Nf/fuXYwZMwbvvvuutF/DxcUFUVFRmDdvHoYNG/bQ16aawSQIERFR9TAJQiZPcHUYIqqEQqEwaEhKXdW+fXtcvXoVV69elXqDnDp1CllZWfDz8zPoXNYlSeP8CuZKmTRpEpYvX17hkvUkPyZBiIiIqodJEDJ9HA5DRBYgJCQEnTp1wtixY7Fs2TI8ePAAr732GoKDgxEYGFjpsVlZWUhLS4NKpcL58+cxf/58PPLII2hfwVwp9vb2mDdvHiZOnFgTVSEjcC6VBBECMGD+XSIiIovGu0YyeYLDYYjIAigUCuzcuRMNGzbEY489hpCQELRs2RIxMTFVHhseHg5PT080b94cY8aMQYcOHfDzzz/Dxqbi70LGjRuHli1bGrMKdcaKFSvg4+MDe3t79OzZE0eOHKmwbL9+/aBQKMptQ4cOrcWIy9P0BClWAffuyxoKERGRSWFPEDJ9HA5DRGZA14poO3bs0HrdokUL7Ny506DzCiGqLDN+/PhyS+ZaW1vj5MmTBl3LFMTExCAyMhKrV69Gz549sWzZMoSGhuLs2bNwc3MrV37btm0oLCyUXt+6dQtdunSRfaU6eyVgawMUPVD3Bqln2PQzREREFot3jWTyNKvDcIlcIiKqypIlSzBhwgSEh4fDz88Pq1evhqOjY4Wr4bi6usLDw0Pa9uzZA0dHR9mTIAoF5wUhIiKqDt41ksmTJkblcBgiIqpEYWEhEhISEBISIu2zsrJCSEgI4uPj9TrHF198gWeffRb16tWrqTD1xiQIERGR4TgchkwfJ0YlIiI93Lx5E8XFxXB3d9fa7+7ujjNnzlR5/JEjR/DPP//giy++qLRcQUEBCgoKpNc5OTnVC7gKTIIQEREZjneNZPKkiVGZBCEiohr0xRdfoFOnTujRo0el5aKjo+Hs7CxtmiWNjU2TBMlmEoSIiEhv7AlCpo/DYYiISA+NGzeGtbU10tPTtfanp6fDw8Oj0mPz8vKwefNmzJ8/v8rrREVFITIyUnqdk5Nj9ERIzuHDUN72BNACN/YnIOPWeaBk5RoA/66ZW8GjorL3Sr8uWwYoX6aCshXGUtl7us5Rdv1fA+JVVHZsVXHqOoc+cerzM9P3Z2Xgz6xcmTJxV/pvW1Es+pTRFUMFx+rze1ruHERERsQkCJk8ToxKRET6sLOzQ0BAAGJjYzFy5EgAgEqlQmxsLCIiIio9dsuWLSgoKMBzzz1X5XWUSiWUSqUxQq5Q5pYtEEndAKf/Q8oP+5D81Sc1ej0i2eibrNGRMKs02VJZ0qfsdYyVKKsohtLHKhSAldW/19WUsbL697UmkVTRayurfxOdpd/XnKN0vUr2KSq5VqXX08Ra9nqlziHFUvZa1a1bmeuVO76S65U7l6aMlZXWa+m55nqlXpcrX6pMnTintbX2z4nKYRKETJ7gErlERKSnyMhIjBs3DoGBgejRoweWLVuGvLw8hIeHAwDCwsLQrFkzREdHax33xRdfYOTIkWjUqJEcYZdTr0MHuF50BbKAYt9OcHEdAGiWQy55FGVeS4Qo/56Ox3LLK1dQVghR8Xu6jq0qztLXraqMrhgqiFdUEoM+8Vb5M9OnjK6ffWV1ray+ZY/V9+eqx7LZdUoFcetTCxOrKZHxVZZo0by2tv73saLnmuSKjudVvS8lZXSct9GwYWgQGFjrPxYmQcj0aeYE4XAYIiKqwujRo5GZmYnZs2cjLS0N/v7+2L17tzRZakpKCqzKJNXPnj2LQ4cO4ddff5UjZJ08wsPhYwvgC8C6dyjaTg+VOyQyQQYllypKRtRkgkuPhF6VcepKHFVVprIYKjmHvomniuKV9pds5V6rVOXLa/aVvr5KpfvY0nUqfT4h1AmjUufTOh7QKqt17bKxlz5e816ZYyusa8n5Kqx3Bees9OemUv1bN1311pRRqf6NveS5KPNa+rmWKi/FraO8dB0d5at7vXL/B6qiOVbzu2fY0TXOsW1bJkGIqkNwdRgiIjJAREREhcNf4uLiyu1r27Zt+RugOoCrw9DDqnAoiOb9WoyFiKpWOtn00ImV4mKpjCguBoqLdT+v6n0dZXUep1JJz1FSxrFDB1l+jkyCkOnjxKhERGSBmAQhIrIsWnOxyB2MCeNX52TyuEQuEZmL8ePHQ1EycZtCoUCjRo0waNAg/P3331IZhUIBe3t7XLlyRevYkSNHYvz48bUcMcmJSRAiIiLD8a6RTB6HwxCRORk0aBBSU1ORmpqK2NhY2NjYYNiwYVplFAoFZs+eLVOEVFcwCUJERGQ43jWS6ePEqERkRpRKJTw8PODh4QF/f3/MnDkTV69eRWZmplQmIiICmzZtwj///CNjpCQ3JkGIiIgMxzlByORxiVwiqowQQP59ea7tYF/hfIN6yc3NxaZNm9C6dWutpVl79+6Nc+fOYebMmdi1a5cRIiVT5FwqCSLEw/2uERERWQomQcj0lQyH4ZwgRKRL/n2g/RB5rn36J8DRwbBjdu3ahfr11Xe3eXl58PT0xK5du8ot2xodHY3OnTvj4MGD6Nu3r7FCJhOi6QlSrALu3QfqGfi7RkREZIl410gmT3B1GCIyI/3790dSUhKSkpJw5MgRhIaGYvDgweUmQvXz80NYWBhmzpwpU6QkN3slYFvydRaHxBAREemHPUHI9HE4DBFVwsFe3SNDrmsbql69emjdurX0eu3atXB2dsaaNWvw7rvvapWdN28eHnnkEezYseMhIyVTpFCoe4PcylInQTybyB0RERFR3cckCJk8weEwRFQJhcLwISl1iUKhgJWVFfLz88u95+XlhYiICLz11lto1aqVDNGR3EonQYiIiKhqvGskkyctkcvhMERkBgoKCpCWloa0tDScPn0akyZNQm5uLp588kmd5aOionDjxg3s3bu3liOluoArxBARERmGSRAyfZolctkThIjMwO7du+Hp6QlPT0/07NkTf/31F7Zs2YJ+/frpLO/q6ooZM2bg/n2ZlsAhWWmSINlMghAREemFw2HI5HGJXCIyFxs2bMCGDRsqLSOEKLcvKioKUVFRNRQV1WVO9dSP7AlCRESkH941kunjcBgiIrJQHA5DRERkGCZByOQJDochIiILxSQIERGRYXjXSKaPw2GIiMhCMQlCRERkGN41ksmTlsjlcBgiIrIwTIIQEREZhkkQMn2aniBMghARkYVhEoSIiMgwTIKQyZPmBFEo5A2EiOoUleZvA9Uo/pzlxSQIERGRYbhELpk8wdVhiKgUOzs7WFlZ4caNG2jSpAns7OygYJLU6IQQKCwsRGZmJqysrGBnZyd3SBaJSRAiIiLDMAlCpo/DYYioFCsrK/j6+iI1NRU3btyQOxyz5+joiBYtWsCKk1PLgkkQIiIiwzAJQiaPw2GIqCw7Ozu0aNECDx48QLFm8mQyOmtra9jY2LCnjYycSyVBhOBHIRERUVVkTYJER0dj27ZtOHPmDBwcHNCrVy8sWrQIbdu2rfCYDRs2IDw8XGufUqnE/fv3azpcqqM4HIaIdFEoFLC1tYWtra3coRDVGE1PkGIVcO8+UM9B3niIiIjqOln7ru7fvx8TJ07En3/+iT179qCoqAhPPPEE8vLyKj3OyckJqamp0nblypVaipjqJE1PECZBiIjIwtgrAduSr7Q4JIaIiKhqsvYE2b17t9brDRs2wM3NDQkJCXjssccqPE6hUMDDw6OmwyMToRkOo+B4dCIisjAKhbo3yK0sdRLEs4ncEREREdVtdequMTs7GwDg6upaabnc3Fx4e3vDy8sLI0aMwMmTJyssW1BQgJycHK2NzIxmvD+TIEREZIE4OSoREZH+6sxdo0qlwtSpU9G7d2907NixwnJt27bFunXrsHPnTmzatAkqlQq9evXCtWvXdJaPjo6Gs7OztHl5edVUFUgmnBOEiIgsGZMgRERE+qszSZCJEyfin3/+webNmystFxQUhLCwMPj7+yM4OBjbtm1DkyZN8Nlnn+ksHxUVhezsbGm7evVqTYRPchJC/cieIEREZIE0SZAsdnYlIiKqUp24a4yIiMCuXbuwb98+NG/e3KBjbW1t0bVrV1y4cEHn+0qlEk5OTlobmRepJwiTIEREpIcVK1bAx8cH9vb26NmzJ44cOVJp+aysLEycOBGenp5QKpV45JFH8NNPP9VStFVr0lD9mHFb3jiIiIhMgax3jUIIREREYPv27fjtt9/g6+tr8DmKi4tx4sQJeHp61kCEZAo4HIaIiPQVExODyMhIzJkzB4mJiejSpQtCQ0ORkZGhs3xhYSEGDhyIy5cvY+vWrTh79izWrFmDZs2a1XLkFfMomQw1LVPeOIiIiEyBrKvDTJw4Ed988w127tyJBg0aIC0tDQDg7OwMBwf1QvdhYWFo1qwZoqOjAQDz58/Ho48+itatWyMrKwsffvghrly5gpdeekm2epDMOByGiIj0tGTJEkyYMAHh4eEAgNWrV+PHH3/EunXrMHPmzHLl161bh9u3b+OPP/6Ara0tAMDHx6c2Q66Se2P1Y9pNeeMgIiIyBbLeNa5atQrZ2dno168fPD09pS0mJkYqk5KSgtTUVOn1nTt3MGHCBLRv3x5DhgxBTk4O/vjjD/j5+clRBaoDOByGiIj0UVhYiISEBISEhEj7rKysEBISgvj4eJ3HfP/99wgKCsLEiRPh7u6Ojh07YuHChSjWrExWB3iUJEHSmQQhIiKqkqw9QYTmG/xKxMXFab1eunQpli5dWkMRkUnSNEQ5HIaIiCpx8+ZNFBcXw93dXWu/u7s7zpw5o/OYS5cu4bfffsPYsWPx008/4cKFC3jttddQVFSEOXPm6DymoKAABQUF0uucnJqdsdSDPUGIiIj0xq/OyeRpkmnsCUJERMamUqng5uaGzz//HAEBARg9ejTefvttrF69usJjoqOj4ezsLG1eXl41GqNnyZwgGbf+/V6AiIiIdONdI5k+DochIiI9NG7cGNbW1khPT9fan56eDg8PD53HeHp64pFHHoF1qd6G7du3R1paGgoLC3UeExUVhezsbGm7evWq8SqhQ+OGgLUVUKwCbt6p0UsRERGZPN41kskTHA5DRER6sLOzQ0BAAGJjY6V9KpUKsbGxCAoK0nlM7969ceHCBahUKmnfuXPn4OnpCTs7O53HKJVKODk5aW01ydoaaOKqfs4hMURERJVjEoRMn2Y4DJMgRERUhcjISKxZswYbN27E6dOn8eqrryIvL09aLSYsLAxRUVFS+VdffRW3b9/GlClTcO7cOfz4449YuHAhJk6cKFcVdOK8IERERPqRdWJUImOQeoIoFPIGQkREdd7o0aORmZmJ2bNnIy0tDf7+/ti9e7c0WWpKSgqsSg2v9PLywi+//IJp06ahc+fOaNasGaZMmYIZM2bIVQWd3JsAOAOkZcodCRERUd3GJAiZPGmJXPYEISIiPURERCAiIkLne2VXpQOAoKAg/PnnnzUc1cPhMrlERET64XAYMn0cDkNERBaOw2GIiIj0wyQImTwOhyEiIkvnUbJMbiqHwxAREVWKSRAyfRwOQ0REFo7DYYiIiPTDJAiZPKFZtpBJECIislAcDkNERKQfJkHI5GmSIAoOhyEiIgulSYLk5QN38+SNhYiIqC5jEoRMn2ZOEPYEISIiC+XoADjVUz9nbxAiIqKKMQlCJk/qCcIkCBERWTB3zgtCRERUJSZByPRp5gThcBgiIrJgmhVi0rhCDBERUYWYBCGTJ7g6DBERkTQvSCp7ghAREVWISRAyeRwOQ0RExJ4gRERE+mAShEyfZjiMFX+diYjIcnlwThAiIqIq8a6RTJ40HIZJECIismCaJAhXhyEiIqoY7xrJ9Gl6gnA4DBERWTB3JkGIiIiqxCQImTxpThD2BCEiIgum6Qly8w5Q9EDeWIiIiOoq3jWS6eNwGCIiIjRyAWxtACGAjFtyR0NERFQ38a6RTJqmFwgADochIiKLZmUFuDVSP+eQGCIiIt2YBCGTppkUFeASuURERJ4ly+TeSJc3DiIiorqKSRAybaV7gigU8sVBRERUB3h5qh+vMQlCRESkE5MgZNJKD4dhTxAiIrJ0Xh7qx6up8sZBRERUVzEJQqaNw2GIiIgkzZkEISIiqhSTIGTSBIfDEBERSTgchoiIqHJMgpBp43AYIiIiiZQESdOeNouIiIjUmAQhk1Z6dRgukUtERJbOswlgbQUUFgEZt+SOhoiIqO5hEoRMW+meIBwOQ0REFs7GGvB0Uz+/miZvLERERHURkyBk0qQ5QdgLhIiICMC/Q2KYBCEiIiqPSRAyaZrhMJwPhIiISK25u/qRK8QQERGVxyQImTZNTxAOhSEiIgKgPTkqERERaWMShEyaZjgMe4IQEZG+VqxYAR8fH9jb26Nnz544cuRIhWU3bNgAhUKhtdnb29ditIbjcBgiIqKKMQlCpo3DYYiIyAAxMTGIjIzEnDlzkJiYiC5duiA0NBQZGRkVHuPk5ITU1FRpu3LlSi1GbDgvD/Ujh8MQERGVJ2sSJDo6Gt27d0eDBg3g5uaGkSNH4uzZs1Uet2XLFrRr1w729vbo1KkTfvrpp1qIluoiweEwRERkgCVLlmDChAkIDw+Hn58fVq9eDUdHR6xbt67CYxQKBTw8PKTN3d29FiM2nCYJkpoBPCiuvCwREZGlkTUJsn//fkycOBF//vkn9uzZg6KiIjzxxBPIy8ur8Jg//vgDY8aMwYsvvohjx45h5MiRGDlyJP75559ajJzqCk6MSkRE+iosLERCQgJCQkKkfVZWVggJCUF8fHyFx+Xm5sLb2xteXl4YMWIETp48Wel1CgoKkJOTo7XVJrdGgJ0tUKwCUjNr9dJERER1nqxJkN27d2P8+PHo0KEDunTpgg0bNiAlJQUJCQkVHvPxxx9j0KBBmD59Otq3b48FCxagW7du+PTTT2sxcqozuEQuERHp6ebNmyguLi7Xk8Pd3R1pabon0Gjbti3WrVuHnTt3YtOmTVCpVOjVqxeuXbtW4XWio6Ph7OwsbV5eXkatR1WsrIBmJVXk5KhERETa6tScINnZ2QAAV1fXCsvEx8drfYMDAKGhoZV+g0PmS+oJYlWnfpWJiMhMBAUFISwsDP7+/ggODsa2bdvQpEkTfPbZZxUeExUVhezsbGm7evVqLUasxnlBiIiIdLOROwANlUqFqVOnonfv3ujYsWOF5dLS0gz6BqegoAAFBQXS69rukko1TNMThEkQIiKqQuPGjWFtbY309HSt/enp6fDw8NDrHLa2tujatSsuXLhQYRmlUgmlUvlQsT4saYUYJkGIiIi01Jk7x4kTJ+Kff/7B5s2bjXpeubukUs3iErlERKQvOzs7BAQEIDY2VtqnUqkQGxuLoKAgvc5RXFyMEydOwNPTs6bCNIrmJTkdDochIiLSVieSIBEREdi1axf27duH5s2bV1rWw8PDoG9w6kKXVKo5HA5DRESGiIyMxJo1a7Bx40acPn0ar776KvLy8hAeHg4ACAsLQ1RUlFR+/vz5+PXXX3Hp0iUkJibiueeew5UrV/DSSy/JVQW9SMNhmAQhIiLSIutwGCEEJk2ahO3btyMuLg6+vr5VHhMUFITY2FhMnTpV2rdnz54Kv8GpC11SqQZxOAwRERlg9OjRyMzMxOzZs5GWlgZ/f3/s3r1bGmqbkpICq1KfKXfu3MGECROQlpaGhg0bIiAgAH/88Qf8/PzkqoJeOByGiIhIN4UQQsh18ddeew3ffPMNdu7cibZt20r7nZ2d4eDgAED9jUyzZs0QHR0NQL1EbnBwMN5//30MHToUmzdvxsKFC5GYmFjpXCIaOTk5cHZ2RnZ2NpycnGqmYlRrco4cwelRo2DfsiW67NsndzhERFQKP3PV5Pg53LwDBDwFKBTA2d2A0q5WLktERCQbfT9vZf36fNWqVcjOzka/fv3g6ekpbTExMVKZlJQUpKb++zVGr1698M033+Dzzz9Hly5dsHXrVuzYsUOvBAiZoZLhMOwJQkRE9K9GLoCjPSAEcCND7miIiIjqDtmHw1QlLi6u3L5Ro0Zh1KhRNRARmRppYlQmQYiIiCQKBdCiKXDmEnD5OuBb+ZRrREREFoN3jmTauDoMERGRTj7N1I+Xr8kbBxERUV3CJAiZNMHhMERERDppen8kX5c3DiIiorqEd45k2jgchoiISCf2BCEiIiqPd45k0jRzgoDDYYiIiLRoeoJcYhKEiIhIwiQImTTNcBj2BCEiItLmU5IEuZ4OFBbJGwsREVFdwTtHMm2aniBMghAREWlxcwXqOag/KlNS5Y6GiIiobuCdI5k0wdVhiIiIdFIoAG/OC0JERKSFSRAybRwOQ0REVCHfkiRIMpMgREREAJgEIRMnLZHLniBERETlaOYFucxlcomIiAAwCUImTggBgD1BiIiIdGlZkgRhTxAiIiI1g+8cd+/ejUOHDkmvV6xYAX9/f/zf//0f7ty5Y9TgiKqk6QnCJAgRkdli26P62BOEiIhIm8F3jtOnT0dOTg4A4MSJE3j99dcxZMgQJCcnIzIy0ugBElVGWiKXw2GIiMwW2x7V51uSBLmRAdwvlDcWIiKiusDG0AOSk5Ph5+cHAPjuu+8wbNgwLFy4EImJiRgyZIjRAySqVMlwGPYEISIyX2x7VJ+rM+BUD8jJA1KuA4/4yh0RERGRvAy+c7Szs8O9e/cAAHv37sUTTzwBAHB1dZW+pSGqLYKrwxARmT22PapPofh3SMwlzgtCRERkeE+QPn36IDIyEr1798aRI0cQExMDADh37hyaN29u9ACJKsPhMERE5o9tj4fj0wz4+yxwmUkQIiIiw3uCfPrpp7CxscHWrVuxatUqNGumXoD+559/xqBBg4weIFGlVCr1I3uCEBGZLbY9Ho5mXpBkTo5KRERkeE+QFi1aYNeuXeX2L1261CgBERlClCRBOByGiMh8se3xcKQVYtgThIiIyPAkSGJiImxtbdGpUycAwM6dO7F+/Xr4+flh7ty5sLOzM3qQRBXSLJHL4TBERGaLbY+H46vuOMOeIERENUgIgQcPHqBYc39CRmdtbQ0bGxsoFIqHOo/BSZD/9//+H2bOnIlOnTrh0qVLePbZZ/Gf//wHW7Zswb1797Bs2bKHCojIEOwJQkRk/tj2eDia4TDpN4G8fKCeg7zxEBGZm8LCQqSmpkqTeFPNcXR0hKen50N9AWJwEuTcuXPw9/cHAGzZsgWPPfYYvvnmG/z+++949tln2RCh2sUkCBGR2WPb4+G4OAGNXIBbWcDFFKBzW7kjIiIyHyqVCsnJybC2tkbTpk1hZ2f30D0VqDwhBAoLC5GZmYnk5GS0adMGVtW8BzQ4CSKEgKrkxnPv3r0YNmwYAMDLyws3b96sVhBE1SU4HIaIyOyx7fHw2vgAt5KAc8lMghARGVNhYSFUKhW8vLzg6OgodzhmzcHBAba2trhy5QoKCwthb29frfMYnDoJDAzEu+++i6+++gr79+/H0KFDAQDJyclwd3evVhBE1cXhMERE5o9tj4f3iI/68dxlOaMgIjJf1e2VQIYxxs/Z4DMsW7YMiYmJiIiIwNtvv43WrVsDALZu3YpevXo9dEBEBuESuUREZo9tj4fHJAgREZGawcNhOnfujBMnTpTb/+GHH8KaQxKolmmGwyj4u0dEZLbY9nh4bX3Vj+cvyxoGERGR7Kr99XlCQgI2bdqETZs2ITExEfb29rC1tTVmbERVY08QIiKLwbZH9Wl6glxLB3K5eAEREdUghUIhbTY2NmjRogUiIyNRUFAgldmwYQMUCgUGDRqkdWxWVhYUCgXi4uJqLD6D7xwzMjLQv39/dO/eHZMnT8bkyZMRGBiIAQMGIDMzsyZiJKoQ5wQhIjJ/xm57rFixAj4+PrC3t0fPnj1x5MgRvY7bvHkzFAoFRo4cafA15ebiBDRxVT9nbxAiIqpp69evR2pqKpKTk7Fy5Up89dVXePfdd7XK2NjYYO/evdi3b1+txmbwneOkSZOQm5uLkydP4vbt27h9+zb++ecf5OTkYPLkyTURI1HFOByGiMjsGbPtERMTg8jISMyZMweJiYno0qULQkNDkZGRUelxly9fxhtvvIG+ffs+TFVkxXlBiIiotH79+mHy5Ml488034erqCg8PD8ydO1erTEpKCkaMGIH69evDyckJzzzzDNLT06s8t4uLCzw8PODl5YVhw4ZhxIgRSExM1CpTr149vPDCC5g5c6Yxq1Ulg5Mgu3fvxsqVK9G+fXtpn5+fH1asWIGff/7ZqMERVYVL5BIRmT9jtj2WLFmCCRMmIDw8HH5+fli9ejUcHR2xbt26Co8pLi7G2LFjMW/ePLRs2bLa9ZAbkyBERLVDCIHie/dk2YQQBsW6ceNG1KtXD4cPH8YHH3yA+fPnY8+ePQAAlUqFESNG4Pbt29i/fz/27NmDS5cuYfTo0QZd49y5c/jtt9/Qs2fPcu/NnTsXJ06cwNatWw0658MweGJUlUqlc/ytra0tVJr5GYhqS8l/cg6HISIyX8ZqexQWFiIhIQFRUVHSPisrK4SEhCA+Pr7C4+bPnw83Nze8+OKLOHjwYJXXKSgo0Br3nJOTo3eMNUmTBOFwGCKimqXKz8fRUon72hR4+jSsHR31Lt+5c2fMmTMHANCmTRt8+umniI2NxcCBAxEbG4sTJ04gOTkZXl5eAIAvv/wSHTp0wF9//YXu3btXeN4xY8bA2toaDx48QEFBAYYNG6b1+avRtGlTTJkyBW+//XatDTc1+M7x8ccfx5QpU3Djxg1p3/Xr1zFt2jQMGDDAqMERVUXqCcIkCBGR2TJW2+PmzZsoLi6Gu7u71n53d3ekpaXpPObQoUP44osvsGbNGr2vEx0dDWdnZ2nTNBzl1sZH/cieIEREpNG5c2et156entIQ0dOnT8PLy0vrc8zPzw8uLi44ffp0peddunQpkpKScPz4cezatQvnzp3D888/r7PsjBkzkJmZWWmvTGMyuCfIp59+iuHDh8PHx0f6YVy9ehUdO3bEpk2bjB4gUWW4RC4RkfmTq+1x9+5dPP/881izZg0aN26s93FRUVGIjIyUXufk5NSJRMgjJcvkpmYC2bmAc3154yEiMldWDg4IrCJJUJPXNkTZnpYKhcIoIzw8PDzQunVrAEDbtm1x9+5djBkzBu+++660X8PFxQVRUVGYN28ehg0b9tDXrorBSRAvLy8kJiZi7969OHPmDACgffv2CAkJMXpwRFXSjHljTxAiIrNlrLZH48aNYW1tXW5Ct/T0dHh4eJQrf/HiRVy+fBlPPvmktE/TMLSxscHZs2fRqlWrcscplUoolUqDYqsNzvUBj8ZA2k31kJjAjnJHRERknhQKhUFDUuqq9u3b4+rVq7h69aqUzD916hSysrLg5+dn0LmsS760zs/P1/n+pEmTsHz5cnz88ccPF7QeDE6CAOp/1IEDB2LgwIHGjofIIFJPECZBiIjMmjHaHnZ2dggICEBsbKw07lilUiE2NhYRERHlyrdr1w4nTpzQ2vfOO+/g7t27+Pjjj+tE7w5DPeLDJAgREeknJCQEnTp1wtixY7Fs2TI8ePAAr732GoKDgxEYGFjpsVlZWUhLS4NKpcL58+cxf/58PPLII1qTnJdmb2+PefPmYeLEiTVRFS16JUGWL1+u9wm5TC7VKg6HISIySzXV9oiMjMS4ceMQGBiIHj16YNmyZcjLy0N4eDgAICwsDM2aNUN0dDTs7e3RsaN2psDFxQUAyu03FW18gANHOS8IERFVTaFQYOfOnZg0aRIee+wxWFlZYdCgQfjkk0+qPFbzuapQKODh4YHHHnsMCxcuhI1NxSmIcePGYfHixTh16pTR6qCLXkmQpUuX6nUyhULBJAjVKsHhMEREZqmm2h6jR49GZmYmZs+ejbS0NPj7+2P37t3SZKkpKSmwMuPPFC6TS0REGnFxceX27dixQ+t1ixYtsHPnToPOq88yvePHj8f48eO19llbW+PkyZMGXas69EqCJCcn18jFDxw4gA8//BAJCQlITU3F9u3bK10WJy4uDv379y+3PzU1VedYXjJ/HA5DRGSeaqrtAQARERE6h78AuhuEpW3YsMH4AdUizeSoZ2vux0tERFSnyXrnmJeXhy5dumDFihUGHXf27FmkpqZKm5ubWw1FSHWeZolcDochIiKqkqYnSOZt4OYdWUMhIiKSRbUmRjWWwYMHY/DgwQYf5+bmJo3JJcsmSmbpZ08QIiKiqtV3BFp6AZeuAv+cB/r1kDsiIiKi2mWSd47+/v7w9PTEwIED8fvvv8sdDslJs4Y1kyBERER66dhG/fjPOXnjICIikoNJ3Tl6enpi9erV+O677/Ddd9/By8sL/fr1Q2JiYoXHFBQUICcnR2sj8yG4OgwREZFBOj6ifjzBJAgREVkgWYfDGKpt27Zo27at9LpXr164ePEili5diq+++krnMdHR0Zg3b15thUi1jMNhiIiIDNOpJAnyz3l54yAiIpJDtZIgWVlZOHLkCDIyMqDSDEcoERYWZpTA9NWjRw8cOnSowvejoqIQGRkpvc7JyYGXl1dthEa1gcNhiIgsQl1qe5i6DiXDYa6lAXeygYbO8sZDRERUmwxOgvzwww8YO3YscnNz4eTkBIVCIb2nUChqvSGSlJQET0/PCt9XKpVQKpW1GBHVJg6HISIyf3Wt7WHqnOsD3k2BKzeAkxeAPgFyR0RERFR7DE6CvP7663jhhRewcOFCODo6PtTFc3NzceHCBel1cnIykpKS4OrqihYtWiAqKgrXr1/Hl19+CQBYtmwZfH190aFDB9y/fx9r167Fb7/9hl9//fWh4iATpvk2kEkQIiKzZcy2B6l1ekSdBDlxjkkQIiKyLAYnQa5fv47JkycbpRFy9OhR9O/fX3qtGbYybtw4bNiwAampqUhJSZHeLywsxOuvv47r16/D0dERnTt3xt69e7XOQZZFmhOk1LeCRERkXozZ9iC1jo8Au+K4QgwREVkegydSCA0NxdGjR41y8X79+kEIUW7bsGEDAGDDhg2Ii4uTyr/55pu4cOEC8vPzcevWLezbt48JEEvH4TBERGbPmG0PUuvEFWKIiCze+PHjoVAopK1Ro0YYNGgQ/v77b6mMQqGAvb09rly5onXsyJEjMX78+FqO2DgM7gkydOhQTJ8+HadOnUKnTp1ga2ur9f7w4cONFhxRVQSHwxARmT22PYyvQ2v145UbQHauep4QIiKyPIMGDcL69esBAGlpaXjnnXcwbNgwrREZCoUCs2fPxsaNG+UK06gMToJMmDABADB//vxy7ykUChSXfDNPVCs4HIaIyOyx7WF8DZ2B5h7qFWJOXQCC/OWOiIiI5KBUKuHh4QEA8PDwwMyZM9G3b19kZmaiSZMmAICIiAgsWbIE06dPR8eOHeUM1ygMToKUXZaOSE6a1WHYE4SIyHyx7VEzOrZRJ0FOnGMShIjImIQA8u/Lc20He6C63w/n5uZi06ZNaN26NRo1aiTt7927N86dO4eZM2di165dRopUPgYnQYjqEmliVCZBiIiIDNLpEWD3QU6OSkRkbPn3gfZD5Ln26Z8ARwf9y+/atQv166vHRObl5cHT0xO7du2ClZX29KHR0dHo3LkzDh48iL59+xoz5FqnVxJk+fLlePnll2Fvb4/ly5dXWnby5MlGCYxIL5pvBzkchojIrLDtUfM6cnJUIiKL179/f6xatQoAcOfOHaxcuRKDBw/GkSNH4O3tLZXz8/NDWFgYZs6cid9//12ucI1CryTI0qVLMXbsWNjb22Pp0qUVllMoFGyIUK0SXB2GiMgsse1R87q0VT9eugrcyVbPE0JERA/PwV7dI0OuaxuiXr16aN26tfR67dq1cHZ2xpo1a/Duu+9qlZ03bx4eeeQR7NixwwiRykevJEhycrLO50Sy43AYIiKzxLZHzWvoDLT2Bi5cARJOAiG95I6IiMg8KBSGDUmpSxQKBaysrJCfn1/uPS8vL0REROCtt95Cq1atZIjOOKyqLkJUd0kTo3I4DBERkcECSyb5/+uEvHEQEZE8CgoKkJaWhrS0NJw+fRqTJk1Cbm4unnzySZ3lo6KicOPGDezdu7eWIzWeak2Meu3aNXz//fdISUlBYWGh1ntLliwxSmBE+uDEqEREloFtj5rRvSOw+Ufg6D9yR0JERHLYvXs3PD09AQANGjRAu3btsGXLFvTr109neVdXV8yYMQNvvfVWLUZpXAYnQWJjYzF8+HC0bNkSZ86cQceOHXH58mUIIdCtW7eaiJGoYpqJUZkEISIyW2x71JzATurHv88C9wsBezt54yEiotqzYcMGbNiwodIyQohy+6KiohAVFVVDUdU8g4fDREVF4Y033sCJEydgb2+P7777DlevXkVwcDBGjRpVEzESVUiaGJXDYYiIzBbbHjXHuynQpCFQWAScOCt3NERERDXP4CTI6dOnERYWBgCwsbFBfn4+6tevj/nz52PRokVGD5CoUuwJQkRk9tj2qDkKBRBQMi8Ih8QQEZElMDgJUq9ePWksrqenJy5evCi9d/PmTeNFRqQHzglCRGT+2PaoWd1LhsQwCUJERJbA4DlBHn30URw6dAjt27fHkCFD8Prrr+PEiRPYtm0bHn300ZqIkahC0nAYKy50RERkrtj2qFmBpXqCqFQAP1KJiMicGZwEWbJkCXJzcwEA8+bNQ25uLmJiYtCmTRvOzk61TzMchi02IiKzxbZHzerQBrBXAlk5wMWrQBtvuSMiIiKqOQYlQYqLi3Ht2jV07twZgLp76urVq2skMCJ9cDgMEZF5Y9uj5tnaAP7tgT+TgKMnmAQhIqoOXauokPEZ4+ds0Nfn1tbWeOKJJ3Dnzp2HvjCRUZQMh2FPECIi81QTbY8VK1bAx8cH9vb26NmzJ44cOVJh2W3btiEwMBAuLi6oV68e/P398dVXXxktlrqC84IQEVWPra0tAODevXsyR2IZND9nzc+9OgweDtOxY0dcunQJvr6+1b4okbFIPUGYBCEiMlvGbHvExMQgMjISq1evRs+ePbFs2TKEhobi7NmzcHNzK1fe1dUVb7/9Ntq1awc7Ozvs2rUL4eHhcHNzQ2ho6EPHU1d0L5kX5M8kQAj1qjFERFQ1a2truLi4ICMjAwDg6OgIBf+IGp0QAvfu3UNGRgZcXFxg/RAjARTCwP4ku3fvRlRUFBYsWICAgADUq1dP630nJ6dqB1MbcnJy4OzsjOzs7DofK1XtaMeOKL57F13i4mDPxBwRUZ1irM9cY7Y9evbsie7du+PTTz8FAKhUKnh5eWHSpEmYOXOmXufo1q0bhg4digULFuhV3hTaHvfygS4jgMIiIO4rwLe53BEREZkOIQTS0tKQlZUldyhmz8XFBR4eHjoTTfp+3hrcE2TIkCEAgOHDh2tdWAgBhUKBYs3wBKJaoFkdBpwThIjIbBmr7VFYWIiEhARERUVJ+6ysrBASEoL4+PgqjxdC4LfffsPZs2exaNEiA2tRtzk6AN06qHuCHDzKJAgRkSEUCgU8PT3h5uaGoqIiucMxW7a2tg/VA0TD4CTIvn37HvqiREajGQ7DLmdERGbLWG2Pmzdvori4GO7u7lr73d3dcebMmQqPy87ORrNmzVBQUABra2usXLkSAwcOrLB8QUEBCgoKpNc5OTkPH3wteCzw3yRI2Ei5oyEiMj3W1tZGuUmnmmVwEsTX1xdeXl7lbjqFELh69arRAiPSh2ZOEPYEISIyX3K3PRo0aICkpCTk5uYiNjYWkZGRaNmyJfr166ezfHR0NObNm1fjcRlb30Dgg7VAfBLwoBiw4UcrERGZIYNnk/T19UVmZma5/bdv3+ZkqVTrNMNhuEQuEZH5Mlbbo3HjxrC2tkZ6errW/vT0dHh4eFR4nJWVFVq3bg1/f3+8/vrrePrppxEdHV1h+aioKGRnZ0ubqXxJ1KE14OIE3M0Dkk7LHQ0REVHNMDgJohl/W1Zubi7s7e2NEhSR3jTjwDkchojIbBmr7WFnZ4eAgADExsZK+1QqFWJjYxEUFKT3eVQqldZwl7KUSiWcnJy0NlNgbQ306aZ+fvCovLEQERHVFL2Hw0RGRgJQz70wa9YsODo6Su8VFxfj8OHD8Pf3N3qARBUpvbARe4IQEZmfmmh7REZGYty4cQgMDESPHj2wbNky5OXlITw8HAAQFhaGZs2aST09oqOjERgYiFatWqGgoAA//fQTvvrqK6xatco4laxj+gQCu+LUSZBp4+WOhoiIyPj0ToIcO3YMgPrG88SJE7Czs5Pes7OzQ5cuXfDGG28YP0KiipRaDYBJECIi81MTbY/Ro0cjMzMTs2fPRlpaGvz9/bF7925pstSUlBRYWf3bUTYvLw+vvfYarl27BgcHB7Rr1w6bNm3C6NGjjVDDuqdvgPox6TSQkws41Zc3HiIiImNTiNJfp+shPDwcH3/8scl07SxL37WDqe5TFRTgr0ceAQAE/P03bJydZY6IiIhKM9ZnLtsetat/GHDpKvD5AiC0j9zREBER6Uffz1uD5wRZv369SXyAkwXgcBgiIovAtkft6lPSG+TAX/LGQUREVBMMToIQ1RWi1HAYLpFLRERkHMHd1Y/7Dmt930BERGQWmAQhk1U6CaJr1QAiIiIyXJ8AwMEeuJ4OnLwgdzRERETGxSQIma7SX0+xJwgREZFR2CuBx0p6g/x6SN5YiIiIjI1JEDJZgqvDEBER1YjQ3urHPb/LGwcREZGxMQlCpqt0EsSKv8pERETG8vijgLUVcOoikJIqdzRERETGwztHMllCpVI/YQKEiIjIqBo6A907q5+zNwgREZkT3j2SydIkQTgUhoiIyPg0Q2I4LwgREZkTJkHIdGmGw7AnCBERkdEN7KN+PHICuJMtbyxERETGIuvd44EDB/Dkk0+iadOmUCgU2LFjR5XHxMXFoVu3blAqlWjdujU2bNhQ43FS3ST1BGEShIiIyOi8PAC/VoBKBcTGyx0NERGRcch695iXl4cuXbpgxYoVepVPTk7G0KFD0b9/fyQlJWHq1Kl46aWX8Msvv9RwpFQnaeYE4XAYIiKiGvFESW+QH/fLGwcREZGx2Mh58cGDB2Pw4MF6l1+9ejV8fX2xePFiAED79u1x6NAhLF26FKGhoTUVJtVRmiVy2ROEiIioZgzrDyzbCBz4Sz0kpqGz3BERERE9HJO6e4yPj0dISIjWvtDQUMTHV9xHs6CgADk5OVobmQcmQYiIiGpWG2+gQxvgQTF7gxARkXkwqbvHtLQ0uLu7a+1zd3dHTk4O8vPzdR4THR0NZ2dnafPy8qqNUKk2CKF+5HAYIiKiGjNigPpxZ6y8cRARERmDSSVBqiMqKgrZ2dnSdvXqVblDIiNhTxAiIqKaN/xxQKEAjvwNXEuTOxoiIqKHY1J3jx4eHkhPT9fal56eDicnJzg4OOg8RqlUwsnJSWsjM8ElcomIiGqcZxOgZxf18x/2yRsLERHRwzKpu8egoCDExmr3xdyzZw+CgoJkiojkJEqGwyg4HIaIiKhGjSyZkm3HXnnjICIieliyJkFyc3ORlJSEpKQkAOolcJOSkpCSkgJAPZQlLCxMKv/KK6/g0qVLePPNN3HmzBmsXLkS3377LaZNmyZH+CQ3DochIiKqFUMeA+xsgTOX1BsREZGpkvXu8ejRo+jatSu6du0KAIiMjETXrl0xe/ZsAEBqaqqUEAEAX19f/Pjjj9izZw+6dOmCxYsXY+3atVwe10IJDochIiKqFc4NgP491c+3/SpvLERERA/DRs6L9+vXTxrSoMuGDRt0HnPs2LEajIpMBofDEBER1ZqnQ4FfDgFbfwHeeFHdM4SIiMjU8Ct0MlnsCUJERFR7Hg8C3BsDt7KAXw/JHQ0REVH18O6RTBaXyCUiIqo9NtbA6MHq59/skjcWIiKi6uLdI5kuzVAqDochIiKqFaOHAAoF8HsicPm63NEQEREZjkkQMlnsCUJERFS7mnsAwT3Uz//H3iBERGSCePdIpkuTBGFPECIiolozdpj6cctuoLBI3liIiIgMxSQImSyhUqmfsCcIERFRrSk9QeovB+WOhoiIyDC8eySTpUmCcDgMERFR7bGxBsYMVT//4jt5YyEiIjIU7x7JdGmWyOVwGCIiMsCKFSvg4+MDe3t79OzZE0eOHKmw7Jo1a9C3b180bNgQDRs2REhISKXlLcVzwwGlLXDsFJBwUu5oiIiI9MckCJks9gQhIiJDxcTEIDIyEnPmzEFiYiK6dOmC0NBQZGRk6CwfFxeHMWPGYN++fYiPj4eXlxeeeOIJXL9u2UujNHEFRg5UP1/zrbyxEBERGYJ3j2S6OCcIEREZaMmSJZgwYQLCw8Ph5+eH1atXw9HREevWrdNZ/uuvv8Zrr70Gf39/tGvXDmvXroVKpUJsbGwtR173vPS0+vGXQ0DKDXljISIi0hfvHslkCa4OQ0REBigsLERCQgJCQkKkfVZWVggJCUF8fLxe57h37x6Kiorg6upaU2GajEd8geDu6u8k1m+TOxoiIiL9MAlCJourwxARkSFu3ryJ4uJiuLu7a+13d3dHWlqaXueYMWMGmjZtqpVIKaugoAA5OTlam7l6aZT6MeYnIDtX3liIiIj0wbtHMl2cE4SIiGrR+++/j82bN2P79u2wt7evsFx0dDScnZ2lzcvLqxajrF19A4G2vkBePrBxu9zREBERVY13j2SyOByGiIgM0bhxY1hbWyM9PV1rf3p6Ojw8PCo99qOPPsL777+PX3/9FZ07d660bFRUFLKzs6Xt6tWrDx17XaVQAK/9n/r52i3A3Tx54yEiIqoKkyBkujgchoiIDGBnZ4eAgACtSU01k5wGBQVVeNwHH3yABQsWYPfu3QgMDKzyOkqlEk5OTlqbOXuyP9DKC8i+y94gRERU9/HukUyWtEQue4IQEZGeIiMjsWbNGmzcuBGnT5/Gq6++iry8PISHhwMAwsLCEBUVJZVftGgRZs2ahXXr1sHHxwdpaWlIS0tDbi4nwNCwtgYinlc/X7MFyL0nbzxERESVYRKETFfJcBj2BCEiIn2NHj0aH330EWbPng1/f38kJSVh9+7d0mSpKSkpSE1NlcqvWrUKhYWFePrpp+Hp6SltH330kVxVqJOGPw74NgeycoAvd8gdDRERUcVs5A6AqLqkOUGYBCEiIgNEREQgIiJC53txcXFary9fvlzzAZkBG2tg8vPAtGjg8xggbCRQ31HuqIiIiMrj3SOZLiEAcDgMERFRXTB8gLo3yJ0cdSKEiIioLmIShEyW4HAYIiKiOsPGGpj+kvr5598C6TfljYeIiEgX3j2SyeISuURERHXLkMeAbh2A/PvA4vVyR0NERFQekyBkurhELhERUZ2iUABvv6J+vmU3cOaSvPEQERGVxbtHMlnSErlMghAREdUZgR2BIcHq7yoWfiZ3NERERNp490imi8NhiIiI6qQZEwBbG2D/ESA2Xu5oiIiI/sUkCJksweEwREREdZJPM+DFp9XPZy9XzxFCRERUF/DukUwXh8MQERHVWZPDgKZuwLU04JOv5I6GiIhIjXePZLKkJXI5HIaIiKjOqecAzJ2kfv75t8D5K/LGQ0REBDAJQiaME6MSERHVbU/0BgYEAUUPgHeWAULIHREREVk63j2S6eKcIERERHWaQgHMmwzYK4E/k4Cvf5A7IiIisnS8eySTJbg6DBERUZ3n5QHMeEn9/L1VQMoNeeMhIiLLxiQImS72BCEiIjIJ458CHu0C3LsPTP/g349wIiKi2sa7RzJZnBOEiIjINFhZAR++CTjaA38eBzZskzsiIiKyVLx7JJPF4TBERESmo0VT4O1X1c/fXwOcvihvPEREZJmYBCHTxeEwREREJmXsk0D/nkBBITBxHpCXL3dERERkaXj3SCaLw2GIiIhMi0IBLIkCPBoDF68C7yzlsrlERFS76sTd44oVK+Dj4wN7e3v07NkTR44cqbDshg0boFAotDZ7e/tajJbqjJLhMOBwGCIiIpPh6gx8MguwtgK27QG27JY7IiIisiSyJ0FiYmIQGRmJOXPmIDExEV26dEFoaCgyMjIqPMbJyQmpqanSduXKlVqMmOoK9gQhIiIyTT06A5Hh6ufvLAP+PitrOEREZEFkv3tcsmQJJkyYgPDwcPj5+WH16tVwdHTEunXrKjxGoVDAw8ND2tzd3WsxYqorpIlRmQQhIiIyOa/9HzAgSD0/yIR3gIzbckdERESWQNa7x8LCQiQkJCAkJETaZ2VlhZCQEMTHx1d4XG5uLry9veHl5YURI0bg5MmTtREu1TWaiVE5HIaIiMjkWFkBy94CWrUA0m4Cr8xWJ0SIiIhqkqxJkJs3b6K4uLhcTw53d3ekpaXpPKZt27ZYt24ddu7ciU2bNkGlUqFXr164du2azvIFBQXIycnR2sg8SMNhmAQhIiIySU71gbXvAk71gISTwNucKJWIiGqYyY0jCAoKQlhYGPz9/REcHIxt27ahSZMm+Oyzz3SWj46OhrOzs7R5eXnVcsRUYzQToyoU8sZBRERE1dbSC/h0trpnyJbdwMdfyh0RERGZM1mTII0bN4a1tTXS09O19qenp8PDw0Ovc9ja2qJr1664cOGCzvejoqKQnZ0tbVevXn3ouKluYE8QIiIi8xDcA3h3ivr50g1AzE+yhkNERGZM1iSInZ0dAgICEBsbK+1TqVSIjY1FUFCQXucoLi7GiRMn4OnpqfN9pVIJJycnrY3MBJMgREREZmPscGDiWPXzqMXAb3/KGw8REZkn2YfDREZGYs2aNdi4cSNOnz6NV199FXl5eQgPV6+bFhYWhqioKKn8/Pnz8euvv+LSpUtITEzEc889hytXruCll16SqwokE8HhMERERGZl+ovAUwOBYhXwyhwgPknuiIiIyNzYyB3A6NGjkZmZidmzZyMtLQ3+/v7YvXu3NFlqSkoKrEotgXrnzh1MmDABaWlpaNiwIQICAvDHH3/Az89PriqQXNgThIiIyKwoFMCi6UB2LhAbD7wQBXz1IRDYUe7IiIjIXMjeEwQAIiIicOXKFRQUFODw4cPo2bOn9F5cXBw2bNggvV66dKlUNi0tDT/++CO6du0qQ9QkN8ElcomIqBpWrFgBHx8f2Nvbo2fPnjhy5EiFZU+ePIn//ve/8PHxgUKhwLJly2ovUAtlZwusnAv0CQDu3QfGzwSOn5E7KiIiMhd1IglCVB2a4TAKDochIiI9xcTEIDIyEnPmzEFiYiK6dOmC0NBQZGRk6Cx/7949tGzZEu+//77ek7bTw7O3A9YsAHp0Bu7mAWPfAI7+I3dURERkDpgEIdPFniBERGSgJUuWYMKECQgPD4efnx9Wr14NR0dHrFu3Tmf57t2748MPP8Szzz4LpVJZy9FaNkcHYH000LMkEfL8dM4RQkRED49JEDJZXCKXiIgMUVhYiISEBISEhEj7rKysEBISgvj4eKNdp6CgADk5OVobVU99R2DjIqBvoHpozLgZ6rlCiIiIqotJEDJdXB2GiIgMcPPmTRQXF0uTr2u4u7sjLS3NaNeJjo6Gs7OztHl5eRnt3JbIwR5Y+x4wIAgoKAQmvANs/lHuqIiIyFQxCUImS5oThD1BiIioDomKikJ2dra0Xb16Ve6QTJ69HfDZfODpUPXyuTM+ApZtBISQOzIiIjI1si+RS1RdoqTlwyQIERHpo3HjxrC2tkZ6errW/vT0dKNOeqpUKjl/SA2wtQE+mgF4NAE+3QQs3QBcvg68/zpgzx83ERHpiT1ByHRphsNY8deYiIiqZmdnh4CAAMTGxkr7VCoVYmNjERQUJGNkpC+FApj+IvDeNMDaCti+B3hmKpCWKXdkRERkKnj3SCZLGg7DJAgREekpMjISa9aswcaNG3H69Gm8+uqryMvLQ3h4OAAgLCwMUVFRUvnCwkIkJSUhKSkJhYWFuH79OpKSknDhwgW5qkAAnhsOfPUh4OIEHD8DDHsFOHxc7qiIiMgU8O6RTJdmIDCHwxARkZ5Gjx6Njz76CLNnz4a/vz+SkpKwe/duabLUlJQUpKamSuVv3LiBrl27omvXrkhNTcVHH32Erl274qWXXpKrClSidzfgh1VAW18g8zbwbCSw4mugZPE4IiIinRRCWNaUUjk5OXB2dkZ2djacnJzkDocewt9PPIH8s2fR7uuv4dynj9zhEBFRGfzMVePPoWbl5QPvLAW27VG/Du6hnjvEzVXeuIiIqHbp+3nLniBksgTnBCEiIrJ49RyAJVHAojcApR2w/wjwRDjw0365IyMiorqId49kurg6DBEREUE9YeqzQ4EfVgN+rYA7OcCrc4GpC4E72XJHR0REdQmTIGSyODEqERERldbWF9i5Cpj4f+qOotv3AAPGAztj/51KjIiILBvvHsl0cTgMERERlWFnC7w5Adi6HGjjDdzKAia/C4ybCVy6Knd0REQkN949kskSHA5DREREFQjoAPy0BogMVydG9h8BnngBiP4MyL0nd3RERCQXJkHIdGmGwzAJQkRERDrY2QJTwoBfvgD69QSKHgCrNwOPjQXWbgHuF8gdIRER1TYmQchkSavDKBTyBkJERER1WksvYEM0sG4h4NNMPURmwUog+Dlg0/dAYZHcERIRUW1hEoRMF4fDEBERkZ4UCmBAELB3A/D+G0BTNyDtJvD2UuDxccDW3cCDYrmjJCKimsYkCJksqScIkyBERESkJ1sbYMxQIO4rYO4koElD4Goq8Poidc+QNd8COblyR0lERDWFSRAyWdISuRwOQ0RERAZS2gHhTwEHvwGi/h/g6gxcSwPeXQU8+gww9xPgynW5oyQiImNjEoRMl0qlfmRPECIiIqomB3vglWeB+Bjg/dfVy+rm5QPrtwHBzwMT3gF++5NDZYiIzIWN3AEQVZcoSYJwThAiIiJ6WPZKYMww4NmhwMGjwBffAXGHgV9/V29NXIH/DARGhQKP+ModLRERVReTIGS6uDoMERERGZlCATzWXb2dvwJ8/T2wMxbIvA18HqPeOrcFnnoCGPIY4N5Y7oiJiMgQHA5DJos9QYiIiKgmtfFWT556eAvw+QIgtA9gYw38fVY9Z0iPUcDIicDqzcBlzh9CRGQS2BOETBeTIERERFQL7GzVCZDQPsCtLOD7WOD7fUDiSeDYKfUW/RnQriUwsDcQ3B3wb69eiYaIiOoW/mkmkyU4HIaIiIhqWSMXIPy/6i39JvDLIfUWfww4c0m9ffIVUN8R6NUV6BuoHlrj3ZRNFiKiuoBJEDJJQghACADsCUJERETycG8MhI1Ub1k5wN4/gLgjwKEE4E7Ov5OqAkBTN6B7p3+3R3wAKw5MJyKqdUyCkGnSLI8LcIlcIiIikp2LE/D0IPWmUgEnzwMHjqq3hH+AGxnqCVZ3xqrLO9UHAjsCXf2ALm2BTm0BV2d560BEZAmYBCGTJA2FAaDg1yhERERUh1hZqZMandoCE8cC9/KBxFPA0X+AI3+rn+fkAr/9qd40mnuoV57p9Ih6fpG2vuoeJBxGQ0RkPEyCkGkq3ROESRAiIiKqwxwdgD4B6g0Aih4Apy8Cf50Ajp8BTpwDLl0FrqWpt5/2/3tsfUegjY86IdLWB3jEV701acjkCBFRdTAJQiZJlEqCcE4QIiIiMiW2NuoeH53b/rsvJ1edDDlxVv147rI6MZJ7798VaEpzbgD4NFNPuOrdDPApefRuxgQJEVFlmAQhk8ThMERERGROnOoDvbupN43CIiD5GnAuGTh7WZ0YOZcMXLkBZN9V9yI5fqb8uRztgRZNAS8PoKk70LQJ4OmmHlrj2UQ9oSuX7yUiS8U/f2SaOByGiIiIzJydbckwGF/gyVL77xeokyNXbgCXrwMpJY9XbqgnYL13/9/lenWxsgLcXNWJEfdGQOOG6tdNXNXPm7gCjV3VPUrslbVSVSKiWsMkCJkkDochIiIiS2WvBNq3Um9lFRSq5xW5fB24ng6kZqoTI6kZwI1MIC1TPSdJ2k31VhWnev8mRBo1BBo6qVfCKfuoee7cALBh04yI6jAmQcg0lRoOw54gRERERGpKO6BVC/Wmi0oFZN4pSYpkAJm31a9v6ngsKAJy8tTbpav6x+BUD3BxVj82qF/yWE89yWuD+urnTmVel97qOXBOEyKqOXUiCbJixQp8+OGHSEtLQ5cuXfDJJ5+gR48eFZbfsmULZs2ahcuXL6NNmzZYtGgRhgwZUosRk9ykOUEUCij4KUlERESkFysr9RAY90aAf/uKywmhTn7cvK1OlNy8o06OZN8F7mQDd3KArLtAVk7J8xz15K7Av4mT6lIo1POaONqrV9ZxtAccSh7rOQAOJe85lLx2LLWv7PtKO3XPGaUdYG+nflTaAexITGS5ZE+CxMTEIDIyEqtXr0bPnj2xbNkyhIaG4uzZs3BzcytX/o8//sCYMWMQHR2NYcOG4ZtvvsHIkSORmJiIjh07ylADkoNmOAyHwhAREREZn0IBONdXbxX1KinrQbE6EaJJkuTkAnfzgNw89WNOnvbru3nA3XsljyVlHxSrEzB5+eoNd2qmfrY2upMjSqXuxInmtZ0NYGurPt7OruSxzD6t17bqzbZkn+Y9O1vt19ZW7P1CVFsUQgghZwA9e/ZE9+7d8emnnwIAVCoVvLy8MGnSJMycObNc+dGjRyMvLw+7du2S9j366KPw9/fH6tWrq7xeTk4OnJ2dkZ2dDScnJ+NVhGpVwfXrSOrVCwqlEj3OnZM7HCIi0qGufubWdg/UuvpzIKprhFDPaZKTB+Tnqyd4zcsH8u+rn9/LL9lKv76vfj9P8zz/3/fuF6onkS0oVG9FD+SuYcUUin8TJjbWgI1NmUdr7dfWOspYWwO2mkebkjIVnKOi9zT7razViRlrK3XvIWvrUs9LXlvp+b61VdXnK/s+UXXo+3kra0+QwsJCJCQkICoqStpnZWWFkJAQxMfH6zwmPj4ekZGRWvtCQ0OxY8eOmgy1UsWFRcg6nyLb9S1RYXo67iscYGVlj3v5ckdDRGQeHOzN/5tI9kAlqrsUCnWPi5pakeZBMVBYqE6OFJRKkJROlEj7KihT9EC9dHHRA6Co6N/XpfcVah4r2VdYpB2bJgFUUFgzdTc1lSVSdCVaFArASgEorNSPVlbVew2F9msrzbkNfK1QVPy+IXGVrVfpc2ueQ6H7vbL7FShJMJV6rtmviRGlnuvaX/baZbeK3isbh6Lk2k1c1b3NapusSZCbN2+iuLgY7u7uWvvd3d1x5oyORc8BpKWl6Syflpams3xBQQEKCgqk1zk5OQ8ZdXk5VzPRLULH9NxUg1oBviW/I5wOhojIKE7/pB5bb86WLFmCCRMmIDw8HACwevVq/Pjjj1i3bp3OHqgff/wxBg0ahOnTpwMAFixYgD179uDTTz/VqwcqEdUdNtaAjUPd+DsnhDopo0mMFJZOqhSq3ysuVj9K2wPtx+JidfniYqCodPlS5Qw5R+lyKpV6n0oFFJdsqtLvafaVlJOel5Sr9H1V1T8fzfFk3qIjgf97supyxib7nCA1LTo6GvPmzavZi7DPFhERUZ1XWz1Qa+MLGCIybQpFyTwhNnUjKVObhNBOiBRXkHAxJNEiRMl5xb+vVaqKX6sEIDSP1ThGc5yo6P0KXmsSQJVdR9drIdR11TwXUJ9beq4pi3/j0rUfpWMv9RxlXmue69ovXVvXdctsKh37Sr+ntJPhFxAyJ0EaN24Ma2trpKena+1PT0+Hh4eHzmM8PDwMKh8VFaXVeMnJyYGXl9dDRq7NxccDp38y6imJiIhqnYO93BHUrNrogQrU0hcwREQmSqEomROE6xuQTGRNgtjZ2SEgIACxsbEYOXIkAPXEqLGxsYiIiNB5TFBQEGJjYzF16lRp3549exAUFKSzvFKphFJZQwMLSygUlpfBJSIiIt1q4wsYIiIiqh7Zh8NERkZi3LhxCAwMRI8ePbBs2TLk5eVJY3XDwsLQrFkzREdHAwCmTJmC4OBgLF68GEOHDsXmzZtx9OhRfP7553JWg4iIiOq42uiBCtTOFzBERERUPbJPZjF69Gh89NFHmD17Nvz9/ZGUlITdu3dLXU9TUlKQmpoqle/Vqxe++eYbfP755+jSpQu2bt2KHTt2cIZ2IiIiqlTpHqgamh6oFfUo1fRALa2yHqhERERUtymEEELuIGqTvmsHExER0cOpi5+5MTExGDduHD777DOpB+q3336LM2fOwN3dvVwP1D/++APBwcF4//33pR6oCxcuNGiJ3Lr4cyAiIjI3+n7eyj4choiIiKi2jB49GpmZmZg9ezbS0tLg7+9frgeqValV3zQ9UN955x289dZbaNOmDXugEhERmTD2BCEiIqIawc9cNf4ciIiIap6+n7eyzwlCRERERERERFQbmAQhIiIiIiIiIovAJAgRERERERERWQQmQYiIiIiIiIjIIjAJQkREREREREQWweKWyNUshpOTkyNzJEREROZN81lrYQvRlcO2BxERUc3Tt91hcUmQu3fvAgC8vLxkjoSIiMgy3L17F87OznKHIRu2PYiIiGpPVe0OhbCwr2dUKhVu3LiBBg0aQKFQGO28OTk58PLywtWrVytdk9jUsZ7mhfU0L5ZST8By6mrq9RRC4O7du2jatCmsrCx3BG5NtD1M/XfDEJZSV9bTvLCe5oX1NA36tjssrieIlZUVmjdvXmPnd3JyMslfGEOxnuaF9TQvllJPwHLqasr1tOQeIBo12fYw5d8NQ1lKXVlP88J6mhfWs+7Tp91huV/LEBEREREREZFFYRKEiIiIiIiIiCwCkyBGolQqMWfOHCiVSrlDqVGsp3lhPc2LpdQTsJy6Wko9yXCW9LthKXVlPc0L62leWE/zYnEToxIRERERERGRZWJPECIiIiIiIiKyCEyCEBEREREREZFFYBKEiIiIiIiIiCwCkyBGsGLFCvj4+MDe3h49e/bEkSNH5A7poURHR6N79+5o0KAB3NzcMHLkSJw9e1arzP379zFx4kQ0atQI9evXx3//+1+kp6fLFLFxvP/++1AoFJg6daq0z1zqef36dTz33HNo1KgRHBwc0KlTJxw9elR6XwiB2bNnw9PTEw4ODggJCcH58+dljLh6iouLMWvWLPj6+sLBwQGtWrXCggULUHrqI1Os64EDB/Dkk0+iadOmUCgU2LFjh9b7+tTp9u3bGDt2LJycnODi4oIXX3wRubm5tViLqlVWz6KiIsyYMQOdOnVCvXr10LRpU4SFheHGjRta5zD1epb1yiuvQKFQYNmyZVr7TaGeVLPY9jDdz+TS2PYwrc/jstjuYLvDFOoJsO1RFpMgDykmJgaRkZGYM2cOEhMT0aVLF4SGhiIjI0Pu0Kpt//79mDhxIv7880/s2bMHRUVFeOKJJ5CXlyeVmTZtGn744Qds2bIF+/fvx40bN/DUU0/JGPXD+euvv/DZZ5+hc+fOWvvNoZ537txB7969YWtri59//hmnTp3C4sWL0bBhQ6nMBx98gOXLl2P16tU4fPgw6tWrh9DQUNy/f1/GyA23aNEirFq1Cp9++ilOnz6NRYsW4YMPPsAnn3wilTHFuubl5aFLly5YsWKFzvf1qdPYsWNx8uRJ7NmzB7t27cKBAwfw8ssv11YV9FJZPe/du4fExETMmjULiYmJ2LZtG86ePYvhw4drlTP1epa2fft2/Pnnn2jatGm590yhnlRz2PYw3c/k0tj2ML3P47LY7mC7wxTqCbDtUY6gh9KjRw8xceJE6XVxcbFo2rSpiI6OljEq48rIyBAAxP79+4UQQmRlZQlbW1uxZcsWqczp06cFABEfHy9XmNV29+5d0aZNG7Fnzx4RHBwspkyZIoQwn3rOmDFD9OnTp8L3VSqV8PDwEB9++KG0LysrSyiVSvG///2vNkI0mqFDh4oXXnhBa99TTz0lxo4dK4Qwj7oCENu3b5de61OnU6dOCQDir7/+ksr8/PPPQqFQiOvXr9da7IYoW09djhw5IgCIK1euCCHMq57Xrl0TzZo1E//884/w9vYWS5culd4zxXqScbHtoWaKn8kabHuY/uexEGx3aLDdYTr1FIJtDyGEYE+Qh1BYWIiEhASEhIRI+6ysrBASEoL4+HgZIzOu7OxsAICrqysAICEhAUVFRVr1bteuHVq0aGGS9Z44cSKGDh2qVR/AfOr5/fffIzAwEKNGjYKbmxu6du2KNWvWSO8nJycjLS1Nq57Ozs7o2bOnSdUTAHr16oXY2FicO3cOAHD8+HEcOnQIgwcPBmBeddXQp07x8fFwcXFBYGCgVCYkJARWVlY4fPhwrcdsLNnZ2VAoFHBxcQFgPvVUqVR4/vnnMX36dHTo0KHc++ZST6oetj1M+zNZg20P8/g8ZrtDje0O06+npbU9bOQOwJTdvHkTxcXFcHd319rv7u6OM2fOyBSVcalUKkydOhW9e/dGx44dAQBpaWmws7OT/gBouLu7Iy0tTYYoq2/z5s1ITEzEX3/9Ve49c6nnpUuXsGrVKkRGRuKtt97CX3/9hcmTJ8POzg7jxo2T6qLr99iU6gkAM2fORE5ODtq1awdra2sUFxfjvffew9ixYwHArOqqoU+d0tLS4ObmpvW+jY0NXF1dTbbe9+/fx4wZMzBmzBg4OTkBMJ96Llq0CDY2Npg8ebLO982lnlQ9bHu4aJU1xb/fbHuYT9uD7Y5/sd2hZqr1tLS2B5MgVKmJEyfin3/+waFDh+QOxeiuXr2KKVOmYM+ePbC3t5c7nBqjUqkQGBiIhQsXAgC6du2Kf/75B6tXr8a4ceNkjs64vv32W3z99df45ptv0KFDByQlJWHq1Klo2rSp2dXVkhUVFeGZZ56BEAKrVq2SOxyjSkhIwMcff4zExEQoFAq5wyGSBdseps9S2h5sd1gGc253AJbZ9uBwmIfQuHFjWFtbl5uxOz09HR4eHjJFZTwRERHYtWsX9u3bh+bNm0v7PTw8UFhYiKysLK3yplbvhIQEZGRkoFu3brCxsYGNjQ3279+P5cuXw8bGBu7u7mZRT09PT/j5+Wnta9++PVJSUgBAqos5/B5Pnz4dM2fOxLPPPotOnTrh+eefx7Rp0xAdHQ3AvOqqoU+dPDw8yk2Y+ODBA9y+fdvk6q1piFy5cgV79uyRvo0BzKOeBw8eREZGBlq0aCH9Xbpy5Qpef/11+Pj4ADCPelL1se2RpVXe1OrNtod5tT3Y7vgX2x1qplhPS2x7MAnyEOzs7BAQEIDY2Fhpn0qlQmxsLIKCgmSM7OEIIRAREYHt27fjt99+g6+vr9b7AQEBsLW11ar32bNnkZKSYlL1HjBgAE6cOIGkpCRpCwwMxNixY6Xn5lDP3r17l1tm8Ny5c/D29gYA+Pr6wsPDQ6ueOTk5OHz4sEnVE1DP5G1lpf1nzdraGiqVCoB51VVDnzoFBQUhKysLCQkJUpnffvsNKpUKPXv2rPWYq0vTEDl//jz27t2LRo0aab1vDvV8/vnn8ffff2v9XWratCmmT5+OX375BYB51JOqj20P0/5MZtvDvNoebHeosd1h2vW0yLaHvPOymr7NmzcLpVIpNmzYIE6dOiVefvll4eLiItLS0uQOrdpeffVV4ezsLOLi4kRqaqq03bt3TyrzyiuviBYtWojffvtNHD16VAQFBYmgoCAZozaO0jO0C2Ee9Txy5IiwsbER7733njh//rz4+uuvhaOjo9i0aZNU5v333xcuLi5i586d4u+//xYjRowQvr6+Ij8/X8bIDTdu3DjRrFkzsWvXLpGcnCy2bdsmGjduLN58802pjCnW9e7du+LYsWPi2LFjAoBYsmSJOHbsmDQ7uT51GjRokOjatas4fPiwOHTokGjTpo0YM2aMXFXSqbJ6FhYWiuHDh4vmzZuLpKQkrb9NBQUF0jlMvZ66lJ2hXQjTqCfVHLY9TPczWRe2PUzn87gstjvY7jCFegrBtkdZTIIYwSeffCJatGgh7OzsRI8ePcSff/4pd0gPBYDObf369VKZ/Px88dprr4mGDRsKR0dH8Z///EekpqbKF7SRlG2ImEs9f/jhB9GxY0ehVCpFu3btxOeff671vkqlErNmzRLu7u5CqVSKAQMGiLNnz8oUbfXl5OSIKVOmiBYtWgh7e3vRsmVL8fbbb2t9WJliXfft26fz/+S4ceOEEPrV6datW2LMmDGifv36wsnJSYSHh4u7d+/KUJuKVVbP5OTkCv827du3TzqHqddTF10NEVOoJ9Ustj1M9zO5LLY9TOfzuCy2O9juMIV6CsG2R1kKIYQwTp8SIiIiIiIiIqK6i3OCEBEREREREZFFYBKEiIiIiIiIiCwCkyBEREREREREZBGYBCEiIiIiIiIii8AkCBERERERERFZBCZBiIiIiIiIiMgiMAlCRERERERERBaBSRAiIiIiIiIisghMghARERERERGRRWAShIjqjPHjx2PkyJFyh0FEREQWgm0PIsvDJAgRERERERERWQQmQYio1m3duhWdOnWCg4MDGjVqhJCQEEyfPh0bN27Ezp07oVAooFAoEBcXBwC4evUqnnnmGbi4uMDV1RUjRozA5cuXpfNpvsWZN28emjRpAicnJ7zyyisoLCyUp4JERERUp7DtQUQaNnIHQESWJTU1FWPGjMEHH3yA//znP7h79y4OHjyIsLAwpKSkICcnB+vXrwcAuLq6oqioCKGhoQgKCsLBgwdhY2ODd999F4MGDcLff/8NOzs7AEBsbCzs7e0RFxeHy5cvIzw8HI0aNcJ7770nZ3WJiIhIZmx7EFFpTIIQUa1KTU3FgwcP8NRTT8Hb2xsA0KlTJwCAg4MDCgoK4OHhIZXftGkTVCoV1q5dC4VCAQBYv349XFxcEBcXhyeeeAIAYGdnh3Xr1sHR0REdOnTA/PnzMX36dCxYsABWVuz0RkREZKnY9iCi0vi/k4hqVZcuXTBgwAB06tQJo0aNwpo1a3Dnzp0Kyx8/fhwXLlxAgwYNUL9+fdSvXx+urq64f/8+Ll68qHVeR0dH6XVQUBByc3Nx9erVGq0PERER1W1sexBRaewJQkS1ytraGnv27MEff/yBX3/9FZ988gnefvttHD58WGf53NxcBAQE4Ouvvy73XpMmTWo6XCIiIjJxbHsQUWlMghBRrVMoFOjduzd69+6N2bNnw9vbG9u3b4ednR2Ki4u1ynbr1g0xMTFwc3ODk5NThec8fvw48vPz4eDgAAD4888/Ub9+fXh5edVoXYiIiKjuY9uDiDQ4HIaIatXhw4excOFCHD16FCkpKdi2bRsyMzPRvn17+Pj44O+//8bZs2dx8+ZNFBUVYezYsWjcuDFGjBiBgwcPIjk5GXFxcZg8eTKuXbsmnbewsBAvvvgiTp06hZ9++glz5sxBREQEx+QSERFZOLY9iKg09gQholrl5OSEAwcOYNmyZcjJyYG3tzcWL16MwYMHIzAwEHFxcQgMDERubi727duHfv364cCBA5gxYwaeeuop3L17F82aNcOAAQO0vp0ZMGAA2rRpg8ceewwFBQUYM2YM5s6dK19FiYiIqE5g24OISlMIIYTcQRARPYzx48cjKysLO3bskDsUIiIisgBsexCZLvbVIiIiIiIiIiKLwCQIEREREREREVkEDochIiIiIiIiIovAniBEREREREREZBGYBCEiIiIiIiIii8AkCBERERERERFZBCZBiIiIiIiIiMgiMAlCRERERERERBaBSRAiIiIiIiIisghMghARERERERGRRWAShIiIiIiIiIgsApMgRERERERERGQR/j+5yFGkNh5/4wAAAABJRU5ErkJggg==",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"low LR: no-BN final 0.688 vs BN final 0.003\n"
]
}
],
"source": [
"# viz: loss curves at high and low LR, with and without BN\n",
"LOW_LR = 0.1\n",
"no_bn_low = train_mlp(bn=False, lr=LOW_LR, steps=BN_STEPS)\n",
"bn_low = train_mlp(bn=True, lr=LOW_LR, steps=BN_STEPS)\n",
"\n",
"def finite(xs): # plot NaNs as a break, not a crash\n",
" return [v if math.isfinite(v) else float(\"nan\") for v in xs]\n",
"\n",
"fig, ax = plt.subplots(1, 2, figsize=(11, 4))\n",
"plot_loss(ax[0], finite(no_bn_high), \"no BN\", \"#C81E1E\"); plot_loss(ax[0], finite(bn_high), \"BN\", \"#1E40FF\")\n",
"ax[0].set_title(f\"high LR = {HIGH_LR} (no-BN diverges)\"); ax[0].legend()\n",
"plot_loss(ax[1], finite(no_bn_low), \"no BN\", \"#C81E1E\"); plot_loss(ax[1], finite(bn_low), \"BN\", \"#1E40FF\")\n",
"ax[1].set_title(f\"low LR = {LOW_LR} (no-BN crawls)\"); ax[1].legend()\n",
"plt.tight_layout(); plt.show()\n",
"print(f\"low LR: no-BN final {no_bn_low[-1]:.3f} vs BN final {bn_low[-1]:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "ba23f50d",
"metadata": {},
"source": [
"> **Interpretation.** Left panel: at the high learning rate the red (no-BN) curve runs off the top or vanishes into `NaN`, while the blue (BN) curve falls smoothly. Right panel: at the low learning rate the red curve barely descends while the blue one still reaches a low loss. Both panels say the same thing the abstract promised: BatchNorm widens the usable learning-rate band. We have reproduced the figure, and the claim survives.\n"
]
},
{
"cell_type": "markdown",
"id": "a8edc239",
"metadata": {},
"source": [
"### Exercise 26.5 — Quantify \"stable\" with one number\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"\"The loss looks stable\" is a vibe. Turn it into the number that goes in the evidence ledger. Fill in `is_stable(losses)`: a run is stable if every value is finite **and** the final loss is no worse than the minimum loss it ever reached plus a small slack (it did not blow up after converging). Return `True`/`False`. Then the check confirms the BN run is stable and the diverged no-BN run is not.\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "fb2cb1f8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:26.663066Z",
"iopub.status.busy": "2026-06-10T20:39:26.662949Z",
"iopub.status.idle": "2026-06-10T20:39:26.666153Z",
"shell.execute_reply": "2026-06-10T20:39:26.665861Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 26.5 stability metric: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def is_stable(losses, slack=0.5):\n",
" \"\"\"A run is stable iff all losses are finite and the final loss <= min(losses) + slack.\"\"\"\n",
" # TODO 1: if any loss is not finite (math.isfinite is False), return False\n",
" # TODO 2: otherwise return whether losses[-1] <= min(losses) + slack\n",
" raise NotImplementedError\n",
"\n",
"def _stable_checks():\n",
" assert is_stable(bn_high) is True, \"the BN high-LR run is stable (finite, no late blow-up)\"\n",
" assert is_stable(no_bn_high) is False, \"the no-BN high-LR run diverged to NaN; not stable\"\n",
" # a run that converges then explodes must be caught by the slack clause, not just the NaN clause\n",
" assert is_stable([1.0, 0.2, 0.1, 5.0]) is False, \"ends 4.9 above its min; the slack clause must catch this\"\n",
" assert is_stable([1.0, 0.5, 0.3, 0.35]) is True, \"small wiggle within slack is still stable\"\n",
"\n",
"check(\"26.5 stability metric\", _stable_checks)"
]
},
{
"cell_type": "markdown",
"id": "babc565d",
"metadata": {},
"source": [
"Hint 1 (conceptual)Two clauses joined by AND. The finiteness clause is `all(math.isfinite(v) for v in losses)`. The no-late-blowup clause compares `losses[-1]` to `min(losses) + slack`. Check finiteness first, because `min` of a list with `NaN` is itself unreliable.\n",
"\n",
"Hint 2 (the lines)\n",
"\n",
"```python\n",
"if not all(math.isfinite(v) for v in losses):\n",
" return False\n",
"return losses[-1] <= min(losses) + slack\n",
"```\n",
"\n",
"\n",
"Help — \"min() of a NaN list gives weird answers\"Exactly why the finiteness check comes first and returns early. If you compute `min(losses)` before filtering `NaN`, the comparison is undefined. Order the two clauses so the finite check short-circuits.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "fa41ff98",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:26.667220Z",
"iopub.status.busy": "2026-06-10T20:39:26.667131Z",
"iopub.status.idle": "2026-06-10T20:39:26.669620Z",
"shell.execute_reply": "2026-06-10T20:39:26.669292Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 26.5 stability metric\n",
"BN high-LR stable: True · no-BN high-LR stable: False\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines is_stable; the checks below re-verify the reference.\n",
"def is_stable(losses, slack=0.5):\n",
" if not all(math.isfinite(v) for v in losses):\n",
" return False\n",
" return losses[-1] <= min(losses) + slack\n",
"\n",
"check(\"26.5 stability metric\", _stable_checks, required=True)\n",
"print(f\"BN high-LR stable: {is_stable(bn_high)} · no-BN high-LR stable: {is_stable(no_bn_high)}\")"
]
},
{
"cell_type": "markdown",
"id": "be27f452",
"metadata": {},
"source": [
"### Experiment log: the BatchNorm claim\n",
"\n",
"The ledger entry for this reproduction. Numbers are from the full run (`NB_FAST` unset); the FAST run shows the same *direction* with fewer steps, so the claim's verdict is unchanged.\n",
"\n",
"| setting | learning rate | BatchNorm | final loss (full) | stable? | reads on the claim |\n",
"|---|---|---|---|---|---|\n",
"| diverge | 3.0 | no | `nan` | no | the failure the paper fixes |\n",
"| tamed | 3.0 | yes | ~0.00 | yes | high LR now works |\n",
"| crawl | 0.1 | no | ~0.69 | yes (but stuck near chance) | cautious LR barely learns |\n",
"| works | 0.1 | yes | ~0.00 | yes | BN learns even at the cautious LR |\n",
"\n",
"> **Key takeaways.** A claim is reproduced by quoting it, building the smallest experiment that tests *that specific promise*, and writing down the deciding number. The BatchNorm abstract promised a wider usable learning-rate band; the toy figure shows exactly that, including the `NaN` divergence that is the claim's reason to exist. The deliberate failure was not a detour; it was the evidence.\n"
]
},
{
"cell_type": "markdown",
"id": "88f626cc",
"metadata": {},
"source": [
"## Part 4 — Re-implement a finding: induction heads\n",
"\n",
"> **Objectives.**\n",
"> - Build a 2-layer attention-only transformer from named parts (a masked attention head, a residual block, token and positional embeddings) with a shape smoke test after each.\n",
"> - Train it on sequences that contain a repeated block, and measure the one number that proves it learned the induction rule: loss on the repeated region versus the rest.\n",
"> - Run the ablation that makes \"two layers\" load-bearing: a 1-layer model on the same data, same budget, and watch the number collapse.\n",
"\n",
"The Induction Heads paper (Olsson et al., 2022) claims that a *2-layer* attention-only transformer learns a \"copying\" circuit: to predict the token after the current one, find the earlier place the current token appeared and copy whatever followed it. That is a two-step lookup (a previous-token head feeding a copy head), so it needs composition across two layers. We will reproduce the *behavioral* signature of that circuit, and the ablation that shows one layer is not enough.\n"
]
},
{
"cell_type": "markdown",
"id": "969fce8d",
"metadata": {},
"source": [
"### Data with a copyable structure\n",
"\n",
"Each sequence is a random block, immediately repeated, then a random tail. The repeated region is the only part that is predictable from earlier context, and the *only* way to predict it is the induction rule. The repeat length varies per sequence, so a model cannot cheat with a fixed positional offset; it must match by content.\n"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "a1bdad25",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:26.670524Z",
"iopub.status.busy": "2026-06-10T20:39:26.670447Z",
"iopub.status.idle": "2026-06-10T20:39:26.674195Z",
"shell.execute_reply": "2026-06-10T20:39:26.673869Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"seq 0: [12, 15, 5, 0, 3, 11, 3, 7, 9, 3, 5, 2, 4, 7, 6, 8, 8, 12, 12, 15, 5, 0, 3, 11, 3, 7, 9, 3, 5, 2, 4, 7, 6, 8, 8, 12, 14, 15, 15, 0, 2, 3, 8, 1, 3, 13, 3, 3]\n",
"repeat mask: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] (1 = second copy of the block)\n"
]
}
],
"source": [
"VOCAB, SEQ = 16, 48 # 16 token types, sequences of length 48\n",
"\n",
"def make_seqs(bs, seqlen=SEQ, vocab=VOCAB, gen=None):\n",
" \"\"\"Each row: a random block of length L, repeated, then random tail.\n",
" Returns (x, is_repeat): is_repeat marks the SECOND copy of the block.\"\"\"\n",
" x = torch.randint(0, vocab, (bs, seqlen), generator=gen) # (bs, seqlen)\n",
" is_repeat = torch.zeros(bs, seqlen, dtype=torch.bool) # (bs, seqlen)\n",
" for b in range(bs):\n",
" L = int(torch.randint(seqlen // 4, seqlen // 2 + 1, (1,), generator=gen).item())\n",
" x[b, L:2 * L] = x[b, :L] # copy the first block into the second slot\n",
" is_repeat[b, L:2 * L] = True # this region is copyable; the rest is random\n",
" return x, is_repeat\n",
"\n",
"g = torch.Generator().manual_seed(SEED)\n",
"xb, rep = make_seqs(2, gen=g)\n",
"print(\"seq 0:\", xb[0].tolist())\n",
"print(\"repeat mask:\", rep[0].int().tolist(), \"(1 = second copy of the block)\")"
]
},
{
"cell_type": "markdown",
"id": "36e9089e",
"metadata": {},
"source": [
"> **Micro-demo.** Look at `seq 0`: the masked region is an exact copy of the block just before it. A model that has learned induction predicts those tokens nearly perfectly; a model that has not is at chance (`ln(16) ≈ 2.77` nats) there.\n"
]
},
{
"cell_type": "markdown",
"id": "c55a1246",
"metadata": {},
"source": [
"### Exercise 26.6 — The causal mask\n",
"`Difficulty 3/5 · ~12 min`\n",
"\n",
"Attention must not look at the future. Fill in `causal_mask(T)`: return a `(T, T)` boolean tensor that is `True` exactly where a query position may **not** attend (strictly future keys), so it can be passed to `masked_fill(mask, -inf)` before the softmax. Position `i` (query) may attend to key `j` only when `j <= i`; mask the `j > i` entries. This is the single sub-skill the attention head depends on, isolated and tested first.\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "d3d5582d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:26.675201Z",
"iopub.status.busy": "2026-06-10T20:39:26.675088Z",
"iopub.status.idle": "2026-06-10T20:39:26.679484Z",
"shell.execute_reply": "2026-06-10T20:39:26.679218Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 26.6 mask shape: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 26.6 mask values: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def causal_mask(T):\n",
" \"\"\"Return a (T, T) bool tensor: True where key j is in the future of query i (j > i).\"\"\"\n",
" # TODO 1: build an upper-triangular boolean mask, offset by 1 (the diagonal is allowed)\n",
" # torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=?) is the move\n",
" raise NotImplementedError\n",
"\n",
"def _mask_shape():\n",
" check_shape(causal_mask(5), (5, 5))\n",
"\n",
"def _mask_values():\n",
" m = causal_mask(4)\n",
" # query 0 may attend only to key 0: positions 1,2,3 are masked (future)\n",
" assert m[0].tolist() == [False, True, True, True], \"query 0 sees only key 0; the rest are future\"\n",
" # query 3 (the last) may attend to everything: nothing masked\n",
" assert m[3].tolist() == [False, False, False, False], \"the last query sees all keys; nothing is future\"\n",
" assert m.dtype == torch.bool, \"the mask must be boolean for masked_fill\"\n",
"\n",
"check(\"26.6 mask shape\", _mask_shape)\n",
"check(\"26.6 mask values\", _mask_values)"
]
},
{
"cell_type": "markdown",
"id": "cb6d88c5",
"metadata": {},
"source": [
"Hint 1 (conceptual)\"Strictly future\" is the strict upper triangle. `torch.triu(..., diagonal=1)` keeps entries strictly above the main diagonal and zeros the rest, which is exactly `j > i`.\n",
"\n",
"Hint 2 (the line)\n",
"\n",
"```python\n",
"return torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)\n",
"```\n",
"`diagonal=1` (not `0`) is what allows a position to attend to itself.\n",
"\n",
"Help — \"query 0 came back all False\"You used `diagonal=0`, which masks the diagonal too, or you inverted the triangle with `tril`. The future is the *strict upper* triangle: `triu` with `diagonal=1`.\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "3918dfe4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:26.680447Z",
"iopub.status.busy": "2026-06-10T20:39:26.680369Z",
"iopub.status.idle": "2026-06-10T20:39:26.683640Z",
"shell.execute_reply": "2026-06-10T20:39:26.683339Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 26.6 mask shape\n",
"[ ok ] 26.6 mask values\n",
"causal_mask(4):\n",
" tensor([[0, 1, 1, 1],\n",
" [0, 0, 1, 1],\n",
" [0, 0, 0, 1],\n",
" [0, 0, 0, 0]], dtype=torch.int32)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines causal_mask; the checks below re-verify the reference.\n",
"def causal_mask(T):\n",
" return torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)\n",
"\n",
"check(\"26.6 mask shape\", _mask_shape, required=True)\n",
"check(\"26.6 mask values\", _mask_values, required=True)\n",
"print(\"causal_mask(4):\\n\", causal_mask(4).int())"
]
},
{
"cell_type": "markdown",
"id": "71b456e6",
"metadata": {},
"source": [
"### The attention head, block, and model\n",
"\n",
"With the mask in hand, the rest is assembly. One head computes masked, scaled dot-product attention. A block runs several heads in parallel and adds the result back to the residual stream (no MLP: this is an *attention-only* model, matching the paper). The model is token + positional embeddings, some blocks, then an unembedding. We define each piece, smoke-test its shape, then build the whole.\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "8d340760",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:26.684741Z",
"iopub.status.busy": "2026-06-10T20:39:26.684641Z",
"iopub.status.idle": "2026-06-10T20:39:26.692076Z",
"shell.execute_reply": "2026-06-10T20:39:26.691590Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"block forward OK, shape (2, 48, 64)\n"
]
}
],
"source": [
"import torch.nn.functional as F\n",
"\n",
"class AttnHead(nn.Module):\n",
" def __init__(self, d, dh):\n",
" super().__init__()\n",
" self.q = nn.Linear(d, dh, bias=False) # query projection\n",
" self.k = nn.Linear(d, dh, bias=False) # key projection\n",
" self.v = nn.Linear(d, dh, bias=False) # value projection\n",
" self.dh = dh\n",
" def forward(self, x): # x: (B, T, d)\n",
" q, k, v = self.q(x), self.k(x), self.v(x) # each (B, T, dh)\n",
" att = (q @ k.transpose(-2, -1)) / math.sqrt(self.dh) # (B, T, T) scores\n",
" att = att.masked_fill(causal_mask(x.shape[1]), float(\"-inf\"))\n",
" return F.softmax(att, dim=-1) @ v # (B, T, dh)\n",
"\n",
"class Block(nn.Module):\n",
" def __init__(self, d, dh, n_heads):\n",
" super().__init__()\n",
" self.heads = nn.ModuleList([AttnHead(d, dh) for _ in range(n_heads)])\n",
" self.proj = nn.Linear(dh * n_heads, d)\n",
" def forward(self, x):\n",
" out = torch.cat([h(x) for h in self.heads], dim=-1) # (B, T, dh*n_heads)\n",
" return x + self.proj(out) # residual add\n",
"\n",
"probe = Block(d=64, dh=16, n_heads=4)(torch.randn(2, SEQ, 64))\n",
"check_shape(probe, (2, SEQ, 64))\n",
"print(\"block forward OK, shape\", tuple(probe.shape))"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "03f38091",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:26.693117Z",
"iopub.status.busy": "2026-06-10T20:39:26.692991Z",
"iopub.status.idle": "2026-06-10T20:39:26.699843Z",
"shell.execute_reply": "2026-06-10T20:39:26.699494Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2-layer model: 38,032 params · logits (2, 48, 16)\n"
]
}
],
"source": [
"class TinyTransformer(nn.Module):\n",
" def __init__(self, vocab=VOCAB, d=64, dh=16, n_heads=4, n_layers=2, seqlen=SEQ):\n",
" super().__init__()\n",
" self.tok = nn.Embedding(vocab, d) # token embedding\n",
" self.pos = nn.Embedding(seqlen, d) # learned positional embedding\n",
" self.blocks = nn.ModuleList([Block(d, dh, n_heads) for _ in range(n_layers)])\n",
" self.unembed = nn.Linear(d, vocab)\n",
" def forward(self, x): # x: (B, T) of token ids\n",
" h = self.tok(x) + self.pos(torch.arange(x.shape[1]))[None] # (B, T, d)\n",
" for blk in self.blocks:\n",
" h = blk(h)\n",
" return self.unembed(h) # (B, T, vocab) logits\n",
"\n",
"m2 = TinyTransformer(n_layers=2)\n",
"logits = m2(make_seqs(2, gen=torch.Generator().manual_seed(SEED))[0])\n",
"check_shape(logits, (2, SEQ, VOCAB))\n",
"print(f\"2-layer model: {sum(p.numel() for p in m2.parameters()):,} params · logits {tuple(logits.shape)}\")"
]
},
{
"cell_type": "markdown",
"id": "6d40c9f1",
"metadata": {},
"source": [
"> **Note:** the model is tiny (a few tens of thousands of parameters) and attention-only by design. We are not chasing a benchmark; we are reproducing a *mechanism*, and the smallest model that can express it is the right one to study.\n"
]
},
{
"cell_type": "markdown",
"id": "28d24a1f",
"metadata": {},
"source": [
"### Train it, and measure the deciding number\n",
"\n",
"Next-token prediction with Adam. The evidence is not the overall loss; it is the loss *split*: loss on the repeated region (where induction should win) versus loss on the rest (random, irreducible at `ln(16)`). We define the split metric, then train.\n",
"\n",
"> **Runtime:** this cell trains two small transformers and takes ~10s on CPU at the full budget (a couple of seconds under `NB_FAST=1`).\n"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "a5b061e1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:26.700904Z",
"iopub.status.busy": "2026-06-10T20:39:26.700821Z",
"iopub.status.idle": "2026-06-10T20:39:33.561607Z",
"shell.execute_reply": "2026-06-10T20:39:33.561166Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2-layer · non-repeat loss 2.952 · repeated-region loss 0.719 · random baseline 2.773\n"
]
}
],
"source": [
"def train_transformer(n_layers, steps, lr=3e-3, bs=64):\n",
" torch.manual_seed(SEED)\n",
" gen = torch.Generator().manual_seed(SEED)\n",
" model = TinyTransformer(n_layers=n_layers)\n",
" opt = torch.optim.Adam(model.parameters(), lr=lr)\n",
" for _ in range(steps):\n",
" x, _ = make_seqs(bs, gen=gen)\n",
" logits = model(x[:, :-1]) # predict next token\n",
" loss = F.cross_entropy(logits.reshape(-1, VOCAB), x[:, 1:].reshape(-1))\n",
" opt.zero_grad(); loss.backward(); opt.step()\n",
" return model\n",
"\n",
"@torch.no_grad()\n",
"def loss_split(model, bs=256):\n",
" \"\"\"Mean next-token loss on the repeated region vs everywhere else (held-out seqs).\"\"\"\n",
" gen = torch.Generator().manual_seed(SEED + 999) # offset seed for eval\n",
" x, is_repeat = make_seqs(bs, gen=gen)\n",
" logits = model(x[:, :-1])\n",
" per_tok = F.cross_entropy(logits.reshape(-1, VOCAB), x[:, 1:].reshape(-1),\n",
" reduction=\"none\").reshape(bs, -1) # (bs, T-1)\n",
" rep = is_repeat[:, 1:] # the token being predicted\n",
" return per_tok[~rep].mean().item(), per_tok[rep].mean().item()\n",
"\n",
"model2 = train_transformer(n_layers=2, steps=IH_STEPS)\n",
"other2, repeat2 = loss_split(model2)\n",
"print(f\"2-layer · non-repeat loss {other2:.3f} · repeated-region loss {repeat2:.3f} · random baseline {math.log(VOCAB):.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "27139cc5",
"metadata": {},
"source": [
"> **Interpretation.** The non-repeat loss sits near the random baseline (`ln(16) ≈ 2.77`), because that region genuinely is unpredictable. The repeated-region loss is far lower: the model is copying. That gap is the behavioral fingerprint of an induction circuit.\n"
]
},
{
"cell_type": "markdown",
"id": "b3b3e2aa",
"metadata": {},
"source": [
"### The ablation: one layer is not enough\n",
"\n",
"The claim is specifically about *two* layers, because the circuit composes two heads. The honest test is to train a 1-layer model with the same budget on the same data and check that the repeated-region loss does *not* collapse. If a 1-layer model could do it, \"two layers\" would be decoration, not mechanism.\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "345b97a6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:33.562426Z",
"iopub.status.busy": "2026-06-10T20:39:33.562346Z",
"iopub.status.idle": "2026-06-10T20:39:37.434127Z",
"shell.execute_reply": "2026-06-10T20:39:37.433805Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1-layer · non-repeat loss 2.881 · repeated-region loss 2.215\n",
"2-layer · non-repeat loss 2.952 · repeated-region loss 0.719\n",
"\n",
"repeated-region loss: 1-layer 2.215 vs 2-layer 0.719 (2-layer is 1.50 nats lower)\n"
]
}
],
"source": [
"model1 = train_transformer(n_layers=1, steps=IH_STEPS)\n",
"other1, repeat1 = loss_split(model1)\n",
"print(f\"1-layer · non-repeat loss {other1:.3f} · repeated-region loss {repeat1:.3f}\")\n",
"print(f\"2-layer · non-repeat loss {other2:.3f} · repeated-region loss {repeat2:.3f}\")\n",
"print(f\"\\nrepeated-region loss: 1-layer {repeat1:.3f} vs 2-layer {repeat2:.3f} \"\n",
" f\"(2-layer is {repeat1 - repeat2:.2f} nats lower)\")"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "2faffdc4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.435258Z",
"iopub.status.busy": "2026-06-10T20:39:37.435125Z",
"iopub.status.idle": "2026-06-10T20:39:37.437384Z",
"shell.execute_reply": "2026-06-10T20:39:37.437021Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] Two layers learn the copying rule; one layer stays near chance. The finding reproduces.\n"
]
}
],
"source": [
"# The finding, as an assertion: the 2-layer model copies the repeat much better than the 1-layer one.\n",
"assert repeat2 < repeat1 - 0.4, (\n",
" f\"induction did not reproduce: 2-layer repeated-region loss {repeat2:.3f} should be well below \"\n",
" f\"the 1-layer {repeat1:.3f}. With more steps the gap widens; check IH_STEPS and the causal mask.\")\n",
"assert repeat2 < other2 - 0.4, (\n",
" f\"the 2-layer model should predict the repeated region far better than the random region; \"\n",
" f\"got repeat {repeat2:.3f} vs non-repeat {other2:.3f}\")\n",
"print(\"[ ok ] Two layers learn the copying rule; one layer stays near chance. The finding reproduces.\")"
]
},
{
"cell_type": "markdown",
"id": "ac763e4b",
"metadata": {},
"source": [
"### The figure: the loss split is the evidence\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "ff33ccb5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.438108Z",
"iopub.status.busy": "2026-06-10T20:39:37.438035Z",
"iopub.status.idle": "2026-06-10T20:39:37.496037Z",
"shell.execute_reply": "2026-06-10T20:39:37.495548Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAGGCAYAAACNCg6xAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAacRJREFUeJzt3XdUFNfbB/DvgvSygFQVQQERFKWoiA1iQ0UFTeyKWH8mdqMmpthSUBNbNLbE2E0ssSQ27L2LvRA1CsSIqAhIEWT3vn8Y5nUFdBfBXeX7OWfPce88M/PMbOHxzt07MiGEABERERG9kp62EyAiIiJ6W7BwIiIiIlITCyciIiIiNbFwIiIiIlITCyciIiIiNbFwIiIiIlITCyciIiIiNbFwIiIiIlITCyciIiIiNbFwIty+fRsymQxLly594/teunQpZDIZbt++/cb3/aZMnDgRMpms1PeTfy5Pnz5d6vt6XSEhIQgJCdF2Gm+l13k/RUVFwdXVtWQTKkEymQwTJ07UdhoAnr1Ha9asqe00SAexcHrLvE1/HJ/37bffYtOmTdpO450wb948rRS5+a5du4axY8fC19cXFhYWcHJyQlhY2Fv3niT6999/MXHiRJw7d07bqbyVsrKyMHHiROzfv1/bqbxRLJzojSiqcOrVqxeys7Ph4uLy5pN6S2m7cPr555/x008/oU6dOpg+fTpGjRqFuLg41K9fH7t379ZaXvT2y87OxhdffPHG9vfvv/9i0qRJLJyKKSsrC5MmTSpzhVM5bSdAZZu+vj709fW1nQZpoFu3bpg4cSLMzc2ltr59+8LLywsTJ05E8+bNtZjd61MqlcjNzYWxsbG2UylzeM6fycrKgqmpqbbToCKwx+kdEBUVBXNzc9y5cwcREREwNzeHnZ0dRo8eDYVCoRKbmpqKqKgoyOVyWFlZoXfv3khNTS2wzaLGoBQ2RkKpVGL27Nnw8fGBsbEx7Ozs0KpVK+nSjUwmQ2ZmJpYtWwaZTAaZTIaoqCgARY9xmjdvHmrUqAEjIyNUqFABgwcPLpBn/hiEK1eu4L333oOpqSkqVqyIadOmFcg7ISEB165de+l5zJecnIx+/frBwcEBxsbGqF27NpYtW6YSkz8u7Pvvv8eiRYvg5uYGIyMj1K1bF6dOnXrp9oODg1G7du1Cl3l6eiI0NLTIdV1dXXH58mUcOHBAOpcvvk45OTkYNWoU7OzsYGZmhg4dOuD+/fsFtrV9+3Y0btwYZmZmsLCwQFhYGC5fvvzS3AEgICBApWgCgPLly6Nx48a4evXqK9cvTG5uLsaPH4+AgADI5XKYmZmhcePG2LdvnxQjhICrqyvCw8MLrP/kyRPI5XL873//k9pycnIwYcIEuLu7w8jICM7Ozhg7dixycnJU1pXJZBgyZAhWrVolved27NhRZK6urq5o27Yt9u/fjzp16sDExAQ+Pj7S/7o3bNggfRYCAgJw9uzZAtvYu3evdO6trKwQHh5e6Lk7fPgw6tatC2NjY7i5uWHhwoVF5rVy5UoEBATAxMQENjY26Nq1KxITE4uMf5Xt27cjODgYFhYWsLS0RN26dbF69WqVmHXr1kn7tLW1Rc+ePXHnzh2VmPzvp7///huhoaEwMzNDhQoVMHnyZAghVGILG+N0584d9O3bFw4ODjAyMkKNGjXwyy+/FMh3zpw5qFGjBkxNTWFtbY06deoUyPd5+/fvR926dQEAffr0kT5PL/bmqvP9ou57rTD532NnzpxBkyZNYGpqis8++0yj7T7/Hvb09JTeewcPHiywP3XOpzqfx9u3b8POzg4AMGnSJOn86coYtVIl6K2yZMkSAUCcOnVKauvdu7cwNjYWNWrUEH379hXz588X77//vgAg5s2bJ8UplUrRpEkToaenJz766CMxZ84c0bRpU1GrVi0BQCxZskSKDQ4OFsHBwQX237t3b+Hi4qLSFhUVJQCI1q1bi1mzZonvv/9ehIeHizlz5gghhFixYoUwMjISjRs3FitWrBArVqwQR48eVTmeW7duSdubMGGCACCaN28u5syZI4YMGSL09fVF3bp1RW5urkqOFSpUEM7OzmL48OFi3rx5omnTpgKA2LZtm0qOwcHBQp23e1ZWlvDy8hIGBgZi5MiR4ocffhCNGzcWAMSsWbOkuFu3bgkAws/PT7i7u4upU6eKadOmCVtbW1GpUiWVPPOPJ99PP/0kAIiLFy+q7PvkyZMCgFi+fHmR+W3cuFFUqlRJVK9eXTqXO3fuVDmXfn5+omnTpmLOnDni448/Fvr6+qJz584q21m+fLmQyWSiVatWYs6cOWLq1KnC1dVVWFlZqbwWmmjQoIGoVq2aWrEvvr/u378vnJycxKhRo8T8+fPFtGnThKenpzAwMBBnz56V4j7//HNhYGAgHj58qLK9tWvXCgDi4MGDQgghFAqFaNmypTA1NRUjRowQCxcuFEOGDBHlypUT4eHhKusCEF5eXsLOzk5MmjRJ/Pjjjyr7fJGLi4vw9PQUTk5OYuLEiWLmzJmiYsWKwtzcXKxcuVJUrlxZTJkyRUyZMkXI5XLh7u4uFAqFtP6uXbtEuXLlRLVq1cS0adPEpEmThK2trbC2tlY59xcuXBAmJiaicuXKIjo6Wnz11VfCwcFB+rw+7+uvvxYymUx06dJFzJs3T9qmq6urePTokRRX2Oe3MEuWLBEymUzUrFlTfPPNN+LHH38U/fv3F7169VKJASDq1q0rZs6cKT799FNhYmJS6D6NjY2Fh4eH6NWrl5g7d65o27atACC+/PLLAq/FhAkTpOdJSUmiUqVKwtnZWUyePFnMnz9ftG/fXgAQM2fOlOIWLVokAIgPPvhALFy4UMyePVv069dPDBs2rMhjTEpKEpMnTxYAxMCBA6XP082bN4UQ6n+/aPJeK0xwcLBwdHQUdnZ2YujQoWLhwoVi06ZNGr+Ha9asKWxtbcXkyZPF1KlThYuLizAxMVH5nlH3fKrzeczIyBDz588XAESHDh2k83f+/PlXHvPbjoXTW6aowgmAmDx5skqsn5+fCAgIkJ5v2rRJABDTpk2T2vLy8qTCoDiF0969ewWAQr+glEql9G8zMzPRu3fvIo8n/w9GcnKyMDQ0FC1btlT5YzN37lwBQPzyyy8qOb5YaOTk5AhHR0fx/vvvq+xH3cJp1qxZAoBYuXKl1JabmyuCgoKEubm5SE9PF0L8f+FUvnx5kZKSIsVu3rxZABB//vmn1PZi4ZSamiqMjY3FJ598orLvYcOGCTMzM5GRkfHSHGvUqFHoa5N/Lps3b65y7keOHCn09fVFamqqEEKIx48fCysrKzFgwACV9ZOSkoRcLi/Qro6DBw8KmUxW4A9hUV58f+Xl5YmcnByVmEePHgkHBwfRt29fqS0uLk4AEPPnz1eJbd++vXB1dZWOe8WKFUJPT08cOnRIJW7BggUCgDhy5IjUBkDo6emJy5cvq5W7i4uLACAV/0IIERMTIwAIExMTER8fL7UvXLhQABD79u2T2nx9fYW9vb1K8Xf+/Hmhp6cnIiMjpbaIiAhhbGyssr0rV64IfX19lffT7du3hb6+vvjmm29U8rx48aIoV66cSrs6hVNqaqqwsLAQgYGBIjs7W2VZ/vnNzc0V9vb2ombNmioxW7ZsEQDE+PHjVfYJQAwdOlRlO2FhYcLQ0FDcv39fan+xcOrXr59wcnISDx48UMmja9euQi6Xi6ysLCGEEOHh4aJGjRovPa7CnDp1qsB3Xz51v180ea8VJn8/CxYsUGnX9D0MQJw+fVpqi4+PF8bGxqJDhw5Sm7rnU93P4/379wu8ZmUBL9W9QwYNGqTyvHHjxvj777+l59u2bUO5cuXw4YcfSm36+voYOnRosff5+++/QyaTYcKECQWWFecn07t370Zubi5GjBgBPb3/f3sOGDAAlpaW2Lp1q0q8ubk5evbsKT03NDREvXr1VI4beNYtL164LFCYbdu2wdHREd26dZPaDAwMMGzYMGRkZODAgQMq8V26dIG1tbX0vHHjxgBQYP/Pk8vlCA8Px6+//irlpFAosGbNGkRERMDMzOyVeb7MwIEDVc5948aNoVAoEB8fDwDYtWsXUlNT0a1bNzx48EB66OvrIzAwUKU7Xh3Jycno3r07qlSpgrFjxxYrZ319fRgaGgJ4duk3JSUFeXl5qFOnDmJjY6W4atWqITAwEKtWrZLaUlJSsH37dvTo0UM67nXr1sHLywvVq1dXOcamTZsCQIFjDA4Ohre3t9r5ent7IygoSHoeGBgIAGjatCkqV65coD3//XD37l2cO3cOUVFRsLGxkeJq1aqFFi1aYNu2bQCevR9iYmIQERGhsj0vL68Cl3I3bNgApVKJzp07qxyro6MjPDw8NH49d+3ahcePH+PTTz8tMOYo//yePn0aycnJ+Oijj1RiwsLCUL169QKfUwAYMmSIynaGDBmC3NzcIn9QIITA77//jnbt2kEIoXJsoaGhSEtLk94bVlZW+Oeff155mVxT6ny/aPpeK4yRkRH69Omj0qbpdoOCghAQECA9r1y5MsLDwxETEwOFQqHR+VT381hWcXD4OyJ/bNHzrK2t8ejRI+l5fHw8nJycCoxP8fT0LPZ+b968iQoVKqj8EXgd+X/cX8zJ0NAQVatWlZbnq1SpUoECzdraGhcuXCj2/j08PFSKNuDZH6zn88v3/B+1/H0DUDnvhYmMjMSaNWtw6NAhNGnSBLt378a9e/fQq1evYuWtSU7Xr18HAOkL+EWWlpZq7yszMxNt27bF48ePcfjwYZX3VkZGBjIyMqTn+vr6Bd6jz1u2bBmmT5+Oa9eu4enTp1J7lSpVVOIiIyMxZMgQxMfHw8XFBevWrcPTp09Vzt3169dx9erVIveXnJys8vzFfbzKi+dYLpcDAJydnQttzz/3Rb2/gWfvsZiYGGRmZuLx48fIzs6Gh4dHgThPT0+pwAKeHasQotBY4Fnhr4mbN28CwEvnMHrZcVSvXh2HDx9WadPT00PVqlVV2qpVqwYARc7hdv/+faSmpmLRokVYtGhRoTH5r+Mnn3yC3bt3o169enB3d0fLli3RvXt3NGzYsMhjUIc63y+avtcKU7FiRalQKe52C3v9q1WrhqysLNy/fx96enpqn09A/c9jWcTC6R1R0r9Mk8lkhfbQvDjYXNuKOm51epe0uf/Q0FA4ODhg5cqVaNKkCVauXAlHR8cS+UXaq3JSKpUAgBUrVsDR0bFAXLly6n0t5ObmomPHjrhw4QJiYmIK/KH9/vvvMWnSJOm5i4tLkX8kV65ciaioKERERGDMmDGwt7eHvr4+oqOjpT/k+bp27YqRI0di1apV+Oyzz7By5UrUqVNH5Y+4UqmEj48PZsyYUej+XixwTExM1DrmfEWdY228H5VKJWQyGbZv317o/l/8j9LbIv992rNnT/Tu3bvQmFq1agF4VnTGxcVhy5Yt2LFjB37//XfMmzcP48ePV3kPakqd11PT91phCnv/lcR2X9weoN751OTzWBaxcCpDXFxcsGfPHmRkZKh8mcbFxRWItba2LvRy04s9Lm5uboiJiUFKSspLe53UvWyXP59TXFycyv9Qc3NzcevWrVL/qbuLiwsuXLgApVKp0uuU/4u8kppvSl9fH927d8fSpUsxdepUbNq0CQMGDFCrAH7dWcjd3NwAAPb29sU+n0qlEpGRkdizZw/Wrl2L4ODgAjGRkZFo1KiR9Pxlxcn69etRtWpVbNiwQeX4CrsEbGNjg7CwMKxatQo9evTAkSNHMGvWLJUYNzc3nD9/Hs2aNXsjs7ar6/n394uuXbsGW1tbmJmZwdjYGCYmJlLv4PNeXNfNzQ1CCFSpUkXqxXkd+e+PS5cuwd3dvdCY54/jxZ7LuLi4Ap8TpVKJv//+WyW/v/76CwCKnMnczs4OFhYWUCgUar1PzczM0KVLF3Tp0kUq6r/55huMGzeuyGkOSuK9UVrvNU23W9h75a+//oKpqanUa6Xu+VT386hLn603iWOcypA2bdogLy8P8+fPl9oUCgXmzJlTINbNzQ3Xrl1T+Rn7+fPnceTIEZW4999/H0KIQv9X9/z/yszMzAqd9uBFzZs3h6GhIX744QeV9RcvXoy0tDSEhYW9chuFUXc6gjZt2iApKQlr1qyR2vLy8jBnzhyYm5sXWiAUV69evfDo0SP873//Q0ZGhspYipdR91wWJTQ0FJaWlvj2229VuuDzFTZ1wYuGDh2KNWvWYN68eejYsWOhMVWrVkXz5s2lx8sum+QXjM+/5idOnMCxY8cKje/VqxeuXLmCMWPGQF9fH127dlVZ3rlzZ9y5cwc//fRTgXWzs7ORmZn5ymMsDU5OTvD19cWyZctUXsNLly5h586daNOmDYBn5yM0NBSbNm1CQkKCFHf16lXExMSobLNjx47Q19fHpEmTCvRsCSHw8OFDjXJs2bIlLCwsEB0djSdPnhTYHgDUqVMH9vb2WLBggcpP47dv346rV68W+jmdO3euynbmzp0LAwMDNGvWrNA89PX18f777+P333/HpUuXCix//n364jEaGhrC29sbQohC3+P58scTvs7nqbTea5pu99ixYyrjjxITE7F582a0bNlSmi9P3fOp7ucxf66p1zl/byP2OJUh7dq1Q8OGDfHpp5/i9u3b8Pb2xoYNG5CWllYgtm/fvpgxYwZCQ0PRr18/JCcnY8GCBahRowbS09OluPfeew+9evXCDz/8gOvXr6NVq1ZQKpU4dOgQ3nvvPWlAaEBAAHbv3o0ZM2agQoUKqFKlijRw9nl2dnYYN24cJk2ahFatWqF9+/aIi4vDvHnzULduXbWLixdFRkbiwIEDr7xkMnDgQCxcuBBRUVE4c+YMXF1dsX79eqlXw8LColj7L4yfnx9q1qwpDQL19/dXa72AgADMnz8fX3/9Ndzd3WFvb1/keKXCWFpaYv78+ejVqxf8/f3RtWtX2NnZISEhAVu3bkXDhg1V/si9aNasWZg3bx6CgoJgamqKlStXqizv0KGDxgPc27Ztiw0bNqBDhw4ICwvDrVu3sGDBAnh7e6uMk8oXFhaG8uXLY926dWjdujXs7e1Vlvfq1Qtr167FoEGDsG/fPjRs2BAKhQLXrl3D2rVrERMTgzp16miUY0n57rvv0Lp1awQFBaFfv37Izs7GnDlzIJfLVebAmTRpEnbs2IHGjRvjo48+kgr4GjVqqIyxcXNzw9dff41x48bh9u3biIiIgIWFBW7duoWNGzdi4MCBGD16tNr5WVpaYubMmejfvz/q1q2L7t27w9raGufPn0dWVhaWLVsGAwMDTJ06FX369EFwcDC6deuGe/fuYfbs2XB1dcXIkSNVtmlsbIwdO3agd+/eCAwMxPbt27F161Z89tlnLx33NmXKFOzbtw+BgYEYMGAAvL29kZKSgtjYWOzevRspKSkAnhV7jo6OaNiwIRwcHHD16lXMnTsXYWFhL/3Murm5wcrKCgsWLICFhQXMzMwQGBio0Tie0nqvabrdmjVrIjQ0FMOGDYORkRHmzZsHACr/qVX3fKr7eTQxMYG3tzfWrFmDatWqwcbGBjVr1nz37/H3xn6/RyWiqOkIzMzMCsS++DN4IYR4+PCh6NWrl7C0tBRyuVz06tVLnD17ttCf5K5cuVJUrVpVGBoaCl9fXxETE1Poz5nz8vLEd999J6pXry4MDQ2FnZ2daN26tThz5owUc+3aNdGkSRNhYmIiAEhTExQ2j5MQz6YfqF69ujAwMBAODg7iww8/VJkbRohnP+Mt7CfIheWo7nQEQghx79490adPH2FraysMDQ2Fj49PgXOTPx3Bd999V2B9vPDz3MJeh3zTpk0TAMS3336rVm5CPJs2ICwsTFhYWAgA0s/6C3tvCCHEvn37CvwkPr89NDRUyOVyYWxsLNzc3ERUVJTKT5oLk//z8qIe6swD9eJ0BEqlUnz77bfCxcVFGBkZCT8/P7Fly5aX/nz+o48+EgDE6tWrC12em5srpk6dKmrUqCGMjIyEtbW1CAgIEJMmTRJpaWlSHAAxePDgV+acz8XFRYSFhRVoL2w7Rb1Pdu/eLRo2bChMTEyEpaWlaNeunbhy5UqBbR44cEAEBAQIQ0NDUbVqVbFgwYIi30+///67aNSokTAzMxNmZmaievXqYvDgwSIuLk6KUXceJyGE+OOPP0SDBg2kHOvVqyd+/fVXlZg1a9YIPz8/YWRkJGxsbESPHj3EP//8oxKT//108+ZNaV4iBwcHMWHCBJUpR/LP4Ys/bb93754YPHiwcHZ2FgYGBsLR0VE0a9ZMLFq0SIpZuHChaNKkiShfvrwwMjISbm5uYsyYMSqvc1E2b94svL29Rbly5VS+BzX5flH3vVaYovajyXbz33srV64UHh4e0mfoxc+8EOqdT00+j0ePHpXeo4W9fu8imRBvaBQtERUwe/ZsjBw5Erdv3y7wSy16uZEjR2Lx4sVISkri7Sl0WFRUFNavX19ozyGVDJlMhsGDB7+0p5hKDsc4EWmJEAKLFy9GcHAwiyYNPXnyBCtXrsT777/PoomI3iiOcSJ6wzIzM/HHH39g3759uHjxIjZv3qztlN4aycnJ2L17N9avX4+HDx9i+PDh2k6JiMoYFk5Eb9j9+/fRvXt3WFlZ4bPPPkP79u21ndJb48qVK+jRowfs7e3xww8/wNfXV9spEVEZwzFORERERGriGCciIiIiNbFwIiIiIlJTmRvjpFQq8e+//8LCwqLMThdPRERE/08IgcePH6NChQoFbvL+ojJXOP37778a3xyRiIiI3n2JiYmoVKnSS2PKXOGUP/1+YmIiLC0ttZwNERERaVt6ejqcnZ3Vuq1WmSuc8i/PWVpasnAiIiIiiTpDeDg4nIiIiEhNLJyIiIiI1MTCiYiIiEhNWh3jNH/+fMyfPx+3b98GANSoUQPjx49H69ati1xn3bp1+PLLL3H79m14eHhg6tSpaNOmzRvKmIiI3iSFQoGnT59qOw16yxkYGEBfX79EtqXVwqlSpUqYMmUKPDw8IITAsmXLEB4ejrNnz6JGjRoF4o8ePYpu3bohOjoabdu2xerVqxEREYHY2FjUrFlTC0dARESlQQiBpKQkpKamajsVekdYWVnB0dHxtedw1Ll71dnY2OC7775Dv379Cizr0qULMjMzsWXLFqmtfv368PX1xYIFC9Tafnp6OuRyOdLS0virOiIiHXX37l2kpqbC3t4epqamnLCYik0IgaysLCQnJ8PKygpOTk4FYjSpDXRmOgKFQoF169YhMzMTQUFBhcYcO3YMo0aNUmkLDQ3Fpk2bitxuTk4OcnJypOfp6eklki8REZUOhUIhFU3ly5fXdjr0DjAxMQEAJCcnw97e/rUu22l9cPjFixdhbm4OIyMjDBo0CBs3boS3t3ehsUlJSXBwcFBpc3BwQFJSUpHbj46Ohlwulx6cNZyISLflj2kyNTXVcib0Lsl/P73umDmtF06enp44d+4cTpw4gQ8//BC9e/fGlStXSmz748aNQ1pamvRITEwssW0TEVHp4eU5Kkkl9X7S+qU6Q0NDuLu7AwACAgJw6tQpzJ49GwsXLiwQ6+joiHv37qm03bt3D46OjkVu38jICEZGRiWbNBEREZVJWu9xepFSqVQZk/S8oKAg7NmzR6Vt165dRY6JIiIiepdFRUUhIiJCa/vv1asXvv32W63tP9+VK1dQqVIlZGZmlvq+tFo4jRs3DgcPHsTt27dx8eJFjBs3Dvv370ePHj0AAJGRkRg3bpwUP3z4cOzYsQPTp0/HtWvXMHHiRJw+fRpDhgzR1iEQERGVSefPn8e2bdswbNgwAM/GDn3yySfw8fGBmZkZKlSogMjISPz7778v3Y6rqytkMlmBx+DBgwEAt2/fLnS5TCbDunXrAADe3t6oX78+ZsyYUboHDS1fqktOTkZkZCTu3r0LuVyOWrVqISYmBi1atAAAJCQkQE/v/2u7Bg0aYPXq1fjiiy/w2WefwcPDA5s2beIcTqQxl/e0nQG9TPw+bWdAVDJyc3NhaGio7TRKxZw5c9CpUyeYm5sDALKyshAbG4svv/wStWvXxqNHjzB8+HC0b98ep0+fLnI7p06dgkKhkJ5funQJLVq0QKdOnQAAzs7OuHv3rso6ixYtwnfffacyYXafPn0wYMAAjBs3DuXKlV55o9Uep8WLF+P27dvIyclBcnIydu/eLRVNALB//34sXbpUZZ1OnTohLi4OOTk5uHTpEmcNJyIinRESEoIhQ4ZgxIgRsLW1RWhoKABgxowZUk+Ms7MzPvroI2RkZEjrLV26FFZWVoiJiYGXlxfMzc3RqlUrlYJBoVBg1KhRsLKyQvny5TF27Fi8OBVjTk4Ohg0bBnt7exgbG6NRo0Y4deqUtHz//v2QyWSIiYmBn58fTExM0LRpUyQnJ2P79u3w8vKCpaUlunfvjqysrCKPU6FQYP369WjXrp3UJpfLsWvXLnTu3Bmenp6oX78+5s6dizNnziAhIaHIbdnZ2cHR0VF6bNmyBW5ubggODgYA6Ovrqyx3dHTExo0b0blzZ6loA4AWLVogJSUFBw4ceNXL9Fp0bowTERFRUXJzc5Gbm6tSMCgUCuTm5iIvL6/EY4tj2bJlMDQ0xJEjR6TJmfX09PDDDz/g8uXLWLZsGfbu3YuxY8eqrJeVlYXvv/8eK1aswMGDB5GQkIDRo0dLy6dPn46lS5fil19+weHDh5GSkoKNGzeqbGPs2LH4/fffsWzZMsTGxsLd3R2hoaFISUlRiZs4cSLmzp2Lo0ePIjExEZ07d8asWbOwevVqbN26FTt37sScOXOKPMYLFy4gLS0NderUeem5SEtLg0wmg5WVlTqnDrm5uVi5ciX69u1b5K/gzpw5g3PnzhWYKNvQ0BC+vr44dOiQWvsqLhZORET01pg5cyZmzpyJ7Oxsqe3EiROYOXMmdu3apRI7d+5czJw5U2Xi49jYWMycORPbt29XiV2wYAFmzpyJBw8eSG0XL14sVo4eHh6YNm0aPD094enpCQAYMWIE3nvvPbi6uqJp06b4+uuvsXbtWpX1nj59igULFqBOnTrw9/fHkCFDVH4QNWvWLIwbNw4dO3aEl5cXFixYALlcLi3PzMzE/PnzpUtY3t7e+Omnn2BiYoLFixer7Ovrr79Gw4YN4efnh379+uHAgQOYP38+/Pz80LhxY3zwwQfYt6/oa+bx8fHQ19eHvb19kTFPnjzBJ598gm7duql9p45NmzYhNTUVUVFRRcYsXrwYXl5eaNCgQYFlFSpUQHx8vFr7Ki4WTkRERCUoICCgQNvu3bvRrFkzVKxYERYWFujVqxcePnyocjnM1NQUbm5u0nMnJyckJycDeNZzc/fuXQQGBkrLy5Urp9Ljc/PmTTx9+hQNGzaU2gwMDFCvXj1cvXpVJZ9atWpJ/3ZwcICpqSmqVq2q0pa/78JkZ2fDyMioyF6hp0+fonPnzhBCYP78+UVu50WLFy9G69atUaFChSL3u3r16kJvywY8myH8ZZcYS4LW53EiIiJS18iRIwE8KwjyBQYGok6dOio/JgIg/eL6+Vh/f3/Url27QOygQYMKxPr4+BQrRzMzM5Xnt2/fRtu2bfHhhx/im2++gY2NDQ4fPox+/fohNzdXmtH6+X0DzyZsLK3byT6/L5lMVui+lUplkevb2toiKyur0MHv+UVTfHw89u7dq3ZvU3x8PHbv3o0NGzYUGbN+/XpkZWUhMjKy0OUpKSkqxWdpYI8TERG9NQwNDWFoaKjS06Gvrw9DQ8MCv6QqidiScObMGSiVSkyfPh3169dHtWrVXvkT/RfJ5XI4OTnhxIkTUlteXh7OnDkjPXdzc5PGVuV7+vQpTp06VeStzIrL19cXAArc6SO/aLp+/Tp2796t0b0GlyxZAnt7e4SFhRUZs3jxYrRv3x52dnaFLr906RL8/PzU3mdxsMepFJxwcdF2CvQqVUv3GjgRUT53d3c8ffoUc+bMQbt27VQGjWti+PDhmDJlCjw8PFC9enXMmDEDqamp0nIzMzN8+OGHGDNmDGxsbFC5cmVMmzYNWVlZRV7aKi47Ozv4+/vj8OHDUhH19OlTfPDBB4iNjcWWLVugUCike8na2NhIPVPNmjVDhw4dVOZgVCqVWLJkCXr37l3kVAI3btzAwYMHsW3btkKX3759G3fu3EHz5s1L8EgLYo8TERFRKapduzZmzJiBqVOnombNmli1ahWio6M13s7HH3+MXr16oXfv3ggKCoKFhQU6dOigEjNlyhS8//776NWrF/z9/XHjxg3ExMTA2tq6pA5H0r9/f6xatUp6fufOHfzxxx/4559/4OvrCycnJ+lx9OhRKe7mzZsqg/CBZ2PAEhIS0Ldv3yL398svv6BSpUpo2bJloct//fVXtGzZEi6l3HkhE6V1AVVHpaenQy6XIy0tTe3rrppij5Pu68weJ53GCTDLtidPnuDWrVuoUqUKjI2NtZ0OFSE7Oxuenp5Ys2aN1m99lpubCw8PD6xevVplcPzzXva+0qQ2YI8TERERaczExATLly8v0HukDQkJCfjss8+KLJpKEsc4ERERUbGEhIRoOwUAz8aRubu7v5F9sceJiIiISE0snIiIiIjUxMKJiIiISE0snIiIiIjUVGYHh+ffBTt/lliFQgGFQgE9PT2Vybdyc3MBPJueXt1YASB/7lmlnh6Enh4gBPSfu9O24r/19BQKyP6bEUIpk0Ho679erL4+IJNBplBA779YIZNBqWksAP3n7h4uxSqV0PtvGn6NYgEo/zuO52Pzz09JxOrl5RU470XFPv8q6ckU0NNTQKnUg1L8/+tZTv/Z65mnMHitWCH0oFA+F6v3FJAJKBTlIP77v4smsTKZEvp6eRBCBoXy/2+ToK/3FDJNY5XlIMR/sVBCXz8PEDLkFTs2DzKZEkqlPpQif9ZlgXL6T/87P4ZqxebmQuU2Dnl5eVAqldDX15dmcxZC4OnTZ9vVJFaTz7I2Y58+fQohBMqVKyfdHkSpVCIvL6/ALTJ0Ibaw865J7POv0fOEENJ3df45y28DoHLrlFfFFtZe3Nj849Pl2BfPjy7EluTrWdzYvLw85ObmFnj/qavMFk4//vgjxowZI90j6MSJEzh06BBq1aqF1q1bS3Fz587F06dPMWjQIOku1LGxsdi7dy+8vb3Rrl07KXbBggXIzs5GXWtrmD16BABI8vTEX++9h/J//w2fHTuk2JNduyLH0hL+69fD8r8bKd53d8fVFi1gnZiI2n/+KcWe+eADZNnYoPamTbD+b5r+h66uuNy6NSzv3oX/xo1S7LkOHfDY3h4+W7ei/H93iH5UsSIutG8PswcPUPe5u3FfaNsWaRUrwjsmBvY3bwIA0h0ccLZjR5ikpiJw9Wop9nKrVkhxcYHnnj1wiosDAGTY2OBMly4wzMhAg+XLpdhrzZrhvrs7PA4eRMVLlwAA2XI5TvboAf2cHDR+7i7dcSEhuFe9OqoePYrK584BAHLNzHCsd2/IFAoEL1woxd5o2BD/+vjA5dQpVDl1CgCQZ2iII/37AwCaLFgA2X8f4FuBgUj084Pz2bNwO3YMACD09HBo4EAAgOHuHOTmPZvHI8DtGOq6H8HFeD8cuvr/E6v1azYb+npKLNv3ETJzLAAAtVxOo0H1/bh2pyb2Xvz/2wL0DpkHI4McrDo4AGlZNgAAb+fzaOK9CzeSPLHzXIQU273JIpgbZ2DNkSg8fOwAAPCocAXNfLYhPrkqtsZ2kmI7N1wCK7NH2HC8B5JSKwEAqtr/hVC/zbiT4ozNJ7tLse/XXwlby2T8caoz/nlYBQBQ2fYWwgLWIznNEeuP9ZZi29ddAyfrO9ge2wG3kqsBAJxs/kFEvV+R8rg8fjvSX4pt4/87nG3jset8W1y/WwMAYGd5Dx80WI70bEusPPChFNui9h+o6nAd+y6F4uo/vgAAa/MH6NboF2TnmmDJ3mFS7Hs1t6NahSs4fLUpLsTXBQBYmKSjV/ACzJ1rgFGjRkmxu3btwoULF9C4cWPpjujZ2dmYM2cOAOCTTz6RYvfv348zZ84gKCgITZo0AfCsAJg5cyaAZ/c6yy+0jhw5gmPHjiEgIEBltuH82KFDh5bKd0Tfvn2lW0ZcvHgRMTEx8PDwQMeOHaXYn3/+Genp6YiMjISTkxMA4OrVq9iyZQtcXFzQtWtXKXbZsmV4+PAhunXrhsqVKwN4Nsvyxo0bUbFiRfTs2VOKXb16NZKSkvDBBx9I9/SKj4/H2rVrYW9vjz59+kix69atQ2JiIsLDw1G9enUAwL///otVq1bB2toaA//7PAHAxo0b8ffff6NNmzbSPd7u37+PpUuXwtzcHIMHD5Zit2zZgri4OLRo0QL+/v4AgEePHuGnn36CkZGRdN844NnNbbOzs2FhYQFzc3MAz/5QJycnQyaTwdHRUYpNT09HVlYWzM3NYWHx7DMrhMC9e/cAQCX28ePHyMzMhJmZmcq8PfmxDg4O0h/gjIwMZGRkwNTUVHqNASA5ORlCCNjb20t/gDMzM/H48WOYmJjAyspKir1//z6USiXs7OykAjkrKwvp6ekwNjZWmaDywYMHUCgUsLW1lQrO7OxspKWlwcjICDY2NlLsw4cPkZeXh/Lly0vv6ydPniA1NRWGhoYqtzxJSUnB06dPYWNjAyMjIwBATk4OHj16BAMDA9ja2kqxjx49Qm5uLqytraU5j3Jzc5GSkoJy5cqp3PIkNTUVOTk5sLKygomJCYBnn7mHDx9CX18f9vb2KrFPnjyBXC6XPlt5eXl48OAB9PT04ODgUOC1t7S0lO7/p1AocP/+/SJf+8LeJwCkzxAAnDx5EqdOnSrwHaEuXqojIiIiUlOZnTn8/v37KF++fKl0w8d6ePBSnY5fqutW+QZ4qU53L9Vd3cZLdbpw+U1bl+qUSqU0w7ORkdE7f6lu0qRJ2Lx5M8791+vOS3WlE5s/c7izszMMDQ1V3n9paWmwsrJSa+bwMnuprrC7YBd2J+znv5DVjZU916anVAL/valUtvFcQSDFCgEU0q5R7HOFUT6ZEIVu443GoojjKOT8vJnY5754hD6UioKv5/N/5Es09rlCozixQugVuj/F68aiJGIL+0qRaRz74seusJt+ymSyQj+fmsSW5Oe+pGOfLzTy6enpFboNXYgt7LxrEvv8a/TkyROV9ue/q9/0La0C4+ML5JDv+T/c+Q4dOoTvvvsOZ86cwd27d7Fx40ZEREQUGpu/3Re3/bJYdXLQ5djC2t90bLly5Qq8LwuLKwov1REREZWQzMxM1K5dGz/++KO2U9GIEAJ5hfwnlApi4URERFRCWrduja+//hodOnQo9jZOnTqFFi1awNbWFnK5HMHBwYiNjZWW9+3bF23btlVZ5+nTp7C3t8fi/358o1QqER0djSpVqsDExAS1a9fG+vXrpfj9+/dDJpNh+/btCAgIgJGREQ4fPlzsnMsSFk5EREQ65PHjx+jduzcOHz6M48ePw8PDA23atMHjx48BAP3798eOHTtw9+5daZ0tW7YgKysLXbp0AQBER0dj+fLlWLBgAS5fvoyRI0eiZ8+eOHDggMq+Pv30U0yZMgVXr15FrVq13txBvsXK7BgnIiIiXdS0aVOV54sWLYKVlRUOHDiAtm3bokGDBvD09MSKFSswduxYAMCSJUvQqVMnmJubIycnB99++y12796NoKAgAEDVqlVx+PBhLFy4EMHBwdK2J0+ejBYtWry5g3sHsMeJiIhIh9y7dw8DBgyAh4cH5HI5LC0tkZGRgYSEBCmmf//+WLJkiRS/fft29O3bF8CzObyysrLQokULmJubS4/ly5fj5n9z9uWrU6fOmzuwdwR7nIiIiHRI79698fDhQ8yePRsuLi4wMjJCUFCQNJ0FAERGRuLTTz/FsWPHcPToUVSpUgWNGzcG8GzCTgDYunUrKlasqLLt/Ikv8+VPLEnqY+FERESkQ44cOYJ58+ahTZs2AIDExEQ8ePBAJaZ8+fKIiIjAkiVLcOzYMZUZ3729vWFkZISEhASVy3JUMlg4ERERlZCMjAzcuHFDen7r1i2cO3cONjY20u1wXsXDwwMrVqxAnTp1kJ6ejjFjxki3Mnle//790bZtWygUCvTu/f+3VLKwsMDo0aMxcuRIKJVKNGrUCGlpaThy5AgsLS1VYklzLJyIiIhKyOnTp/Hee+9Jz/Pvudi7d28sXbpUrW0sXrwYAwcOhL+/P5ydnfHtt99i9OjRBeKaN28OJycn1KhRAxUqVFBZ9tVXX8HOzg7R0dH4+++/YWVlBX9/f3z22WfFPzgCUIZvuaLOtOrF9aZntiXNda4ar+0U6CXi92k7A9Km/FtjVKlSRbrBLBWUkZGBihUrYsmSJSo3iKbCvex9pUltwB4nIiKit4hSqcSDBw8wffp0WFlZoX379tpOqUxh4URERPQWSUhIQJUqVVCpUiUsXbq00Hv/Uenh2SYiInqLuLq6ooyNstEpnACTiIiISE0snIiIiIjUxMKJiIiISE0snIiIiIjUxMKJiIiISE0snIiIiIjUpNXCKTo6GnXr1oWFhQXs7e0RERGBuLi4l66zdOlSyGQylQdnliUiordVVFQUIiIitJ0GqUmr8zgdOHAAgwcPRt26dZGXl4fPPvsMLVu2xJUrV2BmZlbkepaWlioFlkwmexPpEhGRlrm89+qYkqTp7X+io6OxYcMGXLt2DSYmJmjQoAGmTp0KT0/P0kmQ3jitFk47duxQeb506VLY29vjzJkzaNKkSZHryWQyODo6lnZ6REREGiluh4C2KRQKyGQy6OlxBM+rFOsMJSQk4NChQ4iJiUFsbCxycnJKJJm0tDQAgI2NzUvjMjIy4OLiAmdnZ4SHh+Py5cslsn8iIqLXsWPHDkRFRaFGjRqoXbs2li5dioSEBJw5c0ajbTRq1AhWVlYoX7482rZti5s3b0rLmzZtiiFDhqisc//+fRgaGmLPnj0AgJycHIwePRoVK1aEmZkZAgMDsX//fil+6dKlsLKywh9//AFvb28YGRkhISEB+/fvR7169WBmZgYrKys0bNgQ8fG8Kfrz1C6cbt++jU8++QQuLi6oUqUKgoOD0bp1a9SpUwdyuRwtWrTAunXroFQqi5WIUqnEiBEj0LBhQ9SsWbPIOE9PT/zyyy/YvHkzVq5cCaVSiQYNGuCff/4pND4nJwfp6ekqDyIiojdB3Q6B52VmZmLUqFE4ffo09uzZAz09PXTo0EH6+9q/f3+sXr1apdNi5cqVqFixIpo2bQoAGDJkCI4dO4bffvsNFy5cQKdOndCqVStcv35dWicrKwtTp07Fzz//jMuXL8PGxgYREREIDg7GhQsXcOzYMQwcOJDDYV6gVuE0bNgw1K5dG7du3cLXX3+NK1euIC0tDbm5uUhKSsK2bdvQqFEjjB8/HrVq1cKpU6c0TmTw4MG4dOkSfvvtt5fGBQUFITIyEr6+vggODsaGDRtgZ2eHhQsXFhofHR0NuVwuPZydnTXOjYiISFPqdgi86P3330fHjh3h7u4OX19f/PLLL7h48SKuXLkCAOjYsSMAYPPmzdI6S5cuRVRUFGQyGRISErBkyRKsW7cOjRs3hpubG0aPHo1GjRphyZIl0jpPnz7FvHnz0KBBA3h6eiIvLw9paWlo27Yt3Nzc4OXlhd69e6Ny5coldEbeDWqNcTIzM8Pff/+N8uXLF1hmb2+Ppk2bomnTppgwYQJ27NiBxMRE1K1bV+0khgwZgi1btuDgwYOoVKmS+tkDMDAwgJ+fH27cuFHo8nHjxmHUqFHS8/T0dBZPRERU6vI7BA4fPqzRetevX8f48eNx4sQJPHjwQOppSkhIQM2aNWFsbIxevXrhl19+QefOnREbG4tLly7hjz/+AABcvHgRCoUC1apVU9luTk6Oyt9xQ0ND1KpVS3puY2ODqKgohIaGokWLFmjevDk6d+4MJyen4p6Cd5JahVN0dLTaG2zVqpXasUIIDB06FBs3bsT+/ftRpUoVtdfNp1AocPHiRbRp06bQ5UZGRjAyMtJ4u0RERMX1Oh0C7dq1g4uLC3766SdUqFABSqUSNWvWRG5urhTTv39/+Pr64p9//sGSJUvQtGlTuLi4AHg2DlhfXx9nzpyBvr6+yrbNzc2lf5uYmBS4DLdkyRIMGzYMO3bswJo1a/DFF19g165dqF+/vqan4J2l8a/qsrOzIYSAqakpACA+Ph4bN26El5cXQkNDNdrW4MGDsXr1amzevBkWFhZISkoCAMjlcpiYmAAAIiMjUbFiRal4mzx5MurXrw93d3ekpqbiu+++Q3x8PPr376/poRAREZWo1+0QePjwIeLi4vDTTz+hcePGAFBoj5WPjw/q1KmDn376CatXr8bcuXOlZX5+flAoFEhOTpa2oQk/Pz/4+flh3LhxCAoKwurVq1k4PUfjwik8PBwdO3bEoEGDkJqaisDAQBgYGODBgweYMWMGPvzwQ7W3NX/+fABASEiISvuSJUsQFRUF4FnX5PM/j3z06BEGDBiApKQkWFtbIyAgAEePHoW3t7emh0JERFSi1OkQeBlra2uUL18eixYtgpOTExISEvDpp58WGtu/f38MGTIEZmZm6NChg9RerVo19OjRA5GRkZg+fTr8/Pxw//597NmzB7Vq1UJYWFih27t16xYWLVqE9u3bo0KFCoiLi8P169cRGRlZjDPx7tK4cIqNjcXMmTMBAOvXr4eDgwPOnj2L33//HePHj9eocBJCvDLm+Z9PAsDMmTOl/RMREekSdToEXkZPTw+//fYbhg0bhpo1a8LT0xM//PBDge0BQLdu3TBixAh069atwB00lixZgq+//hoff/wx7ty5A1tbW9SvXx9t27Ytct+mpqa4du0ali1bhocPH8LJyQmDBw/G//73v1fmXZbIhDrVy3PyT2zlypXRuXNn1KhRAxMmTEBiYiI8PT2RlZVVWrmWiPT0dMjlcqSlpcHS0rJU9nHiv+vMpLs6V+W8JLpM09ma6d3y5MkT3Lp1C1WqVOEttV7i9u3bcHNzw6lTp+Dv76/tdHTey95XmtQGGk+A6e7ujk2bNiExMRExMTFo2bIlACA5ObnUChEiIiJ65unTp0hKSsIXX3yB+vXrs2h6wzQunMaPH4/Ro0fD1dUVgYGBCAoKAgDs3LkTfn5+JZ4gERER/b8jR47AyckJp06dwoIFC7SdTpmj8RinDz74AI0aNcLdu3dRu3Ztqb1Zs2bSpFxERERUOkJCQtQaI0ylQ+Mep759+8LMzAx+fn4qv3arUaMGpk6dWqLJEREREekSjQunZcuWITs7u0B7dnY2li9fXiJJEREREekitS/VpaenQwgBIQQeP36sMiJdoVBg27ZtsLe3L5UkiYio7CnuTeOJClNS7ye1CycrKyvIZDLIZLIC978BAJlMhkmTJpVIUkREVHYZGhpCT08P//77L+zs7GBoaFjg1iBE6hJCIDc3F/fv34eenh4MDQ1fa3tqF0779u2DEAJNmzbF77//DhsbG2mZoaEhXFxcUKFChddKhoiISE9PD1WqVMHdu3fx77//ajsdekeYmpqicuXKKuOzi0Ptwik4OBjAsynZnZ2dX3vHRERERTE0NETlypWRl5cHhUKh7XToLaevr49y5cqVSM+lxtMR5N99OSsrCwkJCSp3awaAWrVqvXZSREREMpkMBgYGMDAw0HYqRBKNC6f79++jT58+2L59e6HL+T8DIiIieldpfL1txIgRSE1NxYkTJ2BiYoIdO3Zg2bJl8PDwwB9//FEaORIRERHpBI17nPbu3YvNmzejTp060NPTg4uLC1q0aAFLS0tER0cjLCysNPIkIiIi0jqNe5wyMzOl+Zqsra1x//59AICPjw9iY2NLNjsiIiIiHaJx4eTp6Ym4uDgAQO3atbFw4ULcuXMHCxYsgJOTU4knSERERKQrNL5UN3z4cNy9excAMGHCBLRq1QqrVq2CoaEhli5dWtL5EREREekMjQunnj17Sv8OCAhAfHw8rl27hsqVK8PW1rZEkyMiIiLSJRoXTi8yNTWFv79/SeRCREREpNM0LpwUCgWWLl2KPXv2IDk5ucBN8/bu3VtiyRERERHpkmKNcVq6dCnCwsJQs2ZN3niRiIiIygyNC6fffvsNa9euRZs2bUojHyIiIiKdpXHhZGhoCHd399LIhYjonXXiv/t8ku4KjI/Xdgr0FtB4HqePP/4Ys2fPhhCiNPIhIiIi0lka9zgdPnwY+/btw/bt21GjRo0Cd63esGFDiSVHREREpEs0LpysrKzQoUOH0siFiIiISKdpXDgtWbKkNPIgIiIi0nkaj3EiIiIiKqvUKpxatWqF48ePvzLu8ePHmDp1Kn788cfXToyIiIhI16h1qa5Tp054//33IZfL0a5dO9SpUwcVKlSAsbExHj16hCtXruDw4cPYtm0bwsLC8N1335V23kRERERvnFqFU79+/dCzZ0+sW7cOa9aswaJFi5CWlgYAkMlk8Pb2RmhoKE6dOgUvL69STZiIiIhIW9QeHG5kZISePXuiZ8+eAIC0tDRkZ2ejfPnyBaYkICIiInoXafyrunxyuRxyubwkcyEiIiLSafxVHREREZGaWDgRERERqYmFExEREZGaWDgRERERqUnjwikxMRH//POP9PzkyZMYMWIEFi1aVKKJEREREekajQun7t27Y9++fQCApKQktGjRAidPnsTnn3+OyZMnl3iCRERERLpC48Lp0qVLqFevHgBg7dq1qFmzJo4ePYpVq1Zh6dKlGm0rOjoadevWhYWFBezt7REREYG4uLhXrrdu3TpUr14dxsbG8PHxwbZt2zQ9DCIiIiKNaVw4PX36FEZGRgCA3bt3o3379gCA6tWr4+7duxpt68CBAxg8eDCOHz+OXbt24enTp2jZsiUyMzOLXOfo0aPo1q0b+vXrh7NnzyIiIgIRERG4dOmSpodCREREpBGZEEJoskJgYCDee+89hIWFoWXLljh+/Dhq166N48eP44MPPlAZ/6Sp+/fvw97eHgcOHECTJk0KjenSpQsyMzOxZcsWqa1+/frw9fXFggULXrmP9PR0yOVypKWlwdLSsti5vswJF5dS2S6VnM5V47WdAr1E/D5tZ1Dy+L2g+wLj+b1QVmlSG2jc4zR16lQsXLgQISEh6NatG2rXrg0A+OOPP6RLeMWVf/87GxubImOOHTuG5s2bq7SFhobi2LFjr7VvIiIiolfR+JYrISEhePDgAdLT02FtbS21Dxw4EKampsVORKlUYsSIEWjYsCFq1qxZZFxSUhIcHBxU2hwcHJCUlFRofE5ODnJycqTn6enpxc6RiIiIyjaNe5yys7ORk5MjFU3x8fGYNWsW4uLiYG9vX+xEBg8ejEuXLuG3334r9jYKEx0dLd1XTy6Xw9nZuUS3T0RERGWHxoVTeHg4li9fDgBITU1FYGAgpk+fjoiICMyfP79YSQwZMgRbtmzBvn37UKlSpZfGOjo64t69eypt9+7dg6OjY6Hx48aNQ1pamvRITEwsVo5EREREGhdOsbGxaNy4MQBg/fr1cHBwQHx8PJYvX44ffvhBo20JITBkyBBs3LgRe/fuRZUqVV65TlBQEPbs2aPStmvXLgQFBRUab2RkBEtLS5UHERERUXFoPMYpKysLFhYWAICdO3eiY8eO0NPTQ/369RGv4S8SBg8ejNWrV2Pz5s2wsLCQxinJ5XKYmJgAACIjI1GxYkVER0cDAIYPH47g4GBMnz4dYWFh+O2333D69GnOXE5ERESlTuMeJ3d3d2zatAmJiYmIiYlBy5YtAQDJycka9+bMnz8faWlpCAkJgZOTk/RYs2aNFJOQkKAyP1SDBg2wevVqLFq0CLVr18b69euxadOmlw4oJyIiIioJGvc4jR8/Ht27d8fIkSPRtGlT6RLZzp074efnp9G21JlCav/+/QXaOnXqhE6dOmm0LyIiIqLXpXHh9MEHH6BRo0a4e/euNIcTADRr1gwdOnQo0eSIiIiIdInGhRPw7Jdtjo6O0izhlSpVeu3JL4mIiIh0ncZjnJRKJSZPngy5XA4XFxe4uLjAysoKX331FZRKZWnkSERERKQTNO5x+vzzz7F48WJMmTIFDRs2BAAcPnwYEydOxJMnT/DNN9+UeJJEREREukDjwmnZsmX4+eef0b59e6mtVq1aqFixIj766CMWTkRERPTO0vhSXUpKCqpXr16gvXr16khJSSmRpIiIiIh0kcaFU+3atTF37twC7XPnzlX5lR0RERHRu0bjS3XTpk1DWFgYdu/eLc3hdOzYMSQmJmLbtm0lniARERGRrtC4xyk4OBh//fUXOnTogNTUVKSmpqJjx46Ii4uT7mFHRERE9C4q1jxOFSpU4CBwIiIiKnPUKpwuXLig9gZr1apV7GSIiIiIdJlahZOvry9kMtkr7y0nk8mgUChKJDEiIiIiXaNW4XTr1q3SzoOIiIhI56lVOLm4uJR2HkREREQ6T+Nf1RERERGVVSyciIiIiNTEwomIiIhITSyciIiIiNRUrAkwASA3NxfJyclQKpUq7ZUrV37tpIiIiIh0kcaF0/Xr19G3b18cPXpUpV0IwXmciIiI6J2mceEUFRWFcuXKYcuWLXBycoJMJiuNvIiIiIh0jsaF07lz53DmzBlUr169NPIhIiIi0lkaDw739vbGgwcPSiMXIiIiIp2mceE0depUjB07Fvv378fDhw+Rnp6u8iAiIiJ6V2l8qa558+YAgGbNmqm0c3A4ERERves0Lpz27dtXGnkQERER6TyNC6fg4ODSyIOIiIhI5xVr5vBDhw6hZ8+eaNCgAe7cuQMAWLFiBQ4fPlyiyRERERHpEo0Lp99//x2hoaEwMTFBbGwscnJyAABpaWn49ttvSzxBIiIiIl2hceH09ddfY8GCBfjpp59gYGAgtTds2BCxsbElmhwRERGRLtG4cIqLi0OTJk0KtMvlcqSmppZETkREREQ6SePCydHRETdu3CjQfvjwYVStWrVEkiIiIiLSRRoXTgMGDMDw4cNx4sQJyGQy/Pvvv1i1ahVGjx6NDz/8sDRyJCIiItIJGk9H8Omnn0KpVKJZs2bIyspCkyZNYGRkhNGjR2Po0KGlkSMRERGRTtC4cMrLy8Pnn3+OMWPG4MaNG8jIyIC3tzfMzc3x4MED2NralkaeRERERFqn8aW6rl27QggBQ0NDeHt7o169ejA3N8e9e/cQEhJSCikSERER6QaNC6eEhAT0799fpe3u3bsICQlB9erVSywxIiIiIl2jceG0bds2HD16FKNGjQIA/PvvvwgJCYGPjw/Wrl1b4gkSERER6QqNxzjZ2dlh586daNSoEQBgy5Yt8Pf3x6pVq6CnV6w7uBARERG9FYpV6Tg7O2PXrl1YtWoV6tWrh19//RX6+voab+fgwYNo164dKlSoAJlMhk2bNr00fv/+/ZDJZAUeSUlJxTkMIiIiIo2o1eNkbW0NmUxWoD0rKwt//vknypcvL7WlpKSovfPMzEzUrl0bffv2RceOHdVeLy4uDpaWltJze3t7tdclIiIiKi61CqdZs2aVys5bt26N1q1ba7yevb09rKysSj4hIiIiopdQq3Dq3bt3aeehEV9fX+Tk5KBmzZqYOHEiGjZsWGRsTk4OcnJypOfp6elvIkUiIiJ6B2k8OBwAFAoFNm3ahKtXrwIAatSogfbt2xdrnJMmnJycsGDBAtSpUwc5OTn4+eefERISghMnTsDf37/QdaKjozFp0qRSzYuIiIjKBpkQQmiywo0bN9CmTRvcuXMHnp6eAJ6NOXJ2dsbWrVvh5uZWvERkMmzcuBEREREarRccHIzKlStjxYoVhS4vrMfJ2dkZaWlpKuOkStIJF5dS2S6VnM5V47WdAr1E/D5tZ1Dy+L2g+wLj+b1QVqWnp0Mul6tVG2j8q7phw4bBzc0NiYmJiI2NRWxsLBISElClShUMGzas2EkXV7169XDjxo0ilxsZGcHS0lLlQURERFQcGl+qO3DgAI4fPw4bGxuprXz58pgyZcpLxxqVlnPnzsHJyemN75eIiIjKHo0LJyMjIzx+/LhAe0ZGBgwNDTXaVkZGhkpv0a1bt3Du3DnY2NigcuXKGDduHO7cuYPly5cDePbrvipVqqBGjRp48uQJfv75Z+zduxc7d+7U9DCIiIiINKbxpbq2bdti4MCBOHHiBIQQEELg+PHjGDRoENq3b6/Rtk6fPg0/Pz/4+fkBAEaNGgU/Pz+MHz8ewLN74CUkJEjxubm5+Pjjj+Hj44Pg4GCcP38eu3fvRrNmzTQ9DCIiIiKNaTw4PDU1Fb1798aff/4JAwMDAEBeXh7at2+PJUuW6Pz8SpoMACsuDgLVfRwcrts4OJy0gYPDyy5NagONL9VZWVlh8+bNuHHjhjQdgZeXF9zd3YuXLREREdFbQuNLdZMnT0ZWVhbc3d3Rrl07tGvXDu7u7sjOzsbkyZNLI0ciIiIinaBx4TRp0iRkZGQUaM/KyuJEk0RERPRO07hwEkIUesPf8+fPq0xRQERERPSuUXuMk7W1NWQyGWQyGapVq6ZSPCkUCmRkZGDQoEGlkiQRERGRLlC7cJo1axaEEOjbty8mTZoEuVwuLTM0NISrqyuCgoJKJUkiIiIiXaB24dS7d28AQJUqVdCwYUOUK1es+wMTERERvbU0HuMUHBwsFU1hYWG4e/duiSdFREREpIs0Lpyed/DgQWRnZ5dULkREREQ67bUKJyIiIqKy5LUKJxcXF+m2K0RERETvOo0Lp4SEBOTf3u7SpUtwdnYG8Gx+p+dvyEtERET0rtG4cKpSpQru379foD0lJQVVqlQpkaSIiIiIdFGJzRyekZEBY2PjEkmKiIiISBepPRnTqFGjAAAymQxffvklTE1NpWUKhQInTpyAr69viSdIREREpCvULpzOnj0L4FmP08WLF2FoaCgtMzQ0RO3atTF69OiSz5CIiIhIR6hdOO3btw8A0KdPH8yePRuWlpallhQRERGRLtJ4jNO0adOKLJouXrz42gkRERER6SqNCycfHx9s3bq1QPv333+PevXqlUhSRERERLpI48Jp1KhReP/99/Hhhx8iOzsbd+7cQbNmzTBt2jSsXr26NHIkIiIi0gkaF05jx47FsWPHcOjQIdSqVQu1atWCkZERLly4gA4dOpRGjkREREQ6oVi3XHF3d0fNmjVx+/ZtpKeno0uXLnB0dCzp3IiIiIh0isaF05EjR1CrVi1cv34dFy5cwPz58zF06FB06dIFjx49Ko0ciYiIiHSCxoVT06ZN0aVLFxw/fhxeXl7o378/zp49i4SEBPj4+JRGjkREREQ6Qe15nPLt3LkTwcHBKm1ubm44cuQIvvnmmxJLjIiIiEjXaNzjlF803bhxAzExMcjOzgbw/7diISIiInpXaVw4PXz4EM2aNUO1atXQpk0b3L17FwDQr18/3nKFiIiI3mkaF04jR46EgYEBEhISVG7026VLF2zfvr1EkyMiIiLSJcUa4xQTE4NKlSqptHt4eCA+Pr7EEiMiIiLSNRr3OGVmZqr0NOVLSUmBkZFRiSRFREREpIs0LpwaN26M5cuXS89lMhmUSiWmTZuG9957r0STIyIiItIlGl+qmzZtGpo1a4bTp08jNzcXY8eOxeXLl5GSkoIjR46URo5EREREOkHjHqeaNWvir7/+QqNGjRAeHo7MzEx07NgRZ8+ehZubW2nkSERERKQTNO5xAgC5XI7PP/+8pHMhIiIi0mnFKpxSU1Nx8uRJJCcnQ6lUqiyLjIwskcSIiIiIdI3GhdOff/6JHj16ICMjA5aWlpDJZNIymUzGwomIiIjeWRqPcfr444/Rt29fZGRkIDU1FY8ePZIeKSkppZEjERERkU7QuHC6c+cOhg0bVuhcTkRERETvMo0Lp9DQUJw+fbpEdn7w4EG0a9cOFSpUgEwmw6ZNm165zv79++Hv7w8jIyO4u7tj6dKlJZILERER0atoPMYpLCwMY8aMwZUrV+Dj4wMDAwOV5e3bt1d7W5mZmahduzb69u2Ljh07vjL+1q1bCAsLw6BBg7Bq1Srs2bMH/fv3h5OTE0JDQzU9FCIiIiKNyIQQQpMV9PSK7qSSyWRQKBTFS0Qmw8aNGxEREVFkzCeffIKtW7fi0qVLUlvXrl2RmpqKHTt2qLWf9PR0yOVypKWlwdLSsli5vsoJF5dS2S6VnM5VeV9FXRa/T9sZlDx+L+i+QN5vtczSpDbQ+FKdUqks8lHcokldx44dQ/PmzVXaQkNDcezYsVLdLxERERFQzHmctCUpKQkODg4qbQ4ODkhPT0d2djZMTEwKrJOTk4OcnBzpeXp6eqnnSURERO8mjXuc3jbR0dGQy+XSw9nZWdspERER0VvqrSqcHB0dce/ePZW2e/fuwdLSstDeJgAYN24c0tLSpEdiYuKbSJWIiIjeQW/VpbqgoCBs27ZNpW3Xrl0ICgoqch0jIyMYGRmVdmpERERUBmi1xykjIwPnzp3DuXPnADybbuDcuXNISEgA8Ky36PlbuAwaNAh///03xo4di2vXrmHevHlYu3YtRo4cqY30iYiIqIwpVo+TUqnEjRs3Cr3Jb5MmTdTezunTp/Hee+9Jz0eNGgUA6N27N5YuXYq7d+9KRRQAVKlSBVu3bsXIkSMxe/ZsVKpUCT///DPncCIiIqI3QuPC6fjx4+jevTvi4+Px4hRQms7jFBISUmAbzytsVvCQkBCcPXtW7X0QERERlRSNC6dBgwahTp062Lp1K5ycnCCTyUojLyIiIiKdo3HhdP36daxfvx7u7u6lkQ8RERGRztJ4cHhgYCBu3LhRGrkQERER6TSNe5yGDh2Kjz/+GElJSYXe5LdWrVollhwRERGRLtG4cHr//fcBAH379pXaZDIZhBCvdZNfIiIiIl2nceF069at0siDiIiISOdpXDi5uLiURh5EREREOq/Yt1y5cuUKEhISkJubq9Levn37106KiIiISBdpXDj9/fff6NChAy5evCiNbQIgzefEMU5ERET0rtJ4OoLhw4ejSpUqSE5OhqmpKS5fvoyDBw+iTp062L9/fymkSERERKQbNO5xOnbsGPbu3QtbW1vo6elBT08PjRo1QnR0NIYNG8bboRAREdE7S+MeJ4VCAQsLCwCAra0t/v33XwDPBo3HxcWVbHZEREREOkTjHqeaNWvi/PnzqFKlCgIDAzFt2jQYGhpi0aJFqFq1amnkSERERKQTNC6cvvjiC2RmZgIAJk+ejLZt26Jx48YoX7481qxZU+IJEhEREekKjQun0NBQ6d/u7u64du0aUlJSYG1tLf2yjoiIiOhdpPEYp3w3btxATEwMsrOzYWNjU5I5EREREekkjQunhw8folmzZqhWrRratGmDu3fvAgD69euHjz/+uMQTJCIiItIVGhdOI0eOhIGBARISEmBqaiq1d+nSBTt27CjR5IiIiIh0icZjnHbu3ImYmBhUqlRJpd3DwwPx8fEllhgRERGRrtG4xykzM1OlpylfSkoKjIyMSiQpIiIiIl2kceHUuHFjLF++XHouk8mgVCoxbdo0vPfeeyWaHBEREZEu0fhS3bRp09CsWTOcPn0aubm5GDt2LC5fvoyUlBQcOXKkNHIkIiIi0gnFmjn8r7/+wty5c2FhYYGMjAx07NgRgwcPhpOTU2nkSEREVOpceNFEp8Xv03YGz2hcOAGAXC7H559/XtK5EBEREem0YhVOT548wYULF5CcnAylUqmyrH379iWSGBEREZGu0bhw2rFjByIjI/HgwYMCy2QyGRQKRYkkRkRERKRrNP5V3dChQ9GpUyfcvXsXSqVS5cGiiYiIiN5lGhdO9+7dw6hRo+Dg4FAa+RARERHpLI0Lpw8++AD79+8vhVSIiIiIdJvGY5zmzp2LTp064dChQ/Dx8YGBgYHK8mHDhpVYckRERES6ROPC6ddff8XOnTthbGyM/fv3QyaTSctkMhkLJyIiInpnaVw4ff7555g0aRI+/fRT6OlpfKWPiIiI6K2lceWTm5uLLl26sGgiIiKiMkfj6qd3795Ys2ZNaeRCREREpNM0vlSnUCgwbdo0xMTEoFatWgUGh8+YMaPEkiMiIiLSJRoXThcvXoSfnx8A4NKlSyrLnh8oTkRERPSu0bhw2rdPR25PTERERPSGcYQ3ERERkZpYOBERERGpSScKpx9//BGurq4wNjZGYGAgTp48WWTs0qVLIZPJVB7GxsZvMFsiIiIqq7ReOK1ZswajRo3ChAkTEBsbi9q1ayM0NBTJyclFrmNpaYm7d+9Kj/j4+DeYMREREZVVWi+cZsyYgQEDBqBPnz7w9vbGggULYGpqil9++aXIdWQyGRwdHaWHg4PDG8yYiIiIyiqtFk65ubk4c+YMmjdvLrXp6emhefPmOHbsWJHrZWRkwMXFBc7OzggPD8fly5eLjM3JyUF6errKg4iIiKg4tFo4PXjwAAqFokCPkYODA5KSkgpdx9PTE7/88gs2b96MlStXQqlUokGDBvjnn38KjY+OjoZcLpcezs7OJX4cREREVDZo/VKdpoKCghAZGQlfX18EBwdjw4YNsLOzw8KFCwuNHzduHNLS0qRHYmLiG86YiIiI3hUaT4BZkmxtbaGvr4979+6ptN+7dw+Ojo5qbcPAwAB+fn64ceNGocuNjIxgZGT02rkSERERabXHydDQEAEBAdizZ4/UplQqsWfPHgQFBam1DYVCgYsXL8LJyam00iQiIiICoOUeJwAYNWoUevfujTp16qBevXqYNWsWMjMz0adPHwBAZGQkKlasiOjoaADA5MmTUb9+fbi7uyM1NRXfffcd4uPj0b9/f20eBhEREZUBWi+cunTpgvv372P8+PFISkqCr68vduzYIQ0YT0hIgJ7e/3eMPXr0CAMGDEBSUhKsra0REBCAo0ePwtvbW1uHQERERGWETAghtJ3Em5Seng65XI60tDRYWlqWyj5OuLiUynap5HSuyklTdVn8O3gvcX4v6D5+L+i20vxe0KQ2eOt+VUdERESkLSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITTpROP34449wdXWFsbExAgMDcfLkyZfGr1u3DtWrV4exsTF8fHywbdu2N5QpERERlWVaL5zWrFmDUaNGYcKECYiNjUXt2rURGhqK5OTkQuOPHj2Kbt26oV+/fjh79iwiIiIQERGBS5cuveHMiYiIqKzReuE0Y8YMDBgwAH369IG3tzcWLFgAU1NT/PLLL4XGz549G61atcKYMWPg5eWFr776Cv7+/pg7d+4bzpyIiIjKmnLa3Hlubi7OnDmDcePGSW16enpo3rw5jh07Vug6x44dw6hRo1TaQkNDsWnTpkLjc3JykJOTIz1PS0sDAKSnp79m9kXLVCpLbdtUMpR5pff60+srxY+n1vB7Qffxe0G3leb3Qn5NIIR4ZaxWC6cHDx5AoVDAwcFBpd3BwQHXrl0rdJ2kpKRC45OSkgqNj46OxqRJkwq0Ozs7FzNreif8I9d2BvQScr48pA38XtBpb+J74fHjx5C/YkdaLZzehHHjxqn0UCmVSqSkpKB8+fKQyWRazIy0JT09Hc7OzkhMTISlpaW20yEiHcDvhbJNCIHHjx+jQoUKr4zVauFka2sLfX193Lt3T6X93r17cHR0LHQdR0dHjeKNjIxgZGSk0mZlZVX8pOmdYWlpyS9IIlLB74Wy61U9Tfm0Ojjc0NAQAQEB2LNnj9SmVCqxZ88eBAUFFbpOUFCQSjwA7Nq1q8h4IiIiopKi9Ut1o0aNQu/evVGnTh3Uq1cPs2bNQmZmJvr06QMAiIyMRMWKFREdHQ0AGD58OIKDgzF9+nSEhYXht99+w+nTp7Fo0SJtHgYRERGVAVovnLp06YL79+9j/PjxSEpKgq+vL3bs2CENAE9ISICe3v93jDVo0ACrV6/GF198gc8++wweHh7YtGkTatasqa1DoLeMkZERJkyYUOASLhGVXfxeIHXJhDq/vSMiIiIi7U+ASURERPS2YOFEREREpCYWTkRERERqYuFERET0hoWEhGDEiBHaToOKgYUTkZa4urpi1qxZ2k6DiNTEYocAFk5UhuTm5mo7BSLSED+3pGtYOFGpCwkJwbBhwzB27FjY2NjA0dEREydOlJYnJCQgPDwc5ubmsLS0ROfOnVVuqzNx4kT4+vpixYoVcHV1hVwuR9euXfH48eOX7tfV1RVfffUVIiMjYWlpiYEDBwIADh8+jMaNG8PExATOzs4YNmwYMjMzC6zXrVs3mJmZoWLFivjxxx9Vtp2amor+/fvDzs4OlpaWaNq0Kc6fPy8tv3nzJsLDw+Hg4ABzc3PUrVsXu3fvVjkn8fHxGDlyJGQyGe+bSPSfkJAQDBkyBCNGjICtrS1CQ0Nx6dIltG7dGubm5nBwcECvXr3w4MGDAusMGTIEcrkctra2+PLLL1XudJ+Tk4PRo0ejYsWKMDMzQ2BgIPbv3y8tf/jwIbp164aKFSvC1NQUPj4++PXXX6XlUVFROHDgAGbPni19Zm/fvg0Ar8wvMzMTkZGRMDc3h5OTE6ZPn156J5BKHQsneiOWLVsGMzMznDhxAtOmTcPkyZOxa9cuKJVKhIeHIyUlBQcOHMCuXbvw999/o0uXLirr37x5E5s2bcKWLVuwZcsWHDhwAFOmTHnlfr///nvUrl0bZ8+exZdffombN2+iVatWeP/993HhwgWsWbMGhw8fxpAhQ1TW++6776T1Pv30UwwfPhy7du2Slnfq1AnJycnYvn07zpw5A39/fzRr1gwpKSkAgIyMDLRp0wZ79uzB2bNn0apVK7Rr1w4JCQkAgA0bNqBSpUqYPHky7t69i7t3777uKSZ6ZyxbtgyGhoY4cuQIpkyZgqZNm8LPzw+nT5/Gjh07cO/ePXTu3LnAOuXKlcPJkycxe/ZszJgxAz///LO0fMiQITh27Bh+++03XLhwAZ06dUKrVq1w/fp1AMCTJ08QEBCArVu34tKlSxg4cCB69eqFkydPAgBmz56NoKAgDBgwQPrMOjs7IzU19ZX5jRkzBgcOHMDmzZuxc+dO7N+/H7GxsW/gTFKpEESlLDg4WDRq1EilrW7duuKTTz4RO3fuFPr6+iIhIUFadvnyZQFAnDx5UgghxIQJE4SpqalIT0+XYsaMGSMCAwNful8XFxcRERGh0tavXz8xcOBAlbZDhw4JPT09kZ2dLa3XqlUrlZguXbqI1q1bS/GWlpbiyZMnKjFubm5i4cKFReZTo0YNMWfOHJX8Zs6c+dJjICprgoODhZ+fn/T8q6++Ei1btlSJSUxMFABEXFyctI6Xl5dQKpVSzCeffCK8vLyEEELEx8cLfX19cefOHZXtNGvWTIwbN67IXMLCwsTHH3+sktvw4cNVYl6V3+PHj4WhoaFYu3attPzhw4fCxMSkwLbo7cAeJ3ojatWqpfLcyckJycnJuHr1KpydneHs7Cwt8/b2hpWVFa5evSq1ubq6wsLCosD6ALBq1SqYm5tLj0OHDklxderUUdnv+fPnsXTpUpX40NBQKJVK3Lp1S4p78abRQUFBUj7nz59HRkYGypcvr7KdW7du4ebNmwCe9TiNHj0aXl5esLKygrm5Oa5evSr1OBFR0QICAqR/nz9/Hvv27VP5rFWvXh0ApM8bANSvX1/lkndQUBCuX78OhUKBixcvQqFQoFq1airbOXDggLQNhUKBr776Cj4+PrCxsYG5uTliYmJe+Zl9VX43b95Ebm4uAgMDpXVsbGzg6en5+ieKtELr96qjssHAwEDluUwmg1KpLJH127dvr/KlVLFiRenfZmZmKutlZGTgf//7H4YNG1ZgH5UrV1Yrl4yMDDg5OamMj8hnZWUFABg9ejR27dqF77//Hu7u7jAxMcEHH3zAga5Eanj+c5uRkYF27dph6tSpBeKcnJzU2l5GRgb09fVx5swZ6OvrqywzNzcH8Ozy/OzZszFr1iz4+PjAzMwMI0aMeOVn9lX53bhxQ60c6e3Bwom0ysvLC4mJiUhMTJR6na5cuYLU1FR4e3urtQ0LCwuV3qiX8ff3x5UrV+Du7v7SuOPHjxd47uXlJW0jKSkJ5cqVg6ura6HrHzlyBFFRUejQoQOAZ1+u+QNJ8xkaGkKhUKiVN1FZ5e/vj99//x2urq4oV67oP1knTpxQeX78+HF4eHhAX18ffn5+UCgUSE5ORuPGjQtd/8iRIwgPD0fPnj0BAEqlEn/99ZfK91Bhn9lX5efm5gYDAwOcOHFC+s/Zo0eP8NdffyE4OFi9k0A6hZfqSKuaN28OHx8f9OjRA7GxsTh58iQiIyMRHBxc4DJbSfjkk09w9OhRDBkyBOfOncP169exefPmAoPDjxw5gmnTpuGvv/7Cjz/+iHXr1mH48OFSzkFBQYiIiMDOnTtx+/ZtHD16FJ9//jlOnz4NAPDw8MCGDRtw7tw5nD9/Ht27dy/Qw+bq6oqDBw/izp07Kr/AIaL/N3jwYKSkpKBbt244deoUbt68iZiYGPTp00eliElISMCoUaMQFxeHX3/9FXPmzJE+s9WqVUOPHj0QGRmJDRs24NatWzh58iSio6OxdetWAM8+s7t27cLRo0dx9epV/O9//1P5dS/w7DN74sQJ3L59Gw8ePIBSqXxlfubm5ujXrx/GjBmDvXv34tKlS4iKioKeHv/8vq34ypFWyWQybN68GdbW1mjSpAmaN2+OqlWrYs2aNaWyv1q1auHAgQP466+/0LhxY/j5+WH8+PGoUKGCStzHH3+M06dPw8/PD19//TVmzJiB0NBQKedt27ahSZMm6NOnD6pVq4auXbsiPj4eDg4OAIAZM2bA2toaDRo0QLt27RAaGgp/f3+VfUyePBm3b9+Gm5sb7OzsSuV4id52FSpUwJEjR6BQKNCyZUv4+PhgxIgRsLKyUik+IiMjkZ2djXr16mHw4MEYPny4NAUJACxZsgSRkZH4+OOP4enpiYiICJw6dUrqBfriiy/g7++P0NBQhISEwNHRERERESq5jB49Gvr6+vD29oadnR0SEhLUyu+7775D48aN0a5dOzRv3hyNGjVSGcdFbxeZEM9NdEFEcHV1xYgRIzhDMNFbIiQkBL6+vpyJn94I9jgRERERqYmFExEREZGaeKmOiIiISE3scSIiIiJSEwsn0pqHDx/C3t6+wPxGb8Lt27chk8lw7ty5Ut3PlStXUKlSJZWbCBOR7nJ1deUgc3opFk6kNd988w3Cw8OLnETyXeDt7Y369etjxowZ2k6FiNRw6tQplWkMiF7Ewom0IisrC4sXL0a/fv0KXS6EQF5e3hvOqnT06dMH8+fPf2eOh0gXldTtjOzs7GBqaloi26J3Ewsn0opt27bByMgI9evXBwDs378fMpkM27dvR0BAAIyMjHD48GHcvHkT4eHhcHBwgLm5OerWrYvdu3erbMvV1RXffvst+vbtCwsLC1SuXBmLFi1SiTl58iT8/PxgbGyMOnXq4OzZswVyOnDgAOrVqwcjIyM4OTnh008/VSl2QkJCMHToUIwYMQLW1tZwcHDATz/9hMzMTPTp0wcWFhZwd3fH9u3bVbbbokULpKSk4MCBAyV1+ojKvJCQEAwZMgQjRoyAra0tQkNDcenSJbRu3Rrm5uZwcHBAr169VGblf/z4MXr06AEzMzM4OTlh5syZCAkJUZmz7cVLdQkJCQgPD4e5uTksLS3RuXNnlRnFJ06cCF9fX6xYsQKurq6Qy+Xo2rUrHj9+/CZOA2kBCyfSikOHDhU6c+6nn36KKVOm4OrVq6hVqxYyMjLQpk0b7NmzB2fPnkWrVq3Qrl27Ancsnz59ulQQffTRR/jwww8RFxcH4Nl94tq2bQtvb2+cOXMGEydOxOjRo1XWv3PnDtq0aYO6devi/PnzmD9/PhYvXoyvv/5aJW7ZsmWwtbXFyZMnMXToUHz44Yfo1KkTGjRogNjYWLRs2RK9evVCVlaWtI6hoSF8fX1x6NChkjp9RIRnn0dDQ0McOXIEU6ZMQdOmTeHn54fTp09jx44duHfvHjp37izFjxo1CkeOHMEff/yBXbt24dChQ4iNjS1y+0qlEuHh4dJ/fHbt2oW///4bXbp0UYm7efMmNm3ahC1btmDLli04cOAApkyZUmrHTVomiLQgPDxc9O3bV3q+b98+AUBs2rTplevWqFFDzJkzR3ru4uIievbsKT1XKpXC3t5ezJ8/XwghxMKFC0X58uVFdna2FDN//nwBQJw9e1YIIcRnn30mPD09hVKplGJ+/PFHYW5uLhQKhRBCiODgYNGoUSNpeV5enjAzMxO9evWS2u7evSsAiGPHjqnk3KFDBxEVFfXKYyMi9QQHBws/Pz/p+VdffSVatmypEpOYmCgAiLi4OJGeni4MDAzEunXrpOWpqanC1NRUDB8+XGpzcXERM2fOFEIIsXPnTqGvry8SEhKk5ZcvXxYAxMmTJ4UQQkyYMEGYmpqK9PR0KWbMmDEiMDCwJA+XdAh7nEgrsrOzYWxsXKD9xRv7ZmRkYPTo0fDy8oKVlRXMzc1x9erVAj1OtWrVkv4tk8ng6OiI5ORkAJB6r57fX1BQkMr6V69eRVBQEGQymdTWsGFDZGRk4J9//il0P/r6+ihfvjx8fHyktvx71eXvO5+JiYlKLxQRvb7ne63Pnz+Pffv2wdzcXHpUr14dwLMeob///htPnz5FvXr1pHXkcjk8PT2L3P7Vq1fh7OwMZ2dnqc3b2xtWVla4evWq1Obq6goLCwvpuZOTU4HvAHp3lNN2AlQ22dra4tGjRwXazczMVJ6PHj0au3btwvfffw93d3eYmJjggw8+KDAQ1MDAQOW5TCaDUqks8bwL28/zbfmF14v7TklJgZubW4nnQ1SWPf99kZGRgXbt2mHq1KkF4pycnHDjxo1Sy+NNff+QbmCPE2mFn58frly58sq4I0eOICoqCh06dICPjw8cHR01nvfJy8sLFy5cwJMnT6S248ePF4g5duwYxHMT6R85cgQWFhaoVKmSRvsrzKVLl+Dn5/fa2yGiwvn7++Py5ctwdXWFu7u7ysPMzAxVq1aFgYEBTp06Ja2TlpaGv/76q8htenl5ITExEYmJiVLblStXkJqaCm9v71I9HtJdLJxIK0JDQ3H58uVCe52e5+HhgQ0bNuDcuXM4f/48unfvrvH/5Lp37w6ZTIYBAwbgypUr2LZtG77//nuVmI8++giJiYkYOnQorl27hs2bN2PChAkYNWoU9PRe72Ny+/Zt3LlzB82bN3+t7RBR0QYPHoyUlBR069YNp06dws2bNxETE4M+ffpAoVDAwsICvXv3xpgxY7Bv3z5cvnwZ/fr1g56ensol+uc1b94cPj4+6NGjB2JjY3Hy5ElERkYiODi4wLACKjtYOJFW+Pj4wN/fH2vXrn1p3IwZM2BtbY0GDRqgXbt2CA0Nhb+/v0b7Mjc3x59//omLFy/Cz88Pn3/+eYHu/IoVK2Lbtm04efIkateujUGDBqFfv3744osvND62F/36669o2bIlXFxcXntbRFS4ChUq4MiRI1AoFGjZsiV8fHwwYsQIWFlZSf/5mTFjBoKCgtC2bVs0b94cDRs2hJeXV6HjLYFnl9w2b94Ma2trNGnSBM2bN0fVqlWxZs2aN3lopGN4k1/Smq1bt2LMmDG4dOnSa/fq6Krc3Fx4eHhg9erVaNiwobbTIaLnZGZmomLFipg+fXqRk/ESvYiDw0lrwsLCcP36ddy5c0flVyvvkoSEBHz22Wcsmoh0wNmzZ3Ht2jXUq1cPaWlpmDx5MgAgPDxcy5nR24Q9TkREVCacPXsW/fv3R1xcHAwNDREQEIAZM2aoTClC9CosnIiIiIjU9G4OLCEiIiIqBSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITSyciIiIiNTEwomIiIhITf8H4p6NsdxpeucAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: grouped bar chart of the loss split, 1-layer vs 2-layer\n",
"labels = [\"non-repeat\\n(random)\", \"repeated\\nregion\"]\n",
"xpos = np.arange(len(labels))\n",
"fig, ax = plt.subplots(figsize=(6, 4))\n",
"ax.bar(xpos - 0.2, [other1, repeat1], width=0.4, label=\"1 layer\", color=\"#C81E1E\")\n",
"ax.bar(xpos + 0.2, [other2, repeat2], width=0.4, label=\"2 layers\", color=\"#1E40FF\")\n",
"ax.axhline(math.log(VOCAB), ls=\":\", c=\"#888\", label=f\"random ({math.log(VOCAB):.2f})\")\n",
"ax.set_xticks(xpos); ax.set_xticklabels(labels); ax.set_ylabel(\"mean next-token loss (nats)\")\n",
"ax.set_title(\"Induction: only the 2-layer model copies the repeat\"); ax.legend()\n",
"plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "afd0cd96",
"metadata": {},
"source": [
"> **Interpretation.** Both models sit near the dotted random line on the non-repeat region, because nothing could do better there. Only the 2-layer blue bar drops on the repeated region. That single contrast is the reproduction of the paper's central claim at toy scale: induction is a *two-layer* phenomenon.\n"
]
},
{
"cell_type": "markdown",
"id": "098a2d7a",
"metadata": {},
"source": [
"### Exercise 26.7 — Causally verify the copy rule\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"A loss split is correlational evidence. The causal test of an induction circuit: take a held-out repeated sequence, and check that the model's prediction at a position in the *second* copy matches the token that actually followed the same token in the *first* copy. Fill in `induction_accuracy(model)`: for each held-out sequence, look at the last position of the repeated region, and check whether the model's greedy prediction equals the true next token. Return the fraction correct. A 2-layer model should score high; the random baseline is `1/VOCAB`.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "ef16cb66",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.497185Z",
"iopub.status.busy": "2026-06-10T20:39:37.497105Z",
"iopub.status.idle": "2026-06-10T20:39:37.511698Z",
"shell.execute_reply": "2026-06-10T20:39:37.511363Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 26.7 induction accuracy: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"@torch.no_grad()\n",
"def induction_accuracy(model, bs=256):\n",
" \"\"\"Greedy next-token accuracy on positions inside the repeated region.\"\"\"\n",
" gen = torch.Generator().manual_seed(SEED + 1234)\n",
" x, is_repeat = make_seqs(bs, gen=gen)\n",
" logits = model(x[:, :-1]) # (bs, T-1, VOCAB)\n",
" # TODO 1: pred = greedy argmax over the vocab dimension of logits -> (bs, T-1)\n",
" pred = None\n",
" target = x[:, 1:] # (bs, T-1) the true next token\n",
" mask = is_repeat[:, 1:] # (bs, T-1) positions we score\n",
" attempted(pred)\n",
" # TODO 2: return the mean of (pred == target) over the masked positions only.\n",
" # (pred == target)[mask] selects the scored positions; take .float().mean().\n",
" correct = None\n",
" attempted(correct)\n",
" return float(correct)\n",
"\n",
"def _induction_acc():\n",
" acc2 = induction_accuracy(model2)\n",
" acc1 = induction_accuracy(model1)\n",
" base = 1.0 / VOCAB\n",
" assert acc2 > 0.5, f\"2-layer induction accuracy {acc2:.2f} should be well above chance ({base:.2f})\"\n",
" assert acc2 > acc1 + 0.2, f\"2-layer ({acc2:.2f}) should beat 1-layer ({acc1:.2f}) by a clear margin\"\n",
"\n",
"check(\"26.7 induction accuracy\", _induction_acc)"
]
},
{
"cell_type": "markdown",
"id": "19ed84f7",
"metadata": {},
"source": [
"Hint 1 (conceptual)`pred` is `logits.argmax(dim=-1)`. The masked mean is `(pred == target)[mask].float().mean()`: boolean-index the agreement tensor with the repeat mask, then average.\n",
"\n",
"Hint 2 (the lines)\n",
"\n",
"```python\n",
"pred = logits.argmax(dim=-1)\n",
"correct = (pred == target)[mask].float().mean()\n",
"```\n",
"\n",
"\n",
"Help — \"accuracy is near 1/16 for both models\"You are probably averaging over *all* positions, not the masked ones. The non-repeat region is random, so including it drags accuracy to chance. Index with `[mask]` before taking the mean so you only score the copyable positions.\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "39271bc3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.512748Z",
"iopub.status.busy": "2026-06-10T20:39:37.512664Z",
"iopub.status.idle": "2026-06-10T20:39:37.549512Z",
"shell.execute_reply": "2026-06-10T20:39:37.549163Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 26.7 induction accuracy\n",
"induction accuracy: 2-layer 0.82 · 1-layer 0.27 · chance 0.06\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines induction_accuracy; the checks below re-verify the reference.\n",
"@torch.no_grad()\n",
"def induction_accuracy(model, bs=256):\n",
" gen = torch.Generator().manual_seed(SEED + 1234)\n",
" x, is_repeat = make_seqs(bs, gen=gen)\n",
" logits = model(x[:, :-1])\n",
" pred = logits.argmax(dim=-1)\n",
" target = x[:, 1:]\n",
" mask = is_repeat[:, 1:]\n",
" return float((pred == target)[mask].float().mean())\n",
"\n",
"check(\"26.7 induction accuracy\", _induction_acc, required=True)\n",
"print(f\"induction accuracy: 2-layer {induction_accuracy(model2):.2f} · \"\n",
" f\"1-layer {induction_accuracy(model1):.2f} · chance {1/VOCAB:.2f}\")"
]
},
{
"cell_type": "markdown",
"id": "cab663c4",
"metadata": {},
"source": [
"### Experiment log: the induction-head finding\n",
"\n",
"| model | steps (full) | non-repeat loss | repeated-region loss | induction accuracy | reads on the claim |\n",
"|---|---|---|---|---|---|\n",
"| 1 layer | 800 | ~2.8 | ~2.3 | ~0.1 | cannot compose two heads; stays near chance |\n",
"| 2 layers | 800 | ~2.9 | ~0.8 | ~0.8 | learns the copy circuit; the claim reproduces |\n",
"\n",
"> **Key takeaways.** The induction-head finding reproduces at toy scale: a 2-layer attention-only transformer learns to copy from a repeated subsequence, and the 1-layer ablation cannot. The loss split is the correlational evidence; the greedy-copy accuracy is the causal check. \"Two layers\" is load-bearing because the circuit is a two-step lookup, which is exactly the mechanism the paper proposes.\n"
]
},
{
"cell_type": "markdown",
"id": "9b00c3c7",
"metadata": {},
"source": [
"## Safety lens\n",
"\n",
"Reading papers has its own failure modes, and they are about epistemics, not exploits.\n",
"\n",
"**Citation laundering.** A claim originates in a weak or retracted source, gets cited by a stronger-looking paper, then a third, then a survey. By the time you meet it, the trail back to the broken origin is buried under credible-looking citations. The defense is to follow any claim that matters to your work back to its source. We can make the laundering concrete: trace a claim through a small citation chain and find that the \"origin\" is marked unreliable.\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "75bbc0d1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.550487Z",
"iopub.status.busy": "2026-06-10T20:39:37.550411Z",
"iopub.status.idle": "2026-06-10T20:39:37.553062Z",
"shell.execute_reply": "2026-06-10T20:39:37.552697Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"the claim 'Method M is 5x faster' traces to origin_2024, reliable=False\n",
"[ ok ] Following the chain to its origin exposes the laundering the citation count hid.\n"
]
}
],
"source": [
"# A toy citation chain: a survey cites a method paper, which cites a weak origin.\n",
"CHAIN = {\n",
" \"survey_2026\": {\"claim\": \"Method M is 5x faster\", \"cites\": \"method_2025\", \"reliable\": True},\n",
" \"method_2025\": {\"claim\": \"Method M is 5x faster\", \"cites\": \"origin_2024\", \"reliable\": True},\n",
" \"origin_2024\": {\"claim\": \"Method M is 5x faster\", \"cites\": None, \"reliable\": False},\n",
"}\n",
"\n",
"def trace_to_origin(node):\n",
" \"\"\"Walk citations back to the node that cites nothing; report its reliability.\"\"\"\n",
" while CHAIN[node][\"cites\"] is not None:\n",
" node = CHAIN[node][\"cites\"]\n",
" return node, CHAIN[node][\"reliable\"]\n",
"\n",
"origin, reliable = trace_to_origin(\"survey_2026\")\n",
"print(f\"the claim '{CHAIN['survey_2026']['claim']}' traces to {origin}, reliable={reliable}\")\n",
"assert reliable is False, \"the laundered claim's origin is unreliable; the survey inherited it uncritically\"\n",
"print(\"[ ok ] Following the chain to its origin exposes the laundering the citation count hid.\")"
]
},
{
"cell_type": "markdown",
"id": "c330e998",
"metadata": {},
"source": [
"**Hype-cycle reading.** You read in the direction the discourse is flowing (scaling in 2023, RAG in 2024, agents in 2025). The papers you read are a sample of what is fashionable, not what is true, so your worldview lags the field by months and over-weights yesterday's bets. The defense is a deliberate \"old paper\" rotation: read at least one pre-2020 paper a month. The frequency of \"this is what I should have read first\" is humbling.\n",
"\n",
"**LLM-generated papers and reviews.** A growing fraction of arXiv submissions in 2026 are partly machine-written, and the dangerous case is not the empty introduction; it is plausible work with subtly fabricated citations. When you cannot verify a cited paper, do not cite it; when you cannot verify a result, do not condition an important decision on it. This is the new floor of paper-reading hygiene, and it is the same `parse_arxiv_id` guard from Part 1 applied to citations: an id that does not resolve is a citation you cannot use.\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "9b0ffae4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.554044Z",
"iopub.status.busy": "2026-06-10T20:39:37.553963Z",
"iopub.status.idle": "2026-06-10T20:39:37.556524Z",
"shell.execute_reply": "2026-06-10T20:39:37.556112Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1706.03762: resolves=True\n",
"2099.99999: resolves=False\n",
"[ ok ] Unverifiable citations are caught before they enter your knowledge base.\n"
]
}
],
"source": [
"# A fabricated citation fails the same id guard we built in Part 1.\n",
"def citation_resolves(arxiv_id, fetch=canned_fetch):\n",
" \"\"\"A citation is usable only if its id parses AND resolves to real metadata.\"\"\"\n",
" try:\n",
" parse_arxiv_id(arxiv_id)\n",
" fetch(arxiv_id)\n",
" return True\n",
" except (ValueError, KeyError):\n",
" return False\n",
"\n",
"real, fake = \"1706.03762\", \"2099.99999\" # the second is a plausible-looking fabrication\n",
"print(f\"{real}: resolves={citation_resolves(real)}\")\n",
"print(f\"{fake}: resolves={citation_resolves(fake)}\")\n",
"assert citation_resolves(real) and not citation_resolves(fake), \\\n",
" \"a fabricated id should fail to resolve; do not cite what you cannot verify\"\n",
"print(\"[ ok ] Unverifiable citations are caught before they enter your knowledge base.\")"
]
},
{
"cell_type": "markdown",
"id": "8d0a91d6",
"metadata": {},
"source": [
"## Test yourself\n",
"\n",
"Three parts: concept self-checks, two auto-checked problems, and a capstone. Solutions are folded; try before you peek. Every answer is somewhere in this notebook; if unsure, re-run that section.\n"
]
},
{
"cell_type": "markdown",
"id": "1f4e97b8",
"metadata": {},
"source": [
"### Part A — Concepts\n",
"\n",
"1. In Pass 1, why do you read the abstract twice? AnswerThe first read gets you the claim; the second read gets you the qualifiers (by how much, on which eval, vs which baselines, with what compute). The qualifiers are where the work actually lives.\n",
"2. Why is the introduction read last (or skipped) in Pass 1? AnswerThe introduction is the author's marketing pitch, written to convince you to keep reading. Reading it first anchors you to their framing before you have decided whether the framing is right.\n",
"3. \"I have read this paper\" and \"I have re-implemented this paper\": what is the difference? AnswerReading leaves you with a story you could repeat. Re-implementing forces you to confront the choices the paper hand-waved (the part that did not run until you supplied it). One is an opinion; the other is operational knowledge.\n",
"4. In the BatchNorm reproduction, the no-BN net's loss became `nan`. Was that a bug? AnswerNo. It is the exact failure the BatchNorm paper claims to fix: a deep net at a high learning rate diverges without normalization. The `nan` is the negative half of the claim reproducing.\n",
"5. Why does the induction circuit need *two* layers? AnswerThe rule is a two-step lookup: find the earlier occurrence of the current token (a previous-token head), then copy whatever followed it (a copy head). One head must feed the other, which is composition across layers. A single layer cannot compose, so it stays near chance on the repeated region.\n",
"6. In Part 4 the non-repeat loss stayed near `2.77` for both models. Why is that the *right* answer there, not a failure? Answer`ln(16) ≈ 2.77` is the loss of a perfect model on uniformly random tokens. The non-repeat region genuinely is random, so its loss is irreducible; staying at the baseline there means the model is not hallucinating structure that is not present.\n",
"7. Citation count is high for a 2017 paper and low for a 2026 paper. Which should you read? AnswerIt depends on your question. Citation count is a lagging indicator of visibility, not importance. A 2026 mechanistic paper cited 600 times can matter more to a \"how does this work\" question than a 2017 paper cited 95000 times. Read against your question.\n",
"8. You see a claim cited by a survey, by a method paper, and by a blog post. Is it true? AnswerUnknown. Three citations of a laundered claim are three repetitions, not three verifications. Follow the chain to the origin (as in the Safety lens cell) and judge the origin.\n"
]
},
{
"cell_type": "markdown",
"id": "6cf943b2",
"metadata": {},
"source": [
"### Part B — Auto-checked problems\n",
"\n",
"#### B1 — The evidence-strength score\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"Turn the Pass-3 critique into a number. A claim's evidence is stronger when there is an experiment that *could* falsify it and that experiment is in the paper. Fill in `evidence_score(has_falsifier, in_paper, on_new_benchmark)`: start at 0; `+1` if a falsifying experiment exists, `+1` if it is actually in the paper, `+1` if the evaluation is on a benchmark the authors did not pick (a held-out test). Return the integer 0-3. A score of 3 is the gold standard (a falsifier, run, on an independent benchmark).\n"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "99041d44",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.557379Z",
"iopub.status.busy": "2026-06-10T20:39:37.557296Z",
"iopub.status.idle": "2026-06-10T20:39:37.560537Z",
"shell.execute_reply": "2026-06-10T20:39:37.560160Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] B1 evidence score: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def evidence_score(has_falsifier, in_paper, on_new_benchmark):\n",
" \"\"\"Sum three booleans into an evidence strength in 0..3.\"\"\"\n",
" # TODO 1: return the count of conditions that are True (each True is worth 1)\n",
" raise NotImplementedError\n",
"\n",
"def _evidence():\n",
" assert evidence_score(True, True, True) == 3, \"all three signals present is the max score\"\n",
" assert evidence_score(False, False, False) == 0, \"no falsifier, nothing run, no independent eval is 0\"\n",
" assert evidence_score(True, True, False) == 2, \"falsifier in the paper but on the authors' own benchmark is 2\"\n",
" assert evidence_score(True, False, False) == 1, \"a falsifier that exists but was not run is only 1\"\n",
"\n",
"check(\"B1 evidence score\", _evidence)"
]
},
{
"cell_type": "markdown",
"id": "4de36556",
"metadata": {},
"source": [
"Hint 1The cleanest form is `int(has_falsifier) + int(in_paper) + int(on_new_benchmark)`. Booleans are 0/1 under `int`.\n",
"Solution\n",
"\n",
"```python\n",
"def evidence_score(has_falsifier, in_paper, on_new_benchmark):\n",
" return int(has_falsifier) + int(in_paper) + int(on_new_benchmark)\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "474c339f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.561463Z",
"iopub.status.busy": "2026-06-10T20:39:37.561392Z",
"iopub.status.idle": "2026-06-10T20:39:37.563415Z",
"shell.execute_reply": "2026-06-10T20:39:37.563076Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] B1 evidence score\n",
"an independent re-evaluation on a held-out benchmark scores 3 / 3\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines evidence_score; the check below re-verifies the reference.\n",
"def evidence_score(has_falsifier, in_paper, on_new_benchmark):\n",
" return int(has_falsifier) + int(in_paper) + int(on_new_benchmark)\n",
"\n",
"check(\"B1 evidence score\", _evidence, required=True)\n",
"print(\"an independent re-evaluation on a held-out benchmark scores\",\n",
" evidence_score(True, True, True), \"/ 3\")"
]
},
{
"cell_type": "markdown",
"id": "b6e75840",
"metadata": {},
"source": [
"#### B2 — Pick the reproduction target\n",
"`Difficulty 2/5 · ~8 min`\n",
"\n",
"When you re-implement a paper, you reproduce *one* small, decisive number, not the whole benchmark table. Fill in `is_good_target(target)`: given a dict with keys `scope` (`\"single_result\"` or `\"full_benchmark\"`) and `compute` (`\"cpu_minutes\"`, `\"single_gpu\"`, or `\"cluster\"`), return `True` only if the target is a single result reproducible on CPU minutes or a single GPU. A full benchmark, or anything needing a cluster, is not a re-read; it is a project.\n"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "0191e650",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.564125Z",
"iopub.status.busy": "2026-06-10T20:39:37.564058Z",
"iopub.status.idle": "2026-06-10T20:39:37.566803Z",
"shell.execute_reply": "2026-06-10T20:39:37.566508Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] B2 reproduction target: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def is_good_target(target):\n",
" \"\"\"True iff scope is a single result AND compute is cpu_minutes or single_gpu.\"\"\"\n",
" # TODO 1: return the boolean combining the scope and compute conditions\n",
" raise NotImplementedError\n",
"\n",
"def _target():\n",
" assert is_good_target({\"scope\": \"single_result\", \"compute\": \"cpu_minutes\"}) is True\n",
" assert is_good_target({\"scope\": \"single_result\", \"compute\": \"single_gpu\"}) is True\n",
" assert is_good_target({\"scope\": \"full_benchmark\", \"compute\": \"cpu_minutes\"}) is False, \\\n",
" \"the whole benchmark table is a project, not a re-read\"\n",
" assert is_good_target({\"scope\": \"single_result\", \"compute\": \"cluster\"}) is False, \\\n",
" \"a cluster-scale single result is still out of reach for a re-read\"\n",
"\n",
"check(\"B2 reproduction target\", _target)"
]
},
{
"cell_type": "markdown",
"id": "49ff4d72",
"metadata": {},
"source": [
"Hint 1Two conditions joined by `and`: `target[\"scope\"] == \"single_result\"` and `target[\"compute\"] in {\"cpu_minutes\", \"single_gpu\"}`.\n",
"Solution\n",
"\n",
"```python\n",
"def is_good_target(target):\n",
" return target[\"scope\"] == \"single_result\" and target[\"compute\"] in {\"cpu_minutes\", \"single_gpu\"}\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "9a9bfb4d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.567606Z",
"iopub.status.busy": "2026-06-10T20:39:37.567539Z",
"iopub.status.idle": "2026-06-10T20:39:37.569546Z",
"shell.execute_reply": "2026-06-10T20:39:37.569162Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] B2 reproduction target\n",
"induction-head loss split on CPU is a good target: True\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines is_good_target; the check below re-verifies the reference.\n",
"def is_good_target(target):\n",
" return target[\"scope\"] == \"single_result\" and target[\"compute\"] in {\"cpu_minutes\", \"single_gpu\"}\n",
"\n",
"check(\"B2 reproduction target\", _target, required=True)\n",
"print(\"induction-head loss split on CPU is a good target:\",\n",
" is_good_target({\"scope\": \"single_result\", \"compute\": \"cpu_minutes\"}))"
]
},
{
"cell_type": "markdown",
"id": "3570ff67",
"metadata": {},
"source": [
"### Part C — Capstone: reproduce a third claim end to end\n",
"\n",
"You have reproduced two claims. The capstone is to run the full loop yourself on a third, using only the pieces in this notebook. Pick a small, falsifiable claim from a paper you can state in one sentence, then:\n",
"\n",
"**Deliverables**\n",
"1. A one-line claim and the single number that would falsify it (an evidence-ledger row, as in Parts 3 and 4).\n",
"2. The smallest experiment that tests *that specific number*, runnable on CPU in under two minutes, with a fixed seed.\n",
"3. An assertion that decides the claim: `assert` the reproduced number is on the right side of the threshold, with a teaching message.\n",
"4. A two-sentence post-mortem: what the paper did *not* say that you had to figure out. That delta is the actual content of the paper.\n",
"\n",
"**Self-assessment (pass / partial / fail)**\n",
"- (a) the claim is stated as one falsifiable sentence with a specific number, not a vibe;\n",
"- (b) the experiment is the *smallest* one that tests that number (no benchmark table);\n",
"- (c) the seed is fixed and the run is under two minutes on CPU;\n",
"- (d) the deciding `assert` has a message that names the likely failure;\n",
"- (e) the post-mortem names a concrete thing the paper left implicit.\n",
"\n",
"A worked reference follows: the same induction finding, restated as a *positional-generalization* claim and tested with a single new assertion. Try yours before opening it.\n",
"\n",
"My solution (reference)\n",
"\n",
"The claim, restated: *an induction circuit copies by content, so it should still copy when the repeat starts at a position it rarely saw in training.* The falsifier: induction accuracy on sequences whose repeat is forced into the back half of the context should stay well above chance; if the model were cheating with a fixed positional offset, it would fail there.\n",
"\n",
"```python\n",
"@torch.no_grad()\n",
"def late_repeat_accuracy(model, bs=256):\n",
" # force the repeated block to start late, then measure greedy copy accuracy there\n",
" gen = torch.Generator().manual_seed(SEED + 7)\n",
" x = torch.randint(0, VOCAB, (bs, SEQ), generator=gen)\n",
" is_rep = torch.zeros(bs, SEQ, dtype=torch.bool)\n",
" for b in range(bs):\n",
" L = SEQ // 4\n",
" start = SEQ // 2 # repeat starts in the back half\n",
" x[b, start:start + L] = x[b, start - L:start]\n",
" is_rep[b, start:start + L] = True\n",
" logits = model(x[:, :-1])\n",
" pred = logits.argmax(dim=-1)\n",
" mask = is_rep[:, 1:]\n",
" return float((pred == x[:, 1:])[mask].float().mean())\n",
"\n",
"acc = late_repeat_accuracy(model2)\n",
"assert acc > 0.4, (\n",
" f\"content-based induction should generalize to late repeats; got {acc:.2f}. \"\n",
" f\"If this is near chance ({1/VOCAB:.2f}), the model learned a positional shortcut, not the circuit.\")\n",
"print(f\"late-repeat induction accuracy: {acc:.2f} (chance {1/VOCAB:.2f})\")\n",
"```\n",
"\n",
"Post-mortem: the paper says \"copy from the repeat\" but does not spell out that the *evaluation* must vary the repeat's position, or a positional shortcut would pass the easy version of the test. That implicit experimental-design choice is the part you only discover by building the eval yourself.\n"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "67ac903a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.570380Z",
"iopub.status.busy": "2026-06-10T20:39:37.570281Z",
"iopub.status.idle": "2026-06-10T20:39:37.573736Z",
"shell.execute_reply": "2026-06-10T20:39:37.573354Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] C capstone reproduction: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Your capstone scaffold. Fill in a claim, the experiment, and the deciding assert.\n",
"# The reference above shows the shape; yours can target any small claim you can state in one line.\n",
"CLAIM = \"an induction circuit copies by content, so it generalizes to late-positioned repeats\"\n",
"\n",
"@torch.no_grad()\n",
"def my_reproduction(model):\n",
" # TODO: build the smallest experiment that tests YOUR claim's number and return it.\n",
" # (The folded reference implements late_repeat_accuracy; you may adapt it or write your own.)\n",
" raise NotImplementedError\n",
"\n",
"def _capstone():\n",
" val = my_reproduction(model2)\n",
" # TODO: replace this with an assertion that decides YOUR claim, with a teaching message.\n",
" assert val > 1.0 / VOCAB, f\"reproduced value {val:.2f} should beat chance ({1/VOCAB:.2f})\"\n",
"\n",
"# This check is yours to satisfy; it prints [ -- ] until you fill in my_reproduction.\n",
"check(\"C capstone reproduction\", _capstone)"
]
},
{
"cell_type": "markdown",
"id": "5dedde36",
"metadata": {},
"source": [
"Hint — getting unstuckThe shortest path is to copy `late_repeat_accuracy` from the folded reference into `my_reproduction` and return its value, then keep the provided beat-chance assertion. That already satisfies all five self-assessment criteria for the worked claim. The learning is in then *changing the claim* to one of your own and rebuilding the experiment around it.\n",
"\n",
"Solution (the reference, wired to the check)\n",
"\n",
"```python\n",
"@torch.no_grad()\n",
"def my_reproduction(model, bs=256):\n",
" gen = torch.Generator().manual_seed(SEED + 7)\n",
" x = torch.randint(0, VOCAB, (bs, SEQ), generator=gen)\n",
" is_rep = torch.zeros(bs, SEQ, dtype=torch.bool)\n",
" for b in range(bs):\n",
" L, start = SEQ // 4, SEQ // 2\n",
" x[b, start:start + L] = x[b, start - L:start]\n",
" is_rep[b, start:start + L] = True\n",
" pred = model(x[:, :-1]).argmax(dim=-1)\n",
" return float((pred == x[:, 1:])[is_rep[:, 1:]].float().mean())\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "ae45e4e5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:39:37.574377Z",
"iopub.status.busy": "2026-06-10T20:39:37.574314Z",
"iopub.status.idle": "2026-06-10T20:39:37.597669Z",
"shell.execute_reply": "2026-06-10T20:39:37.597344Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] C capstone reproduction\n",
"late-repeat induction accuracy: 0.22 (chance 0.06)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: a working reproduction so the capstone check passes on the canonical run.\n",
"@torch.no_grad()\n",
"def my_reproduction(model, bs=256):\n",
" gen = torch.Generator().manual_seed(SEED + 7)\n",
" x = torch.randint(0, VOCAB, (bs, SEQ), generator=gen)\n",
" is_rep = torch.zeros(bs, SEQ, dtype=torch.bool)\n",
" for b in range(bs):\n",
" L, start = SEQ // 4, SEQ // 2 # repeat forced into the back half\n",
" x[b, start:start + L] = x[b, start - L:start]\n",
" is_rep[b, start:start + L] = True\n",
" pred = model(x[:, :-1]).argmax(dim=-1)\n",
" return float((pred == x[:, 1:])[is_rep[:, 1:]].float().mean())\n",
"\n",
"check(\"C capstone reproduction\", _capstone, required=True)\n",
"print(f\"late-repeat induction accuracy: {my_reproduction(model2):.2f} (chance {1/VOCAB:.2f})\")"
]
},
{
"cell_type": "markdown",
"id": "89ebe221",
"metadata": {},
"source": [
"## Reflection\n",
"\n",
"Write ~150 words, for yourself, on the dumbest bug you hit in this notebook and how you found it. A strong candidate: the induction reproduction is sensitive to the *evaluation*, not just the model. If you score accuracy over all positions instead of only the masked repeated region, both models look like they are at chance, and you might wrongly conclude the circuit did not form. The fix was indexing with the repeat mask before taking the mean. Another candidate: at the high learning rate, `min(losses)` on a list containing `NaN` is unreliable, so the stability check had to test finiteness first. Nobody grades this. Writing it is how you turn a bug you fixed into a bug you will recognize on sight next time, which is the entire return on the Pass-3 habit.\n"
]
},
{
"cell_type": "markdown",
"id": "229d6396",
"metadata": {},
"source": [
"## Going further\n",
"\n",
"- **The Annotated Transformer** (Harvard NLP) — the prototype of paper-as-code: the Vaswani et al. equations interleaved with the implementation until the line between the two disappears. Read it once front to back; the interleave is the lesson.\n",
"- **nanoGPT** (Karpathy) — the same move applied to GPT-2: read, re-implement, ship the re-implementation as the explanation. The model file is small enough to hold in your head.\n",
"- **\"In-context Learning and Induction Heads\"** (Olsson et al., 2022) — the paper Part 4 reproduces in miniature. ARENA's mech-interp track builds the full-scale version with `transformer_lens`.\n",
"- **\"How to Read a Paper\"** (Keshav, 2007) — the three-pass method this notebook operationalizes; two pages, still the cleanest protocol.\n",
"- **Eugene Yan, \"How I read papers\" and the Zettelkasten note-taking writeup** — the most replicable model for a knowledge base you will not abandon in three months.\n",
"- **Connected Papers and Semantic Scholar** — the real citation-graph tools the canned graph in Part 2 stands in for. Paste an arXiv id; read the cluster structure.\n",
"\n",
"## What this enables\n",
"\n",
"- **Independent practice.** You can now decide in five minutes whether a new result deserves your attention and at what depth, and you have the reproduce-a-claim loop to settle the ones that matter.\n",
"- **The whole curriculum behind you.** Every chapter in obvix-learn was a distillation of papers. The triage, three-pass, and re-implementation habits here are how you do that distillation yourself, for the results that have not been written into a curriculum yet.\n",
"- **The gap this leaves.** We reproduced two findings at toy scale. The same loop at *real* scale is harder: you need the compute, the eval discipline of Ch 23, and the mech-interp tooling of Ch 22 to verify a circuit in a model you did not train. Toy-scale reproduction is the rehearsal; the frontier is the performance.\n"
]
},
{
"cell_type": "markdown",
"id": "b96e4039",
"metadata": {},
"source": [
"---\n",
"*Built top-to-bottom. If every check above printed `[ ok ]`, you have reproduced the chapter, including two real paper claims at toy scale. Runtime stamp written by CI.*\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "obvix-nb",
"language": "python",
"name": "obvix-nb"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"obvix": {
"title": "Ch 26 — Reading Papers"
}
},
"nbformat": 4,
"nbformat_minor": 5
}