{
"cells": [
{
"cell_type": "markdown",
"id": "5551960e",
"metadata": {},
"source": [
"# Ch 22 — Mechanistic Interpretability (notebook)\n",
"\n",
"`[← 21 retrieval-augmented-generation]` · **this notebook** · `[23 evaluation-science →]`\n",
"\n",
"Runs top-to-bottom in ~3 min on free Colab CPU. Last verified 2026-06-11.\n",
"\n",
"**What you'll build**\n",
"- A 2-layer attention-only transformer, trained from scratch on synthetic repeat sequences, that learns to copy-by-pattern in seconds on CPU.\n",
"- An induction-stripe detector: the exact attention-pattern signature that says \"this head is an induction head\", computed and asserted against ground truth.\n",
"- An ablation experiment that knocks out the circuit and measures the damage, plus a control that knocks out the wrong heads and measures nothing.\n",
"- An out-of-distribution test that proves the circuit is a real algorithm, not a memorized lookup table, and a from-scratch activation-patch that localizes *where* the information flows.\n",
"- The OV-circuit decomposition that shows, in the vocabulary basis, that the head literally copies whatever token it attends to.\n",
"\n",
"**How this notebook works.** Code cells with a `# TODO` are yours to fill in. Run the cell to grade yourself: `[ ok ]` passed, `[FAIL]` shows what went wrong, `[ -- ]` means not attempted yet. Every exercise has a hint ladder (open only as many as you need) and a folded solution below it. The notebook runs top-to-bottom even if you fill in nothing: the solution cells redefine the functions so the later cells work. See Ch 00 for the full protocol.\n"
]
},
{
"cell_type": "markdown",
"id": "b7a23cce",
"metadata": {},
"source": [
"## Before you start\n",
"\n",
"1. Feed a model the sequence `... A B ... A B ... A` (random tokens, then a partial repeat). It predicts `B` after the final `A`, far above chance. What information does it need at the last position to do this? AnswerIt needs to know \"what token followed the previous occurrence of the current token\". At the last `A`, find the earlier `A`, look one step right, copy that. That is the induction algorithm. It needs a *previous-token* signal at every position plus a way to match the current token against it.\n",
"2. A linear probe reads \"is the model thinking about France?\" off layer 6 with 95% accuracy. Does that prove the model *uses* that information to produce its output? AnswerNo. A probe is correlational. High probe accuracy means the information is decodable, not that any downstream component reads it. The model could represent France as an epiphenomenon it never acts on. Only an intervention experiment (ablate or steer the direction, measure the behavior change) settles \"uses\".\n",
"3. Predict before you run: you ablate the single highest-scoring induction head in a 4-head layer. The model's accuracy on the repeat task barely changes. Bug, or expected? AnswerExpected, and the lesson of this notebook. Small transformers build *redundant* circuits: several heads learn the same induction behavior, so removing one leaves backups. This echoes the \"backup name mover heads\" finding in the IOI circuit. You have to ablate the whole layer to see the damage.\n"
]
},
{
"cell_type": "markdown",
"id": "becb687a",
"metadata": {},
"source": [
"## Setup\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "855a5222",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:05.767874Z",
"iopub.status.busy": "2026-06-10T20:56:05.767789Z",
"iopub.status.idle": "2026-06-10T20:56:06.591443Z",
"shell.execute_reply": "2026-06-10T20:56:06.591058Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"numpy 2.2.6 · torch 2.12.0+cpu\n",
"device cpu (this notebook is CPU-canonical; GPU is not needed)\n"
]
}
],
"source": [
"import numpy as np\n",
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import matplotlib.pyplot as plt\n",
"print(f\"numpy {np.__version__} · torch {torch.__version__}\")\n",
"if np.__version__ < \"2.0\":\n",
" print(\"WARN: written for NumPy 2.x; older versions may shift the last digit\")\n",
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
"print(f\"device {device} (this notebook is CPU-canonical; GPU is not needed)\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "5d0f6963",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:06.592789Z",
"iopub.status.busy": "2026-06-10T20:56:06.592665Z",
"iopub.status.idle": "2026-06-10T20:56:06.600526Z",
"shell.execute_reply": "2026-06-10T20:56:06.600152Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"FAST=False · STEPS=320 · seq_len=64 (half=32) · vocab=50\n"
]
}
],
"source": [
"import os, math, random\n",
"SEED = 0\n",
"FAST = bool(os.environ.get('NB_FAST')) # CI smoke mode: ~3x fewer training steps, same code paths\n",
"# Training-step budget. Induction emerges by ~100 steps at lr=5e-3; we run more for a sharp\n",
"# stripe. The check threshold (induction score > 0.5) holds under BOTH settings (see experiment log).\n",
"STEPS = 120 if FAST else 320\n",
"VOCAB = 50 # small vocab so the model can learn fast; token 0 reserved, draw from [1, VOCAB)\n",
"SEQ_LEN = 64 # sequence length; first half random, second half an exact repeat -> HALF = 32\n",
"D_MODEL = 48 # residual-stream width; 48 = 4 heads x 12 dims per head\n",
"N_HEADS = 4 # heads per layer; redundancy across heads is a feature we will observe\n",
"N_LAYERS = 2 # the minimal depth for induction: prev-token work in L0, matching in L1\n",
"BATCH = 64 # sequences per training step\n",
"LR = 5e-3 # AdamW step size tuned so induction emerges in ~100 steps on CPU\n",
"\n",
"rng = np.random.default_rng(SEED)\n",
"torch.manual_seed(SEED); random.seed(SEED)\n",
"print(f'FAST={FAST} · STEPS={STEPS} · seq_len={SEQ_LEN} (half={SEQ_LEN // 2}) · vocab={VOCAB}')\n",
"\n",
"# ── house self-check harness (identical across all chapter notebooks) ──\n",
"import numpy as _np\n",
"\n",
"def check(label, test_fn, required=False):\n",
" \"\"\"Run one self-check. test_fn raises AssertionError (with a teaching\n",
" message) on failure, NotImplementedError if the stub is unfilled.\n",
" required=True is used only in solution cells; it is what CI grades.\"\"\"\n",
" try:\n",
" test_fn()\n",
" except NotImplementedError:\n",
" if required:\n",
" raise AssertionError(f\"{label}: reference solution incomplete\")\n",
" print(f\"[ -- ] {label}: not attempted yet — fill in the TODO above, then re-run.\")\n",
" return False\n",
" except AssertionError as e:\n",
" if required:\n",
" raise\n",
" print(f\"[FAIL] {label}: {e}\")\n",
" return False\n",
" print(f\"[ ok ] {label}\")\n",
" return True\n",
"\n",
"def attempted(*vals):\n",
" \"\"\"Treat None placeholders as 'not attempted'.\"\"\"\n",
" if any(v is None for v in vals):\n",
" raise NotImplementedError\n",
"\n",
"def check_shape(x, want):\n",
" assert tuple(x.shape) == tuple(want), \\\n",
" f\"shape {tuple(x.shape)}, expected {tuple(want)} — check your reshape/transpose order\"\n",
"\n",
"def check_close(got, want, atol=1e-5, rtol=1e-4, msg=\"\"):\n",
" g, w = _np.asarray(got, dtype=float), _np.asarray(want, dtype=float)\n",
" assert g.shape == w.shape, f\"shape {g.shape} vs expected {w.shape}. {msg}\"\n",
" bad = ~_np.isclose(g, w, atol=atol, rtol=rtol)\n",
" assert not bad.any(), \\\n",
" f\"{bad.mean():.2%} of values wrong (max diff {abs(g - w).max():.3g}). {msg}\""
]
},
{
"cell_type": "markdown",
"id": "da6513f1",
"metadata": {},
"source": [
"> **Note:** seeds make this notebook's printed numbers reproduce on CPU. Library versions and BLAS threading can shift the last digit or two, and on a different seed the *index* of the strongest induction head can move (head 2 here, head 1 elsewhere). The notebook keys every check on a behavioral property (an induction score above a threshold, a copying fraction, an accuracy drop), never on a fragile loss value or a fixed head index. That is deliberate: the old version of this lab pinned a 600-step loss threshold and went red in CI.\n"
]
},
{
"cell_type": "markdown",
"id": "7e4c3552",
"metadata": {},
"source": [
"## The map\n",
"\n",
"> **Part 1 — The interp question.** Why \"look at a neuron\" failed: polysemanticity, superposition, and the residual-stream reframe that makes every component's contribution linear and readable.\n",
"> **Part 2 — A transformer you can see through.** Build a 2-layer attention-only transformer from scratch, with hooks on every attention pattern. Generate the synthetic repeat data with known ground truth.\n",
"> **Part 3 — Train it, and watch induction appear.** The four-line training loop. A deliberate failure (forget to zero the gradients) that prevents the circuit from forming, then the fix.\n",
"> **Part 4 — Find the induction head.** Define the induction stripe, compute the per-head score, and assert the head exists. This is the circuit-discovery move.\n",
"> **Part 5 — Prove it causally.** Ablate the heads and measure the damage; ablate the wrong heads and measure nothing; verify out-of-distribution; localize the information flow with activation patching from scratch.\n",
"> **Part 6 — Read the circuit.** Decompose the OV circuit in the vocabulary basis and watch the head copy tokens. Then the safety lens, and an optional `transformer_lens` + GPT-2 capstone.\n"
]
},
{
"cell_type": "markdown",
"id": "08efcefc",
"metadata": {},
"source": [
"## Part 1 — The interp question, and why \"look at neurons\" was not enough\n",
"\n",
"> **Objectives.** Name what mechanistic interpretability is trying to produce (a *program*, not an average-case explanation). State why single-neuron inspection fails. Write down the residual-stream decomposition that the whole chapter rests on.\n",
"\n",
"Mechanistic interpretability is the project of reverse-engineering a neural network into a description a human can audit: which variables it represents, which operations it composes, what computation it actually runs. Not \"what does it do on average\" (that is evaluation). Not \"which input tokens mattered\" (that is feature attribution). A *program*.\n",
"\n",
"The blunt fact that motivated the field: looking at single neurons in a transformer's MLP does not work. Neurons are **polysemantic**, one neuron's activations span many unrelated concepts. The cause is **superposition**: a network packs more features than it has neurons by representing each feature as a sparse combination of neurons, accepting interference, and using nonlinearities to denoise. Polysemanticity is the symptom; superposition is the disease. Let us see the symptom with our own eyes before we trust the claim.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "18eb49fe",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:06.601487Z",
"iopub.status.busy": "2026-06-10T20:56:06.601380Z",
"iopub.status.idle": "2026-06-10T20:56:06.620982Z",
"shell.execute_reply": "2026-06-10T20:56:06.620550Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"neuron 0 alignment with each of the 5 feature directions:\n",
" feature 0: +0.19\n",
" feature 1: +0.16\n",
" feature 2: +0.74\n",
" feature 3: -0.90\n",
" feature 4: -0.88\n"
]
}
],
"source": [
"# A 3-neuron toy \"layer\" forced to store 5 features in superposition.\n",
"# Each feature is a near-orthogonal random direction; the layer activations are their\n",
"# sparse sum. We then read out one neuron and ask: which features does it respond to?\n",
"rng_local = np.random.default_rng(SEED)\n",
"n_features, n_neurons = 5, 3\n",
"feature_dirs = rng_local.standard_normal((n_features, n_neurons)) # (5, 3): 5 features in 3 neurons\n",
"feature_dirs /= np.linalg.norm(feature_dirs, axis=1, keepdims=True) # unit vectors\n",
"\n",
"# 2000 sparse inputs: usually 1 feature active, occasionally 2\n",
"acts = []\n",
"for _ in range(2000):\n",
" k = 1 if rng_local.random() < 0.8 else 2\n",
" on = rng_local.choice(n_features, size=k, replace=False)\n",
" mag = rng_local.uniform(0.5, 1.0, size=k)\n",
" acts.append((feature_dirs[on] * mag[:, None]).sum(0))\n",
"acts = np.array(acts) # (2000, 3): neuron activations\n",
"\n",
"# Neuron 0 responds to which features? Its alignment is its coordinate in each feature direction.\n",
"print(\"neuron 0 alignment with each of the 5 feature directions:\")\n",
"for f in range(n_features):\n",
" print(f\" feature {f}: {feature_dirs[f, 0]:+.2f}\")"
]
},
{
"cell_type": "markdown",
"id": "f3a1f97c",
"metadata": {},
"source": [
"> **Interpretation.** Neuron 0 has a non-trivial response to *several* of the five features at once, because three neurons cannot give each of five features its own axis. That is polysemanticity in miniature: read neuron 0 in isolation and you cannot say \"this neuron means feature 3\". The features are real and roughly recoverable, but only in the *right linear basis*, not the neuron basis. Sparse autoencoders (covered in the chapter prose, §10) are the modern tool for finding that basis. This notebook takes the other road the field opened first: attention circuits, where the structure is legible without an SAE.\n"
]
},
{
"cell_type": "markdown",
"id": "a12d60c9",
"metadata": {},
"source": [
"### The residual-stream reframe\n",
"\n",
"Recall from Ch 15 that a transformer block does not *transform* the activation `x`, it *adds to* it. Every attention head and every MLP reads from `x` and writes a delta back. The final logits are one linear read of the accumulated stream:\n",
"\n",
"$$\\text{logits} = W_U \\, x_{\\text{final}}, \\qquad x_{\\text{final}} = x_{\\text{embed}} + \\sum_l \\text{attn}_l(x_l) + \\sum_l \\text{mlp}_l(x_l)$$\n",
"\n",
"Every term is a vector in residual-stream space. Because $W_U$ is linear, each component has a *direct, additive* contribution to every logit. That is the lever the whole chapter pulls: to ask \"why did the model output this token\", you decompose $x_{\\text{final}}$ and see who wrote what. We make that decomposition concrete in Part 6. First we need a transformer simple enough to see all the way through, and that is what attention-only buys us.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "32802a8c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:06.621973Z",
"iopub.status.busy": "2026-06-10T20:56:06.621902Z",
"iopub.status.idle": "2026-06-10T20:56:06.625620Z",
"shell.execute_reply": "2026-06-10T20:56:06.625349Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"max difference: 1.9073486328125e-06\n"
]
}
],
"source": [
"# The decomposition is exact, not approximate, when there is no LayerNorm. Verify on random tensors:\n",
"torch.manual_seed(SEED)\n",
"d = 8\n",
"x_embed = torch.randn(d)\n",
"attn0, attn1 = torch.randn(d), torch.randn(d) # per-layer attention writes\n",
"mlp0, mlp1 = torch.randn(d), torch.randn(d) # per-layer mlp writes (our model has none, shown for generality)\n",
"W_U_demo = torch.randn(5, d) # (vocab=5, d_model=8)\n",
"\n",
"x_final = x_embed + attn0 + attn1 + mlp0 + mlp1\n",
"logits_direct = W_U_demo @ x_final\n",
"# Sum of per-component logit contributions must equal the logits computed from the summed stream:\n",
"logits_from_parts = sum(W_U_demo @ c for c in [x_embed, attn0, attn1, mlp0, mlp1])\n",
"print(\"max difference:\", (logits_direct - logits_from_parts).abs().max().item())"
]
},
{
"cell_type": "markdown",
"id": "9951cb2a",
"metadata": {},
"source": [
"> **Interpretation.** The difference is at floating-point noise. The logit is *literally* a sum of per-component contributions, so attributing an output to components is well defined, not a heuristic. (The one wrinkle is the final LayerNorm, which is approximately linear once you fold in its per-position scale. Our model has no LayerNorm, so we get the clean case.)\n",
"\n",
"> **Key takeaways.** Mech interp wants a program, not an average. Single neurons are polysemantic because of superposition, so the neuron basis is the wrong basis. The residual stream makes every component's contribution to the logits an exact additive term, which is the foundation of every tool in this chapter.\n"
]
},
{
"cell_type": "markdown",
"id": "fcc3e6fe",
"metadata": {},
"source": [
"## Part 2 — A transformer you can see through\n",
"\n",
"> **Objectives.** Build a 2-layer attention-only transformer from scratch with no MLPs, no LayerNorm, no biases, the minimal model in which induction can form. Cache every attention pattern so we can inspect it. Generate synthetic repeat data whose ground-truth answer we know exactly.\n",
"\n",
"Attention-only means the only thing each block does is move information between positions. There is no per-token MLP to muddy the picture. This is the model Elhage et al. used to first describe induction heads, and it is small enough to train in seconds on a laptop CPU. We build one attention layer first, smoke-test its shapes on a random input, then assemble the full model in a single cell.\n"
]
},
{
"cell_type": "markdown",
"id": "88e455ae",
"metadata": {},
"source": [
"### One attention layer, with a causal mask\n",
"\n",
"The shapes are how you debug a transformer. Burn them in: `q`, `k`, `v` each become `(batch, n_heads, seq, d_head)` so heads attend in parallel. The pattern is `(batch, n_heads, seq, seq)`. We stash the post-softmax pattern in `self.last_pattern` so the induction detector in Part 4 can read it without re-running the model.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "bee64f10",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:06.626556Z",
"iopub.status.busy": "2026-06-10T20:56:06.626483Z",
"iopub.status.idle": "2026-06-10T20:56:06.631495Z",
"shell.execute_reply": "2026-06-10T20:56:06.631110Z"
}
},
"outputs": [],
"source": [
"class AttentionOnlyLayer(nn.Module):\n",
" \"\"\"One attention block: causal, multi-head, no MLP, no LayerNorm, no biases.\"\"\"\n",
" def __init__(self, d_model, n_heads, max_seq_len):\n",
" super().__init__()\n",
" assert d_model % n_heads == 0, \"d_model must split evenly across heads\"\n",
" self.n_heads, self.d_head = n_heads, d_model // n_heads\n",
" self.W_Q = nn.Linear(d_model, d_model, bias=False) # query projection\n",
" self.W_K = nn.Linear(d_model, d_model, bias=False) # key projection\n",
" self.W_V = nn.Linear(d_model, d_model, bias=False) # value projection\n",
" self.W_O = nn.Linear(d_model, d_model, bias=False) # output projection (concat heads -> d_model)\n",
" causal = torch.triu(torch.ones(max_seq_len, max_seq_len), diagonal=1).bool()\n",
" self.register_buffer(\"mask\", causal.view(1, 1, max_seq_len, max_seq_len))\n",
" self.last_pattern = None # filled on every forward: (batch, n_heads, seq, seq)\n",
"\n",
" def forward(self, x, ablate_heads=()):\n",
" B, T, C = x.shape # (batch, seq, d_model)\n",
" h, dh = self.n_heads, self.d_head\n",
" q = self.W_Q(x).view(B, T, h, dh).transpose(1, 2) # (B, h, T, dh)\n",
" k = self.W_K(x).view(B, T, h, dh).transpose(1, 2) # (B, h, T, dh)\n",
" v = self.W_V(x).view(B, T, h, dh).transpose(1, 2) # (B, h, T, dh)\n",
" scores = q @ k.transpose(-2, -1) / math.sqrt(dh) # (B, h, T, T)\n",
" scores = scores.masked_fill(self.mask[:, :, :T, :T], float(\"-inf\")) # causal: no peeking ahead\n",
" pattern = F.softmax(scores, dim=-1) # (B, h, T, T), rows sum to 1\n",
" self.last_pattern = pattern.detach()\n",
" z = pattern @ v # (B, h, T, dh): attention output per head\n",
" if ablate_heads: # zero specified heads (used in Part 5)\n",
" z = z.clone()\n",
" for head in ablate_heads:\n",
" z[:, head] = 0.0\n",
" z = z.transpose(1, 2).contiguous().view(B, T, C) # (B, T, d_model): concat heads\n",
" return self.W_O(z)"
]
},
{
"cell_type": "markdown",
"id": "d998264e",
"metadata": {},
"source": [
"> **Predict:** what shape comes out of a forward pass on a `(2, 10, 48)` input? Figure it out before running.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "e60dc20a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:06.632380Z",
"iopub.status.busy": "2026-06-10T20:56:06.632311Z",
"iopub.status.idle": "2026-06-10T20:56:06.637110Z",
"shell.execute_reply": "2026-06-10T20:56:06.636797Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"output: (2, 10, 48) · pattern: (2, 4, 10, 10)\n",
"pattern row sums all ~1: True\n"
]
}
],
"source": [
"# randn smoke test: a module is not trusted until its shapes check out on garbage input.\n",
"_layer = AttentionOnlyLayer(d_model=D_MODEL, n_heads=N_HEADS, max_seq_len=SEQ_LEN)\n",
"_out = _layer(torch.randn(2, 10, D_MODEL))\n",
"print(\"output:\", tuple(_out.shape), \"· pattern:\", tuple(_layer.last_pattern.shape))\n",
"# the attention pattern is a probability distribution over keys: each query row sums to 1\n",
"row_sums = _layer.last_pattern.sum(dim=-1)\n",
"print(\"pattern row sums all ~1:\", torch.allclose(row_sums, torch.ones_like(row_sums), atol=1e-5))"
]
},
{
"cell_type": "markdown",
"id": "89133804",
"metadata": {},
"source": [
"> **Interpretation.** Output is `(2, 10, 48)`, same shape as the input, because attention writes a delta of the same width. The pattern is `(2, 4, 10, 10)`: for each of 4 heads, a 10x10 matrix whose rows are softmax distributions. The causal mask makes the matrix lower-triangular (a query at position `t` can only attend to keys `<= t`).\n"
]
},
{
"cell_type": "markdown",
"id": "da307f71",
"metadata": {},
"source": [
"### Assemble the full model\n",
"\n",
"Now the whole transformer in one cell: token + positional embeddings, two attention layers added into the residual stream, an unembed. We keep references to `tok_emb`, `pos_emb`, and `unembed` because Part 6 reads their weights directly to decompose the OV circuit. The `ablate` argument lets us knock out heads in a chosen layer, which Part 5 needs.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "664b264e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:06.638094Z",
"iopub.status.busy": "2026-06-10T20:56:06.638022Z",
"iopub.status.idle": "2026-06-10T20:56:06.642768Z",
"shell.execute_reply": "2026-06-10T20:56:06.642487Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"model built · 26,304 parameters · 2 layers x 4 heads\n"
]
}
],
"source": [
"class TinyTransformer(nn.Module):\n",
" def __init__(self, vocab, d_model, n_heads, n_layers, max_seq_len):\n",
" super().__init__()\n",
" self.tok_emb = nn.Embedding(vocab, d_model) # W_E: (vocab, d_model)\n",
" self.pos_emb = nn.Embedding(max_seq_len, d_model) # learned positional embedding\n",
" self.layers = nn.ModuleList(\n",
" [AttentionOnlyLayer(d_model, n_heads, max_seq_len) for _ in range(n_layers)])\n",
" self.unembed = nn.Linear(d_model, vocab, bias=False) # W_U: (vocab, d_model)\n",
" self.n_layers, self.n_heads = n_layers, n_heads\n",
" self.d_model, self.d_head = d_model, d_model // n_heads\n",
"\n",
" def forward(self, idx, ablate=None):\n",
" # idx: (batch, seq) of token ids. ablate: optional {layer_index: [head_indices]} to zero.\n",
" B, T = idx.shape\n",
" x = self.tok_emb(idx) + self.pos_emb(torch.arange(T, device=idx.device)) # (B, T, d_model)\n",
" for l, layer in enumerate(self.layers):\n",
" heads = tuple(ablate.get(l, ())) if ablate else ()\n",
" x = x + layer(x, ablate_heads=heads) # residual add: every layer writes a delta\n",
" return self.unembed(x) # (B, T, vocab)\n",
"\n",
"torch.manual_seed(SEED)\n",
"model = TinyTransformer(VOCAB, D_MODEL, N_HEADS, N_LAYERS, SEQ_LEN)\n",
"n_params = sum(p.numel() for p in model.parameters())\n",
"print(f\"model built · {n_params:,} parameters · {N_LAYERS} layers x {N_HEADS} heads\")"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "42f213d5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:06.643666Z",
"iopub.status.busy": "2026-06-10T20:56:06.643591Z",
"iopub.status.idle": "2026-06-10T20:56:06.648238Z",
"shell.execute_reply": "2026-06-10T20:56:06.647940Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 2.0 model output shape\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# layer-summary smoke test: dummy forward, confirm logits shape before any training.\n",
"_dummy = torch.randint(1, VOCAB, (2, SEQ_LEN))\n",
"_logits = model(_dummy)\n",
"check(\"2.0 model output shape\", lambda: check_shape(_logits, (2, SEQ_LEN, VOCAB)))"
]
},
{
"cell_type": "markdown",
"id": "57f24a5f",
"metadata": {},
"source": [
"> **Interpretation.** A few tens of thousands of parameters, output shape `(2, 64, 50)`: one logit vector per position. Tiny by any standard, and that is the point. We can hold the entire computation in our heads.\n"
]
},
{
"cell_type": "markdown",
"id": "7078946d",
"metadata": {},
"source": [
"### Exercise 22.1 — Generate the synthetic repeat data\n",
"`Difficulty 1/5 · ~5 min`\n",
"\n",
"The data is the experiment. We build sequences whose second half is an exact copy of the first half: `[t1, t2, ..., t_H, t1, t2, ..., t_H]`. On the second half, the optimal next-token prediction is \"copy what came after this token last time\", which is exactly the induction algorithm, and the ground-truth answer is known. Fill in `make_batch` so the second half repeats the first.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "2cfe7787",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:06.649022Z",
"iopub.status.busy": "2026-06-10T20:56:06.648951Z",
"iopub.status.idle": "2026-06-10T20:56:06.652243Z",
"shell.execute_reply": "2026-06-10T20:56:06.651979Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 22.1 repeat structure: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def make_batch(batch_size, seq_len, vocab, gen):\n",
" \"\"\"Return (batch_size, seq_len) int64 tokens; second half is an exact repeat of the first.\n",
" Tokens drawn from [1, vocab) so 0 is never used (a free 'pad-like' id we keep unused).\"\"\"\n",
" half = seq_len // 2\n",
" # TODO 1: draw `first` of shape (batch_size, half) with torch.randint(1, vocab, ..., generator=gen)\n",
" first = None\n",
" attempted(first)\n",
" # TODO 2: concatenate `first` with a copy of itself along the sequence axis (dim=-1)\n",
" return torch.cat([first, first.clone()], dim=-1)\n",
"\n",
"def _check_repeat():\n",
" g = torch.Generator().manual_seed(SEED)\n",
" b = make_batch(8, SEQ_LEN, VOCAB, g)\n",
" check_shape(b, (8, SEQ_LEN))\n",
" half = SEQ_LEN // 2\n",
" assert (b[:, :half] == b[:, half:]).all(), \\\n",
" \"second half must equal the first half exactly — that is the repeat structure induction exploits\"\n",
" assert b.min() >= 1, \"token 0 is reserved; draw from [1, vocab)\"\n",
"\n",
"check(\"22.1 repeat structure\", _check_repeat)"
]
},
{
"cell_type": "markdown",
"id": "8a8d85d0",
"metadata": {},
"source": [
"Hint 1 (conceptual)`torch.randint(low, high, size, generator=gen)` draws the random first half. The second half is not random, it is a copy. `torch.cat([a, b], dim=-1)` joins along the sequence axis.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"half = seq_len // 2\n",
"first = torch.randint(1, vocab, (batch_size, half), generator=gen)\n",
"# return first concatenated with a copy of itself\n",
"```\n",
"\n",
"\n",
"Help — \"Expected all tensors to be on the same device\" or a generator errorPass the `generator=gen` keyword (not positional) to `torch.randint`. The generator is a seeded `torch.Generator()`, which is how we make the data reproducible without touching the global RNG.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "b7dbae60",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:06.653027Z",
"iopub.status.busy": "2026-06-10T20:56:06.652962Z",
"iopub.status.idle": "2026-06-10T20:56:06.655713Z",
"shell.execute_reply": "2026-06-10T20:56:06.655356Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 22.1 repeat structure\n",
"one length-12 sequence (note halves match):\n",
" first half: [12, 39, 43, 24, 16, 17]\n",
" second half: [12, 39, 43, 24, 16, 17]\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines make_batch; the check below re-verifies the reference.\n",
"def make_batch(batch_size, seq_len, vocab, gen):\n",
" half = seq_len // 2\n",
" first = torch.randint(1, vocab, (batch_size, half), generator=gen) # (batch, half)\n",
" return torch.cat([first, first.clone()], dim=-1) # (batch, seq_len)\n",
"\n",
"check(\"22.1 repeat structure\", _check_repeat, required=True)\n",
"_g = torch.Generator().manual_seed(SEED)\n",
"_demo = make_batch(1, 12, VOCAB, _g)[0]\n",
"print(\"one length-12 sequence (note halves match):\")\n",
"print(\" first half: \", _demo[:6].tolist())\n",
"print(\" second half:\", _demo[6:].tolist())"
]
},
{
"cell_type": "markdown",
"id": "1f44e621",
"metadata": {},
"source": [
"> **Interpretation.** The two halves are identical token-for-token. A model that has not seen this exact sequence in training can still predict the second half perfectly *if* it has learned the copy-by-pattern algorithm, because the rule \"look back to the matching token and copy its successor\" is sequence-independent. That generalization is what we test out-of-distribution in Part 5.\n",
"\n",
"> **Key takeaways.** Attention-only blocks only move information between positions, which keeps the circuit legible. The repeat dataset has a known ground-truth next token on its second half, so every later claim is assertable. Shapes are the contract: `(batch, n_heads, seq, seq)` patterns, `(batch, seq, vocab)` logits.\n"
]
},
{
"cell_type": "markdown",
"id": "aef27cc8",
"metadata": {},
"source": [
"## Part 3 — Train it, and watch induction appear\n",
"\n",
"> **Objectives.** Write the canonical four-line training loop (forward, backward, update, track). Train on the second-half positions only. First do it *wrong* on purpose, forget to zero the gradients, watch the circuit fail to form, then fix it and watch loss collapse.\n",
"\n",
"The loss is next-token cross-entropy, but only on the second half: positions `< half` are genuinely unpredictable random tokens, and asking the model to predict noise just adds gradient variance. We predict token `half` from the logits at position `half-1`, and so on to the end.\n"
]
},
{
"cell_type": "markdown",
"id": "175dd007",
"metadata": {},
"source": [
"### A deliberate failure: forget to zero the gradients\n",
"\n",
"The single most common training bug. PyTorch *accumulates* gradients across `.backward()` calls. If you never zero them, every step adds the new gradient on top of the stale sum, the effective step direction is garbage, and the model never learns the circuit. We run it broken first, on purpose, so you recognize the symptom when it bites you for real.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "e8866d53",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:06.656654Z",
"iopub.status.busy": "2026-06-10T20:56:06.656579Z",
"iopub.status.idle": "2026-06-10T20:56:08.025341Z",
"shell.execute_reply": "2026-06-10T20:56:08.024946Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"broken final loss after 100 steps: 4.756 (a trained model reaches < 0.1)\n"
]
}
],
"source": [
"# BROKEN ON PURPOSE: the optimizer step uses accumulated (stale + new) gradients.\n",
"torch.manual_seed(SEED)\n",
"broken = TinyTransformer(VOCAB, D_MODEL, N_HEADS, N_LAYERS, SEQ_LEN)\n",
"opt_b = torch.optim.AdamW(broken.parameters(), lr=LR)\n",
"gen_b = torch.Generator().manual_seed(SEED)\n",
"half = SEQ_LEN // 2\n",
"broken_losses = []\n",
"for step in range(100):\n",
" batch = make_batch(BATCH, SEQ_LEN, VOCAB, gen_b)\n",
" logits = broken(batch) # forward\n",
" pred = logits[:, half - 1:-1, :] # predict positions [half, end)\n",
" target = batch[:, half:]\n",
" loss = F.cross_entropy(pred.reshape(-1, VOCAB), target.reshape(-1))\n",
" loss.backward() # backward\n",
" opt_b.step() # update <-- BUG: no opt_b.zero_grad() anywhere\n",
" broken_losses.append(loss.item())\n",
"print(f\"broken final loss after 100 steps: {broken_losses[-1]:.3f} (a trained model reaches < 0.1)\")"
]
},
{
"cell_type": "markdown",
"id": "06d239a7",
"metadata": {},
"source": [
"> **Help — \"my loss is stuck / bounces around and never drops\"**. Symptom: loss plateaus high or oscillates. Cause: gradients accumulating across steps because `zero_grad()` is missing (or the lr is far too high). Diagnostic: print `next(model.parameters()).grad.abs().mean()` right after `backward()` on two consecutive steps; if it keeps growing without bound, you are accumulating. The fix is one line.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "debc3c2e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:08.026515Z",
"iopub.status.busy": "2026-06-10T20:56:08.026364Z",
"iopub.status.idle": "2026-06-10T20:56:10.614955Z",
"shell.execute_reply": "2026-06-10T20:56:10.614634Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"fixed final loss after 320 steps: 0.0003\n"
]
}
],
"source": [
"# THE FIX: zero the gradients before each backward. One line, total behavior change.\n",
"def train(model, n_steps, seed):\n",
" opt = torch.optim.AdamW(model.parameters(), lr=LR)\n",
" gen = torch.Generator().manual_seed(seed)\n",
" half = SEQ_LEN // 2\n",
" losses = []\n",
" for step in range(n_steps):\n",
" batch = make_batch(BATCH, SEQ_LEN, VOCAB, gen)\n",
" logits = model(batch) # forward\n",
" pred = logits[:, half - 1:-1, :] # predict second-half tokens\n",
" target = batch[:, half:]\n",
" loss = F.cross_entropy(pred.reshape(-1, VOCAB), target.reshape(-1))\n",
" opt.zero_grad() # <-- THE FIX: clear stale gradients\n",
" loss.backward() # backward\n",
" opt.step() # update\n",
" losses.append(loss.item()) # track stats\n",
" return losses\n",
"\n",
"torch.manual_seed(SEED)\n",
"model = TinyTransformer(VOCAB, D_MODEL, N_HEADS, N_LAYERS, SEQ_LEN)\n",
"losses = train(model, STEPS, seed=SEED)\n",
"print(f\"fixed final loss after {STEPS} steps: {losses[-1]:.4f}\")"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "17c16d0f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.615965Z",
"iopub.status.busy": "2026-06-10T20:56:10.615885Z",
"iopub.status.idle": "2026-06-10T20:56:10.690626Z",
"shell.execute_reply": "2026-06-10T20:56:10.690165Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAFUCAYAAADYjN+CAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAedtJREFUeJzt3Xd4FNXXwPHvpveEhFQISSD00Jt0pFcpIgooXUBAmijwU5AiAgoiTQELWBCVKtJ7L9J7DQGCBEJNSCfZ+/6xb1aWhJAlZbPhfJ5nn+zeuTNz5mZ2c3L3zh2NUkohhBBCCCGEmbEwdQBCCCGEEEK8CElkhRBCCCGEWZJEVgghhBBCmCVJZIUQQgghhFmSRFYIIYQQQpglSWSFEEIIIYRZkkRWCCGEEEKYJUlkhRBCCCGEWZJEVgghhBBCmCVJZEWO0mg0DBo0yNRhPFePHj0IDAzM9u2OGzcOjUaT7dsVOWPHjh1oNBp27NiRYb3U3+vdu3dzJ7A8IDAwkNatW5s6DCDn3q+ZERgYSI8ePbK0jQ0bNlCxYkXs7OzQaDQ8fPgwW2LLKXk1XlOeByLvkERWZNm+ffsYN25cnvlwE0K8mLNnzzJu3DiuXr1q6lDMhrFtdu/ePTp16oS9vT1z587ll19+wdHRMWeDzAJzi1e8fKxMHYAwf/v27WP8+PH06NEDNzc3U4fzQr777ju0Wm22b/eTTz5h1KhR2b5dIXLC2bNnGT9+PA0aNMjTPV059X7NjAsXLmBh8V8fkLFtdujQIR49esTEiRNp3LhxDkaaPfJyvKY8D0TeIT2yQgDW1tbY2tpm+3atrKyws7PL9u3mBXFxcaYOQS8hIUH+oL1EMvN+TU5OJikpKdv3bWtri7W19QuvHxkZCZCpf/rzwnvMmHgzKzY2Nlu2k1Of28K8SCIrsmTcuHF8+OGHAAQFBaHRaNBoNGm+Zlu1ahUhISHY2tpStmxZNmzYkGZb//77L7169cLb21tf78cff8xUHKljcZcuXUqZMmWwt7enZs2anDp1CoD58+cTHByMnZ0dDRo0SBNfemOtfv/9d6pUqYKzszMuLi6UK1eOmTNn6pc/fvyY8ePHU7x4cezs7PDw8KBOnTps3rzZoH2eHiObGmtm2mTHjh1UrVoVOzs7ihUrxvz58zM17nbRokX638XTjwYNGhjU/fXXX6lSpQr29va4u7vz1ltvER4eblCnQYMGhISEcOTIEerVq4eDgwP/+9//AN0fut69e+Pt7Y2dnR0VKlTgp59+yjC+Z5k7dy5FixbF3t6e6tWrs3v3bho0aGAQc+o41t9//51PPvmEQoUK4eDgQHR0NPfv32fEiBGUK1cOJycnXFxcaNGiBSdOnEizrxs3btCuXTscHR3x8vJi2LBhJCYmvlDcANeuXSM4OJiQkBBu374NwMOHDxk6dCj+/v7Y2toSHBzM1KlT0yTdWq2Wr7/+mrJly2JnZ4e3tzf9+vXjwYMHBvVSx6lu2rRJP2axTJkyrFixwqBeZs7Npy1atIg33ngDgFdffVV/vjw9XnjPnj1Ur14dOzs7ihYtys8//5xmW5k97mdZv3499evX17/3qlWrxm+//aZf/vT79erVq2g0GqZNm8bXX39NsWLFsLW15ezZswCcP3+eTp064enpib29PSVLluTjjz9+5vZSpfdee3KMbGbbLFWDBg3o3r07ANWqVUOj0ei3ldX32JNtkPo+cnBwoGnTpoSHh6OUYuLEiRQuXBh7e3vatm3L/fv3M/w9ZBQvwNKlS/WfHQULFuTtt9/m33//NdhGjx49cHJyIjQ0lJYtW+Ls7EzXrl2B7P/cfrINFixYoD8PqlWrxqFDh9IcX+p+7ezsCAkJYeXKlTLu1gzJ0AKRJR06dODixYssWbKEGTNmULBgQQA8PT31dfbs2cOKFSsYMGAAzs7OzJo1i9dff53r16/j4eEBwO3bt3nllVf0H2yenp6sX7+e3r17Ex0dzdChQ58by+7du1m9ejUDBw4EYPLkybRu3ZqPPvqIb775hgEDBvDgwQO++OILevXqxbZt2565rc2bN9O5c2caNWrE1KlTATh37hx79+5lyJAhgO6P3OTJk+nTpw/Vq1cnOjqaw4cPc/ToUZo0aZJhrJlpk2PHjtG8eXN8fX0ZP348KSkpTJgwwaBtn6VevXr88ssvBmXXrl3jk08+wcvLS182adIkxowZQ6dOnejTpw937txh9uzZ1KtXj2PHjhn0wty7d48WLVrw1ltv8fbbb+Pt7U18fDwNGjTg8uXLDBo0iKCgIJYuXUqPHj14+PChvq0y49tvv2XQoEHUrVuXYcOGcfXqVdq1a0eBAgUoXLhwmvoTJ07ExsaGESNGkJiYiI2NDWfPnmXVqlW88cYbBAUFcfv2bebPn0/9+vU5e/Ysfn5+AMTHx9OoUSOuX7/O4MGD8fPz45dffsnwnMhIaGgoDRs2xN3dnc2bN1OwYEHi4uKoX78+//77L/369aNIkSLs27eP0aNHExERwddff61fv1+/fixatIiePXsyePBgwsLCmDNnDseOHWPv3r0GPYCXLl3izTffpH///nTv3p2FCxfyxhtvsGHDBv159yLnZr169Rg8eDCzZs3if//7H6VLlwbQ/wS4fPkyHTt2pHfv3nTv3p0ff/yRHj16UKVKFcqWLQtg1HGnZ9GiRfTq1YuyZcsyevRo3NzcOHbsGBs2bKBLly4Zrrtw4UISEhLo27cvtra2uLu7c/LkSerWrYu1tTV9+/YlMDCQ0NBQ/v77byZNmpTh9p4nM232pI8//piSJUuyYMECJkyYQFBQEMWKFdMvz4732OLFi0lKSuL999/n/v37fPHFF3Tq1ImGDRuyY8cORo4cyeXLl5k9ezYjRozIsLMgo3hTz9dq1aoxefJkbt++zcyZM9m7d2+az47k5GSaNWtGnTp1mDZtGg4ODvpl2fm5neq3337j0aNH9OvXD41GwxdffEGHDh24cuWK/r20du1a3nzzTcqVK8fkyZN58OABvXv3plChQs/dvshjlBBZ9OWXXypAhYWFpVkGKBsbG3X58mV92YkTJxSgZs+erS/r3bu38vX1VXfv3jVY/6233lKurq4qLi4uwxgAZWtraxDD/PnzFaB8fHxUdHS0vnz06NFp4u3evbsKCAjQvx4yZIhycXFRycnJz9xnhQoVVKtWrTKM69NPP1VPv80y2yZt2rRRDg4O6t9//9WXXbp0SVlZWaXZ5vPEx8erKlWqKD8/PxUREaGUUurq1avK0tJSTZo0yaDuqVOnlJWVlUF5/fr1FaDmzZtnUPfrr79WgPr111/1ZUlJSapmzZrKycnJoN0zkpiYqDw8PFS1atXU48eP9eWLFi1SgKpfv76+bPv27QpQRYsWTXNeJCQkqJSUFIOysLAwZWtrqyZMmJAm7j///FNfFhsbq4KDgxWgtm/fnmG8qb/XO3fuqHPnzik/Pz9VrVo1df/+fX2diRMnKkdHR3Xx4kWDdUeNGqUsLS3V9evXlVJK7d69WwFq8eLFBvU2bNiQpjwgIEABavny5fqyqKgo5evrqypVqqQvy8y5mZ6lS5c+8/hT971r1y59WWRkpLK1tVUffPCB0cednocPHypnZ2dVo0YNFR8fb7BMq9Xqnz/9fg0LC1OAcnFxUZGRkQbr1atXTzk7O6tr165lenup0nv/BgQEqO7du+tfZ9Rm6Vm4cKEC1KFDhwzKs/oeS20DT09P9fDhQ33d1M+7ChUqGLy3OnfurGxsbFRCQoLR8SYlJSkvLy8VEhJi8Htas2aNAtTYsWP1Zd27d1eAGjVqVJptZ/fndmobeHh4GLwX//rrLwWov//+W19Wrlw5VbhwYfXo0SN92Y4dOxSQ7rkg8i4ZWiByXOPGjQ16HcqXL4+LiwtXrlwBQCnF8uXLadOmDUop7t69q380a9aMqKgojh49+tz9NGrUyOAroRo1agDw+uuv4+zsnKY8df/pcXNzIzY2NsOvYt3c3Dhz5gyXLl16bmxPe16bpKSksGXLFtq1a6fvRQQIDg6mRYsWRu9vwIABnDp1iuXLl+Pj4wPAihUr0Gq1dOrUyaDNfXx8KF68ONu3bzfYhq2tLT179jQoW7duHT4+PnTu3FlfZm1tzeDBg4mJiWHnzp2Ziu/w4cPcu3ePd999Fyur/74o6tq1KwUKFEh3ne7du2Nvb58mxtQLcVJSUrh37x5OTk6ULFnS4Bxat24dvr6+dOzYUV/m4OBA3759MxVvqtOnT1O/fn0CAwPZsmWLQaxLly6lbt26FChQwKB9GzduTEpKCrt27dLXc3V1pUmTJgb1qlSpgpOTU5rfg5+fH+3bt9e/dnFxoVu3bhw7doxbt24BWTs3M1KmTBnq1q2rf+3p6UnJkiUN3kuZPe70bN68mUePHjFq1Kg0Y8szM43d66+/bvCNxZ07d9i1axe9evWiSJEiRm8vt2XHe+yNN97A1dVV/zr18+7tt982eG/VqFGDpKSkNEMBMuPw4cNERkYyYMAAg99Tq1atKFWqFGvXrk2zznvvvZfutrLzczvVm2++afBeTD1nU9e9efMmp06dolu3bjg5Oenr1a9fn3Llyj13+yJvkaEFIsc9/QcEoECBAvrxf3fu3OHhw4csWLCABQsWpLuN1AsOjNlP6oe5v79/uuVPjz980oABA/jzzz9p0aIFhQoVomnTpnTq1InmzZvr60yYMIG2bdtSokQJQkJCaN68Oe+88w7ly5c3OlYwbJPIyEji4+MJDg5OUy+9sozMnz+fhQsXMn/+fF555RV9+aVLl1BKUbx48XTXe/qClkKFCmFjY2NQdu3aNYoXL25wFTf899XqtWvXMhVjar2nj83KyuqZ49WCgoLSlGm1WmbOnMk333xDWFgYKSkp+mWpQzZS9xccHJwmmSlZsmSm4k3Vpk0bvL292bhxo8EfRNC178mTJ585FCT1nL506RJRUVEGQz7Sq5cqvbhLlCgB6MYI+vj4ZOnczMjzztvU48nMcacnNDQUgJCQkBeK7+lzIjVxedHt5bbseI9l5+fgs6TuM733S6lSpdizZ49BmZWVVbrDg3Iq3qe3mZrUpq77rM+b1LLMdJyIvEMSWZHjLC0t0y1XSgHoLwB5++239RcWPC0zf4CftZ/n7T89Xl5eHD9+nI0bN7J+/XrWr1/PwoUL6datm/4ii3r16hEaGspff/3Fpk2b+P7775kxYwbz5s2jT58+LxRrRjG9iH/++YchQ4bQp0+fNL2NWq0WjUbD+vXr043n6cTs6d5PU0svns8//5wxY8bQq1cvJk6ciLu7OxYWFgwdOjRHZjV4/fXX+emnn1i8eDH9+vUzWKbVamnSpAkfffRRuuumJp9arRYvLy8WL16cbr3MjIl+WlbOzYxk5rzN7HHnhBc9R5/VO/vkP0K5ITveY9n5OZhdnvym5Gk5Ea8pj1XkPklkRZZl9Ss6T09PnJ2dSUlJyVPzFNrY2NCmTRvatGmDVqtlwIABzJ8/nzFjxuj/k3d3d6dnz5707NmTmJgY6tWrx7hx47KULIAukbazs+Py5ctplqVXlp47d+7QsWNHKlasyNy5c9MsL1asGEopgoKCXji5CAgI4OTJk2i1WoM/VOfPn9cvz+x2QHdsr776qr48OTmZq1evZroncdmyZbz66qv88MMPBuUPHz7UX4iYur/Tp0+jlDI4fy9cuJCp/aT68ssvsbKy0l+09+TFSMWKFSMmJua553SxYsXYsmULtWvXzlQic/ny5TRxX7x4EcCg9/pFzs3s+Lo9s8f9rHVBN2TD2G8e0lO0aFH99jJSoECBdG/okplvFHJ6iEJ2vceyOybQvV8aNmxosOzChQsmickYT37ePC2zn68i75AxsiLLUu/y8qJ39rK0tOT1119n+fLl6f7BuXPnTlbCeyH37t0zeG1hYaFPplKnaHq6jpOTE8HBwVmawimVpaUljRs3ZtWqVdy8eVNffvnyZdavX//c9VNSUnjrrbdISkpi+fLlab6uBN2ME5aWlowfPz5NT4VSKs3xpadly5bcunWLP/74Q1+WnJzM7NmzcXJyon79+s/dBkDVqlXx8PDgu+++Izk5WV++ePFio776tLS0THMsS5cuTTMOsGXLlty8eZNly5bpy+Li4p45tOVZNBoNCxYsoGPHjnTv3p3Vq1frl3Xq1In9+/ezcePGNOs9fPhQf5ydOnUiJSWFiRMnpqmXnJyc5n118+ZNVq5cqX8dHR3Nzz//TMWKFfXjn1/03Mzqexkyf9zpadq0Kc7OzkyePJmEhASDZS/Sm+bp6Um9evX48ccfuX79+jO3V6xYMaKiojh58qS+LCIiwqCdnyU72iwj2fUey05Vq1bFy8uLefPmGZxT69ev59y5c7Rq1SrXYzKGn58fISEh/Pzzz8TExOjLd+7cqZ/6S5gP6ZEVWValShVAN1XLW2+9hbW1NW3atDHqNoZTpkxh+/bt1KhRg3fffZcyZcpw//59jh49ypYtW54732F269OnD/fv36dhw4YULlyYa9euMXv2bCpWrKgfm1amTBkaNGhAlSpVcHd35/DhwyxbtoxBgwZlSwzjxo1j06ZN1K5dm/fee4+UlBTmzJlDSEgIx48fz3DdefPmsW3bNvr375/mYiFvb2+aNGlCsWLF+Oyzzxg9erR+qitnZ2fCwsJYuXIlffv2ZcSIERnup2/fvsyfP58ePXpw5MgRAgMDWbZsGXv37uXrr782uFgjIzY2NowbN47333+fhg0b0qlTJ65evcqiRYsoVqxYpnu9WrduzYQJE+jZsye1atXi1KlTLF68WN8zl+rdd99lzpw5dOvWjSNHjuDr68svv/xiMC1QZllYWPDrr7/Srl07OnXqxLp162jYsCEffvghq1evpnXr1vopqmJjYzl16hTLli3j6tWrFCxYkPr169OvXz8mT57M8ePHadq0KdbW1ly6dImlS5cyc+ZMg4vSSpQoQe/evTl06BDe3t78+OOP3L59m4ULF+rrvOi5WbFiRSwtLZk6dSpRUVHY2trSsGHDZ47fTU9mjzs9Li4uzJgxgz59+lCtWjW6dOlCgQIFOHHiBHFxcS80P/GsWbOoU6cOlStXpm/fvgQFBXH16lXWrl2rfx+99dZbjBw5kvbt2zN48GDi4uL49ttvKVGixHPHS2ZHm2Uku95j2cna2pqpU6fSs2dP6tevT+fOnfXTbwUGBjJs2LBcj8lYn3/+OW3btqV27dr07NmTBw8e6D9fn0xuhRnI7WkSRP40ceJEVahQIWVhYWEwRQqgBg4cmKb+01PYKKXU7du31cCBA5W/v7+ytrZWPj4+qlGjRmrBggXP3X96+0mdiuXLL780KE+dvmnp0qX6sqencVm2bJlq2rSp8vLyUjY2NqpIkSKqX79++qmrlFLqs88+U9WrV1dubm7K3t5elSpVSk2aNEklJSXp6zxr+q3MtsnWrVtVpUqVlI2NjSpWrJj6/vvv1QcffKDs7OwybI/U/ab3eHIqK6WUWr58uapTp45ydHRUjo6OqlSpUmrgwIHqwoUL+jr169dXZcuWTXdft2/fVj179lQFCxZUNjY2qly5cmrhwoUZxvcss2bNUgEBAcrW1lZVr15d7d27V1WpUkU1b95cXye931+qhIQE9cEHHyhfX19lb2+vateurfbv36/q16+f5rivXbumXnvtNeXg4KAKFiyohgwZop/yypjpt1LFxcWp+vXrKycnJ3XgwAGllFKPHj1So0ePVsHBwcrGxkYVLFhQ1apVS02bNs3gPFFKqQULFqgqVaooe3t75ezsrMqVK6c++ugjdfPmTX2dgIAA1apVK7Vx40ZVvnx5ZWtrq0qVKpWmLTJzbj7Ld999p4oWLaosLS0N2iJ1309Lr22NOe70rF69WtWqVUvZ29srFxcXVb16dbVkyRL98mdNu/T0ez3V6dOnVfv27ZWbm5uys7NTJUuWVGPGjDGos2nTJhUSEqJsbGxUyZIl1a+//pqp6beUenabpSej6bey8h4z5vMuozgyG69SSv3xxx+qUqVKytbWVrm7u6uuXbuqGzduGNTp3r27cnR0THfb2f25ndF5AKhPP/3UoOz3339XpUqVUra2tiokJEStXr1avf7666pUqVLpxivyJo1SMvpZCHPSrl27HJlaKS/SarV4enrSoUMHvvvuO1OHY3KBgYGEhISwZs0aU4ciRL5UsWJFPD09M5x6UeQtMkZWiDwsPj7e4PWlS5dYt25dmtvM5gcJCQlpxkH+/PPP3L9/P18erxDCdB4/fpxmvPaOHTs4ceKEfN6YGRkjK0QeVrRoUXr06EHRokW5du0a3377LTY2Ns+c2igvun//PklJSc9cbmlpiaenJwcOHGDYsGG88cYbeHh4cPToUX744QdCQkL097MXQojs8O+//9K4cWPefvtt/Pz8OH/+PPPmzcPHx4f+/fubOjxhBElkhcjDmjdvzpIlS7h16xa2trbUrFmTzz///Jk3MciLOnTokOEdvgICArh69SqBgYH4+/sza9Ys7t+/j7u7O926dWPKlCnpzroghBAvqkCBAlSpUoXvv/+eO3fu4OjoSKtWrZgyZYrBzVNE3idjZIUQOerIkSMZTqFlb29P7dq1czEiIYQQ+YUkskIIIYQQwizJxV5CCCGEEMIsmfUYWa1Wy82bN3F2ds7x2wQKIYQQQoicp5Ti0aNH+Pn5GdyaOT1mncjevHkTf39/U4chhBBCCCGyWXh4OIULF86wjlknsqm35gsPD8fFxcXE0QghhBBCiKyKjo7G398/U7dgNutENnU4gYuLiySyQgghhBD5SGaGjcrFXkIIIYQQwixJIiuEEEIIIcySSRPZcePGodFoDB6lSpUyZUhCCCGEEMJMmHyMbNmyZdmyZYv+tZWVyUMSQgghhBBmwORZo5WVFT4+PqYOQwghhBBCmBmTj5G9dOkSfn5+FC1alK5du3L9+nVThySEEEIIIcyASXtka9SowaJFiyhZsiQRERGMHz+eunXrcvr06XTnDktMTCQxMVH/Ojo6OjfDFUIIIYQQeYhGKaVMHUSqhw8fEhAQwFdffUXv3r3TLB83bhzjx49PUx4VFSXzyAqRCUqrRZuUgKWdg6lDEUIIIdIVHR2Nq6trpvI7kw8teJKbmxslSpTg8uXL6S4fPXo0UVFR+kd4eHguRyiEebv6/Tj2tfbj0fkjpg5FCCGEyLI8lcjGxMQQGhqKr69vusttbW31d/GSu3kJYRxt8mNurVkI2hTu7V1r6nCEEEKILDNpIjtixAh27tzJ1atX2bdvH+3bt8fS0pLOnTubMiwh8qWoE3tIjnkIQMylE6YNRgghhMgGJr3Y68aNG3Tu3Jl79+7h6elJnTp1OHDgAJ6enqYMS4h86d7u1frnMZeOoZTK1H2shRBCiLzKpIns77//bsrdC/HSUFot9/au0b9+/OAOSfduYVsw/WE8QgghhDnIU2NkhRA549H5wyTdu4WlgzP2hYsBEHPpuGmDEkIIIbJIElkhXgK31iwEoECNpjiXrg7IOFkhhBDmTxJZIfK5mNBT3N64GIBCrw/AqURFAGKlR1YIIYSZM+kYWSFEzlJKETbvY1CKgq++jkuZ6qjkx4D0yAohhDB/0iMrRD52/8AGHh7ZjsbahqA+4wBwDC4PQGLkDZIe3jFhdEIIIUTWSCIrRD6lTUrgytxRgG5IgZ1vIABWDs7YFw4GIDb0tKnCE0IIIbJMhhYIkU/dWDqbhJtXsPHwwf/tjwyWlfpkIdbu3th4+JgoOiGEECLrJJEVIh+Ku36R8MXTAAjq9xlWDs4Gy1Mv+BJCCCHMmSSyQuQTt9Yu4tban/Br348bf85CmxCHa6X6eDbqZOrQhBBCiBwhiawQ+UD0mYNcmjEUtClcmHwYAGu3gpQc/Z3chlYIIUS+JRd7CWHmkmOiOD+pN2hTcAyugMbaBjQaSoxaILegFUIIka9Jj6wQZkwpxeWvh5F46xq2PgGU/2otKfExJMdE4xhU2tThCSGEEDlKElkhzNjtDb9wZ9tSsLCk1Cc/YuXkipWTK7aehUwdmhBCCJHjZGiBEGYq/mYYobNGABDYeywuZaqbOCIhhBAid0kiK4SZ+nfpHLSJ8bhWqEPhN4eaOhwhhBAi10kiK4QZSo55yO2NiwHwf2ckGgt5KwshhHj5yF8/IczQrXU/o02IxSGoDG6V6ps6HCGEEMIkJJEVwgzc3vArx/rV5f6BDSTdv83NFfMAKNThPZknVgghxEtLZi0QIo+4uXI+cdcvUmzQF2gsLfXlKYnxXJn3P5KjH3Dm405YObmR/OgB1u7eeDZ+04QRCyGEEKYlPbJC5AGPo+4ROnckEX8tIPrsQYNld3euIjn6ARprW1CK5EcPcCwaQvnpa7C0tTdRxEIIIYTpSY+sEHnAvX3rQJsCQOzlU7iWq6VfduvvHwAo8vZH2BcpQWLEVfw69MfCxs4ksQohhBB5hSSyQuQBd3et0j+PCT0JwKPzR4g+fYDoMwfBwhKflt2w8fAxUYRCCCFE3iOJrBAmlhzzkIdHtutfx14+xcMTezg1rIW+zKNOa0lihRBCiKdIIiuEid3buxaV/BgrVw+So+4RG3aWO1v/BMC+SAncKr9K4TeHmDhKIYQQIu+RRFYIE4gLv8T9/eu5f2AjUSf3AOD7Wh9uLv+GlLhHRG7+HYCgvhPxqNXSlKEKIYQQeZYkskLksjs7VnL+sx6g1erLXCvWpVCH94g6vovoU/vRJsZjYWOHW+UGJotTCCGEyOskkRUiG6iUZC5+OQAbd2+C+k58Zr3YK2e4+EV/0GpxKVeLgnVfw6NOa+x8AgBwLFae6FP7AXCtVA9LO4dciV8IIYQwR5LICpENHh7fTeSmJQAUrNcO51JV0tRJvHOTs2M6o02Iw61KQ0KmrDC48QGAU3A5/XP3V1o8vQkhhBBCPEESWSGywb09a/TPbyydTcmR87m3fz3x1y+QHPMQh8AyhC+eRkJEGHa+gZT65Mc0SSyAY3B5/XP3V5rlSuxCCCGEuZJE1kjbV53CzsGamk1LmToUkUcopbi/f53+9d2dK4kPv0zs5RNp6tr5BlJu+hqsXT3S3ZZTsfJ4NnwD6wJe2Hn751jMQgghRH4giawRVvxwmA9+rYi7RSTrS9/Hy9/d1CGJPCD20gkSI29gYeeAc8nKRJ3YQ+zlE1g5F8Cjdiss7ByJOX8EjbU1pT7+EVuvws/clsbSklKf/JiL0QshhBDmSxJZI7zaujheS25yK6UI7w0+wldTtVw+d496LYtjbW1h6vDyNaUUGo3mhdZNjo3m1t8/4latEU7Fyj1/hUy6f2Ajtzf8ijYpAYACVRtRqNP7nPqgDfb+wZSZsAR7v6Bs258QQgghDGmUUsrUQbyo6OhoXF1diYqKwsXFJVf2eXjHJd4aX5jH2OvLqrsf5adFZXFwts2VGMxN9LlD3Nv9N4XeeB+bAp4GyxJuhxPx1wJUSgo2Hj64lKmOU8nKWFjbALoE9vpPn3NzxTwC+47Hp1VP/l02h6S7NwnqOxGN5X//iyXHPeLKnI+IOrGHEh99i2uFOiREXOXMx52Iu3oOK5cCVF6wL8Me0cxKjoni0NvlSY6+ry8rMXIe3s268jjqHlbOBdBYyD83QgghhLGMye8kkX0BP0zbx4S1tdCQggUppGBDeeeT/PZzMM5u+X+6JGN6R5NjHnLonYokR93DzjeIkKkrsC8cDMDjqLscH9iIhJtXDNaxcvWgyDujcCpenpsr5nF350rdAgtLvJq8SeTG3wAoOfo7vJq8BUDUyb1c/HIgCf+GAmDp6ELhToP5d9lckh890G/bJeQVyn21Dgsr6yy1wdUfJxL+6xfYeBbCwsoajZU1Feduw8rJLUvbFUIIIV52ksjmgtBT/+Lh48zh7ZcZ+G0JEnCilsdBfl1SBUvr/Dti4+6uv7g0bRCOxUIo8s4oXCvU1veKxoad5fykXti4eVL8g9nY+QYSOmckN1d8o1/fysWd0mN/xiGwNOfGdSX69AFsvYtQsH47Em6GEXVqH8lR9wz2qbG0wqlERR6dO2xQbl84mAqzNxM6Z6T+lq62Xv7YeHgb1HUqWZmi733OmY87kRIbjV+HARQbNDXd41NKERt6CseiIc/sUU288y+Hu1dGmxBH6XG/UrBeW+MbUgghhBDpkkQ2l21beYp3Z5UgGVvaB27j81m1cHC2M1k8LyLxbgRWji5Y2jumWRZz+SSPzh4i6f4trv/6hcEdqTTWNtj7F8epeAXu7f6blLhHgK5H1L1GM+7sWAHaFEr+73v+Xf4tMReOgIUFFlY2aJMSsHR0pcLsLTgG6maBUCnJ3Fr7E+G/TUP7OBHXcrXxe/09nIIrcOL9xsReOY1fh/eI3PI7ydEPsHIuoOtx1Wjwad2TwD7j0GgsOD2qPfE3LhPQ/WN8X+uNxtKKu7v+4ty4twEo+t5knEpWwsrJDceiZfXHE/bdp9xY8hWFOg2maP9JBu2QdE93/Lc3LkabEIdz6apUmLPthcfuCiGEECIts0xkp0yZwujRoxkyZAhff/11ptbJK4kswM+zjzBmhW4SfGfNA14tfZMq5W1o/1ZRXF3TzheaF6TExxL+23TubF9Ows0r2PkGUWH2FqwLeJIQcRWNRsPtTUu4/ssUg+TVu8U7WNjY/39CF2uwTdcKdVDJj4k+c1Bf5lG7NWUmLiElMZ7LXw/VDw1wLl2VYu9PS/fmAenGmxBH3NVzOJWsTPhv07j2wwQAbL0KU3rcrwbbUf8f79O9qtd//ZJrP074r8DCgnLT1+JWoQ6xoac52q8OaFPQWFpR+YcDOBQpCcC/K77l2g8TSImPAXTzvZb83w/6BFwIIYQQ2cPsEtlDhw7RqVMnXFxcePXVV80ykQX44etjfPO3F3e1hfRlhWxv8McCN/yLOJkwMkhJjOfBwU1EndxL7JUzWDm6EHPpBImR4Qb1nEpUwsLOgeiTew3KXSvWxdLOEbeqDfFr3x+NRoPSakmMvEFs6EkenTuMtWtBfNv1RWNhwd09a0iKvIFKSca7ZTesXXRTlSmluH9gA5b2TrhWqPPiMxHEPeLM6I5YOTpTfMQ32Lh7ZWo9pRSXZwzh1tpFWDm5kfzoATaehagwazMXPutJ9JmDaCytUCnJFKjelJApy7m3dw1nx3QGwLlUVQL7jMO1Uj3piRVCCCFygFklsjExMVSuXJlvvvmGzz77jIoVK5ptIguQFBPLqm+3cPAEbLtZjfvKDy/rCBbPcqJEKecc37/2cRL3968n4fZ1PBu8jsbKmvDfphO5cTHJMQ/T1Lf1LkJQ3wnYFyrGqY/a6q/C11haobG2wdLeiaL9J+kvqsovVEoK2qR4jvato79ADMDS3okyk/7k9EdtUcmPKdJtNLc3/EJi5A38OrxH0QFTZDYCIYQQIgeZVSLbvXt33N3dmTFjBg0aNMgwkU1MTCQxMVH/Ojo6Gn9//zyVyD7pwr7TvPOJE7dVILaaeHo0u8e/16JJiHtMhaJx1KtlR0itYKwcspbgKqWIvXySyM1LiNzyB48f3gVAY22LhbWNftyqrZc/7rVa4lyyEikJ8VhYWeHZqBOWdrqZFqJO7+fi5L44laxMUL/PXoo7Sz06f4QTQ5qiHidhXziYoH6f4VG7FdcXf6kfugBg6xNAlR//0beVEEIIIXKG2SSyv//+O5MmTeLQoUPY2dk9N5EdN24c48ePT1OeVxNZgLAjZxk0OorTj2umu9yPC/g6PcTD142Bfd0pX9GN+/vWkRBxFesCnth6FsbWqzAJN68Qd/0itp6FsC8cjJWTK3E3LnNv99/c37/eYIiAjYcPtp6FeXRed+W+U4lKBPQaQ4GqjaQ3MR3x/15BaVNw8C9uUH5z1QJC53wIWi1lJv2JR80WJopQCCGEeHmYRSIbHh5O1apV2bx5M+XLlwfIdz2yqRLu3OLLwUs5cqsYwZ4PcChYkJM3vTkZVYYUbPT1bInhXfuPcYy/iiVJFOUodpq4TO3DwsYO91ea49WsC+7Vm4CFJVEn9pASG4V7zZaSwL6gmEsneBx1jwJVG5o6FCGEEOKlYBaJ7KpVq2jfvj2Wlv9d0Z+SkoJGo8HCwoLExESDZenJi2Nkn0VptSQ/uo+1a0F9WXQM7Np5j4jjJ/l7vysnYisbrGNBCoWtr+Cbcppgp2tUCIgiIHEvKvIiKbHRWDm74V6zBR61WuFWuYF87S2EEEIIs2cWieyjR4+4du2aQVnPnj0pVaoUI0eOJCQk5LnbSD3QO3fu4OHhob+KPCUlhZSUFCwsLLCy+u/mBElJSQBYW1tna93Hjx+jlMLKygqL/+/51Gq1JCcno9FosLa2fm7dhMRkpsx6xLr9Tri7WxMdY0HkvcdoNIoUrRVK6epaW2opVyKZNq9qaN3QCi8PjdExJCcno9VqsbS01P+zYExdpRSPHz8GwMbGJkfqptfuxtQ15e8+J8+T3Kxrqt99Vs+T3Prd5/ZnhKl/93ntPJHPCNPXlc8I+YzIqc+I2NhY3N3dM5XImuwWVM7OzmmSVUdHRzw8PDKVxD5p7ty5fPjhhzg46HokDx48yO7duylfvjwtWvw3rnHOnDk8fvyY/v374+rqCsDRo0fZtm0bZcqUoU2bNvq68+bNIz4+nl69euHp6QnAqVOn2LhxI8WLF6dDhw76ut9//z3R0dF069YNX19fAM6dO8eaNWsICAjgrbf+u+L/p59+4t69e3Tu3JkiRYoAcPnyZVauXEmJQoWYsOrt/7b7w2/cu3sLB8+OnLlWjKNnwVpd45WAPzl33Ivx3/TEuyCUKgplPJeiSQnHoWBbPLxKEeQPhT1u8ttviylQoAB9+/bVb3flypVcuXKFli1bUq5cOQDu3LnDokWLcHJyYuDAgfq6a9as4cKFCzRp0oTKlXU9xg8ePOC7777D1taWoUOH6utu3LiR06dP06BBA2rUqAHoZqX45ptvsLCw4MMPP9TX3bZtG8eOHaN27drUqVMH0A0dmTlzJgAjRozQn/C7du3in3/+oXr16rz66quA7g0zY8YMAIYMGYKdne4GFPv372fv3r1UqlSJpk2b6vc3c+ZMtFotAwYMwNlZd3Hd4cOH2bFjByEhIbRq1Upf95tvviExMZF3330Xd3fdtGEnTpxg8+bNlCxZknbt2unrLliwgJiYGHr06IG3tzcAZ8+eZd26dRQtWpQ33nhDX3fhwoU8ePCArl27UrhwYQAuXrzIX3/9hb+/P126dNHX/fXXX4mMjKRTp04EBQUBEBYWxrJly/Dx8aF79+76un/88Qf//vsv7du3p0SJEgDcuHGDJUuW4OHhQZ8+ffR1ly9fzrVr12jdujVly+puBHH79m1+/vlnXFxceO+99/R1V69ezaVLl2jWrBkVK1YE4O7du/z444/Y29szePBgfd3169dz9uxZGjZsSLVq1QDdP5rz5s3D2tqa4cOH6+tu3ryZkydPUrduXWrVqgVAfHw8s2fPBmDkyJH6ujt27ODIkSPUrFmTevXqAboP29Tf/bBhw/Qfgnv37mX//v1UqVKFxo0b67eRWvf99983+8+IQoUK8fbb/31G/Pbbb9y6dYuOHTtSrFgxAK5du8aff/6Jl5cXPXv21NddunQp4eHhtG3bllKldPMf37x5k8WL5TMC5DMilXxGyGdEXviMSD1HMiP/3ks1H0i9023L+vB+MVAKjhyHrZvAzla37PZd3cOtOhRyh5WbIfS2blmf9jwxAlcIIYQQIn8x+fRbWZGfhha8SN3Ex9ZcCIPzoXAx7DEJSYrHyVbci7Jgx0HQoOW1hsm8UlEDWHP3ITSvC0ULy9eGOfm7z2vniXxtKF8bGltXhhbIZ4R8Rpj+PDG3372phhbki0TWHC72ym1L1sCo6WnLPdxg5RwIKJR2mRBCCCGEqRmT38nQgnyqc2vw84JNeyH8FlhZwtUbEBoOPUbD6L5QIggCJaEVQgghhJmSHtmXyO270G4g3Iz8r6x+NRjSHaqUNV1cQgghhBCpjMnvjJ4l/6effmLt2rX61x999BFubm7UqlUrzXRaIm/xLgi/TYcOTaBcCbC0gJ2HoMMgGDsL4uJNHaEQQgghROYZnch+/vnn2NvbA7ppTObOncsXX3xBwYIFGTZsWLYHKLJXUGGY8T9YMx92/AJvNNeV/7QS2g+CB1GmjU8IIYQQIrOMTmTDw8MJDg4GdHfnev311+nbty+TJ09m9+7d2R6gyDlF/GDaSPj1S/B0h/NXoNtIeBRr6siEEEIIIZ7P6ETWycmJe/fuAbBp0yaaNGkCgJ2dHfHx8t20OapbVTfkwN0VTl6AfmPhcbKpoxJCCCGEyJjRiWyTJk3o06cPffr04eLFi7Rs2RKAM2fOEBgYmN3xiVxSIhB+/gIc7WHvURg329QRCSGEEEJkzOhEdu7cudSsWZM7d+6wfPlyPDw8ADhy5AidO3fO9gBF7ilXAmZ+DBoN/LoaPp8HySmmjkoIIYQQIn0y/ZZIY8GfMOlb3fPq5eG7ieAmzSuEEEKIXJCj029t2LCBPXv26F/PnTuXihUr0qVLFx48eGB8tCLP6dsJvvkUnBzgn5PQ63+QkGjqqIQQQgghDBmdyH744YdER0cDcOrUKT744ANatmxJWFgYw4cPz/YAhWm0agDLZ4OLExw5A+9PhBQZZiCEEEKIPMToRDYsLIwyZcoAsHz5clq3bs3nn3/O3LlzWb9+fbYHKEynVFH4YRLYWutudTvzZ1NHJIQQQgjxH6MTWRsbG+Li4gDYsmULTZs2BcDd3V3fUyvyj+rlYcqHuuczf4at+00bjxBCCCFEKqMT2Tp16jB8+HAmTpzIP//8Q6tWrQC4ePEihQsXzvYAhel1aALd2umef/QlJCSZNBwhhBBCCOAFEtk5c+ZgZWXFsmXL+PbbbylUqBAA69evp3nz5tkeoMgbxgwAX0+4+wDW7TR1NEIIIYQQMv2WMMKsX2D6j1C5LKycY+pohBBCCJEfGZPfWb3IDlJSUli1ahXnzp0DoGzZsrz22mtYWlq+yOaEmXirFcz6GY6egdOXIKS4qSMSQgghxMvM6KEFly9fpnTp0nTr1o0VK1awYsUK3n77bcqWLUtoaGhOxCjyCC93aFFP93z2L2C+fflCCCGEyA+MTmQHDx5MsWLFCA8P5+jRoxw9epTr168TFBTE4MGDcyJGkYf0eQMsLWDDbpi+0NTRCCGEEOJlZvQYWUdHRw4cOEC5cuUMyk+cOEHt2rWJiYnJ1gAzImNkTeP3tTBymu75nDHQpqFp4xFCCCFE/pGjt6i1tbXl0aNHacpjYmKwsbExdnPCDL3VCgZ00T2fJUMMhBBCCGEiRieyrVu3pm/fvhw8eBClFEopDhw4QP/+/XnttddyIkaRB/XvDA52cPEq7D1q6miEEEII8TIyOpGdNWsWxYoVo2bNmtjZ2WFnZ0ft2rUJDg5m5syZORGjyINcnaDj/08bvHCFaWMRQgghxMvpheeRvXTpEufPnwegdOnSBAcHZ2tgmSFjZE0r9Do07A4aDWz/GYLkxm5CCCGEyKIcn0cWoHjx4hQvLhOJvsyKFYGGr8C2A7obJcwZa+qIhBBCCPEyyVQiO3z48Exv8KuvvnrhYIT5+bA3bD8If2+HXh2hchlTRySEEEKIl0WmEtljx45lamMajSZLwQjzUyYYOjaDpRvgs29g+WzdUAMhhBBCiJyWqUR2+/btOR2HMGMjeul6ZI+cgX9OQo0Kpo5ICCGEEC8Do2ctEOJpPp7QtpHu+R/rTBuLEEIIIV4eL3yxlxBPerOlLolduxPGvQ8uTqaOSAjxMkhJSeHx48emDkMIYQRra2ssLS2zZVuSyIpsUbkMBAfA5Wvw9zboKvfGEELkIKUUt27d4uHDh6YORQjxAtzc3PDx8cny9VWSyIpsodHoemUnfQt/rJdEVgiRs1KTWC8vLxwcHORiYyHMhFKKuLg4IiMjAfD19c3S9oxOZGNjY3F0dMzSTkX+1KEJTF0AJ87DuVAoXczUEQkh8qOUlBR9Euvh4WHqcIQQRrK3twcgMjISLy+vLA0zMPpiL29vb3r16sWePXteeKcifypYABrX0j2Xi76EEDkldUysg4ODiSMRQryo1PdvVse4G53I/vrrr9y/f5+GDRtSokQJpkyZws2bN7MUhMg/3mql+7lyCyQmmTYWIUT+JsMJhDBf2fX+NTqRbdeuHatWreLff/+lf//+/PbbbwQEBNC6dWtWrFhBcnJyprf17bffUr58eVxcXHBxcaFmzZqsX7/e2JBEHlKvKvh6wsNo2CSd9kIIIYTIQS88j6ynpyfDhw/n5MmTfPXVV2zZsoWOHTvi5+fH2LFjiYuLe+42ChcuzJQpUzhy5AiHDx+mYcOGtG3bljNnzrxoWMLELC2hY3Pd82WbTBuLEELkNQ0aNGDo0KE5su3AwEC+/vrrHNl2RpKSkggODmbfvn25vu+XzaJFi3Bzc9O/njdvHm3atDFdQHnACyeyt2/f5osvvqBMmTKMGjWKjh07snXrVqZPn86KFSto167dc7fRpk0bWrZsSfHixSlRogSTJk3CycmJAwcOvGhYIg94raHu576jEBtv2liEEELkrHnz5hEUFEStWrVMHcpLp1evXhw9epTdu3ebOhSTMXrWghUrVrBw4UI2btxImTJlGDBgAG+//bbBfwi1atWidOnSRm03JSWFpUuXEhsbS82aNdOtk5iYSGJiov51dHS0seGLXFA8APx9ITwC9h6BpnVMHZEQQpivpKQkbGxsTB1GupRSzJkzhwkTJpg6FKOZql2zc782NjZ06dKFWbNmUbdu3WzZprkxuke2Z8+e+Pn5sXfvXo4fP86gQYMMklgAPz8/Pv7440xt79SpUzg5OWFra0v//v1ZuXIlZcqUSbfu5MmTcXV11T/8/f2NDV/kAo0GGr6ie75VOteFEMJAcnIygwYNwtXVlYIFCzJmzBiUUvrlgYGBTJw4kW7duuHi4kLfvn0BWL58OWXLlsXW1pbAwECmT5+e4X6+//573Nzc2Lp1KwCnT5+mRYsWODk54e3tzTvvvMPdu3f19Rs0aMDgwYP56KOPcHd3x8fHh3HjxmW4jyNHjhAaGkqrVq30ZVevXkWj0bBixQpeffVVHBwcqFChAvv37zdY19jjCQwMRKPRpHmkCg8Pp1OnTri5ueHu7k7btm25evWqfnmPHj1o164dkyZNws/Pj5IlSwK6PKRhw4bY29vj4eFB3759iYmJyTCWVMnJyQwePBg3Nzc8PDwYOXIk3bt3N/hWukGDBgwaNIihQ4dSsGBBmjVrBsBXX31FuXLlcHR0xN/fnwEDBqTZ76JFiyhSpAgODg60b9+ee/fupYmhTZs2rF69mvj4l/QrUGWk2NhYY1fJUGJiorp06ZI6fPiwGjVqlCpYsKA6c+ZMunUTEhJUVFSU/hEeHq4AFRUVla0xiazbcVCpIg2Uqvq6UlqtqaMRQuQn8fHx6uzZsyo+Pl5fptVqVXJcjEkeWiM+5OrXr6+cnJzUkCFD1Pnz59Wvv/6qHBwc1IIFC/R1AgIClIuLi5o2bZq6fPmyunz5sjp8+LCysLBQEyZMUBcuXFALFy5U9vb2auHChQbrzZgxQyml1NSpU5WHh4c6ePCgUkqpBw8eKE9PTzV69Gh17tw5dfToUdWkSRP16quvGsTm4uKixo0bpy5evKh++uknpdFo1KZNm555PF999ZUqVaqUQVlYWJgCVKlSpdSaNWvUhQsXVMeOHVVAQIB6/PixUkpl6nieFhkZqSIiIlRERIS6ceOGeuWVV1TdunWVUkolJSWp0qVLq169eqmTJ0+qs2fPqi5duqiSJUuqxMREpZRS3bt3V05OTuqdd95Rp0+fVqdPn1YxMTHK19dXdejQQZ06dUpt3bpVBQUFqe7duz/3d6mUUp999plyd3dXK1asUOfOnVP9+/dXLi4uqm3btgbt6uTkpD788EN1/vx5df78eaWUUjNmzFDbtm1TYWFhauvWrapkyZLqvffe06934MABZWFhoaZOnaouXLigZs6cqdzc3JSrq6tBDLGxscrCwkJt3749UzHnFem9j1NFRUVlOr/TKPXEv4GZlJKSwsqVKzl37hwApUuXpl27dlhZZf1GYY0bN6ZYsWLMnz//uXWjo6NxdXUlKioKFxeXLO9bZJ+EJKjUFuISYM18KFfC1BEJIfKLhIQEwsLCCAoKws7ODoCU+Fj2tfIxSTy11t7C0j5zNwpq0KABkZGRnDlzRt+bOGrUKFavXs3Zs2cBXc9jpUqVWLlypX69rl27cufOHTZt+u8q2o8++oi1a9fqL5AODAxk6NChRERE8Msvv7B582bKli0LwGeffcbu3bvZuHGjfv0bN27g7+/PhQsXKFGiBA0aNCAlJcVgvGX16tVp2LAhU6ZMSfd4hg4dyqlTp/S9vqDrkQ0KCuL777+nd+/eAJw9e5ayZcty7tw5SpUqlanjyciQIUP466+/OHToEJ6envz666989tlnnDt3Tt+uSUlJuLm5sWrVKpo2bUqPHj3YsGED169f13+1/9133zFy5EjCw8P1N3tat24dbdq04ebNm3h7e2cYh4+PDyNGjGDEiBGALj8qWrQolSpVYtWqVYDudx4dHc3Ro0cz3NayZcvo37+/vpe8S5cuREVFsXbtWn2dt956iw0bNqS5NbO7uzszZsyge/fuz227vCK993EqY/I7o4cWnDlzhuLFi9O9e3dWrlzJypUr6dGjB8WLF+f06dPGbi4NrVZrMA5WmCc7G6hTRfd89TbTxiKEEHnJK6+8YvCVeM2aNbl06RIpKSn6sqpVqxqsc+7cOWrXrm1QVrt27TTrTZ8+ne+++449e/bok1iAEydOsH37dpycnPSPUqVKARAaGqqvV758eYN9+Pr66m8lmp74+Pg0SUh620q9DWnqtjJ7POlZsGABP/zwA6tXr8bT01N/fJcvX8bZ2Vl/fO7u7iQkJBgcX7ly5QzGp547d44KFSoY3LG0du3aaLVaLly4kGEcUVFR3L59m+rVq+vLLC0tqVKlSpq66ZVt2bKFRo0aUahQIZydnXnnnXe4d++eftanc+fOUaNGDYN1nnUNkb29faZmi8qPjO5C7dOnDyEhIRw5coQCBQoA8ODBA3r06EHfvn2Nmn5j9OjRtGjRgiJFivDo0SN+++03duzYYfAfozBfnVrApr2waAW83QYCCpk6IiFEfmVh50CttbdMtu/s9qK3gq9bty5r167lzz//ZNSoUfrymJgY2rRpw9SpU9Os8+S97q2trQ2WaTQatFrtM/dXsGBBTp06le6yJ7eVmrhntK3M2L59O++//z5LliwxSJRjYmKoUqUKixcvTrNOarILL96uWfX0fq9evUrr1q157733mDRpEu7u7uzZs4fevXuTlJRk9F3r7t+/b3CcLxOjE9njx49z+PBhfRILUKBAASZNmkS1atWM2lZkZCTdunUjIiICV1dXypcvz8aNG2nSpImxYYk8qHEtqFsVdh+GCd/AD5NMHZEQIr/SaDSZ/nrf1A4ePGjw+sCBAxQvXjzD+82XLl2avXv3GpTt3buXEiVKGKxXvXp1Bg0aRPPmzbGystJ/5V25cmWWL19OYGBgtgwDTFWpUiW+/fZblFJG3akps8fzpMuXL9OxY0f+97//0aFDB4NllStX5o8//sDLy8uooYalS5dm0aJFxMbG6pPNvXv3YmFhob8Y7FlcXV3x9vbm0KFD1KtXD9ANLTh69CgVK1bMcN0jR46g1WqZPn06Fha6L8f//PPPNLGld648LTQ0lISEBCpVqpThPvMro4cWlChRgtu3b6cpj4yMJDg42Kht/fDDD1y9epXExEQiIyPZsmWLJLH5iEYD494HK0vYsg8OHDd1REIIYXrXr19n+PDhXLhwgSVLljB79myGDBmS4ToffPABW7duZeLEiVy8eJGffvqJOXPm6BPVJ9WqVYt169Yxfvx4/Q0SBg4cyP379+ncuTOHDh0iNDSUjRs30rNnz+d+lZ+RV199lZiYGKNvZGTM8YBuCEObNm2oVKkSffv25datW/oH6MYQFyxYkLZt27J7927CwsLYsWMHgwcP5saNG8+Mo2vXrtjZ2dG9e3dOnz6t7/F95513njs+FuD9999n8uTJ/PXXX1y4cIEhQ4bw4MGD5yb1wcHBPH78mNmzZ3PlyhV++eUX5s2bZ1Bn8ODBbNiwgWnTpnHp0iXmzJnDhg0b0mxr9+7dFC1alGLFij033vzI6ER28uTJDB48mGXLlnHjxg1u3LjBsmXLGDp0KFOnTiU6Olr/ECK4CLyum2mEVVtMG4sQQuQF3bp1Iz4+nurVqzNw4ECGDBmin2LrWSpXrsyff/7J77//TkhICGPHjmXChAn06NEj3fp16tRh7dq1fPLJJ8yePVs/bWZKSgpNmzalXLlyDB06FDc3N32P4Ivw8PCgffv26X6ln53Hc/v2bc6fP8/WrVvx8/PD19dX/wBwcHBg165dFClShA4dOlC6dGl69+5NQkJChj20Dg4ObNy4kfv371OtWjU6duxIo0aNmDNnTqaOY+TIkXTu3Jlu3bpRs2ZNnJycaNas2TPHDaeqUKECX331FVOnTiUkJITFixczefJkgzqvvPIK3333HTNnzqRChQps2rSJTz75JM22lixZwrvvvpupePMjo2ctePKET/2PI3UTT77WaDRZ+i8vM2TWAvOw5wh0HQHurnBoua6HVgghXlRGVzuL3Hfy5EmaNGlCaGgoTk5Opg7HpLRaLaVLl6ZTp05MnDgxx/d35swZGjZsyMWLF3F1dc3x/WWn7Jq1wOiBMtu3bzd2FfGSe6WiLom9H6UbXlAn7cWbQgghzFT58uWZOnUqYWFhlCtXztTh5Kpr166xadMm6tevT2JiInPmzCEsLIwuXbrkyv4jIiL4+eefzS6JzU5GJ7L169fPiThEPmZlCc3qwJK1sG6nJLJCCJHfPGtIgLnLqId5/fr1BAYGsmjRIkaMGIFSipCQELZs2ULp0qVzJb7GjRvnyn7yshe6dPHhw4f88MMP+hsilC1bll69er3U/xGIjLVqoEtkN+yGCUNkeIEQQoi87/jx489cVqhQIezt7dPMviByl9GJ7OHDh2nWrBn29vb6SYC/+uorJk2axKZNm6hcuXK2BynM3ysVwdUZ7j2EE+ehStnnrSGEEEKYlrGzMYncZ/SlisOGDeO1117j6tWrrFixghUrVhAWFkbr1q0ZOnRoDoQo8gNrK6j1/1Pc7Ttm2liEEEIIkT8YncgePnyYkSNHGkyobGVlxUcffcThw4ezNTiRv+gT2YxvNy2EEEIIkSlGJ7IuLi5cv349TXl4eDjOzs7ZEpTIn2r9/6iTI6chIcm0sQghhBDC/BmdyL755pv07t2bP/74g/DwcMLDw/n999/p06cPnTt3zokYRT5RzB+8C0LiY10yK4QQQgiRFUZf7DVt2jQ0Gg3dunUjOTkZAGtra9577z2mTJmS7QGK/EOj0Q0vWLlZN062tlwXKIQQQogsMKpHNiUlhQMHDjBu3DgePHjA8ePHOX78OPfv32fGjBnY2trmVJwin0gdJ7tXxskKIV5CSin69u2Lu7s7Go0GNze3HL9Qety4cVSsWDFH9yGEqRjVI2tpaUnTpk05d+4cQUFBL90dPETWpd4M4cR5uH0PvD1MG48QQuSmDRs2sGjRInbs2EHRokWxsLDA3t7e1GEJYbaMHiMbEhLClStXciIW8RLw84LKZUGrhTVyt2MhxEsmNDQUX19fatWqhY+PD15eXnKhtBBZYHQi+9lnnzFixAjWrFlDREQE0dHRBg8hnqddI93PVVtMG4cQQuSmHj168P7773P9+nU0Gg2BgYE0aNBAP7Tg/PnzODg48Ntvv+nX+fPPP7G3t+fs2bOA7s6affr0wdPTExcXFxo2bMiJEycM9jNlyhS8vb1xdnamd+/eJCQk5NoxCpHbjL7Yq2XLlgC89tpraDQafblSCo1GQ0pKSvZFJ/Kl1q/C+Dlw8gJcCYei/qaOSAhh7pSCeBPla/Z2uotZn2fmzJkUK1aMBQsWcOjQISwtLXnjjTf0y0uVKsW0adMYMGAAderUwcLCgv79+zN16lTKlCkDwBtvvIG9vT3r16/H1dWV+fPn06hRIy5evIi7uzt//vkn48aNY+7cudSpU4dffvmFWbNmUbRo0Zw6fCFMyuhEdvt2+T5YZI2HG9StBjsO6nplh/c0dURCCHMXnwClW5pm3+fWgUMmhrm6urri7OyMpaUlPj4+6dYZMGAA69at4+2338bGxoZq1arx/vvvA7Bnzx7++ecfIiMj9RdXT5s2jVWrVrFs2TL69u3L119/Te/evenduzeg+xZ1y5Yt0isr8i2jE9mgoCD8/f0NemNB1yMbHh6ebYGJ/K1do/8S2WE9MtebIYQQL4Mff/yREiVKYGFhwZkzZ/R/b0+cOEFMTAweHoZXycbHxxMaGgrAuXPn6N+/v8HymjVrSieUyLdeKJGNiIjAy8vLoPz+/fsEBQXJ0AKRKU3r6L6Ou3YTjp+DSmVMHZEQwpzZ2+l6Rk217+x04sQJYmNjsbCwICIiAl9fXwBiYmLw9fVlx44dadZxc3PL3iCEMBNGJ7KpY2GfFhMTg51dNr+bRb7laA9Na8NfW3W9spLICiGyQqPJ3Nf7ed39+/fp0aMHH3/8MREREXTt2pWjR49ib29P5cqVuXXrFlZWVgQGBqa7funSpTl48CDdunXTlx04cCCXohci92U6kR0+fDgAGo2GMWPG4ODgoF+WkpLCwYMHZcJlYZR2jXWJ7JodMGYgWFmaOiIhhDCt/v374+/vzyeffEJiYiKVKlVixIgRzJ07l8aNG1OzZk3atWvHF198QYkSJbh58yZr166lffv2VK1alSFDhtCjRw+qVq1K7dq1Wbx4MWfOnJGLvUS+lelE9tixY4CuR/bUqVPY2Njol9nY2FChQgVGjBiR/RGKfKtuVXB3hbsPYO8RqF/d1BEJIYTp/Pzzz6xbt45jx45hZWWFlZUVv/76K3Xq1KF169a0aNGCdevW8fHHH9OzZ0/u3LmDj48P9erVw9vbG4A333yT0NBQPvroIxISEnj99dd577332Lhxo4mPToicoVFKKWNW6NmzJzNnzsTFxSWnYsq06OhoXF1diYqKyhPxCON9PAN+XQ2dW8EU+T9ICJEJCQkJhIWFERQUJEPahDBTGb2PjcnvjL4hwsKFCyVpFNmm4Su6n7uP6OaBFEIIIYTILKMv9oqNjWXKlCls3bqVyMhItFqtwXK5fa0wxisVwdoKbtzSzWAQWMjUEQkhhBDCXBidyPbp04edO3fyzjvv4Ovrm+4MBkJklqM9VC4DB0/CniOSyAohhBAi84xOZNevX8/atWupXbt2TsQjXkJ1quoS2d2H4e3XTB2NEEIIIcyF0WNkCxQogLu7e07EIl5Sdavofu47BnI/DSGEEEJkltGJ7MSJExk7dixxcXE5EY94CZUvCS6OEB0DJy+YOhohhBBCmAujhxZMnz6d0NBQvL29CQwMxNra2mD50aNHsy048XKwtISalWDjHjhwQu7yJYQQQojMMTqRbdeuXQ6EIV521crpEtl/TsJ7nU0djRBCCCHMgdGJ7KeffpoTcYiXXPXyup+HT4NWCxZGD3oRQgghxMvmhdKFhw8f8v333zN69Gju378P6IYU/Pvvv9kanHh5lC0ODna6cbIXwkwdjRBC5F07duxAo9Hw8OFDfdmqVasIDg7G0tKSoUOHmiw2c5Zeu5qLF409KSmJ4OBg9u3bl63xzJs3jzZt2mTrNp/F6ET25MmTlChRgqlTpzJt2jR9o61YsYLRo0dnd3ziJWFlCZXL6p7/c9K0sQghRF5Wq1YtIiIicHV11Zf169ePjh07Eh4ezsSJE00YnTAn8+bNIygoiFq1aqVZlpiYSMWKFdFoNBw/ftxg2Z9//knFihVxcHAgICCAL7/80mB5r169OHr0KLt3787J8IEXSGSHDx9Ojx49uHTpksG9cVu2bMmuXbuyNTjxcqleTvfz0CnTxiGEEHmZjY0NPj4++hsSxcTEEBkZSbNmzfDz88PZ2fmFtpuUlJSdYYo8TinFnDlz6N27d7rLP/roI/z8/NKUr1+/nq5du9K/f39Onz7NN998w4wZM5gzZ46+jo2NDV26dGHWrFk5Fn8qoxPZQ4cO0a9fvzTlhQoV4tatW0Zta/LkyVSrVg1nZ2e8vLxo164dFy7I/Esvq2r/P072n1OglGljEUKInBAYGMjXX39tUFaxYkXGjRunf63RaPj+++9p3749Dg4OFC9enNWrV+uXP/k18o4dO/SJa8OGDdFoNOzYsQOA5cuXU7ZsWWxtbQkMDGT69OlpYpk4cSLdunXDxcWFvn37smjRItzc3FizZg0lS5bEwcGBjh07EhcXx08//URgYCAFChRg8ODBpDxn4u/z589Tp04d7OzsKFOmDFu2bEGj0bBq1ao0x5Hq+PHjaDQarl69qi/bs2cPdevWxd7eHn9/fwYPHkxsbKx++TfffEPx4sWxs7PD29ubjh076pctW7aMcuXKYW9vj4eHB40bNzZY93me14Y5ue+sSP09bty4kdKlS+Pk5ETz5s2JiIjQ1zly5AihoaG0atUqzfrr169n06ZNTJs2Lc2yX375hXbt2tG/f3+KFi1Kq1atGD16NFOnTkU98ce7TZs2rF69mvj4+Jw5yP9ndCJra2tLdHR0mvKLFy/i6elp1LZ27tzJwIEDOXDgAJs3b+bx48c0bdo0137RIm+pVBqsreD2XQi7YepohBDmKCkpiaSkJIM/qCkpKSQlJZGcnJztdXPK+PHj6dSpEydPnqRly5Z07dpVf03Kk2rVqqXvAFq+fDkRERHUqlWLI0eO0KlTJ9566y1OnTrFuHHjGDNmDIsWLTJYf9q0aVSoUIFjx44xZswYAOLi4pg1axa///47GzZsYMeOHbRv355169axbt06fvnlF+bPn8+yZcueGX9KSgrt2rXDwcGBgwcPsmDBAj7++GOj2yE0NJTmzZvz+uuvc/LkSf744w/27NnDoEGDADh8+DCDBw9mwoQJXLhwgQ0bNlCvXj0AIiIi6Ny5M7169eLcuXPs2LGDDh06GPwOM/K8NszufTs5OWX46N+/v1FtFxcXx7Rp0/jll1/YtWsX169fZ8SIEfrlu3fvpkSJEml68G/fvs27777LL7/8goODQ5rtJiYmGnwjD2Bvb8+NGze4du2avqxq1aokJydz8OBBo+I2mjJS7969Vbt27VRSUpJycnJSV65cUdeuXVOVKlVSQ4YMMXZzBiIjIxWgdu7cman6UVFRClBRUVFZ2q/IO7p8oFSRBkp985upIxFC5FXx8fHq7NmzKj4+Ps2yKVOmqClTpqjY2Fh92d69e9WUKVPUunXrDOpOnz5dTZkyRT18+FBf9s8//6gpU6ao1atXG9SdOXOmmjJlioqMjNSXHTt2zOjYAwIC1IwZMwzKKlSooD799FP9a0B98skn+tcxMTEKUOvXr1dKKbV9+3YFqAcPHiillHrw4IEC1Pbt2/XrdOnSRTVp0sRgPx9++KEqU6aMQSzt2rUzqLNw4UIFqMuXL+vL+vXrpxwcHNSjR4/0Zc2aNVP9+vV75nGuX79eWVlZqYiICH3Z5s2bFaBWrlyZ7nEopWtTQIWFhSmldDlH3759Dba9e/duZWFhoeLj49Xy5cuVi4uLio6OThPDkSNHFKCuXr36zDif9HQ8z2vD7Ny3UkpdunQpw8ft27czHXt6v8e5c+cqb29v/eshQ4aohg0bGmxHq9Wq5s2bq4kTJyqllAoLC1OAwbk+f/585eDgoLZs2aJSUlLUhQsXVKlSpRSg9u3bZ7C9AgUKqEWLFqUbc0bvY2PyO6N7ZKdPn05MTAxeXl7Ex8dTv359goODcXZ2ZtKkSVlKqqOiogDkFrgvsZa6f2ZZt9O0cQghhCmVL19e/9zR0REXFxciIyMzvf65c+eoXbu2QVnt2rW5dOmSQU9y1apV06zr4OBAsWLF9K9Tb4Dk5ORkUJYaz+eff27Qc3j9+nUuXLiAv78/Pj4++nWqV6+e6fhTnThxgkWLFhlsv1mzZmi1WsLCwmjSpAkBAQEULVqUd955h8WLF+vvPFqhQgUaNWpEuXLleOONN/juu+948OBBpvf9vDbM7n0HBwdn+PDy8jKq7Z7+Pfr6+hqcQ/Hx8Wl6VmfPns2jR48yvHj/3XffZdCgQbRu3RobGxteeeUV3nrrLQAsnpo7097ePsfvBGv0PLKurq5s3ryZvXv3cuLECWJiYqhcuTKNGzfOUiBarZahQ4dSu3ZtQkJC0q2TmJhIYmKi/nV6QxyEeWtaBz7+Wner2hu3oLDPc1cRQgi9YcOGARjcdbJGjRpUrVo1zR/Z1K+nn6xbuXJlKlSokKZu6te6T9YtV66c0fFZWFik+Xr58ePHaeo9fddMjUaDVqs1en/P4+jomKl9ZxRP//796dSpk35ZehcIpSe1jZ9sj6fbIiYmhn79+jF48OA06xcpUgQbGxuOHj3Kjh072LRpE2PHjmXcuHEcOnQINzc3Nm/ezL59+9i0aROzZ8/m448/5uDBgwQFBWUqxow4Oztn676f/EchPW+//Tbz5s3LdHzp/c6ebOuCBQty6pTh1dXbtm1j//792NraGpRXrVqVrl278tNPP6HRaJg6dSqff/45t27dwtPTk61btwJQtGhRg/Xu379v9LBTYxmdyKaqXbs2AQEB+Pr6YmlpmeVABg4cyOnTp9mzZ88z60yePJnx48dneV8i7/J0181ecPAkbNgNfd4wdURCCHNiY2OTpszS0jLdv1PZUddYnp6eBhfcREdHExaW/ZNnly5dmr179xqU7d27lxIlSmTL3+wnubu7p/kmtWTJkoSHh3P79m28vb0B3cXiT0pNcCIiIihQoABAmmmeKleuzNmzZwkODn7m/q2srGjcuDGNGzfm008/xc3NjW3bttGhQwc0Gg21a9emdu3ajB07loCAAFauXMnw4cOfe1yZacPs3PfTx/40FxeX58ZsjEqVKvHtt9+ilNLPgDFr1iw+++wzfZ2bN2/SrFkz/vjjD2rUqGGwvqWlJYUKFQJgyZIl1KxZ0yBpDQ0NJSEhgUqVKmVr3E974UQWoEyZMhw/fjxNBm6sQYMGsWbNGnbt2kXhwoWfWW/06NEGJ0B0dDT+/v5Z2rfIe1rU1yWy63ZKIiuEyF8aNmzIokWLaNOmDW5ubowdOzbbE0uADz74gGrVqjFx4kTefPNN9u/fz5w5c/jmm2+yfV/padKkCcWKFaN79+588cUXPHr0iE8++QRAnzQFBwfj7+/PuHHjmDRpEhcvXkwzK8DIkSN55ZVXGDRoEH369MHR0ZGzZ8+yefNm5syZw5o1a7hy5Qr16tWjQIECrFu3Dq1WS8mSJTl48CBbt26ladOmeHl5cfDgQe7cuUPp0qUzdQzPa8Ps3ndGyXpOePXVV4mJieHMmTP6b8KLFCliUCe1l7hYsWL6/Ozu3bssW7aMBg0akJCQwMKFC1m6dCk7dxqOCdy9ezdFixY1GN6QE7J0I9Cnvx55kfUHDRrEypUr2bZt23O7+m1tbXFxcTF4iPynWR3dz2PnICrGtLEIIUR2Gj16NPXr16d169a0atWKdu3a5cgf+sqVK/Pnn3/y+++/ExISwtixY5kwYQI9evTI9n2lx9LSklWrVhETE0O1atXo06ePftaC1HGZ1tbWLFmyhPPnz1O+fHmmTp1q0BsIurHCO3fu5OLFi9StW5dKlSoxduxY/fAFNzc3VqxYQcOGDSldujTz5s1jyZIllC1bFhcXF3bt2kXLli0pUaIEn3zyCdOnT6dFixaZOobntWFO7js3eHh40L59exYvXmz0uj/99BNVq1aldu3anDlzhh07dqQZA71kyRLefffd7Ar3mTQqC9mos7MzJ06ceOEe2QEDBvDbb7/x119/UbJkSX25q6sr9vb2z10/OjoaV1dXoqKiJKnNZxq8o5uC68fPoVFNU0cjhMhLEhISCAsLIygoKM3FKiLv2rt3L3Xq1OHy5cs53ksnMufkyZM0adKE0NDQ547RNcaZM2do2LAhFy9eNLgD3ZMyeh8bk99lqUf2f//7X5ZmGPj222+JioqiQYMG+Pr66h9//PFHVsIS+UCNCrqfB06YNg4hhBAvZuXKlWzevJmrV6+yZcsW+vbtS+3atSWJzUNSe8Kze5x2REQEP//88zOT2OyUpTGyo0ePJiUlhePHjxMQEKAfrJ1ZWR2aIPKvVyrA72vhwHFTRyKEEOJFPHr0iJEjR3L9+nUKFixI48aN04yBFaaXE8NNsjqTlTGM7pEdOnQoP/zwA6C7c0f9+vWpXLky/v7++tviCZFVr1TU/Tx9CR7Jjd6EEMLsdOvWjYsXL5KQkMCNGzdYtGgRHh4epg5L5DNGJ7LLli2jQgXd975///03V65c4fz58wwbNuyFbj8nRHp8PSHAD7RaOHTq+fWFEEII8fIxOpG9e/eu/k4d69ato1OnTpQoUYJevXqlmVhXiKxI7ZWV4QVCiPTI8DQhzFd2vX+NTmS9vb05e/YsKSkpbNiwgSZNmgAQFxeXI3PhiZfXK/9/wdc/J00bhxAib0m9Y1FO3/pSCJFzUt+/T9+BzFhGX+zVs2dPOnXqhK+vLxqNRj+g9+DBg5QqVSpLwQjxpKr/f/fH05cgIRHsbDOuL4R4OVhaWuLm5qa/b7yDg4N+kn0hRN6mlCIuLo7IyEjc3Nyy3AlqdCI7btw4QkJCCA8P54033tDfj9fS0pJRo0ZlKRghnuTvo7tl7Z37cOoiVDP+tuZCiHwqdYhbajIrhDAvbm5u+vdxVrzQ9FsdO3Y0eP3w4UO6d++e5WCEeJJGA1XKwobdcOS0JLJCiP9oNBp8fX3x8vLi8ePHpg5HCGEEa2vrbBuOanQiO3XqVAIDA3nzzTcB6NSpE8uXL8fX15d169ZRvnz5bAlMCIAqIf+fyJ4xdSRCiLzI0tJSrs8Q4iVm9MVe8+bNw9/fH4DNmzezefNm1q9fT/PmzRkxYkS2ByheblXK6n4eOQNygbIQQgghnmR0j+ytW7f0ieyaNWvo1KkTTZs2JTAwkBo1amR7gOLlFlIcbK3h3kO4+i8EFTZ1REIIIYTIK4zukS1QoADh4eEAbNiwQT9rgVKKlJSU7I1OvPRsbaBcSd3zI6dNG4sQQggh8hajE9kOHTrQpUsXmjRpwr1792jRogUAx44dIzg4ONsDFCL1Iq9lm2R4gRBCCCH+Y3QiO2PGDAYNGkSZMmXYvHkzTk5OAERERDBgwIBsD1CIt18DG2vYfwz2HDF1NEIIIYTIKzTKjO/xFx0djaurK1FRUbi4uJg6HJGDJsyFH5bpxsz+PQ8sjP4XTAghhBDmwJj87oXSgdDQUN5//30aN25M48aNGTx4MFeuXHmhYIXIjIFdwclBd5evnYdMHY0QQggh8gKjE9mNGzdSpkwZ/vnnH8qXL0/58uU5ePCgfqiBEDnBww3aNtI9333YpKEIIYQQIo8wevqtUaNGMWzYMKZMmZKmfOTIkTRp0iTbghPiSa9UhMV/w8ETpo5ECCGEEHmB0T2y586do3fv3mnKe/XqxdmzZ7MlKCHSU+P/bxp3NhSiY0wbixBCCCFMz+hE1tPTk+PHj6cpP378OF5eXtkRkxDp8i4IgYVAq4VDp0wdjRBCCCFMzeihBe+++y59+/blypUr1KpVC4C9e/cydepUhg8fnu0BCvGkGhV0d/g6eAIa1TR1NEIIIYQwJaMT2TFjxuDs7Mz06dMZPXo0AH5+fowbN47Bgwdne4BCPOmVCvDHOhknK4QQQggjE9nk5GR+++03unTpwrBhw3j06BEAzs7OORKcEE+rUUH389RFiInTTcklhBBCiJeTUWNkrays6N+/PwkJCYAugZUkVuSmQt5Q2BtStHDivKmjEUIIIYQpGX2xV/Xq1Tl27FhOxCJEplQqo/t5TCbJEEIIIV5qRo+RHTBgAB988AE3btygSpUqODo6GiwvX758tgUnRHoqlYG/t8Oxc6aORAghhBCmZHQi+9ZbbwEYXNil0WhQSqHRaEhJScm+6IRIx5M9skqBRmPaeIQQQghhGkYnsmFhYTkRhxCZVjYYbKzh3kMIj4AifqaOSAghhBCmYHQiGxAQkBNxCJFptja6ZPbYOTh6VhJZIYQQ4mVl9MVekydP5scff0xT/uOPPzJ16tRsCUqI55ELvoQQQghhdCI7f/58SpUqlaa8bNmyzJs3L1uCEuJ5UhPZo5LICiGEEC8toxPZW7du4evrm6bc09OTiIiIbAlKiOepGqL7eeoinLls2liEEEIIYRpGJ7L+/v7s3bs3TfnevXvx85PBiiJ3+HnBaw11sxZMmW/qaIQQQghhCkZf7PXuu+8ydOhQHj9+TMOGDQHYunUrH330ER988EG2ByjEs3zYG9bvgl2HYc8RqFPF1BEJIYQQIjcZnch++OGH3Lt3jwEDBpCUlASAnZ0dI0eOZPTo0dkeoBDPUsQPuraBRSthzmJJZIUQQoiXjUYppV5kxZiYGM6dO4e9vT3FixfH1tY2u2N7rujoaFxdXYmKisLFxSXX9y9ML/wW1OkMFhZweDl4uJk6IiGEEEJkhTH5ndFjZFM5OTlRrVo1QkJCTJLECgHg7wPlSoBWC5vSDt0WQgghRD72wolsdti1axdt2rTBz88PjUbDqlWrTBmOMFPN6+p+rt9l2jiEEEIIkbtMmsjGxsZSoUIF5s6da8owhJlrUU/3c99RiIoxbSxCCCGEyD1GX+yVnVq0aEGLFi1MGYLIB4oVgeIBcOkabN0HHZqaOiIhhBBC5AaT9sgaKzExkejoaIOHEACtX9X9/HG5bm5ZIYQQQuR/ZpXITp48GVdXV/3D39/f1CGJPOKdtuBgp7vT17YDpo5GCCGEELnBrBLZ0aNHExUVpX+Eh4ebOiSRR3i4Qff2uuczFkmvrBBCCPEyMKtE1tbWFhcXF4OHEKne7fRfr+zuw6aORgghhBA5zawSWSEy4uH234Vea3aYMhIhhBBC5AaTJrIxMTEcP36c48ePAxAWFsbx48e5fv26KcMSZix1Kq7NeyE5xbSxCCGEECJnmTSRPXz4MJUqVaJSpUoADB8+nEqVKjF27FhThiXMWI0K4OYC96Pg8ClTRyOEEEKInGTSeWQbNGiAkqtyRDaytoImtWDpBtiwG16paOqIhBBCCJFTZIysyHea1dH93LhHZi8QQggh8jNJZEW+U7eqbvaCm5G6GQyEEEIIkT9JIivyHTtbeLWG7vmGXaaNRQghhBA5RxJZkS81q6v7uX63DC8QQggh8itJZEW+1PAVsLGGK+Fw6ZqpoxFCCCFETpBEVuRLzo5Qu7Lu+cY9po1FCCGEEDlDElmRb6UOL1i7XYYXCCGEEPmRJLIi32paW3fh17kr8Mc6U0cjhBBCiOwmiazItzzc4INeuueTvoXbd00ajhBCCCGymSSyIl/r9TpUKAXRsTB5gamjEUIIIUR2kkRW5GtWljDufd3z9bsgLt608QghhBAi+0giK/K9SqXB3xcSEmH7P6aORgghhBDZRRJZke9pNNCynu75+p2mjUUIIYQQ2UcSWfFSaNVA93Prfl3PrBBCCCHMnySy4qVQviQU8oa4BNh+0NTRCCGEECI7SCIrXgoazX+9spPnQ9Qjk4YjhBBCiGwgiax4aQzoDIW94dpNGDIJtFpTRySEEEKIrJBEVrw0CrjCvAlga6MbXrBohakjEkIIIURWSCIrXirlSsCYAbrn036EiDumjUcIIYQQL04SWfHS6doGKpeF2Hj4dLapoxFCCCHEi5JEVrx0LCxg8nDdXb827oblG00dkRBCCCFehCSy4qVUqigM7qZ7/snXcPm6ScMRQgghxAuQRFa8tAZ1hVqVdHPLDhwvN0oQQgghzI0ksuKlZWkJMz+BggXg/BUYP8fUEQkhhBDCGJLIipealzvM/Fh3w4Tf1sDqbaaOSAghhBCZJYmseOnVqQIDu+qej54OV/81bTxCCCGEyBxJZIUAhvWA6uUhJg4GjIfEJFNHJIQQQojnkURWCHRTcc36BAq4wJlLMPRzSHps6qiEEEIIkRFJZIX4f76eumTW2grW7YR+YyFBemaFEEKIPEsSWSGeUK8afD8JbG1g2wEYMQW0WlNHJYQQQoj0SCIrxFMaVIcfP9f1zP69HSbM1Y2dFUIIIUTeolFKKVMH8aKio6NxdXUlKioKFxcXU4cj8pkVm2DYZN1zO1uoUQGKB8CbLaFEoElDE0IIIfItY/I76ZEV4hk6NIXpIyGosO6uXzv/ge+XQodBcOK8qaMTQgghhPTICvEcSsHpi3DqIizdCEfPgIsjzPgfNKqpu5mCEEIIIbKHMfmdJLJCGCE2Hrp9BIdP615XLw9d2+guErO1AQc7SWyFEEKIrJBEVogcFBsPs36Ghcsh8am5ZgP8oP9bUL86FCygS26FEEIIkXlmN0Z27ty5BAYGYmdnR40aNfjnn39MHZIQz+RoD6P7wY5fYUg3CCz037JrN2H0V1DrLSjVAl5/H75dArsPw+17umEKQgghhMgeJu+R/eOPP+jWrRvz5s2jRo0afP311yxdupQLFy7g5eWV4brSIyvyiqTHEJ8IyzfAL6shPAIeJ6et5+aim/GgRCC4OoOtNZQsCqWLgsZCN2dtcgqk/P/DuyB4uOXywQghhBAmZFZDC2rUqEG1atWYM2cOAFqtFn9/f95//31GjRqV4bqSyIq8Sin49zZs3Q/7jsGlqxD274vdXMHFSdcL7GAHAYV0wxfc3eDGLTh5Hgr5QP1quoRXKXgUCyla3e12ra10CbWLky6JjnoE0THg7KibUgzAzkaXVIPuTmaJibqfySm69b0LgqWFbrsO9uBZANDoEu3kFLDQ6GKztPzv2LVa0P7/T6XS74lOM5Y4nbHFaapoMn6dbp3n7TeT2xXmR6vV/S7l9ymEeTGbRDYpKQkHBweWLVtGu3bt9OXdu3fn4cOH/PXXXwb1ExMTSUxM1L+Ojo7G399fEllhFhKSIPQ6XAiDy9cgLl53o4VTFyEsHCwswcpSlxBaWeoSsLsPzWc4gkZjPrFmh0wl1S+yznMS8XQT/udsN7087rnrZOJ4ni54oePJoVji4nXj2S0sdGPVbW3A2vIZAZix/Jak57vjMXUA2WjKCHi1Ru7sy5hE1ip3Qkrf3bt3SUlJwdvb26Dc29ub8+fTTtQ5efJkxo8fn1vhCZGt7GygbLDukVnxCRB+CxKTdD2pYTd0Pb13H4C7K1QsDZevw6GTurluNRpdb6tGAw+idT2zVpa6ntj7UeDmrOudjYvXDYXQaHT7ePhI17NqZ6v7g29jDVZWuv1G3tP1bDk76hKDpMfpx/oyJbGQ9nhftuM3F1qt7hyPTzB1JEKYt4TE59cxBZMmssYaPXo0w4cP179O7ZEVIr+ytzO8i1jtyrkfQ2qCltrjGhOne57ae5yi1SXGKSm63i+NRpcUayx0Py0s0vZKPJ3zpZcEPi9RTDdvfN466e0nh2JJs51siCVTsT0vjkzEkpnjya5YnttOLxCLUrphME4OuvMzMUn3ePyMf8JeZvL/V1ryT2n6/H1NHUH6TJrIFixYEEtLS27fvm1Qfvv2bXx8fNLUt7W1xdbWNrfCE0Jg+FVfao/vk6zR9TYLIYQQuc2k02/Z2NhQpUoVtm7dqi/TarVs3bqVmjVrmjAyIYQQQgiR15l8aMHw4cPp3r07VatWpXr16nz99dfExsbSs2dPU4cmhBBCCCHyMJMnsm+++SZ37txh7Nix3Lp1i4oVK7Jhw4Y0F4AJIYQQQgjxJJPPI5sVMo+sEEIIIUT+Yna3qBVCCCGEEMJYksgKIYQQQgizJImsEEIIIYQwS5LICiGEEEIIs2TyWQuyIvU6tejoaBNHIoQQQgghskNqXpeZ+QjMOpF99OgRgNymVgghhBAin3n06BGurq4Z1jHr6be0Wi03b97E2dkZjebpu7nnjOjoaPz9/QkPD5cpv3KQtHPukHbOHdLOuUPaOXdIO+eOl7mdlVI8evQIPz8/LCwyHgVr1j2yFhYWFC5c2CT7dnFxeelOLFOQds4d0s65Q9o5d0g75w5p59zxsrbz83piU8nFXkIIIYQQwixJIiuEEEIIIcySJLJGsrW15dNPP8XW1tbUoeRr0s65Q9o5d0g75w5p59wh7Zw7pJ0zx6wv9hJCCCGEEC8v6ZEVQgghhBBmSRJZIYQQQghhliSRFUIIIYQQZkkSWSPMnTuXwMBA7OzsqFGjBv/884+pQzJr48aNQ6PRGDxKlSqlX56QkMDAgQPx8PDAycmJ119/ndu3b5swYvOwa9cu2rRpg5+fHxqNhlWrVhksV0oxduxYfH19sbe3p3Hjxly6dMmgzv379+natSsuLi64ubnRu3dvYmJicvEo8r7ntXOPHj3SnN/Nmzc3qCPt/HyTJ0+mWrVqODs74+XlRbt27bhw4YJBncx8Vly/fp1WrVrh4OCAl5cXH374IcnJybl5KHlaZtq5QYMGac7p/v37G9SRds7Yt99+S/ny5fVzw9asWZP169frl8u5bDxJZDPpjz/+YPjw4Xz66accPXqUChUq0KxZMyIjI00dmlkrW7YsERER+seePXv0y4YNG8bff//N0qVL2blzJzdv3qRDhw4mjNY8xMbGUqFCBebOnZvu8i+++IJZs2Yxb948Dh48iKOjI82aNSMhIUFfp2vXrpw5c4bNmzezZs0adu3aRd++fXPrEMzC89oZoHnz5gbn95IlSwyWSzs/386dOxk4cCAHDhxg8+bNPH78mKZNmxIbG6uv87zPipSUFFq1akVSUhL79u3jp59+YtGiRYwdO9YUh5QnZaadAd59912Dc/qLL77QL5N2fr7ChQszZcoUjhw5wuHDh2nYsCFt27blzJkzgJzLL0SJTKlevboaOHCg/nVKSory8/NTkydPNmFU5u3TTz9VFSpUSHfZw4cPlbW1tVq6dKm+7Ny5cwpQ+/fvz6UIzR+gVq5cqX+t1WqVj4+P+vLLL/VlDx8+VLa2tmrJkiVKKaXOnj2rAHXo0CF9nfXr1yuNRqP+/fffXIvdnDzdzkop1b17d9W2bdtnriPt/GIiIyMVoHbu3KmUytxnxbp165SFhYW6deuWvs63336rXFxcVGJiYu4egJl4up2VUqp+/fpqyJAhz1xH2vnFFChQQH3//fdyLr8g6ZHNhKSkJI4cOULjxo31ZRYWFjRu3Jj9+/ebMDLzd+nSJfz8/ChatChdu3bl+vXrABw5coTHjx8btHmpUqUoUqSItHkWhIWFcevWLYN2dXV1pUaNGvp23b9/P25ublStWlVfp3HjxlhYWHDw4MFcj9mc7dixAy8vL0qWLMl7773HvXv39MuknV9MVFQUAO7u7kDmPiv2799PuXLl8Pb21tdp1qwZ0dHR+p4wYejpdk61ePFiChYsSEhICKNHjyYuLk6/TNrZOCkpKfz+++/ExsZSs2ZNOZdfkJWpAzAHd+/eJSUlxeDEAfD29ub8+fMmisr81ahRg0WLFlGyZEkiIiIYP348devW5fTp09y6dQsbGxvc3NwM1vH29ubWrVumCTgfSG279M7l1GW3bt3Cy8vLYLmVlRXu7u7S9kZo3rw5HTp0ICgoiNDQUP73v//RokUL9u/fj6WlpbTzC9BqtQwdOpTatWsTEhICkKnPilu3bqV7zqcuE4bSa2eALl26EBAQgJ+fHydPnmTkyJFcuHCBFStWANLOmXXq1Clq1qxJQkICTk5OrFy5kjJlynD8+HE5l1+AJLLCZFq0aKF/Xr58eWrUqEFAQAB//vkn9vb2JoxMiKx766239M/LlStH+fLlKVasGDt27KBRo0YmjMx8DRw4kNOnTxuMpRfZ71nt/OT47XLlyuHr60ujRo0IDQ2lWLFiuR2m2SpZsiTHjx8nKiqKZcuW0b17d3bu3GnqsMyWDC3IhIIFC2JpaZnmysHbt2/j4+NjoqjyHzc3N0qUKMHly5fx8fEhKSmJhw8fGtSRNs+a1LbL6Fz28fFJcxFjcnIy9+/fl7bPgqJFi1KwYEEuX74MSDsba9CgQaxZs4bt27dTuHBhfXlmPit8fHzSPedTl4n/PKud01OjRg0Ag3Na2vn5bGxsCA4OpkqVKkyePJkKFSowc+ZMOZdfkCSymWBjY0OVKlXYunWrvkyr1bJ161Zq1qxpwsjyl5iYGEJDQ/H19aVKlSpYW1sbtPmFCxe4fv26tHkWBAUF4ePjY9Cu0dHRHDx4UN+uNWvW5OHDhxw5ckRfZ9u2bWi1Wv0fLmG8GzducO/ePXx9fQFp58xSSjFo0CBWrlzJtm3bCAoKMliemc+KmjVrcurUKYN/HDZv3oyLiwtlypTJnQPJ457Xzuk5fvw4gME5Le1sPK1WS2JiopzLL8rUV5uZi99//13Z2tqqRYsWqbNnz6q+ffsqNzc3gysHhXE++OADtWPHDhUWFqb27t2rGjdurAoWLKgiIyOVUkr1799fFSlSRG3btk0dPnxY1axZU9WsWdPEUed9jx49UseOHVPHjh1TgPrqq6/UsWPH1LVr15RSSk2ZMkW5ubmpv/76S508eVK1bdtWBQUFqfj4eP02mjdvripVqqQOHjyo9uzZo4oXL646d+5sqkPKkzJq50ePHqkRI0ao/fv3q7CwMLVlyxZVuXJlVbx4cZWQkKDfhrTz87333nvK1dVV7dixQ0VEROgfcXFx+jrP+6xITk5WISEhqmnTpur48eNqw4YNytPTU40ePdoUh5QnPa+dL1++rCZMmKAOHz6swsLC1F9//aWKFi2q6tWrp9+GtPPzjRo1Su3cuVOFhYWpkydPqlGjRimNRqM2bdqklJJz+UVIImuE2bNnqyJFiigbGxtVvXp1deDAAVOHZNbefPNN5evrq2xsbFShQoXUm2++qS5fvqxfHh8frwYMGKAKFCigHBwcVPv27VVERIQJIzYP27dvV0CaR/fu3ZVSuim4xowZo7y9vZWtra1q1KiRunDhgsE27t27pzp37qycnJyUi4uL6tmzp3r06JEJjibvyqid4+LiVNOmTZWnp6eytrZWAQEB6t13303zj6+08/Ol18aAWrhwob5OZj4rrl69qlq0aKHs7e1VwYIF1QcffKAeP36cy0eTdz2vna9fv67q1aun3N3dla2trQoODlYffvihioqKMtiOtHPGevXqpQICApSNjY3y9PRUjRo10iexSsm5/CI0SimVe/2/QgghhBBCZA8ZIyuEEEIIIcySJLJCCCGEEMIsSSIrhBBCCCHMkiSyQgghhBDCLEkiK4QQQgghzJIkskIIIYQQwixJIiuEEEIIIcySJLJCCCGEEMIsSSIrhBDZKDAwkK+//jrT9Xfs2IFGo+Hhw4c5FpMQQuRXcmcvIcRLrUGDBlSsWNGo5DMjd+7cwdHREQcHh0zVT0pK4v79+3h7e6PRaLIlBmPt2LGDV199lQcPHuDm5maSGIQQ4kVYmToAIYTI65RSpKSkYGX1/I9MT09Po7ZtY2ODj4/Pi4YmhBAvNRlaIIR4afXo0YOdO3cyc+ZMNBoNGo2Gq1ev6r/uX79+PVWqVMHW1pY9e/YQGhpK27Zt8fb2xsnJiWrVqrFlyxaDbT49tECj0fD999/Tvn17HBwcKF68OKtXr9Yvf3powaJFi3Bzc2Pjxo2ULl0aJycnmjdvTkREhH6d5ORkBg8ejJubGx4eHowcOZLu3bvTrl27Zx7rtWvXaNOmDQUKFMDR0ZGyZcuybt06rl69yquvvgpAgQIF0Gg09OjRAwCtVsvkyZMJCgrC3t6eChUqsGzZsjSxr127lvLly2NnZ8crr7zC6dOnX/A3IoQQxpFEVgjx0po5cyY1a9bk3XffJSIigoiICPz9/fXLR40axZQpUzh37hzly5cnJiaGli1bsnXrVo4dO0bz5s1p06YN169fz3A/48ePp1OnTpw8eZKWLVvStWtX7t+//8z6cXFxTJs2jV9++YVdu3Zx/fp1RowYoV8+depUFi9ezMKFC9m7dy/R0dGsWrUqwxgGDhxIYmIiu3bt4tSpU0ydOhUnJyf8/f1Zvnw5ABcuXCAiIoKZM2cCMHnyZH7++WfmzZvHmTNnGDZsGG+//TY7d+402PaHH37I9OnTOXToEJ6enrRp04bHjx9nGI8QQmQLJYQQL7H69eurIUOGGJRt375dAWrVqlXPXb9s2bJq9uzZ+tcBAQFqxowZ+teA+uSTT/SvY2JiFKDWr19vsK8HDx4opZRauHChAtTly5f168ydO1d5e3vrX3t7e6svv/xS/zo5OVkVKVJEtW3b9plxlitXTo0bNy7dZU/HoJRSCQkJysHBQe3bt8+gbu/evVXnzp0N1vv999/1y+/du6fs7e3VH3/88cxYhBAiu8gYWSGEeIaqVasavI6JiWHcuHGsXbuWiIgIkpOTiY+Pf26PbPny5fXPHR0dcXFxITIy8pn1HRwcKFasmP61r6+vvn5UVBS3b9+mevXq+uWWlpZUqVIFrVb7zG0OHjyY9957j02bNtG4cWNef/11g7iedvnyZeLi4mjSpIlBeVJSEpUqVTIoq1mzpv65u7s7JUuW5Ny5c8/cthBCZBdJZIUQ4hkcHR0NXo8YMYLNmzczbdo0goODsbe3p2PHjiQlJWW4HWtra4PXGo0mw6QzvfoqixPM9OnTh2bNmrF27Vo2bdrE5MmTmT59Ou+//3669WNiYgBYu3YthQoVMlhma2ubpViEECK7yBhZIcRLzcbGhpSUlEzV3bt3Lz169KB9+/aUK1cOHx8frl69mrMBPsXV1RVvb28OHTqkL0tJSeHo0aPPXdff35/+/fuzYsUKPvjgA7777jtA1wap20lVpkwZbG1tuX79OsHBwQaPJ8cRAxw4cED//MGDB1y8eJHSpUtn6TiFECIzpEdWCPFSCwwM5ODBg1y9ehUnJyfc3d2fWbd48eKsWLGCNm3aoNFoGDNmTIY9qznl/fffZ/LkyQQHB1OqVClmz57NgwcPMpyHdujQobRo0YISJUrw4MEDtm/frk82AwIC0Gg0rFmzhpYtW2Jvb4+zszMjRoxg2LBhaLVa6tSpQ1RUFHv37sXFxYXu3bvrtz1hwgQ8PDzw9vbm448/pmDBghnOoCCEENlFemSFEC+1ESNGYGlpSZkyZfD09MxwvOtXX31FgQIFqFWrFm3atKFZs2ZUrlw5F6PVGTlyJJ07d6Zbt27UrFkTJycnmjVrhp2d3TPXSUlJYeDAgZQuXZrmzZtTokQJvvnmGwAKFSrE+PHjGTVqFN7e3gwaNAiAiRMnMmbMGCZPnqxfb+3atQQFBRlse8qUKQwZMoQqVapw69Yt/v77b30vrxBC5CS5s5cQQpg5rVZL6dKl6dSpExMnTsy1/codwYQQpiZDC4QQwsxcu3aNTZs2Ub9+fRITE5kzZw5hYWF06dLF1KEJIUSukqEFQghhZiwsLFi0aBHVqlWjdu3anDp1ii1btsgFVkKIl44MLRBCCCGEEGZJemSFEEIIIYRZkkRWCCGEEEKYJUlkhRBCCCGEWZJEVgghhBBCmCVJZIUQQgghhFmSRFYIIYQQQpglSWSFEEIIIYRZkkRWCCGEEEKYJUlkhRBCCCGEWfo/skLIjkTmq5sAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: loss curves, broken vs fixed\n",
"fig, ax = plt.subplots(figsize=(7, 3.5))\n",
"ax.plot(broken_losses, color=\"#C2410C\", label=\"broken (no zero_grad)\")\n",
"ax.plot(losses, color=\"#1E40FF\", label=\"fixed\")\n",
"ax.axhline(math.log(VOCAB - 1), ls=\":\", c=\"#888\", label=f\"uniform-guess loss = ln({VOCAB - 1})\")\n",
"ax.set_xlabel(\"training step\"); ax.set_ylabel(\"cross-entropy loss\"); ax.legend()\n",
"ax.set_title(\"the missing zero_grad keeps the circuit from forming\")\n",
"plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "2e95847c",
"metadata": {},
"source": [
"> **Interpretation.** The fixed run dives well below the uniform-guess line (`ln(49) ≈ 3.89`, the loss of a model that has learned nothing) toward near-zero: it has learned to copy. The broken run sits at or above the uniform line the whole time. Same architecture, same data, same seed; one missing line. This is why the four-comment skeleton (`# forward / # backward / # update / # track`) is worth making muscle memory.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "949eedf9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.691763Z",
"iopub.status.busy": "2026-06-10T20:56:10.691687Z",
"iopub.status.idle": "2026-06-10T20:56:10.699252Z",
"shell.execute_reply": "2026-06-10T20:56:10.698879Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"on held-out repeat sequences: loss 0.0004 · next-token accuracy 100.0%\n"
]
}
],
"source": [
"# The model now predicts the second half almost perfectly. Quantify it.\n",
"@torch.no_grad()\n",
"def eval_metrics(model, batch, ablate=None):\n",
" half = batch.shape[1] // 2\n",
" model.eval()\n",
" logits = model(batch, ablate=ablate)\n",
" pred = logits[:, half - 1:-1, :]\n",
" target = batch[:, half:]\n",
" loss = F.cross_entropy(pred.reshape(-1, logits.shape[-1]), target.reshape(-1)).item()\n",
" acc = (pred.argmax(-1) == target).float().mean().item()\n",
" return loss, acc\n",
"\n",
"gen_eval = torch.Generator().manual_seed(SEED + 100) # offset seed: eval data disjoint from training\n",
"eval_batch = make_batch(BATCH, SEQ_LEN, VOCAB, gen_eval)\n",
"clean_loss, clean_acc = eval_metrics(model, eval_batch)\n",
"print(f\"on held-out repeat sequences: loss {clean_loss:.4f} · next-token accuracy {clean_acc:.1%}\")"
]
},
{
"cell_type": "markdown",
"id": "6ae90fd5",
"metadata": {},
"source": [
"> **Interpretation.** Near-perfect accuracy on sequences it never trained on. The model did not memorize, it learned the copy rule. Now we go find the mechanism that implements it.\n",
"\n",
"> **Key takeaways.** Train only on predictable positions. The four-comment loop is a checklist: a missing `zero_grad()` silently breaks training. A held-out eval batch (offset seed) confirms generalization, not memorization, before any interpretability work begins.\n"
]
},
{
"cell_type": "markdown",
"id": "814a8d60",
"metadata": {},
"source": [
"## Part 4 — Find the induction head\n",
"\n",
"> **Objectives.** Define the induction stripe precisely. Compute a per-head induction score and assert at least one head in layer 1 is an induction head. Visualize the attention pattern and see the stripe.\n",
"\n",
"The induction algorithm has a signature in the attention pattern. On a length-`2H` repeat sequence, a query at position `t` in the second half (`t >= H`) wants the token that followed the *previous* occurrence of the current token. The current token at `t` equals the token at `t - H`, whose successor sits at `t - H + 1`. So an induction head attends from `t` to `t - H + 1`: a diagonal stripe, offset by `H - 1` from the main diagonal.\n"
]
},
{
"cell_type": "markdown",
"id": "fb68d0c4",
"metadata": {},
"source": [
"### Micro-demo: the stripe, on a hand-built pattern\n",
"\n",
"Before scoring a real head, trace the index arithmetic on a tiny made-up pattern so the offset is unambiguous. We build a 6-position pattern (`H = 3`) where position `t` attends exactly to `t - H + 1`, then read the induction score off it. If our score function is right, this synthetic perfect-induction pattern scores 1.0.\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "16762891",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.700243Z",
"iopub.status.busy": "2026-06-10T20:56:10.700163Z",
"iopub.status.idle": "2026-06-10T20:56:10.702817Z",
"shell.execute_reply": "2026-06-10T20:56:10.702519Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"perfect induction pattern (rows = query t, cols = key):\n",
"[[0 0 0 0 0 0]\n",
" [0 0 0 0 0 0]\n",
" [0 0 0 0 0 0]\n",
" [0 1 0 0 0 0]\n",
" [0 0 1 0 0 0]\n",
" [0 0 0 1 0 0]]\n"
]
}
],
"source": [
"# A perfect induction pattern for seq_len=6, half=3: position t attends to t-half+1 = t-2.\n",
"seq6, half6 = 6, 3\n",
"perfect = torch.zeros(1, 1, seq6, seq6) # (batch=1, heads=1, 6, 6)\n",
"for t in range(half6, seq6): # second-half query positions 3,4,5\n",
" perfect[0, 0, t, t - half6 + 1] = 1.0 # attend to t-2\n",
"print(\"perfect induction pattern (rows = query t, cols = key):\")\n",
"print(perfect[0, 0].int().numpy())\n",
"# the stripe sits on positions (3->1), (4->2), (5->3)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "b334460b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.703559Z",
"iopub.status.busy": "2026-06-10T20:56:10.703485Z",
"iopub.status.idle": "2026-06-10T20:56:10.706913Z",
"shell.execute_reply": "2026-06-10T20:56:10.706517Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 4.0 perfect stripe scores 1.0\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def induction_score(pattern, seq_len):\n",
" \"\"\"pattern: (batch, n_heads, seq, seq). Mean attention from second-half query t\n",
" to its induction target t - half + 1. Returns one score per head, in [0, 1].\"\"\"\n",
" half = seq_len // 2\n",
" queries = torch.arange(half, seq_len) # second-half query positions\n",
" targets = queries - half + 1 # their induction targets\n",
" # gather pattern[:, :, t, t-half+1] for each t, average over t and batch\n",
" striped = pattern[:, :, queries, targets] # (batch, n_heads, len(queries))\n",
" return striped.mean(dim=(0, 2)) # (n_heads,)\n",
"\n",
"_score6 = induction_score(perfect, seq6)\n",
"check(\"4.0 perfect stripe scores 1.0\",\n",
" lambda: check_close(_score6.item(), 1.0, msg=\"a perfect stripe must score exactly 1\"))"
]
},
{
"cell_type": "markdown",
"id": "832b9cbc",
"metadata": {},
"source": [
"> **Interpretation.** The synthetic pattern that attends exactly along the induction stripe scores 1.0, which validates the index arithmetic (`t - half + 1`, not `t - half`). Off-by-one errors here are the classic induction-detector bug; the micro-demo catches them before we touch the real model.\n"
]
},
{
"cell_type": "markdown",
"id": "86fbba92",
"metadata": {},
"source": [
"### Score every real head\n",
"\n",
"Now run the trained model on the eval batch, cache each layer's pattern, and score all heads. We expect layer-1 heads to light up (induction needs two layers to compose), and we assert at least one clears a 0.5 threshold.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "cc8d9060",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.707852Z",
"iopub.status.busy": "2026-06-10T20:56:10.707783Z",
"iopub.status.idle": "2026-06-10T20:56:10.714665Z",
"shell.execute_reply": "2026-06-10T20:56:10.714304Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"induction score per head (rows = layer, cols = head):\n",
" layer 0: h0=0.61 h1=0.40 h2=0.56 h3=0.44\n",
" layer 1: h0=0.63 h1=0.75 h2=0.79 h3=0.61\n",
"strongest induction head: layer 1, head 2, score 0.79\n"
]
}
],
"source": [
"@torch.no_grad()\n",
"def all_head_scores(model, batch):\n",
" model.eval()\n",
" model(batch) # populates each layer's last_pattern\n",
" seq_len = batch.shape[1]\n",
" scores = torch.zeros(model.n_layers, model.n_heads)\n",
" for l, layer in enumerate(model.layers):\n",
" scores[l] = induction_score(layer.last_pattern, seq_len)\n",
" return scores\n",
"\n",
"scores = all_head_scores(model, eval_batch)\n",
"print(\"induction score per head (rows = layer, cols = head):\")\n",
"for l in range(model.n_layers):\n",
" print(f\" layer {l}: \" + \" \".join(f\"h{h}={scores[l, h]:.2f}\" for h in range(model.n_heads)))\n",
"best_layer, best_head = np.unravel_index(int(scores.argmax()), scores.shape)\n",
"best_layer, best_head = int(best_layer), int(best_head)\n",
"print(f\"strongest induction head: layer {best_layer}, head {best_head}, score {scores.max():.2f}\")"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "0eafbcde",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.715656Z",
"iopub.status.busy": "2026-06-10T20:56:10.715576Z",
"iopub.status.idle": "2026-06-10T20:56:10.722306Z",
"shell.execute_reply": "2026-06-10T20:56:10.721875Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 4.1 induction head exists in layer 1\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Assert the circuit exists. Keyed on a behavioral property with a wide margin, not a fragile value.\n",
"def _induction_exists():\n",
" s = all_head_scores(model, eval_batch)\n",
" assert s[1].max() > 0.5, \\\n",
" f\"expected an induction head in layer 1 (score > 0.5); got max {s[1].max():.2f}. \" \\\n",
" \"Did training converge? Check the loss curve dropped below ~0.5.\"\n",
"check(\"4.1 induction head exists in layer 1\", _induction_exists)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "2d67c5c5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.723103Z",
"iopub.status.busy": "2026-06-10T20:56:10.723027Z",
"iopub.status.idle": "2026-06-10T20:56:10.811523Z",
"shell.execute_reply": "2026-06-10T20:56:10.811016Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAG3CAYAAAB7fRYTAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAgE1JREFUeJzt3XdcE+cfB/DPJUDYS6bKco+KA5XiqFZRtG5ttXXhqHUhIlpHq+JoxVEVWwfVVtFWf1r3rAtFrVvUunGh4AAcbGQlz+8PSjTmgjkSCIHvu6971Tx57u57Scg3d/cMjjHGQAghhJAySaTrAAghhBCiGiVqQgghpAyjRE0IIYSUYZSoCSGEkDKMEjUhhBBShlGiJoQQQsowStSEEEJIGUaJmhBCCCnDKFETQgghZRgl6hIWEREBjuPw6NEjXYdS6oYMGQJzc3Ndh6Ggbdu2aNu2ra7DUOLu7o6uXbuW+H5K4vh1+ZpyHIdZs2bpZN8loSJ/XxDVKFETlTIyMhASEoJOnTrB1tYWHMchIiJC12GVuKysLKxYsQIdO3aEs7MzLCws0LhxY6xatQpSqbTY27116xZmzZpFX8ICHThwQC+T8cqVKyvE3wspeZSoiUovX77EnDlzcPv2bTRs2FDX4ZSahw8fYty4cWCMITg4GD/99BM8PDwwZswYDBs2rNjbvXXrFmbPnq3TRH348GEcPnxYZ/svjgMHDmD27Nm8z7158wbTp08v5YjUU5xEPWjQILx58wZubm4lExTRSwa6DoDoVmZmJszMzHifc3Z2xvPnz+Hk5IRLly6hWbNmpRydbjg5OeH69euoX7++vGzkyJEYNmwY1q1bhxkzZqBGjRo6jLD4jIyMdB2CVhkbG+s6BK0o/DsUi8UQi8W6DoeUMXRGrQO7d+9Gly5dULlyZUgkElSvXh1z585VuKwaEhICQ0NDvHjxQmn9b775BtbW1sjOzpaX/f3332jdujXMzMxgYWGBLl264ObNmwrrFd4zfvDgAT777DNYWFhgwIABKuOUSCRwcnLS+HifPn2Knj17wtzcHPb29pg0aZLSJWSZTIawsDDUr18fxsbGcHR0xMiRI5GcnKxQT53XrtDq1atRvXp1mJiYoHnz5jh16pRa8drZ2Skk6UK9evUCANy+fVuh/MGDB3jw4EGR24yIiMAXX3wBAPj000/BcRw4jkNUVJRCvX/++QfNmzeHsbExqlWrhg0bNihtKyUlBUFBQXBxcYFEIkGNGjWwYMECyGSyDx7b+/eTo6KiwHEc/vrrL/z444+oWrUqjI2N0b59e9y/f19pfXVeU1X3WQv39f4xnz9/Hp999hlsbGxgZmYGT09PLFu2DEDBZ3bFihUAIH/NOI6Tr8t3j/rKlSvo3LkzLC0tYW5ujvbt2+PcuXO8MZ4+fRrBwcGwt7eHmZkZevXqxfs3976EhAQMHToUVatWhUQigbOzM3r06CE/Znd3d9y8eRMnTpyQx1z4uhfu+8SJExgzZgwcHBxQtWpVla9dYfuFw4cPo1GjRjA2Nka9evWwY8cOpbg0+WyQsovOqHUgIiIC5ubmCA4Ohrm5OY4dO4aZM2ciLS0NixYtAlBwCWzOnDnYsmULAgIC5Ovm5uZi27Zt6NOnj/xs4o8//oC/vz/8/PywYMECZGVlYdWqVWjVqhWuXLkCd3d3+fr5+fnw8/NDq1at8NNPP8HU1LREj1UqlcLPzw/e3t746aefcPToUSxevBjVq1fH6NGj5fVGjhyJiIgIDB06FIGBgYiNjcXy5ctx5coVnD59GoaGhmq/dgDw+++/Y+TIkWjRogWCgoLw8OFDdO/eHba2tnBxcSnWsSQkJAAoSOTvat++PQAUeUn7k08+QWBgIH7++Wd89913qFu3LgDI/w8A9+/fx+eff47hw4fD398fa9euxZAhQ+Dl5SX/4ZCVlYU2bdrg6dOnGDlyJFxdXXHmzBlMmzYNz58/R1hYWLGObf78+RCJRJg0aRJSU1OxcOFCDBgwAOfPn5fXKYnX9MiRI+jatSucnZ0xfvx4ODk54fbt29i3bx/Gjx+PkSNH4tmzZzhy5Aj++OOPD27v5s2baN26NSwtLTF58mQYGhri119/Rdu2bXHixAl4e3sr1B83bhxsbGwQEhKCR48eISwsDAEBAdiyZUuR++nTpw9u3ryJcePGwd3dHUlJSThy5Aji4uLg7u6OsLAwjBs3Dubm5vj+++8BAI6OjgrbGDNmDOzt7TFz5kxkZmYWub979+6hX79+GDVqFPz9/bFu3Tp88cUXOHjwIDp06ACg5D4bpAxgpEStW7eOAWCxsbHysqysLKV6I0eOZKampiw7O1te5uPjw7y9vRXq7dixgwFgx48fZ4wxlp6ezqytrdmIESMU6iUkJDArKyuFcn9/fwaATZ06VfBxXLx4kQFg69atU3udwv3NmTNHobxx48bMy8tL/vjUqVMMANu4caNCvYMHDyqVq/Pa5ebmMgcHB9aoUSOWk5Mjr7d69WoGgLVp00btYyiUk5PD6tWrxzw8PFheXp7Cc25ubszNze2D29i6davCe/f+NgCwkydPysuSkpKYRCJhEydOlJfNnTuXmZmZsbt37yqsP3XqVCYWi1lcXFyRMbRp00bh+I8fP84AsLp16yq8VsuWLWMA2PXr1xljwl5Tvs/8u/sqPP78/Hzm4eHB3NzcWHJyskJdmUwm//fYsWOZqq8qACwkJET+uGfPnszIyIg9ePBAXvbs2TNmYWHBPvnkE6UYfX19FfY1YcIEJhaLWUpKCu/+GGMsOTmZAWCLFi1SWYcxxurXr8/7WSvcd6tWrVh+fj7vc+++doWfje3bt8vLUlNTmbOzM2vcuLG8TNPPBim76NK3DpiYmMj/nZ6ejpcvX6J169bIysrCnTt35M8NHjwY58+fV7isunHjRri4uKBNmzYACs5IUlJS8NVXX+Hly5fyRSwWw9vbG8ePH1fa/7tnsqVh1KhRCo9bt26Nhw8fyh9v3boVVlZW6NChg8IxeHl5wdzcXOEY1HntLl26hKSkJIwaNUrhnuyQIUNgZWVVrGMICAjArVu3sHz5chgYKF6IevTokVYaiNWrVw+tW7eWP7a3t0ft2rWVXqvWrVvDxsZG4bXy9fWFVCrFyZMni7XvoUOHKrxWhXEU7rskXtMrV64gNjYWQUFBsLa2Vnju3cvb6pJKpTh8+DB69uyJatWqycudnZ3Rv39//PPPP0hLS1NY55tvvlHYV+vWrSGVSvH48WOV+zExMYGRkRGioqKUbs0IMWLECLXvR1euXFl+6wUALC0tMXjwYFy5ckV+paekPhtE9+jStw7cvHkT06dPx7Fjx5S+OFJTU+X/7tevH4KCgrBx40bMnDkTqamp2LdvHyZMmCD/crl37x4AoF27drz7srS0VHhsYGAgvx9WGoyNjWFvb69QZmNjo/AFd+/ePaSmpsLBwYF3G0lJSfJ/q/PaFX7J1qxZU+F5Q0NDhS9wdS1atAhr1qzB3Llz8dlnnwleX12urq5KZXyv1bVr15Re00Lvvlaa7NvGxgYA5PvW9msKQP4D9KOPPirW+u978eIFsrKyULt2baXn6tatC5lMhvj4eIX2Bx86bj4SiQQLFizAxIkT4ejoiI8//hhdu3bF4MGDBbXp8PDwULtujRo1lH681KpVC0DBD0UnJ6cS+2wQ3aNEXcpSUlLQpk0bWFpaYs6cOahevTqMjY1x+fJlTJkyRaHRh42NDbp27SpP1Nu2bUNOTg4GDhwor1NY/48//uD9knj/7E8ikUAkKr0LKeqcMchkMjg4OGDjxo28zxd+8Qh57bQlIiICU6ZMwahRo0q8G5Cq14oxJv+3TCZDhw4dMHnyZN66hV/eJbFvdak6G9akD3pJKe5xBwUFoVu3bti1axcOHTqEGTNmIDQ0FMeOHUPjxo3V2ve7V4e0oaQ+G0T3KFGXsqioKLx69Qo7duzAJ598Ii+PjY3lrT948GD06NEDFy9exMaNG9G4cWOFM4Lq1asDABwcHODr61uywZeQ6tWr4+jRo2jZsmWRX17qvnaFfVDv3buncKUhLy8PsbGxavcJ3717N77++mv07t1b3vJYE8W5nPu+6tWrIyMjo9TfayGvaeFZaUpKisI23r+cXPjZvXHjRpHHo+7rZm9vD1NTU8TExCg9d+fOHYhEomI3euNTvXp1TJw4ERMnTsS9e/fQqFEjLF68GH/++aeguNVx//59MMYUtnn37l0AkDcW1dVng5Q8ukddygp/wb/7iz03NxcrV67krd+5c2fY2dlhwYIFOHHihMLZNAD4+fnB0tIS8+bNQ15entL66nQ10bW+fftCKpVi7ty5Ss/l5+fLv/DVfe2aNm0Ke3t7hIeHIzc3V14eERGhlDxUOXnyJL788kt88skn2LhxY5FXIdTpngVA3l9d3Rj49O3bF2fPnsWhQ4eUnktJSUF+fn6xt10UIa9pYQJ+956oVCrF6tWrFeo1adIEHh4eCAsLU9rGu++xuq+bWCxGx44dsXv3boU2A4mJidi0aRNatWqldCuoOLKyshS6RgIFx2xhYYGcnByFuDV5r9/17Nkz7Ny5U/44LS0NGzZsQKNGjeRX0nT12SAlj86oS1mLFi1gY2MDf39/BAYGguM4/PHHHyovtRkaGuLLL7/E8uXLIRaL8dVXXyk8b2lpiVWrVmHQoEFo0qQJvvzyS9jb2yMuLg779+9Hy5YtsXz58mLHu3z5cqSkpODZs2cAgL179+LJkycACrq2FLch0bvatGmDkSNHIjQ0FFevXkXHjh1haGiIe/fuYevWrVi2bBk+//xztV87Q0ND/PDDDxg5ciTatWuHfv36ITY2FuvWrVPrfurjx4/RvXt3cByHzz//HFu3blV43tPTE56envLH6nTPAoBGjRpBLBZjwYIFSE1NhUQiQbt27VTem+fz7bffYs+ePejatau861ZmZiauX7+Obdu24dGjR0rdx7RByGtav359fPzxx5g2bRpev34NW1tbbN68WSlRiEQirFq1Ct26dUOjRo0wdOhQODs7486dO7h586Y84Xh5eQEAAgMD4efnB7FYjC+//JI3zh9++AFHjhxBq1atMGbMGBgYGODXX39FTk4OFi5cqJXX4u7du2jfvj369u2LevXqwcDAADt37kRiYqJCXF5eXli1ahV++OEH1KhRAw4ODirbknxIrVq1MHz4cFy8eBGOjo5Yu3YtEhMTsW7dOnkdXX02SCnQWXvzCoKvu8Xp06fZxx9/zExMTFjlypXZ5MmT2aFDh1R23blw4QIDwDp27KhyP8ePH2d+fn7MysqKGRsbs+rVq7MhQ4awS5cuyev4+/szMzMzQfEXdg3hW97vfvM+VfsLCQnh7W6zevVq5uXlxUxMTJiFhQVr0KABmzx5Mnv27Jm8jpDXbuXKlczDw4NJJBLWtGlTdvLkSaXuSXwKuxGpWt7tDlT4GqnTPYsxxtasWcOqVavGxGKxQsxubm6sS5cuSvX54k1PT2fTpk1jNWrUYEZGRszOzo61aNGC/fTTTyw3N7fI/avqnrV161aFerGxsbzd8dR9TR88eMB8fX2ZRCJhjo6O7LvvvmNHjhzhfZ/++ecf1qFDB2ZhYcHMzMyYp6cn++WXX+TP5+fns3HjxjF7e3vGcZzCZ4fv/bh8+TLz8/Nj5ubmzNTUlH366afszJkzCnUK/y4vXryoUP5+FzI+L1++ZGPHjmV16tRhZmZmzMrKinl7e7O//vpLoV5CQgLr0qULs7CwUOjCpmrf7z73fvesLl26sEOHDjFPT08mkUhYnTp1lN4zxjT7bJCyi2OsGK1FSKn6999/0ahRI2zYsAGDBg3SdTiEkFLk7u6Ojz76CPv27dN1KERH6B61HlizZg3Mzc3Ru3dvXYdCCCEV1smTJ9GtWzdUrlwZHMdh165dH1wnKioKTZo0kQ/pWpwZ1ShRl2F79+7FggULsHr1aowYMULl5BmEEEJKXmZmJho2bKh2L5DY2Fh06dIFn376Ka5evYqgoCB8/fXXvA3+ikKXvsswd3d3JCYmws/PD3/88QcsLCx0HRIhpJTRpe+yieM47Ny5Ez179lRZZ8qUKdi/fz9u3LghL/vyyy+RkpKCgwcPqr0vavVdhuly3mJCSNlA3wPKsrOzFboJFhd7r286UDAolEQi0XjbAHD27Fmlfu1+fn4ICgoStB1K1IQQQvRGdnY2PDyqICHhtcbbMjc3R0ZGhkJZSEiI0tSpxZWQkKA0a5qjoyPS0tLw5s0btUeno0RNCCFEb+Tm5iIh4TUexW6GpWXxp+lNS8uCu8eXiI+PVxgIR1tn09qkF4l6xYoVWLRoERISEtCwYUP88ssvaN68uVrrymQyPHv2DBYWFlod0o8QQoh6GGNIT09H5cqVtTbXgKWlKSwtNW9ga2lpqZUR6/g4OTkhMTFRoSwxMRGWlpaCxnov84l6y5YtCA4ORnh4OLy9vREWFgY/Pz/ExMSoNaLTs2fPtDq+LyGEkOKJj4/X3ux9MlnBosn6JczHxwcHDhxQKDty5Ah8fHwEbafMJ+olS5ZgxIgRGDp0KAAgPDwc+/fvx9q1azF16tQPrl/YUjr20V9Kl0kq2XbXfsCEEELeUzCwn1Z7ruggUWdkZOD+/fvyx7Gxsbh69SpsbW3h6uqKadOm4enTp9iwYQMAYNSoUVi+fDkmT56MYcOG4dixY/jrr7+wf/9+Qfst04k6NzcX0dHRmDZtmrxMJBLB19cXZ8+e5V0nJydHYWD89PR0AKouk9ClcEIIKR3KLaw12xwrWDRZX6BLly7h008/lT8ODg4GAPj7+yMiIgLPnz9HXFyc/HkPDw/s378fEyZMwLJly1C1alX89ttv8PPzE7TfMp2oX758CalUyttq7s6dO7zrhIaGYvbs2aURHiGEkAqkbdu2Rc5VzjfqWNu2bXHlyhWN9lvuRiabNm0aUlNT5Ut8fLyuQyKEEKJtMvb28nexFv0Z66tMn1Hb2dlBLBbztpornIP1fdrsrE4IIaSM0oPGZNpSphO1kZERvLy8EBkZKR+mTSaTITIyEgEBAYK2VdBwTPH+SF7+UaV6hga+SmWEEEKIrpTpRA0U3Kz39/dH06ZN0bx5c4SFhSEzM1PeCpwQQkgFRGfUZUe/fv3w4sULzJw5EwkJCWjUqBEOHjyo1MCMEEJIBUKJumwJCAgQfKmbEEJIOcY0TNRMfxJ1uWv1TQghhJQnenFGTQghhLyLYzJwGpwVa7JuaaNETQghRP9UoHvUdOmbEEIIKcPojJoQQoj+kTHNRhejkckIIYSQElSBLn1ToiaEEKJ/KFGXRwbg3htClG+40PzMffxrm3UtkagIIYSQolSgRE0IIaTcYEyzQUs0mcu6lFGiJoQQon8q0KVv6p5FCCGElGF0Rk0IIUT/UPes8igf7L3GZCKRuVItA7NuvGtL769WKpPUnsy/J2mK8PAIIYSojy59E0IIIaQsqEBn1IQQQsqNCjTNJSVqQggheoeTycBpkKg1Wbe0UaImhBCifxjTrC+0HvWjpnvUhBBCSBlWoc+oPazaK5U9TDnEW1dc4xulMkvT2rx1X83srlRmOHWDwOgIIYSoVIFafVfoRE0IIURPVaBETZe+CSGEkDKMzqgJIYToHxqZjBBCCCnDKtCl7wqdqB8k79Zo/bSsGP4nJq1QLqPGZIQQoj0ypmGi1p8zarpHTQghhJRhFfqMmhBCiJ6qQAOeUKImhBCifyrQPWq69E0IIYSUYXRGTQghRP8wDbtn0aVv/ZWXe5i33NCoI08px1/XwFeLERFCCFFSgS59U6ImhBCifypQoqZ71IQQQkgZRmfUhBBC9A8NIUoIIYSUYUxWsGiyvp6o0IlaGr1YqcxA0pW3rlhsqVTGWD5vXZksi6eUv+EZoPyrjuOM+GuyXBXbIIQQUl5V6ERNCCFET9Glb0IIIaQMq0CtvilRE0II0T8V6IyaumcRQgghZRidURNCCNE/FWg+6gqdqJmDvXKZipbVYpG1UlluXpKQvalfU2Xrbr6W4/rzYSOEEK2hS9+EEEIIKQsq9Bk1IYQQfaXhgCegVt+EEEJIyaFL36Xj5MmT6NatGypXrgyO47Br1y6F5xljmDlzJpydnWFiYgJfX1/cu3dPN8ESQggpOwoTtSaLntDpGXVmZiYaNmyIYcOGoXfv3krPL1y4ED///DPWr18PDw8PzJgxA35+frh16xaMjY013r+By2ClsmZWI3nrXkz9VanMyqwub93UzDtqx3C5bYBSWZOoX3jrisUWSmWOFo146z5LOal2DIQQQsounSbqzp07o3PnzrzPMcYQFhaG6dOno0ePHgCADRs2wNHREbt27cKXX35ZmqESQggpSyrQyGRlttV3bGwsEhIS4OvrKy+zsrKCt7c3zp49q3K9nJwcpKWlKSyEEELKmQp06bvMJuqEhAQAgKOjo0K5o6Oj/Dk+oaGhsLKyki8uLi4lGichhBBSkspsoi6uadOmITU1Vb7Ex8frOiRCCCHaVoHOqMts9ywnJycAQGJiIpydneXliYmJaNSokcr1JBIJJBJJSYdHCCFElyrQPeoym6g9PDzg5OSEyMhIeWJOS0vD+fPnMXr0aMHbszCpCY4TK5SlZcUo1eNr3a1KWtYD3vIq1m2Vyp6lnOKtezPFUqmsuk0P3roPkvcplWXkJvLWNTeprlz3DX+8hBCidxgrWDRZX0/oNFFnZGTg/v378sexsbG4evUqbG1t4erqiqCgIPzwww+oWbOmvHtW5cqV0bNnT90FTQghhJQinSbqS5cu4dNPP5U/Dg4OBgD4+/sjIiICkydPRmZmJr755hukpKSgVatWOHjwoFb6UBNCCNFjFWhkMp0m6rZt24IVcfmB4zjMmTMHc+bMKcWoCCGElHkVKFGXu1bfhBBCSElZsWIF3N3dYWxsDG9vb1y4cKHI+mFhYahduzZMTEzg4uKCCRMmIDs7W9A+y2xjMm1Lf3Mf/PM5q8fNppNS2ePkg7x1n6WeVirztB7AW3fQ1R+Vyq5+qjysKAA0Oi5VKpOxfN66WTnPecsJIaRcYBq2+i7GzFtbtmxBcHAwwsPD4e3tjbCwMPj5+SEmJgYODg5K9Tdt2oSpU6di7dq1aNGiBe7evYshQ4aA4zgsWbJE7f3SGTUhhBD9o4N+1EuWLMGIESMwdOhQ1KtXD+Hh4TA1NcXatWt56585cwYtW7ZE//794e7ujo4dO+Krr7764Fn4+yhRE0IIqbDeH3I6JyeHt15ubi6io6MVhrUWiUTw9fVVOax1ixYtEB0dLU/MDx8+xIEDB/DZZ58JirHCXPomhBBSjsigYWOygv+9P8x0SEgIZs2apVT95cuXkEqlvMNa37nDP2Ni//798fLlS7Rq1QqMMeTn52PUqFH47rvvBIUqKFHLZDKcOHECp06dwuPHj5GVlQV7e3s0btwYvr6+NK42IYSQ0qGlVt/x8fGwtHw78JQ2R7aMiorCvHnzsHLlSnh7e+P+/fsYP3485s6dixkzZqi9HbUufb958wY//PADXFxc8Nlnn+Hvv/9GSkoKxGIx7t+/j5CQEHh4eOCzzz7DuXPnin1QhBBCiDqYjGm8AIClpaXCoipR29nZQSwWIzFRcTTIxMRE+ZDX75sxYwYGDRqEr7/+Gg0aNECvXr0wb948hIaGQiagIZxaZ9S1atWCj48P1qxZgw4dOsDQ0FCpzuPHj7Fp0yZ8+eWX+P777zFixAi1gygdmvWZW1+3nlJZ2zP8rb5HVp6iVBb+dK7a+2pygr9hAh8hw4LyDSsKADKZcsvxrJzHam+XEELKOyMjI3h5eSEyMlI+OqZMJkNkZCQCAvh76mRlZUEkUjwfFosLhrIuagyR96mVqA8fPoy6desWWcfNzQ3Tpk3DpEmTEBcXp3YAhBBCiGA6GOs7ODgY/v7+aNq0KZo3b46wsDBkZmZi6NChAIDBgwejSpUqCA0NBQB069YNS5YsQePGjeWXvmfMmIFu3brJE7Y61ErUH0rS7zI0NET16vxnboQQQohW6GBksn79+uHFixeYOXMmEhIS0KhRIxw8eFDewCwuLk7hDHr69OngOA7Tp0/H06dPYW9vj27duuHHH5XHzyhKsVp9Z2dn49q1a0hKSlK6zt69e/fibJIQQggp8wICAlRe6o6KilJ4bGBggJCQEISEhGi0T8GJ+uDBgxg8eDBevnyp9BzHcZBKlUfPIoQQQrSqAo31LThRjxs3Dl988QVmzpyp1J+sLPumymQYiRRn3Voer34Dr7ZnliqVcRz/LF4HMs+rvV0zYw+lsszsWN66Bga2SmUijv8tdDZvolSWmHmNt66FcRWlMhnL462bnfuMt5wQQkoVJWrVEhMTERwcrFdJmhBCSDlTgRK14CFEP//8c6Xr8IQQQggpGYLPqJcvX44vvvgCp06dQoMGDZT6VAcGBmotOEIIIYQPY28HLSnu+vpCcKL+3//+h8OHD8PY2BhRUVHguLdTR3IcR4maEEJIyatAl74FJ+rvv/8es2fPxtSpU5VGXCGEEEKIdglO1Lm5uejXr5/eJenVTxcC4BTKfKzGKtU7m7pC7W0yls1b/jzjstrb4G/hzfGUAfn5r9Xe7uNk5eFNRSJz3rrW4o+Vyl7kXuSte7ntOKWyJlG/qB0XIYRoRQU6oxacbf39/bFly5aSiIUQQghRT2Gi1mTRE4LPqKVSKRYuXIhDhw7B09NTqTHZkiVLtBYcIYQQwksHY33riuBEff36dTRu3BgAcOPGDYXn3m1YRgghhBDNCU7Ux48fL4k4CCGEELUxWcGiyfr6oliTchR68uQJAKBq1apaCaa08TUcUzUsKH/DMf5pyvLylcdBVyX/5ValMgO7L9ReXwiZLIO3/F7yDrW3QQ3HCCFlAjUmU00mk2HOnDmwsrKCm5sb3NzcYG1tjblz5yrNpEUIIYQQzRSrH/Xvv/+O+fPno2XLlgCAf/75B7NmzUJ2drbgeTYJIYQQwSrQGbXgRL1+/Xr89ttvCvNOe3p6okqVKhgzZgwlakIIISWO7lEX4fXr16hTp45SeZ06dfD6tfoDchBCCCHFxjQ8o9aj7lmC71E3bNgQy5cvVypfvnw5GjZsqJWgCCGEEFJA8Bn1woUL0aVLFxw9ehQ+Pj4AgLNnzyI+Ph4HDhzQeoDaY6TUz5uxXKVaqoYF5WNgYMVb7mD+kVLZs5RT/NvgaeHNN7QpwN9KfWm9Gbx1g2+FKpUx5PPW5dPUegRv+aWUNWpvQ9VQqID+/JIlhJRRsv8WTdbXE4LPqNu0aYO7d++iV69eSElJQUpKCnr37o2YmBi0bt26JGIkhBBCFDAZ03jRF4LOqPPy8tCpUyeEh4dTozFCCCGkFAhK1IaGhrh27VpJxUIIIYSohy59qzZw4ED8/vvvJRELIYQQoh6mhUVPCG5Mlp+fj7Vr1+Lo0aPw8vKCmZmZwvM0exYhhJCSpul95nJ7jxoomDGrSZMmAIC7d+8qPFeWZ8+qatUKIk5xSs64lMNK9TgVLwlfi2kTo0q8dZ+lnCxGhG/xte5WZcKtuRrtS5UhVZx4yy+lKJdN9pjJW3dh7BwtRkQIIRWTWon62rVr+OijjyASiWj2LEIIIbpH96gVNW7cGC9fFswIVa1aNbx69apEgyKEEEKKUjiEqCaLvlArUVtbWyM2NhYA8OjRI5olixBCCCklal367tOnD9q0aQNnZ2dwHIemTZtCLOafi/nhw4daDZAQQghRUoEufauVqFevXo3evXvj/v37CAwMxIgRI2BhYVHSsWnVk9RjeH9Iy++qKTeCylAxyubPccoNo9Kz7gmIgL+hnanEVaksK+cx/xY4Y7X3JmQoVED5R1fATf5GasZGlZXKVDUaM5W48Za/yU1UjkDEf2z50hTeckJIxUazZ/Ho1KkTACA6Ohrjx4/Xu0RNCCGkHGHQ7KxYf3pnCe+etW7dupKIgxBCCCE8BCdqQgghRNcY02xKaT2ajlr4EKLaFBoaimbNmsHCwgIODg7o2bMnYmJiFOpkZ2dj7NixqFSpEszNzdGnTx8kJirf4ySEEFJxUPesUnLixAmMHTsW586dw5EjR5CXl4eOHTsiMzNTXmfChAnYu3cvtm7dihMnTuDZs2fo3bu3DqMmhBBCSg/HWNm5APDixQs4ODjgxIkT+OSTT5Camgp7e3ts2rQJn3/+OQDgzp07qFu3Ls6ePYuPP/74g9tMS0uDlZUVClo2v9/yWrNDVzXcaO78/kplhlM38NY1MLBVKrM0UW4JDgCv06/yRsFP+djMTarz1sx480DFNkoKX8xl5mNICNG6gpZfqampsLS01GhLhd/pz4b2haWRUfG3k5uLyuv+0kpMJU3wPeo9e/bwlnMcB2NjY9SoUQMeHh7FCiY1NRUAYGtbkLyio6ORl5cHX19feZ06derA1dVVZaLOyclBTk6O/HFaWlqxYiGEEFJ2UfesIvTs2RMcx+H9E/HCMo7j0KpVK+zatQs2NjZqb1cmkyEoKAgtW7bERx99BABISEiAkZERrK2tFeo6OjoiISGBdzuhoaGYPXu2sIMihBBCyijB96iPHDmCZs2a4ciRI0hNTUVqaiqOHDkCb29v7Nu3DydPnsSrV68wadIkQdsdO3Ysbty4gc2bNwsNScG0adPkcaWmpiI+Pl6j7RFCCCl7Clt9a7LoC8Fn1OPHj8fq1avRokULeVn79u1hbGyMb775Bjdv3kRYWBiGDRum9jYDAgLkSb5q1arycicnJ+Tm5iIlJUXhrDoxMRFOTvzTMEokEkgkEqGHRQghRJ/IuIJFk/X1hOBE/eDBA94b75aWlvJxvmvWrCmfbasojDGMGzcOO3fuRFRUlNK9bS8vLxgaGiIyMhJ9+vQBAMTExCAuLg4+Pj4CI+f7+cQ3XrlUxfrKbyrfHNUAYDj1D7Wjys9/rVT2Ol25TDX1fxb6mnzBW/5VXeVj63c5VEAMqvCPB1/f5kulspvJG3nrmhkrt3fIzH6kYn969BOZEKKRinSPWvClby8vL3z77bd48eKFvOzFixeYPHkymjVrBgC4d+8eXFxcPritsWPH4s8//8SmTZtgYWGBhIQEJCQk4M2bNwAAKysrDB8+HMHBwTh+/Diio6MxdOhQ+Pj4qNXimxBCCNF3gs+of//9d/To0QNVq1aVJ+P4+HhUq1YNu3fvBgBkZGRg+vTpH9zWqlWrAABt27ZVKF+3bh2GDBkCAFi6dClEIhH69OmDnJwc+Pn5YeXKlULDJoQQUo4wxoGx4l++1mTd0iY4UdeuXRu3bt3C4cOHcffuXXlZhw4dIBIVnKD37NlTrW2p04Xb2NgYK1aswIoVK4SGSgghpJyqSJe+izXWt0gkQqdOneQzahFCCCGkZBQrUUdGRiIyMhJJSUmQyRR/lqxdu1YrgRFCCCGqMKbhGbUetT0VnKhnz56NOXPmoGnTpnB2dgbH6ct1fg7KLbeVW3irGhaUv4U3f6tm/pbj6g/1WVJ2vZ6volz5OAzE1rx186UpSmU5M5SHTAUAydxNvOW3UrbzB8hDKstRKjM0qMRb10BsrFT2JueJ2vsihOgPukddhPDwcERERGDQoEElEQ8hhBDyYTIOrIL0oxbcPSs3N1dhsBNCCCGElBzBifrrr7/Gpk38lzQJIYSQ0kBDiBYhOzsbq1evxtGjR+Hp6QlDQ0OF55csWaK14AghhBA+dI+6CNeuXUOjRo0AADdu3FB4rmw3LFP++SQWKw+FKpUKmRZT1XCjysZW5R8AZsWTuTylJdXwjL/xG8cpl3exHMVbN1um3KhOMnexiv3xHwdj2Srq8+wv97lSmYHYireuiDPkLSeEEH0mOFEfP368JOIghBBC1MY0bEymUUO0UlasftSEEEKILml6n7nc3aPu3bs3IiIiYGlpid69exdZd8eOHVoJjBBCCFGF7lG/x8rKSn7/2cqK//4gIYQQQrRPrUS9bt063n8TQgghuiCTcZBpcJ9Zk3VLW4W+Ry2shTffm6r+TY4VT34QsK+SunnC30qdMeXyS+w8b92nqeo3JuQ45SE9C/anfqtv2V9BSmXivr/w1s3kGd6UEFI+0T3q9zRu3FjtrleXL1/WKCBCCCGEvKVWon53funs7GysXLkS9erVg4+PDwDg3LlzuHnzJsaMGVMiQRJCCCHvosZk7wkJCZH/++uvv0ZgYCDmzp2rVCc+Pl670RFCCCE8KlKiFjzW99atWzF48GCl8oEDB2L7dvWnLySEEEKKS8Y4jZfiWLFiBdzd3WFsbAxvb29cuHChyPopKSkYO3YsnJ2dIZFIUKtWLRw4cEDQPgU3JjMxMcHp06dRs2ZNhfLTp0/D2Ji/8VBZ0MEqCIacRKHsQMpCAVvQrOWBqmEv+eZ3LguepqjfaOzNxK94y00W/0/jOER9l2q8DX6aNQ4khFQ8W7ZsQXBwMMLDw+Ht7Y2wsDD4+fkhJiYGDg4OSvVzc3PRoUMHODg4YNu2bahSpQoeP34Ma2trQfsVnKiDgoIwevRoXL58Gc2bNwcAnD9/HmvXrsWMGTOEbo4QQggRTBdDiC5ZsgQjRozA0KFDAQDh4eHYv38/1q5di6lTpyrVX7t2LV6/fo0zZ87IJ7Byd3cXvF/BiXrq1KmoVq0ali1bhj///BMAULduXaxbtw59+/YVHAAhhBAilLa6Z6WlKXbTlUgkkEgkSvVzc3MRHR2NadOmyctEIhF8fX1x9uxZ3n3s2bMHPj4+GDt2LHbv3g17e3v0798fU6ZMgVjMP0kSn2L1o+7bty8lZUIIIXrPxcVF4XFISAhmzZqlVO/ly5eQSqVwdHRUKHd0dMSdO3d4t/3w4UMcO3YMAwYMwIEDB3D//n2MGTMGeXl5Co20P6TYA57k5uYiKSkJMplModzV1bW4mySEEELUIkPxG4QVrg8A8fHxsLR8O+Ux39l0sfchk8HBwQGrV6+GWCyGl5cXnj59ikWLFpVsor537x6GDRuGM2fOKJQzxsBxHKRS9edoJoQQQopDW92zLC0tFRK1KnZ2dhCLxUhMTFQoT0xMhJOTE+86zs7OMDQ0VLjMXbduXSQkJCA3NxdGRkZqxSo4UQ8ZMgQGBgbYt28fnJ2d1R6xTNeOpIaBv6WvutRvJexk1UqpLCH1H966phI3pbLsvBe8dWWyLNXhqcHIULlVIgDk8u5P/Zs/qlp3GxjY8pZLpcrHwVieiq1r9sPv9YABvOW2GzdqtF1CiG4xDbpYFa4vhJGREby8vBAZGSkfBEwmkyEyMhIBAQG867Rs2RKbNm2CTCaDSFTQG/ru3btwdnZWO0kDxUjUV69eRXR0NOrUqSN0VUIIIURvBQcHw9/fH02bNkXz5s0RFhaGzMxMeSvwwYMHo0qVKggNDQUAjB49GsuXL8f48eMxbtw43Lt3D/PmzUNgYKCg/QpO1PXq1cPLly+FrkYIIYRojS5GJuvXrx9evHiBmTNnIiEhAY0aNcLBgwflDczi4uLkZ85AQUO1Q4cOYcKECfD09ESVKlUwfvx4TJkyRdB+BSfqBQsWYPLkyZg3bx4aNGgg7xtWSJ1r/YQQQogmZP8tmqxfHAEBASovdUdFRSmV+fj44Ny5c8XcWwHBidrX1xcA0L59e4VyakxGCCGEaJ/gRH38uPpDS+qj/J+H8ZYbBK5VexuqGo7xsTR2USrLynms9vpC5OYlaWEr6jeqk8n4550WMh+1ply2n/lwJUKI3qlIk3IITtRt2rQpiTgIIYQQtckYNOtHrUdD+wuePQsATp06hYEDB6JFixZ4+vQpAOCPP/7AP/+ofyZJCCGEkA8TnKi3b98OPz8/mJiY4PLly8jJyQEApKamYt68eVoPkBBCCHlf4aVvTRZ9IThR//DDDwgPD8eaNWsUWny3bNkSly9f1mpwhBBCCJ+CS9+aLfpC8D3qmJgYfPLJJ0rlVlZWSElJ0UZMhBBCSJGoMVkRnJyccP/+faU5Nf/55x9Uq1ZNW3HpjKrW3RJD5bFcc/ISNN6fkBbipam37TTe8h2vQ9XehkyWo3EcO5oqz/Ha+9IiFbWVuwZmZsdqHIOQlu5NrUcolV1KWaOFGAghFZXgS98jRozA+PHjcf78eXAch2fPnmHjxo2YNGkSRo8eXRIxEkIIIQpk4DRe9IXgM+qpU6dCJpOhffv2yMrKwieffAKJRIJJkyZh3LhxJREjIYQQooCxgkWT9fWF4ETNcRy+//57fPvtt7h//z4yMjJQr149mJubl0R8hBBCSIUm+NL3sGHDkJ6eDiMjI9SrVw/NmzeHubk5MjMzMWwY/6hehBBCiDbJ/pvmUpNFX3CMCbsAIBaL8fz5czg4KM5t/PLlSzg5OSE/P1+rAWoqLS0NVlZWMDFyB8cp/i55k/NUqT6D+vGLRKa85XzzRttaNOKt+zr9qlLZFI+ZvHUXxM5VKhvs+D1v3T8S5yuVqT425Q+sSGTGW1Mmy1CxDc1wKi7uMJ5GWwZiC966+dJU3i2UDFV/5Mr74zj+eWcZy9ViPISUZQyADKmpqRpP3FT4nR7ZIgBmBpJibyczPwftzyzXSkwlTe1L32lpaWCMgTGG9PR0GBsby5+TSqU4cOCAUvImhBBCSgLdo+ZhbW0NjuPAcRxq1aql9DzHcZg9e7ZWgyOEEEIqOrUT9fHjx8EYQ7t27bB9+3bY2trKnzMyMoKbmxsqV65cIkESQggh79L0PrM+3aNWO1EXzpoVGxsLV1dXcJzyQcbFxcHV1VXtna9atQqrVq3Co0ePAAD169fHzJkz0blzZwBAdnY2Jk6ciM2bNyMnJwd+fn5YuXIlHB0d1d4HIYSQ8oeBA9OgL7Qm65Y2wa2+q1WrhhcvXiiVv3r1Ch4eHoK2VbVqVcyfPx/R0dG4dOkS2rVrhx49euDmzZsAgAkTJmDv3r3YunUrTpw4gWfPnqF3795CQyaEEEL0luBW3yKRCAkJCUoNxx4/fox69eohMzNTo4BsbW2xaNEifP7557C3t8emTZvw+eefAwDu3LmDunXr4uzZs/j444/V2l5hC0FX604QcYYKzz1KPqBU30BszbudfGmKUpmVWV3euqmZt9WKDeBv7Syk5bmZMf+Po8zsRzylJdN6QlUMWdnxvOV/eX2rVPZFNP/QpJq+PsLw/8LmawEvpPW7SMQ/xoBMxve3okctXAhRm/Zbff/tHahxq+/O538uX62+g4ODARQ0Gps5cyZMTd92TZJKpTh//jwaNWpU7ECkUim2bt2KzMxM+Pj4IDo6Gnl5efD19ZXXqVOnDlxdXQUlakIIIeUP3aPmceXKFQAAYwzXr1+HkdHbvqFGRkZo2LAhJk2aJDiA69evw8fHB9nZ2TA3N8fOnTtRr149XL16FUZGRrC2tlao7+joiIQE1ZNh5OTkyOfIBgp+fRFCCClfKtI9akGtvgFg6NChWLZsmdYuFdSuXRtXr15Famoqtm3bBn9/f5w4caLY2wsNDaVuYoQQQsoNwY3J1q1bp9Xr+UZGRqhRowa8vLwQGhqKhg0bYtmyZXByckJubq7SHNeJiYlwclKecrLQtGnTkJqaKl/i4/nvkxJCCNFfMqb5oi8ET8oBAJcuXcJff/2FuLg45OYqDoO4Y8cOjQKSyWTIycmBl5cXDA0NERkZiT59+gAAYmJiEBcXBx8fH5XrSyQSSCTKDQyuPAyApaVioyBDA+XGZHyNxlQR0mhMFU0bRmlnvmXNCI1BVcMxPiXXcIx/b3w0HTa1pIZdJaQio0vfRdi8eTMGDx4MPz8/HD58GB07dsTdu3eRmJiIXr16CdrWtGnT0LlzZ7i6uiI9PR2bNm1CVFQUDh06BCsrKwwfPhzBwcGwtbWFpaUlxo0bBx8fH2pIRgghpMIQnKjnzZuHpUuXYuzYsbCwsMCyZcvg4eGBkSNHwtnZWdC2kpKSMHjwYDx//hxWVlbw9PTEoUOH0KFDBwDA0qVLIRKJ0KdPH4UBTwghhFRsml6+1qdL34L7UZuZmeHmzZtwd3dHpUqVEBUVhQYNGuD27dto164dnj9/XlKxFkthn7tXr/fxXPr2VbEWIYQQ7dF+P+rtXsEa96PuE71EL/pRC25MZmNjg/T0dABAlSpVcOPGDQBASkoKsrKUp3ckhBBCtI1pYdEXgi99f/LJJzhy5AgaNGiAL774AuPHj8exY8dw5MgRtG/fviRiJIQQQioswYl6+fLlyM7OBgB8//33MDQ0xJkzZ9CnTx9Mnz5d6wFqSyXb7nh/iEjp3V+V6olrjSyliPRTyQ3pqaoFpvLvXr4YtBeHeoQMNUsI0T4GzUYmK9etvt+d3lIkEmHq1KlaDYgQQgj5ENl/iybr6wu17lELnWhD04k5SlMul48fHf/CfaOy1QiOEEIIAdRM1DVq1MD8+fOLbNHNGMORI0fQuXNn/Pzzz1oLsKRlc7l4YJSAoCq/UbImhBA9wRin8aIv1Lr0HRUVhe+++w6zZs1Cw4YN0bRpU1SuXBnGxsZITk7GrVu3cPbsWRgYGGDatGkYOVJ/7vNaykwR9vRrBFX5DUFVfoNpAzGyrkt1HRYhhJAiVKRL32ol6tq1a2P79u2Ii4vD1q1bcerUKZw5cwZv3ryBnZ0dGjdujDVr1qBz584Qi8UlHbPWvZus60Wm49+Gach7rk+N9wkhpGKhAU/KkcLO8QVX+Yu+1GFgw2H9yyB8wT4D915dGhyFEEKKS/sDnvzZeBJMxcUf8CRLmoOBV34qnwOelGf5yQx9WRdw4HAS5/EAj3UdEiGEEB6Fk3JosugLStQ8pJDiT9FOBIlmU7ImhJAyqCJNc0mJmocYYvwk+x4OsKNkTQghRKcoUatgCQsslc2QJ2uT+vRSEUJIWVGRLn0LHpmsvHu/0ZiBDQePcFM8uPwn7GGrWFfSScVW+D4AenSdhRBCyriK1Oq7WIk6JSUFFy5cQFJSEmQyxd5ogwcP1kpgZUV+MsO9fpmwz7FFCtKQzKXCg7noOixCCKnQKFEXYe/evRgwYAAyMjJgaWkJjnt79shxXLlL1O9aLv4D50X/Iiz/e1RnbroOhxBCSAUg+MbrxIkTMWzYMGRkZCAlJQXJycny5fXr1yURY5kRKPWHA6uEIIMf8YCjBmaEEKIrFeketeBE/fTpUwQGBsLU1LQk4inTLGGOpfnfy5O1aQP9G4WNEELKA6Zh1yx9GupLcKL28/PDpUuXSiIWvVCYrJ2YHUw9KVETQggpWYLvUXfp0gXffvstbt26hQYNGsDQ0FDh+e7du2stOF0wMLDlLVdq4S0G8p/+D1gMJHPpsGEWb7fh9FVJhkgIIRUeTcpRhBEjRgAA5syZo/Qcx3GQSivIzFP/HeYek3+w2nwPliWPR/X8KrqNiRBCKghNp6rUp2kuBV/6lslkKpcKk6Tf0Ta7MRykNhhvswwPDJ7qOhxCCCHlDA23pSFLZoaw5EBK1oQQUopkWlj0RbES9YkTJ9CtWzfUqFEDNWrUQPfu3XHq1Cltx6Y33k3WG8z+1nU4hBBS7lWkSTkE36P+888/MXToUPTu3RuBgYEAgNOnT6N9+/aIiIhA//79tR5kacrPV78v+PuNxgxsOMiyGfLyj0IGGUTv/A6i+awJIUR7GDQbmFmP8rTwRP3jjz9i4cKFmDBhgrwsMDAQS5Yswdy5c/U+UWsiP7ngrX+EJ5gpWowQWRCqg0YwI4QQUnyCL30/fPgQ3bp1Uyrv3r07YmNjtRKUvrOFFQxhSFNkEkJICSm4fM1psOj6CNQnOFG7uLggMjJSqfzo0aNwcaHJKgDlKTIpWRNCiHYxLSz6QvCl74kTJyIwMBBXr15FixYtABTco46IiMCyZcu0HqC+KkzWE0RzMU20AJwhwPJ0HRUhhBB9IzhRjx49Gk5OTli8eDH++usvAEDdunWxZcsW9OjRQ+sB6rPCZP0ECdiZN1rX4RBCSLlB01x+QK9evdCrVy9tx1Ju8LXwzs7cjWWGf6BnfntUf2c+awOzrqUZGpHjG6e94g3YQ4i+qkhDiNKAJ6UkGzm4Kb6P8caheMDF6zocQgjRa4xpvugLtRK1ra0tXr58CQCwsbGBra2tyoXwM4cpwrKnwoHZUrImhBCiNrUufS9duhQWFhbyf3Oc/gxmXpZYwhxh2VMRZDwf441D8Xu28sQmhBBCPoyBgwwaTMqhwbqlTa1E7e/vL//3kCFDSiqWCqEwWe8xOA57RlcgCCGkODS9fF3cdVesWIFFixYhISEBDRs2xC+//ILmzZt/cL3Nmzfjq6++Qo8ePbBr1y5B+xTcmEwsFuP58+dwcHBQKH/16hUcHBwq5Axa6uBrNDYEv+JU/iLYw1ZhBDMabrQ00OeUECLMli1bEBwcjPDwcHh7eyMsLAx+fn6IiYlRyonvevToESZNmoTWrVsXa7+CG5MxFT9DcnJyYGRkVKwgKiwOWC/aSoOiEEKIQLqYPWvJkiUYMWIEhg4dinr16iE8PBympqZYu3atynWkUikGDBiA2bNno1q1asXYq4Az6p9//hkAwHEcfvvtN5ibmysEcvLkSdSpU6dYQVRYDFggm4YJorkIEs1GmCyExgYnhBA1aKsfdVpamkK5RCKBRCJRqp+bm4vo6GhMmzZNXiYSieDr64uzZ8+q3M+cOXPg4OCA4cOHF3uWSbUT9dKlSwEUnFGHh4dDLH7bD9XIyAju7u4IDw8vVhAV2bsjmBUma0IIIUXT1uxZ7w99HRISglmzZinVf/nyJaRSKRwdHRXKHR0dcefOHd59/PPPP/j9999x9epVDSIVkKgLJ9z49NNPsWPHDtjY2Gi0Y/JWYbKeL1oFExjrOhxCCKkw4uPjYWlpKX/MdzZdHOnp6Rg0aBDWrFkDOzs7jbYluDHZ8ePHNdoh4WcJC8yTTQYAiK04GDlzeHNHn8bOIYSQ0qOtS9+WlpYKiVoVOzs7iMViJCYmKpQnJibCyclJqf6DBw/w6NEjhdkmZbKC73QDAwPExMSgevXqasWqVqIODg7G3LlzYWZmhuDg4CLrLlmyRK0dkwJ8Lbx/yOmOfwyuYFnmt6guqyovN7DuXZqhaYzj+K8OMJZdypEQQsqb0u6eZWRkBC8vL0RGRqJnz54AChJvZGQkAgIClOrXqVMH169fVyibPn060tPTsWzZMkGzTaqVqK9cuYK8vDz5v1WhgVC0Y0z257hr9hjjzRYhLHMSasho+lBCCNG14OBg+Pv7o2nTpmjevDnCwsKQmZmJoUOHAgAGDx6MKlWqIDQ0FMbGxvjoo48U1re2tgYApfIPUStRv3u5my59lzxLZo6wzIkIMluMILOflM6sCSGkotPFpBz9+vXDixcvMHPmTCQkJKBRo0Y4ePCgvIFZXFwcRCLtT6HBMVUdo9WUlpaGY8eOoU6dOmWye1ZaWhqsrKxQ0GVcP87481N2AADSuAxMMFuCHjlt0D2vDV36JoToKQZAhtTUVLXuBxel8Dt9evVpMBYXv/FttjQbPzwI1UpMJU1w6u/bty+WL18OAHjz5g2aNm2Kvn37okGDBti+fXuxA5k/fz44jkNQUJC8LDs7G2PHjkWlSpVgbm6OPn36KN3IL88smTnCM75D97w2AAADW/34oUEIISWNaWHRF4JbfZ88eRLff/89AGDnzp1gjCElJQXr16/HDz/8gD59+ggO4uLFi/j111/h6empUD5hwgTs378fW7duhZWVFQICAtC7d2+cPn1a8D74SAyVW+oNdRjBWzf86Q8CtizkI6CcfPnOnO0GGqHRHUvcapeBrBvqXLTR7GMoEpnylstkWTyl/D8gVJ05822bf7vCTPGYqVS2IFbVxCfKMXMcf7cMxnJ41uabz5ofUzFcqZGhvVJZbl6Siq2o+pHG9z7z1+U4Q+W1Wa6K7SoTicx5y2WyTLW3oV9fj4SUDYLPqFNTU+XTWR48eBB9+vSBqakpunTpgnv37gkOICMjAwMGDMCaNWsU+manpqbi999/x5IlS9CuXTt4eXlh3bp1OHPmDM6dOyd4P/ouZX8ecuJlqHfMHKYf0TTihJCKrbB7liaLvhD8je/i4oKzZ88iMzMTBw8eRMeOHQEAycnJMDYWfr9g7Nix6NKlC3x9FbspRUdHIy8vT6G8Tp06cHV1LXK4tpycHKSlpSks5UF+MsNt30xK1oQQgoJpKjVd9IXgb/ugoCAMGDAAVatWReXKldG2bVsABZfEGzRoIGhbmzdvxuXLlxEaGqr0XEJCAoyMjOTN2Qs5OjoiISFB5TZDQ0NhZWUlX4T0VSvr3k3WLj+a6DocQgghpUDwPeoxY8agefPmiI+PR4cOHeRN0atVq4YfflD/Pm58fDzGjx+PI0eOFOtMXJVp06YpDMqSlpZWLpM1y9ej6zaEEKJlDJpdvtanb1DBiRoAmjZtiqZNm4IxBsYYOI5Dly5dBG0jOjoaSUlJaNKkibyscBau5cuX49ChQ8jNzUVKSorCWbWq4doKqZr5pDzJTy74iEmqiVBrqyke+Gep2cCMEELKB20NIaoPipWoN2zYgEWLFskbj9WqVQvffvstBg0apPY22rdvrzS82tChQ1GnTh1MmTIFLi4uMDQ0RGRkpLwleUxMDOLi4uDj41OcsJXk5ClfQg9/Olcr21afkE+LYl1psgyNG9dAlWuvEJY/E9XhKn/O0KijRlEJa4Ut7BOvjRbefFS38OajHLOQ/t0M+QL2xU91C2/+PWpaV0gLbz4yWYZG6xOiTdqaPUsfCE7US5YswYwZMxAQEICWLVsCKJjKa9SoUXj58iUmTJig1nYsLCyUhlEzMzNDpUqV5OXDhw9HcHAwbG1tYWlpiXHjxsHHxwcff/yx0LDLpfxkhqX5MzDBYC6CDOYgLH8GzWdNCCHljOBE/csvv2DVqlUYPHiwvKx79+6oX78+Zs2apXaiVsfSpUshEonQp08f5OTkwM/PDytXrtTa9ssDS5jLk/UUgwXYlL8MRlDuL0sIIeUJXfouwvPnz9GiRQul8hYtWuD58+caBRMVFaXw2NjYGCtWrMCKFSs02m55V5isH3JxlKQJIRUC++8/TdbXF4K7Z9WoUQN//fWXUvmWLVtQs2ZNrQRFhLOEORqxepBBhuWiDTBtoP7IWYQQQsouwWfUs2fPRr9+/XDy5En5PerTp08jMjKSN4FXRIYGdkplefkvBWxB/Y74hkadFR6LLYB6UWaoF2mO2+3fIOv629bgqhpA8Q0NKZO9UTsGqBgi09ioMm95dq76V15EIuX+4jKZ8pCe/z2jVCJkWFAzY3feutl5r5TKpFL+gXTcbT5TKnuccoy3rsTQVqnMQMzfPz7jzQPecr6JT4RNesL/WeN73ZlM/cZoqoZN5dtuW4uRvHWPpS5Ve3+k4qlIl74Fn1H36dMH58+fh52dHXbt2oVdu3bBzs4OFy5cQK9evUoiRiKANB247ZuJ3HiGupEmMG1AI5gRQsofmpTjA7y8vPDnn39qOxaiJfnJwC3fLNQ7aoq6kSa40TwLOY/06WNJCCFFq0hn1MVK1FKpFDt37sTt27cBAPXq1UOPHj1gYFCszZESIP0vWTsHGiEnTo8+kYQQQhQIzqw3b95E9+7dkZCQgNq1awMAFixYAHt7e+zdu1epbzTRHWky8GR2wX1FyzZi5L2S0QhmhJBygbGCRZP19YXgG5hff/016tevjydPnuDy5cu4fPky4uPj4enpiW+++aYkYiRaUHWuEeods6BZtwgh5YJMC4u+4BgT9rvCxMQEly5dQv369RXKb9y4gWbNmuHNGyGthUteWloarKysUPCbRH+mNdM2AxsO/ZMaIEmUjGVpwagurfr2ObsvBGyJ7zXUo5+mROs0b3lOyj8GQIbU1FRYWlpqtKXC7/TRLtMgERV/QqccWTZWxYdqJaaSJvj0qlatWkhMTFQqT0pKQo0aNbQSFNG+/GSGsLRgOMhsMN5yCR6In+g6JEIIKbbCxmSaLPpCcKIODQ1FYGAgtm3bhidPnuDJkyfYtm0bgoKCsGDBAqSlpckXUrZYMjOEpQWjbr47DBgNiEII0WPs7X3q4iz6dCFQcGOyrl27AgD69u0Ljiu4DFp49bxbt27yxxzHQSrlH/SA6I4lM8Oi9EAAQBay8UKUrOOICCGEFEVwoj5+/HhJxEF0YJXZdhw3ioZpAzGyrtOPKkKI/tC0QZg+NSYTnKjbtGlTEnGQUmBg96XiYxug7lEztLnkgGWXe6JGxtuhT0Ud+Ofl5jjlST80nee4aEIaAJbUtSzl2wQcx3/rQMhrwXFGGq1fVojFpkpl+fnUmIyULOqeRSqE/OSC4UYdsi0wvsku3DcXMh45IYToTkXqnkWJuoLLTwaWXe4Bh2wLRNtQS3BCCClraMxPAst8Y4Rf6gOJrODjkCFWNTsVIYSUDYwxCBwGRGl9fUGJmgCAPEkfd7iPxbVPwPQjEQ03SggpsyrSpByCL32HhITg8ePHJRELKQO8XleFfY45DTdKCCFlhOAz6t27d+PHH39EmzZtMHz4cPTp0wcSiaQkYiNax98F6/0W3gY2HAYk1If91RQsSxqD6nmV5c+J3YaUZIBKDA0qKZXl56fw1mXIL6EolF83xjTvzqaPLbz55Oe/1nUIpALSdMwSPTqhFn5GffXqVVy8eBH169fH+PHj4eTkhNGjR+PixYslER/RgfxkhrCk0XCQWmO59W5dh0MIIUpoCNEPaNy4MX7++Wc8e/YMv//+O548eYKWLVvC09MTy5YtQ2pqqrbjJKXMUmaGsKTRmPVqMACA6dXvT0JIeUeJWk2MMeTl5SE3NxeMMdjY2GD58uVwcXHBli1btBUj0RFLmRmsZGZIFCfjG8cw3Dd8puuQCCGkwilWoo6OjkZAQACcnZ0xYcIENG7cGLdv38aJEydw7949/PjjjwgMDNR2rERHTJgRpJwUQQ4rYdqAJvMghOhewT1qTf7TH4IbkzVo0AB37txBx44d8fvvv6Nbt24QixW/vL/66iuMHz9ea0GWHJpbWZX3G40Z2HCoe9Qcn/xbBWH536M6c5M/ZyjpVGJx5OXTaGmEEGXUPasIffv2xaNHj7B//3707NlTKUkDgJ2dHWQy6oNbnuQnM9z2zYADq4RJBvORDRoUhRBCSoOgRJ2Xl4eIiAiaa7qCyk9mWJr/Pb7LHw1jUJc8QojuaDIXtaYTepQ2QYna0NAQ2dk0K05FZglzNGOeYGD4TfwXHnBxug6JEFIBMTDINFj06S614EvfY8eOxYIFC5CfX1KDSxB98AY5OMtdQZDBDzSCGSGk1FWkM2rBjckuXryIyMhIHD58GA0aNICZmZnC8zt27NBacKTsMoUxluZ/jwkGP6LesVTcapdOY4MTQkgJEJyora2t0adPn5KIRQf06CdVGfF+C28DGw79XzWG3bWXCJfNQxU4va1r4Fva4RFCKghN55TWp9MKwYl63bp1JREH0VP5yQxLZTOwidsNByiPy00IISWhIk1zWaybi/n5+Th69Ch+/fVXpKenAwCePXuGjIwMrQZH9IMlLDCKDYQhDHETd/EANLsaIYRoi+Az6sePH6NTp06Ii4tDTk4OOnToAAsLCyxYsAA5OTkIDw8viTiJHmBgWC3ahIeIQ5gsRNfhEELKMRrwpAjjx49H06ZNkZycDBMTE3l5r169EBkZqdXgiH7hwGGubCIcYIcg0WwabpQQUmI06ZpVuOgLwWfUp06dwpkzZ2BkZKRQ7u7ujqdPn2otMKI/3m80pjDcqCwE1eFWZH1CCBGKQbMuVvqTpotxRi2TySCVSpXKnzx5AgsLC60ERfRb4XCjNeEOmV61rSSEkLJHcKLu2LEjwsLC5I85jkNGRgZCQkLw2WefaTM2osfykxmWyGaiJjyQg1zE47muQyKElCMV6dK34ES9ePFinD59GvXq1UN2djb69+8vv+y9YMGCkoiR6Lk13CaMEX1PrcEJIVpDI5MVoWrVqvj333+xefNmXLt2DRkZGRg+fDgGDBig0LiMkEKDWR9c4W4hSDSbWoMTQohAghM1ABgYGGDgwIHajoWUU5awwFLZDEwQzS1oDf6RiIYbJYRoRNPL1/p06Vtwot6wYUORzw8ePLjYwZDyRVVr8OX/jsIg1rvIuoQQUhQZ0zBR69G1b8GJevz48QqP8/LykJWVBSMjI5iamlKiJirlJzPcbJWOgem9AABZeANT0O0SQohwTMOpKsv1NJfJyckKS0ZGBmJiYtCqVSv873//K4kYSTkie1MwMMppXEJ/0ThqYEYIIR+glYmEa9asifnz5yudbROiSgPURiXYIkg0m5I1IUQwhrczaBVn0Z/zaS0laqCggdmzZ88ErTNr1ixwHKew1KlTR/58dnY2xo4di0qVKsHc3Bx9+vRBYmKitkImOlTYwEw+3OhHWvsoEkIqgIrUj1rwPeo9e/YoPGaM4fnz51i+fDlatmwpOID69evj6NGjbwMyeBvShAkTsH//fmzduhVWVlYICAhA7969cfr0acH7IWXPu63BPVa8xs02NPsaIYS8T3Ci7tmzp8JjjuNgb2+Pdu3aYfHixcIDMDCAk5OTUnlqaip+//13bNq0Ce3atQNQMBd23bp1ce7cOXz88ceC9/U+sdhSqUwqTdN4u/w4FeWa/arjOwagJI9Dc3ytwZ+FD0Wl9SZgYODeea3E/it4tqBqsg/lLl+ciroM+eqGWyaIROa85TJZ2fxxw6n4auF73YXULWqPfFvQvK429kdKAmMaNiYrz62+ZTLt9n+9d+8eKleuDGNjY/j4+CA0NBSurq6Ijo5GXl4efH3ffqnXqVMHrq6uOHv2rMpEnZOTg5ycHPnjtLSym7BIgfxkhkrZJnhl/AYhn5zGhAtNUT3FWtdhEULKsIrUj1qnNwa9vb0RERGBgwcPYtWqVYiNjUXr1q2Rnp6OhIQEGBkZwdraWmEdR0dHJCQkqNxmaGgorKys5IuLi0sJHwXRFkOZCFmGeRjfMRIPrFN0HQ4hpAyje9RFCA4OVrvukiVLiny+c+fO8n97enrC29sbbm5u+Ouvv4o9HOm0adMUYkxLS6NkrScscyUIO9IOQR2OYXzHSCw73F7XIRFCiM4JTtRXrlzBlStXkJeXh9q1awMA7t69C7FYjCZNmsjrcZyqe7KqWVtbo1atWrh//z46dOiA3NxcpKSkKJxVJyYm8t7TLiSRSCCRSATvm5QN7ybrCR2OQWQGyDJ1HRUhpKwpPC/WZP3iWLFiBRYtWoSEhAQ0bNgQv/zyC5o3b85bd82aNdiwYQNu3LgBAPDy8sK8efNU1ldFcKLu1q0bLCwssH79etjY2AAoGARl6NChaN26NSZOnCh0k3IZGRl48OABBg0aBC8vLxgaGiIyMhJ9+vQBAMTExCAuLg4+Pj7F3se7hDW44muUpDwvt2olc5lFG43GOM5IqYyxXI23K8T7jcYMbDhYtDRA/nrlKziiz5eq2Irya1zajcbsLJoolb1Mv6z2+qoaVglrNKaqsZ2Qz6tmhLzu2nmPhPx9aeNvkW8bun/dKxJd3KPesmULgoODER4eDm9vb4SFhcHPzw8xMTFwcHBQqh8VFYWvvvoKLVq0gLGxMRYsWICOHTvi5s2bqFKlitr75ZjApm9VqlTB4cOHUb9+fYXyGzduoGPHjoL6Uk+aNAndunWDm5sbnj17hpCQEFy9ehW3bt2Cvb09Ro8ejQMHDiAiIgKWlpYYN24cAODMmTNq7yMtLQ1WVlYouB0v/Cz/LU0TddlVFhK1KtJtE7Ch9jW0fO6CGmm2AIQl6tJWUolaWDKjhKEb9LqrVjA8SWpqKiwt+XuqqKvwO93HcgwMuOJfPc1nOTibtlJQTN7e3mjWrBmWL18OoKBxtYuLC8aNG4epU6d+cH2pVAobGxssX75c0HDbgs+o09LS8OLFC6XyFy9eID09XdC2njx5gq+++gqvXr2Cvb09WrVqhXPnzsHe3h4AsHTpUohEIvTp0wc5OTnw8/PDypUrhYZM9FiOWIqoKo/wV82bWHaykzxZE0IqNm2dUb/fM0jV7dPc3FxER0dj2rRp8jKRSARfX1+cPXtWrX1mZWUhLy8PtrbCvscEt/ru1asXhg4dih07duDJkyd48uQJtm/fjuHDh6N3794f3sA7Nm/ejGfPniEnJwdPnjzB5s2bUb16dfnzxsbGWLFiBV6/fo3MzEzs2LGjyPvTpPwxlhpg2alOcMgyw/hPDuK+5Wtdh0QIKQNkWvgPAFxcXBR6CoWGhvLu7+XLl5BKpXB0dFQo/1BPpHdNmTIFlStXVuh2rA7BZ9Th4eGYNGkS+vfvj7y8vIKNGBhg+PDhWLRokdDNEfJBlnkSLDvVCeNbH8T4Tw7CuKYI2fdoPmtCiObi4+MVLn2XVGPk+fPnY/PmzYiKioKxsbGgdQUnalNTU6xcuRKLFi3CgwcPAADVq1eHmZmZ0E0RorbCZL2u7lUcjKPx3gmp6BjHwDhNWn0XXPq2tLRU6x61nZ0dxGKx0nwTH+qJBAA//fQT5s+fj6NHj8LT01NwrIITdSEzM7Ni7bAskRgqv7h5UlWta5U/EDJZltr70k4joZLCdwdE/WE6Vd9BEdKIhr+hn+hz5b74TlatYNQ0G0wG5N16+8s05c1Dpbo5eS95t2toYK1UlpuXxB8ZT2M7G/N6vHWFNBzje41VfR4sTGvylqdn3eMpVfW6C2kQqfshMg0MlO/j5eeXzK2PAQ7f85ZvTPpRxRp8rw//aykSmSqVCfnuIPyYhveohQ4/amRkBC8vL0RGRsqH0pbJZIiMjERAQIDK9RYuXIgff/wRhw4dQtOmTYsVa7ETNSG6ZDsnAUae2Xje3V0hWRNCKgYZZOA06EctK8a6wcHB8Pf3R9OmTdG8eXOEhYUhMzMTQ4cOBQAMHjwYVapUkd/nXrBgAWbOnIlNmzbB3d1dfi/b3Nwc5ub84/fzoURN9FLSsKpw2v0YznseUbImhJSKfv364cWLF5g5cyYSEhLQqFEjHDx4UN7ALC4uDiLR2yuMq1atQm5uLj7//HOF7YSEhGDWrFlq75cSNdFLshQDJPRwU0jWiNZ1VISQ0qKrkckCAgJUXuqOiopSePzo0aNi7eN9Op2UgxBNFCbr3DsSiEypFTghFYmMk2m86AvBI5Ppm6JHJlNuECIS8U8GQo0/yr43SRuRKHqNqrK3Q/kZOPTTYUSEkALaH5nM09ofYp6GnuqSslxcS1mvlZhKGp1Rk3Ljd7O9GGWzEPfFT3QdCiGkhGlrwBN9QImalBv9szrCQWqDIOswStaElHOUqAnRQ5bMDGGpQZSsCakAChuTabLoC0rUpFx5N1kfl1AzcEKI/qPuWaTcsWRmWJ4yESaQYDQ2QWQMyLJ1HRUhRJtkkILTYApRmR5NP1rBE7Vyg3dq3a2/+Fp4n81fgnmi5Vgk+x7V4SYvNzQQNnsNIaRsYf8NIqrJ+vqCLn2Tcq0WPGADawSJZuMBHus6HEIIEYwSNSnXLGGBpbIZcIAdJWtCypGKNOAJJWpS7r2brBeJftWrS16EEH4ySDVe9EUFv0dNKorCZP0GOeBUTKlJCNEnmnax0p8zakrUpFzjazRmYMMhUNYWY2SDqIEZIaTMo0vfpEJKQRrdsyZEj8mYVONFX1CiJhVOfjKjBmaE6DkamYyQcu7dBmYTRHOQjkxdh0QIIbzoHjWpsAqTdTRuwAJmug6HECIAgxRMg3NNpketvumMmlRolrDAp/ABADgFSmD6Ef1JEKIPKtLsWXRGTbRAVXensttf+f0W3pwEGPSmMZLwCmH5M1Edrm/rGnUs7fAIIR9AQ4gSUsGwHGBp/gw4oBKCDOZQAzNCSJlBiZqQ/1jC/J1kPReP8VTXIRFCVGBMqvGiLyhRE/KOwmT9qexj2KOSrsMhhKhQke5RU6Im5D2WMEew7GuYwhiP8RSmDcS6DokQUoFRYzKiBfrTKKMofI3G6uwzxyf/VkGYLISGGyWkDCnonlX8cfupexYh5cT9QZk0ghkhZRBjMo0XfUGJmpAi0HCjhJRNdI+aECJXOIKZG6ogDRm6DocQUsHQPWpC1GAJC/wimwMOHKSQQuImQs5j/flFTkh5w5iG96ipexYh5Q/335fCBm4HGly0oOFGCdGhwpHJir/oTyPYCn1GXc/mK6WyW8n/00EkBFDVBap4v3o5joO1tTUsLMzBcZol1BrVhys8NrDi0O1gA9T95w2+O+MD1zRL+XP1pmzUaF8Vk/4NQUveYkyG9PQMpKSkgDF6z0pChU7UpHyyt7fH6NEj0bRpUxgYGIAr/tUxlVyemuO1cTYS3Rhy3xjDQFbwYyA8vIX2d0ZIGcYYkJ+fj4sXLyI8fDVevHhRSvuVaXjpW39uXVGiJuWKgYEBFi9eBA8PDxgbS0psP9Wq2MKNY0gwz4RxvgEqvTEGAGRxJbfP8ovOqMuDjh07ombNmhg27Gvk5+eXwh6lGn5C9OceNSVqUq44OzvDzq5SiSZpADA2KvjT8ci1gohxgBGlFVKxGRtLYGdXCU5OTnjy5ImuwylXKFGTckUkEoEriWvdqvbHCvb1xiAfz8wzIDLhIHtDKZtUTBzHQSwunSF3Cy5d06Xvco8ajpUl+nMZCgCiY5MUHnNiDsa1RDCvZwRXOEOCt2f00dExpR2enqEfNkS4ipSoqX8JIVrApAzZd2UwhAHi8Bw5yNF1SISUazQyGSFEMCZlcIEzDGGA50gS1E9z5MiRWLx4sUb7f/bsGZo1a4aYmJI9g9+7dy8+/fTTEt1HRTFr1ixMmjRJo21ER0ejWbNmSE9Pl5dFRUWhV69e8Pb21vhzRXSPEjUhWiSGGC5wRmU4yQdIUcfChQsxatSoEoyseLp3745NmzYplHXo0AHbt2/Xyb51Td2YmjVrhqioqJIPSIXQ0FC0a9cO+/btK9bn6vLly5gwYQI6d+6s82NRhSblKEVPnz7FwIEDUalSJZiYmKBBgwa4dOmS/HnGGGbOnAlnZ2eYmJjA19cX9+7d02HEhBRNDDGMYAgpZHiC5xCZfDhhW1lZwczMrBSi05yxsTFsbW11HYba8vLydB1CqcrKysLr16/h4+MDe3t73s/V3r17MXLkSJXbePPmDWrVqoXJkyeXZKgaYUyq8aIvdJqok5OT0bJlSxgaGuLvv//GrVu3sHjxYtjY2MjrLFy4ED///DPCw8Nx/vx5mJmZwc/PD9nZ2TqMnBB1MORDCuNa4g8m6/cvfXfv3h3r1q3DnDlz0KZNG3Tt2hU7duxQWOfmzZsYMGAAWrZsicGDBytd8ua7RB0VFYVmzZoplJ08eRKDBw9Gy5Yt4evri2+//VYe0/Pnz7F06VI0a9ZMvh7fdrdt24aePXvCx8cHffr0wYEDBxSeb9asGXbt2oVvv/0WrVq1Qu/evXHixIkiXw++faekpOD777/HZ599hlatWuHLL7/EoUOHlNZduHAhFi9eDF9fX4wbNw4AcOLECfTu3RstW7bEqFGjsG/fPqVLxlevXsWIESPQqlUrdOnSBT/99BPevHlTZEzv6969OwDg22+/RbNmzeSPi/LHH3+gU6dO8PX1xYIFCxT6IR84cACDBw9GmzZt4Ofnh+nTp+P169e824mOjkabNm0AAKNHj0azZs0QHR39wf2/r2XLlhg9ejTd4igjdNrqe8GCBXBxccG6devkZR4eHvJ/M8YQFhaG6dOno0ePHgCADRs2wNHREbt27cKXX35Z6jEXl1hsyVsulabxlKo/AATHGfHX5P21WDK/IM2MPXjLM7PjVKxRmr9kC17LVMNXSDN8/U6pGCb55rDLdUYel4sEk8dKE8m7ZNUAACRKniBXrPjD0DbHEWZSC6QbpCDF6KW8PP72fUikpnDIqVKwHzFgXVcCg9oc7OKtYJjztutKTEaiyngBYOPGjRg5chSGDh2GyMijWLBgAZo0aQJ3d3dkZWVhwoQJaN68OebMmYNnz54V617kP//8g8mTJ2Po0KGYPXs28vLycPr0aQAFP5L79++PXr16oWfPniq3cfz4cSxevBjBwcFo3rw5/vnnH8yZMwcODg5o2rSpvN6aNWsQGBiIwMDx2LJlC2bOnIk9e/bCykr5b0PVvnNzc1GnTh0MHjwYZmZmOH36NEJCQlC1alXUr/+RvN7+/fvRp8/n+O23tQCAp0+fY+rUqfjyy6/Qo0d33L17F8uWLVPY55MnTxAYGIhRo0ZhxowZSE5OxqJFi7Bw4SKEhIRg4cJF78TUQ+XrsX79enTs2BEzZ86Ej4/PB7srXbp0CXZ2dggPD0d8fDy+++471KpVC7169QJQMOrXyJGj4ObmhuTkZCxduhSzZ89Wih8APD09sW3bNnz++edYsGABPD09YWVlVeT+3ycR23y4UhlQ0Aak+JevaaxvNe3Zswd+fn744osvcOLECVSpUgVjxozBiBEjAACxsbFISEiAr6+vfB0rKyt4e3vj7NmzvIk6JycHOTlvW9ympfElQlLRnLY/gIOVFcfhbvqqPfxjpyLF6AUW1hujtM7Pl/4GAGz0WIxH5ncUnhv08Fs0e90OV2xOYZvbSoXn6qQ2wZh78wAATArYxVnipWsaXrqkwjHWBiKpeveuW7RogS+++AIA4O/vj//973+Ijo6Gu7s7Dh48CJlMhhkzZkAikaB69epISkrC/Pnz1XtB/rN27Vp06NBB4TJorVq1ABT8rYnFYpiamsLOzk7lNv7880907dpVHqubmxtu3LiBP//8UyFRd+3aFX5+nQAAY8eOxZYtm3Hz5k20aOGjtE1V+3ZwcMCgQYPkj/v164dz587hyJGjConaxcUFgYHj5Y9/+eVnuLm5Yfz4IAAyuLu748GDB1i7dq28TkREBDp16oT+/fsDAFxdXTFp0rcYOfIbTJ069b+YRB98PQqvCFpYWBRZr5ClpSW+/fZbiMViuLu7o1WrVrh48aI8UXfv/vZHQdWqVTFp0iT4+w9GVlYWTE1NFbZlaGgovy1hZWWl1v71lab3mPXpHrVOE/XDhw+xatUqBAcH47vvvsPFixcRGBgIIyMj+Pv7IyEhAQDg6OiosJ6jo6P8ufeFhoZi9uzZJR470S8tX3yGBikfyx8XnlEDgHWuPSbfWql0Rl1oQOxE3jNqAGic3BoemXUVnpNIFb88RTIOdnGWyDbPVTtJA0DNmjXfxstxqFSpkvyS56NHj1CzZk1IJG/7azdo0EDtbRe6e/dukWfL6nj06JE8qRTy9PTE5s2bFcrePR4TExOYmZkhOZn/Eq4qUqkU69atw9GjR/HixQvk5eUhNzcXxsbGCvXq1FF8T+LiHqNevfoKZfXq1VN4fPfuXdy/fx8HDx6UlzHGIJPJ8OzZM4WrfUIlJCSgb9++8sdDhw7F0KFDAQDVqlVTOOuuVKkSHjx4IH98+/ZtrF69Gvfu3UV6ejpkMpl8m9WqVSvW/qVSKfLz8/HJJ5/wxqQPKFGXEplMhqZNm2LevIKzj8aNG+PGjRsIDw+Hv79/sbY5bdo0BAcHyx+npaXBxcVFK/ES/WWVVwlWeZXkj7l3ZusyZEZwyaoJBv7xiR1zqqrcrkW+NSzyrd8p4U/EIhkH07SCpJpplQOjbAMgo+iYxWLlP08hsxOJRCKl+u+Pwfx+gitJBgaKx8NxHGQyYZcf//jjD2zevBnBwcGoUaMGTExMsGTJEqUGYyYmJoLje/PmDXr37o1+/fq9GyUAwMnJSfD23mVnZ4eNG99e0bG0fHu5n/91kcljGjcuAB9/7IO5c3+AjY0NEhISMG5cgKBGcu/v//jx4zh27Bjmzp3LGxMpW3SaqJ2dnZV+1datW1fe9aPwjyMxMRHOzs7yOomJiWjUqBHvNiUSicJZBiFlCgdk2mQjzUAKUVbxhxt1d3fHgQMHkJOTI/+837hxQ6GOtbU1srKy8ObNG3niunv3rkKdGjVq4OLFiyobPBkaGsqTRlGx/Pvvv+jatau87Nq1a2qf7anCt+9///0Xbdq0wWeffQag4Md+XFzcB892XV3dcObMaYWyW7duKTyuXbs2Hj58+N4Pe8UfXuq8HkBB8n23noGBQbFOGB49eoTU1FQEBATIvw/fj1sd7+/fxsYGEolEr09imIYDlmi6fmnSaavvli1bKrVUvXv3Ltzc3AAUNCxzcnJCZGSk/Pm0tDScP38ePj7K97WE4jhjpaWkSKVpvAs/pmLhqclyeZeCBlvvLyUjMzuWd+GPoeTi4DgjcJwhCr5cORR8vPlfS4Z8pUXYvgx5F759xWQkvl3SE5FyOwd5OTJY1DXER/Uc4VWnCixMJXCwNYfie636ve/UqRM4jsOPP/6Ihw8f4vTp0/jzzz8V6nz00UcwNjbGihUr8OTJExw8eBD79u1TqDNixAgcPnwYv/76K2JjY3H//n2sX79e/ryzszOuXLmCpKQkpKSk8L4WgwYNwr59+7Bt2zbExcVh48aNOH78OAYOHMhT+/1jUv1DhW/frq6uOH/+PP7991/ExsZi3rx5ePXq1XvbLvy3TL707t0Ljx49wi+/LMPjx49x5MgR+WtRODa8v78/rl27hoULFyImJgZxcXE4cSIKCxcukG+7IKbLRb4eAFC5cmVcuHABL1++1KidjJOTEwwNDfHXX1vw5Ek8TpyIwu+//1bs7akjR5qM5PSnuHbrHK7dOgegYDCdmJgYlbccdYH6UZeSCRMm4Ny5c5g3bx7u37+PTZs2YfXq1Rg7diyAgj+goKAg/PDDD9izZw+uX7+OwYMHo3LlyhrfVyNEVwqHGzVgIsQbJyOXEz4loKmpKZYsWYL79+9j4MCBWLlyJQICAhTqWFlZYc6cOThz5oy8G1NhQ81CXl5eCA0NxcmTJzFgwACMHj0aN2/elD9f2CWpV69e6NChA28sbdu2xcSJE/Hnn3+iX79+2LFjB2bOnAkvLy/Bx/Uuvn0PGzYMderUkbfOrlSpEtq2bfvBbVWpUgXz58/H8ePH0b9/f2zfvh3Dhg0DUHCWDBTcQ//1118RFxeHb775BgMHDsSvv/4Ke3t7Qa8HAIwfPx4XLlxA165dVfxgUY+NjQ1CQkIQGRmJfv36Yf369Rg/fvyHV9TQ7du3MXDgQHnsS5cuxcCBAxEeHl7i+ybKOCbkplcJ2LdvH6ZNm4Z79+7Bw8MDwcHBCl8mjDGEhIRg9erVSElJQatWrbBy5Up5y9QPSUtL+697ggjvX8biO4NmjPpn6yuOM4KbmytWrVr2X2tXDpp03yh6X4a85Yypf9+wUZ3KeCFJh0OOBUT/fTaj7zzVSnzkw9auXYvt27dj//79ug6l3Hj58iVGjRqLx48fv/dMwRWO1NRUje+FF36nGxo4geOKf67JmAx5+Qlaiamk6Xz2rK5duyrc23ofx3GYM2cO5syZU4pREVLyxODglFPwBZHHSSHj9Kdfpz7aunUr6tWrBysrK1y7dg1//PGHQktoom80/RGuP5e+dZ6oCSFAkiQdb0R5NJ91CYqPj8fatWuRlpYGJycnDBgwAEOGDNF1WKSYqHsWIaRUOWVbId7kNYxriZF9V0rJugQEBwcrdN0kRF9U6ERN96PLl4IW73n4UEt57exL84ke3r8fzYk52DYygUE9KVyZMyR4Ozxs9OW7769OSIVWkbpnVehETcqfgm4X+nk2yqQMLswZT7gE5CFfIVETog8YY6V2Sbng71yDsb716HuCEjUpV169eo3cXP2d1lAMMVxZZXDg/uvtLYUh/ZkSPZGbm4eXL199uCIRROfzUROiTZmZmdizZy9SUlJ1HUqxZGfnIic7D9nZuUjMeYUHOU+Qlv2BsUYJKQNSUlKxZ89eZGVlldIeVQ2oJGTRD/RTnZQ769ZFAAC6d+8GIyND+chT+iD20dsxyBlkeI1UJHFJyE7NhxZuixOidYwx5ObmYc+evfK/vdLZrwyqpwRWZ339ufSt8wFPSlpRA57wU38uaG0oD4Ou/FhrJm/593f5+74bGNgqleXnC5tFiY/EUHHiBFNTE1hYMbUHRbAwdectz3ijPK+26vtw6v9Kf3diEPl231vfwIpDtbWmMHIW4cGQLLy5W/yzAHX2V0gsNlcqk0p1f2bPdwyA6uNQX9FzRitTf3/VrDorlT1M/VtFbb7vH/7vHiHvp5CExnF8k8Go/pXImAwvX76Sn0lznHLbioI0k63VAU84zlSjH+EF99OzaMATQnQpK+sNklNfqF3fyox/Mpe0LL5EreoLUUii5vlC5Blz/HEHDnWPmiOnlxSPxxT/sqK6+wMAsVj5i0v12PSlh+8YANXHob6SS9QmNso/Qh8nvz9yVyEhiVr991NYolYeda9g/gCiK5SoCSnj8pMZbrXLgDSjXF/8IkQYDS99Q48uJlOiJkQPSFP150uFkNLANLwdqen6pYlafRNCCCFlWLlP1G/byqma41m3S0GDBsVF1zEJXbKl2byL6mPmmxdWG6+lZttlTKpiKZn3iO+/Ev2sCdhfWf1cqvpP13EVtUhZntJSlj8/mr73qteHlltay7SwCLdixQq4u7vD2NgY3t7euHDhQpH1t27dijp16sDY2BgNGjTAgQMHhO+UlXPx8fG6/0ulhRZaaKGFxcfHa/ydnpqa+t/2DBgHw2IvgAEDwFJTU9Xe9+bNm5mRkRFbu3Ytu3nzJhsxYgSztrZmiYmJvPVPnz7NxGIxW7hwIbt16xabPn06MzQ0ZNevXxd0zOW+e5ZMJsOzZ89gYWGB9PR0uLi4ID4+vsw3xxcqLS2Njk1Plefjo2PTT9o+NsYY0tPTUblyZYhEml3IfdvlVgyNGpOBAZAK6p7l7e2NZs2aYfny5QAK8ouLiwvGjRuHqVOnKtXv168fMjMzsW/fPnnZxx9/jEaNGiE8PFztSMt9YzKRSISqVasCgLzPnaWlZbn7wypEx6a/yvPx0bHpJ20eW0Fy1TbNzzPT0hS7HUokEkgkyl01c3NzER0djWnTpsnLRCIRfH19cfbsWd5tnz17VmnGNj8/P+zatUtQjOX+HjUhhJDyw8jICE5OTtDGEKLm5uZwcXGBlZWVfAkNDeXd78uXLyGVSuHo6KhQ7ujoiISEBN51EhISBNVXpdyfURNCCCk/jI2NERsbi9xczQdhYYwpjW7GdzataxUqUUskEoSEhJTJN0JTdGz6qzwfHx2bfirrx2ZsbAxjY+Xhl0uSnZ0dxGIxEhMTFcoTExP/O8NX5uTkJKi+KuW+MRkhhBCiDd7e3mjevDl++eUXAAWNyVxdXREQEKCyMVlWVhb27t0rL2vRogU8PT2pMRkhhBCibcHBwfD390fTpk3RvHlzhIWFITMzE0OHDgUADB48GFWqVJHf5x4/fjzatGmDxYsXo0uXLti8eTMuXbqE1atXC9ovJWpCCCFEDf369cOLFy8wc+ZMJCQkoFGjRjh48KC8wVhcXJxC97MWLVpg06ZNmD59Or777jvUrFkTu3btwkcffSRov3TpmxBCCCnDqHsWIYQQUoZVmEQtdHzWsurkyZPo1q0bKleuDI7jlDrOM8Ywc+ZMODs7w8TEBL6+vrh3755ughUgNDQUzZo1g4WFBRwcHNCzZ0/ExMQo1MnOzsbYsWNRqVIlmJubo0+fPkotKsuqVatWwdPTUz6AhI+PD/7++2/58/p8bO+bP38+OI5DUFCQvExfj2/WrFngOE5hqVOnjvx5fT2udz19+hQDBw5EpUqVYGJiggYNGuDSpUvy5/X1O6U8qRCJesuWLQgODkZISAguX76Mhg0bws/PD0lJSboOTbDMzEw0bNgQK1as4H1+4cKF+PnnnxEeHo7z58/DzMwMfn5+yM7OLuVIhTlx4gTGjh2Lc+fO4ciRI8jLy0PHjh2RmZkprzNhwgTs3bsXW7duxYkTJ/Ds2TP07t1bh1Grr2rVqpg/fz6io6Nx6dIltGvXDj169MDNmzcB6PexvevixYv49ddf4enpqVCuz8dXv359PH/+XL78888/8uf0+bgAIDk5GS1btoShoSH+/vtv3Lp1C4sXL4aNjY28jr5+p5QrgkYG11PNmzdnY8eOlT+WSqWscuXKLDQ0VIdRaQ4A27lzp/yxTCZjTk5ObNGiRfKylJQUJpFI2P/+9z8dRFh8SUlJDAA7ceIEY6zgOAwNDdnWrVvldW7fvs0AsLNnz+oqTI3Y2Niw3377rdwcW3p6OqtZsyY7cuQIa9OmDRs/fjxjTL/fu5CQENawYUPe5/T5uApNmTKFtWrVSuXz5ek7RZ+V+zPqwvFZfX195WUfGp9VX8XGxiIhIUHhWK2srODt7a13x5qamgoAsLW1BQBER0cjLy9P4djq1KkDV1dXvTs2qVSKzZs3IzMzEz4+PuXm2MaOHYsuXbooHAeg/+/dvXv3ULlyZVSrVg0DBgxAXFwcAP0/LgDYs2cPmjZtii+++AIODg5o3Lgx1qxZI3++PH2n6LNyn6iLMz6rvio8Hn0/VplMhqCgILRs2VLejSEhIQFGRkawtrZWqKtPx3b9+nWYm5tDIpFg1KhR2LlzJ+rVq1cujm3z5s24fPky7zjJ+nx83t7eiIiIwMGDB7Fq1SrExsaidevWSE9P1+vjKvTw4UOsWrUKNWvWxKFDhzB69GgEBgZi/fr1AMrPd4q+o37UpMwZO3Ysbty4oXAvsDyoXbs2rl69itTUVGzbtg3+/v44ceKErsPSWHx8PMaPH48jR46U+rCOJa1z587yf3t6esLb2xtubm7466+/YGJiosPItEMmk6Fp06aYN28eAKBx48a4ceMGwsPD4e/vr+PoSKFyf0ZdnPFZ9VXh8ejzsQYEBGDfvn04fvy4fHpSoODYcnNzkZKSolBfn47NyMgINWrUgJeXF0JDQ9GwYUMsW7ZM748tOjoaSUlJaNKkCQwMDGBgYIATJ07g559/hoGBARwdHfX6+N5lbW2NWrVq4f79+3r/vgGAs7Mz6tWrp1BWt25d+eX98vCdUh6U+0RtZGQELy8vREZGystkMhkiIyPh4+Ojw8i0z8PDA05OTgrHmpaWhvPnz5f5Y2WMISAgADt37sSxY8fg4eGh8LyXlxcMDQ0Vji0mJgZxcXFl/thUkclkyMnJ0ftja9++Pa5fv46rV6/Kl6ZNm2LAgAHyf+vz8b0rIyMDDx48gLOzs96/bwDQsmVLpW6Qd+/ehZubGwD9/k4pV3Tdmq00bN68mUkkEhYREcFu3brFvvnmG2Ztbc0SEhJ0HZpg6enp7MqVK+zKlSsMAFuyZAm7cuUKe/z4MWOMsfnz5zNra2u2e/dudu3aNdajRw/m4eHB3rx5o+PIizZ69GhmZWXFoqKi2PPnz+VLVlaWvM6oUaOYq6srO3bsGLt06RLz8fFhPj4+OoxafVOnTmUnTpxgsbGx7Nq1a2zq1KmM4zh2+PBhxph+Hxufd1t9M6a/xzdx4kQWFRXFYmNj2enTp5mvry+zs7NjSUlJjDH9Pa5CFy5cYAYGBuzHH39k9+7dYxs3bmSmpqbszz//lNfR1++U8qRCJGrGGPvll1+Yq6srMzIyYs2bN2fnzp3TdUjFcvz4cQZAafH392eMFXSnmDFjBnN0dGQSiYS1b9+excTE6DZoNfAdEwC2bt06eZ03b96wMWPGMBsbG2Zqasp69erFnj9/rrugBRg2bBhzc3NjRkZGzN7enrVv316epBnT72Pj836i1tfj69evH3N2dmZGRkasSpUqrF+/fuz+/fvy5/X1uN61d+9e9tFHHzGJRMLq1KnDVq9erfC8vn6nlCc01jchhBBShpX7e9SEEEKIPqNETQghhJRhlKgJIYSQMowSNSGEEFKGUaImhBBCyjBK1IQQQkgZRomaEEIIKcMoURONtW3bFkFBQboOQ6tmzZqFRo0aFVnn0aNH4DgOV69eLfF4cnNzUaNGDZw5c6bE96VNUVFR4DhOaTxsodzd3REWFqaVmN41depUjBs3TuvbJUSbKFETwmPSpEkK4xsPGTIEPXv2VKjj4uKC58+fy6fiLEnh4eHw8PBAixYt1F5H1Y8NjuOwa9cu7QVXBvC9P+qYNGkS1q9fj4cPH2o/KEK0hBI1ITzMzc1RqVKlIuuIxWI4OTnBwKBkZ4tljGH58uUYPnx4ie6nIrKzs4Ofnx9WrVql61AIUYkSNdG6/fv3w8rKChs3bgRQMF9x3759YW1tDVtbW/To0QOPHj0CAJw8eRKGhoZKk9AHBQWhdevWKvfBcRxWrVqFzp07w8TEBNWqVcO2bdsU6ly/fh3t2rWDiYkJKlWqhG+++QYZGRny56OiotC8eXOYmZnB2toaLVu2xOPHjwEono3OmjUL69evx+7du8FxHDiOQ1RUFO+l7xMnTqB58+aQSCRwdnbG1KlTkZ+fL3++bdu2CAwMxOTJk2FrawsnJyfMmjWryNczOjoaDx48QJcuXRTKp0yZglq1asHU1BTVqlXDjBkzkJeXBwCIiIjA7Nmz8e+//8pjjoiIgLu7OwCgV69e4DhO/hgAdu/ejSZNmsDY2BjVqlXD7NmzFWLnOA6//fYbevXqBVNTU9SsWRN79uxRiOnAgQOoVasWTExM8Omnn8rf53f9888/aN26NUxMTODi4oLAwEBkZmbKn09KSkK3bt1gYmICDw8P+edIFVXvD/DhzwAAdOvWDZs3by5yH4TolI7HGiflwLsTMGzcuJFZWFiwvXv3MsYYy83NZXXr1mXDhg1j165dY7du3WL9+/dntWvXZjk5OYwxxmrVqsUWLlwo315ubi6zs7Nja9euVblPAKxSpUpszZo1LCYmhk2fPp2JxWJ269YtxhhjGRkZzNnZmfXu3Ztdv36dRUZGMg8PD/nkJXl5eczKyopNmjSJ3b9/n926dYtFRETIZyELCQlhDRs2ZIwVzFjWt29f1qlTJ/msXjk5OSw2NpYBYFeuXGGMMfbkyRNmamrKxowZw27fvs127tzJ7OzsWEhIiMJrZWlpyWbNmsXu3r3L1q9frzCLFp8lS5awOnXqKJXPnTuXnT59msXGxrI9e/YwR0dHtmDBAsYYY1lZWWzixImsfv36CjORJSUlySc7ef78uXwWqJMnTzJLS0sWERHBHjx4wA4fPszc3d3ZrFmzFF7zqlWrsk2bNrF79+6xwMBAZm5uzl69esUYYywuLo5JJBIWHBzM7ty5w/7880/m6OjIALDk5GTGGGP3799nZmZmbOnSpezu3bvs9OnTrHHjxmzIkCHy/XTu3Jk1bNiQnT17ll26dIm1aNGCmZiYsKVLl/K+Pqrenw99Bgrdvn2bAWCxsbEq3wNCdIkSNdFYYaJevny5fKrKQn/88QerXbs2k8lk8rKcnBxmYmLCDh06xBhjbMGCBaxu3bry57dv387Mzc1ZRkaGyn0CYKNGjVIo8/b2ZqNHj2aMMbZ69WpmY2OjsI39+/czkUjEEhIS2KtXrxgAhVjf9W6iZowxf39/1qNHD4U67yfq7777TulYV6xYwczNzZlUKpW/Vq1atVLYTrNmzdiUKVNUHuv48eNZu3btVD5faNGiRczLy0vlMRQCwHbu3KlQ1r59ezZv3jyFsj/++IM5OzsrrDd9+nT544yMDAaA/f3334wxxqZNm8bq1aunsI0pU6YoJOrhw4ezb775RqHOqVOnmEgkYm/evGExMTEMALtw4YL8+cJEqipRM8b//nzoM1AoNTW1yM8CIbpWsjfXSIWxbds2JCUl4fTp02jWrJm8/N9//8X9+/dhYWGhUD87OxsPHjwAUNAQaPr06Th37hw+/vhjREREoG/fvjAzMytyn+9PXO/j4yO/DH379m00bNhQYRstW7aETCZDTEwMPvnkEwwZMgR+fn7o0KEDfH190bdvXzg7Oxf7Nbh9+zZ8fHzAcZzCPjMyMvDkyRO4uroCADw9PRXWc3Z2RlJSksrtvnnzBsbGxkrlW7Zswc8//4wHDx4gIyMD+fn5sLS0LFbs//77L06fPo0ff/xRXiaVSpGdnY2srCyYmpoqxW5mZgZLS0t57Ldv34a3t7fCdt9/j/79919cu3ZN4XI2YwwymQyxsbG4e/cuDAwM4OXlJX++Tp06sLa2FnxMH/oMODo6AgBMTEwAAFlZWYL3QUhpoERNtKJx48a4fPky1q5di6ZNm8qTVUZGBry8vHjvM9rb2wMAHBwc0K1bN6xbtw4eHh74+++/5fcYS9K6desQGBiIgwcPYsuWLZg+fTqOHDmCjz/+uET3a2hoqPCY4zjIZDKV9e3s7HD9+nWFsrNnz2LAgAGYPXs2/Pz8YGVlhc2bN2Px4sXFiikjIwOzZ89G7969lZ5790eC0Nj59jNy5EgEBgYqPefq6oq7d+8KiFo7Xr9+DeDt55GQsoYSNdGK6tWrY/HixWjbti3EYjGWL18OAGjSpAm2bNkCBweHIs/2vv76a3z11VeoWrUqqlevjpYtW35wn+fOncPgwYMVHjdu3BgAULduXURERCAzM1N+RnX69GmIRCLUrl1bvk7jxo3RuHFjTJs2DT4+Pti0aRNvojYyMoJUKi0ynrp162L79u1gjMl/qJw+fRoWFhaoWrXqB49HlcaNG2PVqlUK2z1z5gzc3Nzw/fffy+sVNoT7UMyGhoZK5U2aNEFMTAxq1KhR7Djr1q2r1Ljs3LlzSvu5deuWyv3UqVMH+fn5iI6Oll+ZiYmJ+WA/bL5jVfczcOPGDRgaGqJ+/fpqHSchpY1afROtqVWrFo4fP47t27fLB0AZMGAA7Ozs0KNHD5w6dQqxsbGIiopCYGAgnjx5Il/Xz88PlpaW+OGHHzB06FC19rd161asXbsWd+/eRUhICC5cuICAgAD5fo2NjeHv748bN27g+PHjGDduHAYNGgRHR0fExsZi2rRpOHv2LB4/fozDhw/j3r17qFu3Lu++3N3dce3aNcTExODly5fy1tXvGjNmDOLj4zFu3DjcuXMHu3fvRkhICIKDgyESFf9P7dNPP0VGRgZu3rwpL6tZsybi4uKwefNmPHjwAD///DN27typFHNsbCyuXr2Kly9fIicnR14eGRmJhIQEJCcnAwBmzpyJDRs2YPbs2bh58yZu376NzZs3Y/r06WrHOWrUKNy7dw/ffvstYmJisGnTJkRERCjUmTJlCs6cOYOAgABcvXoV9+7dw+7du+XvW+3atdGpUyeMHDkS58+fR3R0NL7++mv55WlV+N6fD30GCp06dUreCp2QMknH98hJOfBuq2/GGLt16xZzcHBgwcHBjDHGnj9/zgYPHszs7OyYRCJh1apVYyNGjGCpqakK25kxYwYTi8Xs2bNnH9wnALZixQrWoUMHJpFImLu7O9uyZYtCnWvXrrFPP/2UGRsbM1tbWzZixAiWnp7OGGMsISGB9ezZkzk7OzMjIyPm5ubGZs6cKW/09X5DrKSkJNahQwdmbm7OALDjx48rNSZjjLGoqCjWrFkzZmRkxJycnNiUKVNYXl6eyteKMcZ69Oih1BL5fX379mVTp05VKPv2229ZpUqVmLm5OevXrx9bunQps7Kykj+fnZ3N+vTpw6ytreUtvRljbM+ePaxGjRrMwMCAubm5yesfPHhQ3sLa0tKSNW/enK1evVrhNX+/EZqVlZV8u4wxtnfvXlajRg0mkUhY69at2dq1axUakzHG2IULF+SvpZmZGfP09GQ//vij/Pnnz5+zLl26MIlEwlxdXdmGDRuYm5tbkY3J+N4fxor+DBSqXbs2+9///qdy24ToGscYYzr8nUCI3PDhw/HixQuly6d8OI7Dzp07izUalT66du0aOnTogAcPHsDc3FzX4ZQbf//9NyZOnIhr166V+MA1hBQXfTKJzqWmpuL69evYtGmTWkm6IvL09MSCBQsQGxuLBg0a6DqcciMzMxPr1q2jJE3KNPp0Ep3r0aMHLly4gFGjRqFDhw66DqfMGjJkiK5DKHc+//xzXYdAyAfRpW9CCCGkDKNW34QQQkgZRomaEEIIKcMoURNCCCFlGCVqQgghpAyjRE0IIYSUYZSoCSGEkDKMEjUhhBBShlGiJoQQQsowStSEEEJIGfZ/isxFjF8ebQkAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: the attention pattern of the strongest induction head, with the stripe marked.\n",
"with torch.no_grad():\n",
" model(eval_batch)\n",
"pat = model.layers[best_layer].last_pattern[0, best_head].numpy() # (seq, seq) for one sequence\n",
"fig, ax = plt.subplots(figsize=(5, 5))\n",
"im = ax.imshow(pat, cmap=\"magma\", aspect=\"equal\")\n",
"ax.set_xlabel(\"key position (attended to)\"); ax.set_ylabel(\"query position (attending from)\")\n",
"ax.set_title(f\"layer {best_layer} head {best_head}: the induction stripe\")\n",
"# overlay the expected stripe line: query t -> key t - half + 1\n",
"hf = SEQ_LEN // 2\n",
"ax.plot([t - hf + 1 for t in range(hf, SEQ_LEN)], list(range(hf, SEQ_LEN)),\n",
" color=\"#39FF14\", lw=1, ls=\"--\", label=\"induction target t-half+1\")\n",
"ax.legend(loc=\"lower right\"); plt.colorbar(im, ax=ax, fraction=0.046)\n",
"plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "8104b2d7",
"metadata": {},
"source": [
"> **What is the interpretation of this plot?** AnswerThe bright diagonal stripe sits exactly on the dashed green line, the induction target `t - half + 1`. For every second-half query, the head puts almost all its attention on the position right after the previous occurrence of the current token. That is the induction head doing its job, visible in raw attention weights. The faint bright column on the far left is attention to the first position, a common \"rest position\" sink.\n",
"\n",
"> **Key takeaways.** An induction head has a geometric signature: attention along the `t - half + 1` stripe on a repeat sequence. The score is the mean weight on that stripe. Layer 1 carries it because induction is a two-layer composition. We assert a *property* (score > 0.5), not a value, so the check is seed-robust.\n"
]
},
{
"cell_type": "markdown",
"id": "9bb2a8af",
"metadata": {},
"source": [
"### Exercise 22.2 — A general offset-attention score\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"Induction is \"attend at offset `half - 1`\". A *previous-token* head is \"attend at offset 1\" (every position looks one step back). Generalize the detector: `offset_score(pattern, offset)` returns the mean attention each head pays to the key exactly `offset` positions behind the query, averaged over all valid queries. Then we can score induction (`offset = half - 1`) and prev-token (`offset = 1`) with one function.\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "ab7b454e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.812566Z",
"iopub.status.busy": "2026-06-10T20:56:10.812453Z",
"iopub.status.idle": "2026-06-10T20:56:10.816361Z",
"shell.execute_reply": "2026-06-10T20:56:10.815981Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 22.2 offset score: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def offset_score(pattern, offset):\n",
" \"\"\"pattern: (batch, n_heads, seq, seq). For each query position t >= offset, take the\n",
" attention to key (t - offset). Average over valid t and batch. Return (n_heads,).\"\"\"\n",
" seq_len = pattern.shape[-1]\n",
" # TODO 1: build the query positions that have a valid key at t-offset (t from `offset` to seq_len-1)\n",
" queries = None\n",
" attempted(queries)\n",
" # TODO 2: the matching key positions are queries - offset\n",
" keys = queries - offset\n",
" # TODO 3: gather pattern[:, :, queries, keys] and average over batch and query axes\n",
" striped = pattern[:, :, queries, keys]\n",
" return striped.mean(dim=(0, 2))\n",
"\n",
"# A pure shift-by-1 pattern: EVERY query t>=1 attends one step back (a clean prev-token head).\n",
"shift_pat = torch.zeros(1, 1, 5, 5)\n",
"for t in range(1, 5):\n",
" shift_pat[0, 0, t, t - 1] = 1.0\n",
"\n",
"def _offset_checks():\n",
" # On the shift-by-1 pattern, offset=1 (prev-token) must score 1.0:\n",
" check_close(offset_score(shift_pat, 1)[0].item(), 1.0, msg=\"offset 1 on a pure shift pattern is 1.0\")\n",
" # offset 2 on that same pattern scores 0 (it never attends two-back):\n",
" check_close(offset_score(shift_pat, 2)[0].item(), 0.0, atol=1e-6, msg=\"shift pattern pays nothing at offset 2\")\n",
"check(\"22.2 offset score\", _offset_checks)"
]
},
{
"cell_type": "markdown",
"id": "b5c93cef",
"metadata": {},
"source": [
"Hint 1 (conceptual)This is `induction_score` with the offset as a parameter instead of hard-coded to `half - 1`. The valid queries start at `t = offset` (you need a key `offset` steps back to exist).\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"queries = torch.arange(offset, seq_len)\n",
"keys = queries - offset\n",
"striped = pattern[:, :, queries, keys] # (batch, n_heads, n_queries)\n",
"return striped.mean(dim=(0, 2))\n",
"```\n",
"\n",
"\n",
"Help — \"index out of range\" or shape mismatchIf `queries` starts at 0, then `queries - offset` goes negative and wraps around (PyTorch allows negative indexing, which is the wrong key). Start queries at `offset`.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "dc9981fc",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.817099Z",
"iopub.status.busy": "2026-06-10T20:56:10.817027Z",
"iopub.status.idle": "2026-06-10T20:56:10.823395Z",
"shell.execute_reply": "2026-06-10T20:56:10.823063Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 22.2 offset score\n",
"layer 0: best induction-offset score 0.59 · best prev-token-offset score 0.06\n",
"layer 1: best induction-offset score 0.80 · best prev-token-offset score 0.07\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines offset_score; the check re-verifies the reference.\n",
"def offset_score(pattern, offset):\n",
" seq_len = pattern.shape[-1]\n",
" queries = torch.arange(offset, seq_len) # queries with a valid key offset-back\n",
" keys = queries - offset # the offset-behind key positions\n",
" striped = pattern[:, :, queries, keys] # (batch, n_heads, n_queries)\n",
" return striped.mean(dim=(0, 2))\n",
"\n",
"check(\"22.2 offset score\", _offset_checks, required=True)\n",
"with torch.no_grad():\n",
" model(eval_batch)\n",
"hf = SEQ_LEN // 2\n",
"for l in range(model.n_layers):\n",
" p = model.layers[l].last_pattern\n",
" ind = offset_score(p, hf - 1).max().item() # induction offset\n",
" prev = offset_score(p, 1).max().item() # prev-token offset\n",
" print(f\"layer {l}: best induction-offset score {ind:.2f} · best prev-token-offset score {prev:.2f}\")"
]
},
{
"cell_type": "markdown",
"id": "c141256a",
"metadata": {},
"source": [
"> **Interpretation.** Layer 1 has the high induction-offset scores. The prev-token-offset scores are modest here: in this minimal 2-layer model the layer-0 heads do the \"look-back\" work in a more distributed way than a single clean previous-token head, which is common at this scale. The textbook clean split (one pure prev-token head in L0, one pure induction head in L1) sharpens with more heads, more width, and more training. The mechanism is the same; the tidiness is a function of scale.\n"
]
},
{
"cell_type": "markdown",
"id": "da0cbf71",
"metadata": {},
"source": [
"## Part 5 — Prove it causally\n",
"\n",
"> **Objectives.** Move from correlation to causation. Ablate the induction heads and measure the accuracy drop; ablate non-induction heads as a control and measure (almost) nothing; verify the circuit generalizes out-of-distribution; localize the information flow with an activation patch built from scratch.\n",
"\n",
"Finding a head whose pattern *looks* like induction is correlational. To claim the head *causes* the copying, we intervene: zero its contribution and see if the behavior breaks. This is the single most important move in mech interp, the line between \"the model represents X\" and \"the model uses X\".\n"
]
},
{
"cell_type": "markdown",
"id": "42d228e6",
"metadata": {},
"source": [
"### Ablate the circuit, and a control\n",
"\n",
"We already wired `ablate={layer: [heads]}` into the model: it zeros those heads' outputs before the output projection. First the lesson from \"Before you start\" question 3: ablating the *single* strongest head barely moves accuracy, because the layer has redundant induction heads (the same backup-head phenomenon as the IOI circuit). To see the damage we ablate the whole induction layer.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "4f16b430",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.824476Z",
"iopub.status.busy": "2026-06-10T20:56:10.824387Z",
"iopub.status.idle": "2026-06-10T20:56:10.836071Z",
"shell.execute_reply": "2026-06-10T20:56:10.835722Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"layer-1 induction heads (score > 0.5): [0, 1, 2, 3]\n",
"\n",
"clean: loss 0.000 · acc 100.0%\n",
"ablate single best head: loss 0.011 · acc 99.9% (redundancy absorbs it)\n",
"ablate ALL layer-1 ind heads: loss 0.901 · acc 77.4% (circuit destroyed)\n"
]
}
],
"source": [
"# Which layer-1 heads are induction heads (score > 0.5)? Ablate the lot.\n",
"l1_scores = scores[1]\n",
"induction_heads = [h for h in range(model.n_heads) if l1_scores[h] > 0.5]\n",
"print(f\"layer-1 induction heads (score > 0.5): {induction_heads}\")\n",
"\n",
"clean_loss, clean_acc = eval_metrics(model, eval_batch)\n",
"single_loss, single_acc = eval_metrics(model, eval_batch, ablate={1: [best_head]})\n",
"all_loss, all_acc = eval_metrics(model, eval_batch, ablate={1: induction_heads})\n",
"print(f\"\\nclean: loss {clean_loss:.3f} · acc {clean_acc:.1%}\")\n",
"print(f\"ablate single best head: loss {single_loss:.3f} · acc {single_acc:.1%} (redundancy absorbs it)\")\n",
"print(f\"ablate ALL layer-1 ind heads: loss {all_loss:.3f} · acc {all_acc:.1%} (circuit destroyed)\")"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "d37dd33f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.837091Z",
"iopub.status.busy": "2026-06-10T20:56:10.836987Z",
"iopub.status.idle": "2026-06-10T20:56:10.845706Z",
"shell.execute_reply": "2026-06-10T20:56:10.845404Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 5.1 ablation destroys the circuit\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Assert the causal claim: knocking out the induction layer wrecks accuracy; the clean model is near-perfect.\n",
"def _ablation_breaks_it():\n",
" cl, ca = eval_metrics(model, eval_batch)\n",
" al, aa = eval_metrics(model, eval_batch, ablate={1: induction_heads})\n",
" assert ca > 0.95, f\"clean accuracy should be near-perfect, got {ca:.1%}\"\n",
" assert aa < ca - 0.15, \\\n",
" f\"ablating the induction heads should drop accuracy by >15 points; \" \\\n",
" f\"got clean {ca:.1%} -> ablated {aa:.1%}. If not, you ablated the wrong heads.\"\n",
"check(\"5.1 ablation destroys the circuit\", _ablation_breaks_it)"
]
},
{
"cell_type": "markdown",
"id": "afcda015",
"metadata": {},
"source": [
"> **Interpretation.** Removing one induction head barely dents accuracy: the layer built backups. Removing the whole induction layer collapses accuracy. The redundancy is itself a finding, the field calls these \"backup heads\", and it is why single-component ablations can mislead you into thinking a component does not matter when it does.\n"
]
},
{
"cell_type": "markdown",
"id": "2099ec88",
"metadata": {},
"source": [
"### Exercise 22.3 — A control ablation in the other layer\n",
"`Difficulty 2/5 · ~8 min`\n",
"\n",
"A causal claim needs a control: ablating something *less relevant* should do less. Layer 1 carries induction; ablating it hurts. Ablate the head in **layer 0** with the *lowest* induction-offset score and confirm the accuracy drop is smaller than ablating the whole induction layer. Return `(control_acc_drop, induction_acc_drop)` so we can compare.\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "b06209ba",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.846736Z",
"iopub.status.busy": "2026-06-10T20:56:10.846657Z",
"iopub.status.idle": "2026-06-10T20:56:10.856018Z",
"shell.execute_reply": "2026-06-10T20:56:10.855675Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 22.3 control ablation: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def control_vs_induction(model, batch, induction_heads):\n",
" \"\"\"Return (control_drop, induction_drop): accuracy lost by ablating the weakest\n",
" layer-0 head vs ablating all layer-1 induction heads.\"\"\"\n",
" with torch.no_grad():\n",
" model(batch)\n",
" l0 = offset_score(model.layers[0].last_pattern, SEQ_LEN // 2 - 1) # induction-offset scores in L0\n",
" weak_head = int(l0.argmin())\n",
" _, clean_acc = eval_metrics(model, batch)\n",
" # TODO 1: ablated accuracy when zeroing layer-0 head `weak_head` (ablate={0: [weak_head]})\n",
" control_acc = None\n",
" attempted(control_acc)\n",
" # TODO 2: ablated accuracy when zeroing all layer-1 induction heads (ablate={1: induction_heads})\n",
" induction_acc = None\n",
" attempted(induction_acc)\n",
" return clean_acc - control_acc, clean_acc - induction_acc\n",
"\n",
"def _control():\n",
" cd, idd = control_vs_induction(model, eval_batch, induction_heads)\n",
" assert idd > cd, \\\n",
" f\"ablating the induction layer ({idd:.1%} drop) should hurt more than the control \" \\\n",
" f\"({cd:.1%} drop). If not, the 'induction head' you found is not load-bearing.\"\n",
"check(\"22.3 control ablation\", _control)"
]
},
{
"cell_type": "markdown",
"id": "43228618",
"metadata": {},
"source": [
"Hint 1 (conceptual)You already have `eval_metrics(model, batch, ablate=...)`. The `ablate` dict maps a layer index to a list of head indices. Call it twice with different dicts and read the accuracy (second return value).\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"_, control_acc = eval_metrics(model, batch, ablate={0: [weak_head]})\n",
"_, induction_acc = eval_metrics(model, batch, ablate={1: induction_heads})\n",
"```\n",
"\n",
"\n",
"Help — \"the control drop is as big as the induction drop\"In this minimal model, layer-0 heads also contribute to the copy (they route the prev-token signal). The test only requires the induction-layer drop to be *larger*, which holds robustly. If they tie, re-run training: the loss curve must have reached < 0.1.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "af6e900e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.856994Z",
"iopub.status.busy": "2026-06-10T20:56:10.856918Z",
"iopub.status.idle": "2026-06-10T20:56:10.882435Z",
"shell.execute_reply": "2026-06-10T20:56:10.882007Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 22.3 control ablation\n",
"control ablation accuracy drop: 0.1%\n",
"induction ablation accuracy drop: 22.6%\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines control_vs_induction; the check re-verifies the reference.\n",
"def control_vs_induction(model, batch, induction_heads):\n",
" with torch.no_grad():\n",
" model(batch)\n",
" l0 = offset_score(model.layers[0].last_pattern, SEQ_LEN // 2 - 1)\n",
" weak_head = int(l0.argmin())\n",
" _, clean_acc = eval_metrics(model, batch)\n",
" _, control_acc = eval_metrics(model, batch, ablate={0: [weak_head]})\n",
" _, induction_acc = eval_metrics(model, batch, ablate={1: induction_heads})\n",
" return clean_acc - control_acc, clean_acc - induction_acc\n",
"\n",
"check(\"22.3 control ablation\", _control, required=True)\n",
"cd, idd = control_vs_induction(model, eval_batch, induction_heads)\n",
"print(f\"control ablation accuracy drop: {cd:.1%}\")\n",
"print(f\"induction ablation accuracy drop: {idd:.1%}\")"
]
},
{
"cell_type": "markdown",
"id": "27b111c6",
"metadata": {},
"source": [
"> **Interpretation.** The induction-layer ablation costs more accuracy than the control. That asymmetry is the causal evidence: the heads we identified by their *pattern* are also the heads that, when removed, *break the behavior*. Correlation plus intervention.\n"
]
},
{
"cell_type": "markdown",
"id": "41e073f3",
"metadata": {},
"source": [
"### Out-of-distribution: is it an algorithm or a lookup table?\n",
"\n",
"The strongest test that a circuit is real: it should generalize to inputs unlike anything in training, as long as they share the structure it exploits. We feed the model repeat sequences built from a fresh, far-away seed (different random tokens, same repeat structure). If it copies them perfectly, it learned the algorithm, not a memorized table.\n"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "471ad8b3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.883292Z",
"iopub.status.busy": "2026-06-10T20:56:10.883208Z",
"iopub.status.idle": "2026-06-10T20:56:10.893914Z",
"shell.execute_reply": "2026-06-10T20:56:10.893447Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OOD repeat sequences: acc 100.0% (algorithm generalizes)\n",
"non-repeating sequences: acc 1.7% (nothing to copy -> chance ~2.0%)\n"
]
}
],
"source": [
"# OOD repeat sequences: brand-new random tokens, same repeat structure.\n",
"gen_ood = torch.Generator().manual_seed(99999) # far from training seeds\n",
"ood_batch = make_batch(BATCH, SEQ_LEN, VOCAB, gen_ood)\n",
"ood_loss, ood_acc = eval_metrics(model, ood_batch)\n",
"\n",
"# Negative control: NON-repeating sequences (both halves independently random). The copy rule\n",
"# cannot help here, so accuracy on the \"second half\" should be near chance.\n",
"torch.manual_seed(SEED)\n",
"nonrep = torch.randint(1, VOCAB, (BATCH, SEQ_LEN))\n",
"nonrep_loss, nonrep_acc = eval_metrics(model, nonrep)\n",
"print(f\"OOD repeat sequences: acc {ood_acc:.1%} (algorithm generalizes)\")\n",
"print(f\"non-repeating sequences: acc {nonrep_acc:.1%} (nothing to copy -> chance ~{1/(VOCAB-1):.1%})\")"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "5e01bfdb",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.894779Z",
"iopub.status.busy": "2026-06-10T20:56:10.894703Z",
"iopub.status.idle": "2026-06-10T20:56:10.904294Z",
"shell.execute_reply": "2026-06-10T20:56:10.903959Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 5.2 circuit generalizes OOD and fails on non-repeats\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def _ood_generalizes():\n",
" _, oa = eval_metrics(model, make_batch(BATCH, SEQ_LEN, VOCAB, torch.Generator().manual_seed(99999)))\n",
" assert oa > 0.95, \\\n",
" f\"a real induction circuit copies unseen repeat sequences; OOD accuracy {oa:.1%} is too low\"\n",
" torch.manual_seed(SEED)\n",
" _, na = eval_metrics(model, torch.randint(1, VOCAB, (BATCH, SEQ_LEN)))\n",
" assert na < 0.2, \\\n",
" f\"on non-repeating data there is nothing to copy; accuracy {na:.1%} should be near chance\"\n",
"check(\"5.2 circuit generalizes OOD and fails on non-repeats\", _ood_generalizes)"
]
},
{
"cell_type": "markdown",
"id": "46b8fbca",
"metadata": {},
"source": [
"> **Interpretation.** Perfect copying on never-seen repeat sequences, chance-level on non-repeating sequences. That double dissociation is the signature of a learned *algorithm*: it fires exactly when its structural precondition (a prior occurrence to copy from) is present, and not otherwise. A lookup table would not generalize to fresh tokens; a circuit does.\n"
]
},
{
"cell_type": "markdown",
"id": "974f0276",
"metadata": {},
"source": [
"### Activation patching, from scratch\n",
"\n",
"Ablation asks \"does this component matter?\". Activation patching asks the sharper question \"*where* does the information that determines the answer live?\". The recipe:\n",
"\n",
"1. A **clean** prompt the model gets right, and a **corrupt** prompt where one token is changed so the answer flips.\n",
"2. Cache the clean run's residual stream.\n",
"3. Run the corrupt prompt but **patch in** the clean activation at one site, and measure how much the logit difference recovers.\n",
"\n",
"We build a clean/corrupt pair on our repeat data: take a repeat sequence (clean), then change the single first-half token that is the induction *source* for one query position. That flips the model's prediction at that position. Patching the clean residual back in at the right layer and position should recover it.\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "6bb809f4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.905271Z",
"iopub.status.busy": "2026-06-10T20:56:10.905187Z",
"iopub.status.idle": "2026-06-10T20:56:10.908523Z",
"shell.execute_reply": "2026-06-10T20:56:10.908171Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"query position 40: clean answer token 22, corrupt flips the source to 23\n"
]
}
],
"source": [
"# Build a clean/corrupt pair around one query position.\n",
"torch.manual_seed(SEED)\n",
"hf = SEQ_LEN // 2\n",
"gpair = torch.Generator().manual_seed(42)\n",
"first = torch.randint(1, VOCAB, (1, hf), generator=gpair)\n",
"clean_seq = torch.cat([first, first.clone()], dim=-1) # (1, seq_len)\n",
"q = 40 # a second-half query position\n",
"src = q + 1 - hf # the first-half token it copies from\n",
"correct = clean_seq[0, q + 1].item() # clean next token at q (== first[src])\n",
"new = (correct % (VOCAB - 1)) + 1 # a different token id\n",
"corrupt_seq = clean_seq.clone()\n",
"corrupt_seq[0, src] = new # corrupt the induction source\n",
"incorrect = new\n",
"print(f\"query position {q}: clean answer token {correct}, corrupt flips the source to {incorrect}\")"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "bdd5fd13",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.909190Z",
"iopub.status.busy": "2026-06-10T20:56:10.909111Z",
"iopub.status.idle": "2026-06-10T20:56:10.912819Z",
"shell.execute_reply": "2026-06-10T20:56:10.912564Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"clean logit diff: +23.97 (model prefers the correct token)\n",
"corrupt logit diff: -20.07 (the swap flips it)\n"
]
}
],
"source": [
"# Cache clean residual-stream-after-each-layer, then measure clean/corrupt logit diffs.\n",
"@torch.no_grad()\n",
"def run_capturing_resid(model, idx):\n",
" B, T = idx.shape\n",
" x = model.tok_emb(idx) + model.pos_emb(torch.arange(T))\n",
" resid = [] # resid[l] = stream entering layer l\n",
" for layer in model.layers:\n",
" resid.append(x.clone())\n",
" x = x + layer(x)\n",
" resid.append(x.clone()) # final stream (entering unembed)\n",
" return model.unembed(x), resid\n",
"\n",
"model.eval()\n",
"clean_logits, clean_resid = run_capturing_resid(model, clean_seq)\n",
"corrupt_logits, _ = run_capturing_resid(model, corrupt_seq)\n",
"def logit_diff(logits): # logit(correct) - logit(incorrect) at q\n",
" return (logits[0, q, correct] - logits[0, q, incorrect]).item()\n",
"print(f\"clean logit diff: {logit_diff(clean_logits):+.2f} (model prefers the correct token)\")\n",
"print(f\"corrupt logit diff: {logit_diff(corrupt_logits):+.2f} (the swap flips it)\")"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "85bf5575",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.913674Z",
"iopub.status.busy": "2026-06-10T20:56:10.913604Z",
"iopub.status.idle": "2026-06-10T20:56:10.917588Z",
"shell.execute_reply": "2026-06-10T20:56:10.917348Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"recovery from patching the clean residual at the source position:\n",
" patch after layer 0 at position 9: recovery +36%\n",
" patch after layer 1 at position 9: recovery +0%\n"
]
}
],
"source": [
"# Patch the clean residual at (layer l, position src) into the corrupt run; measure recovery.\n",
"@torch.no_grad()\n",
"def patch_resid(model, idx, layer_to_patch, pos, clean_value):\n",
" B, T = idx.shape\n",
" x = model.tok_emb(idx) + model.pos_emb(torch.arange(T))\n",
" for l, layer in enumerate(model.layers):\n",
" x = x + layer(x)\n",
" if l == layer_to_patch:\n",
" x = x.clone(); x[:, pos, :] = clean_value # overwrite this position's stream\n",
" return model.unembed(x)\n",
"\n",
"cd_lp, kd_lp = logit_diff(clean_logits), logit_diff(corrupt_logits)\n",
"print(\"recovery from patching the clean residual at the source position:\")\n",
"for l in range(model.n_layers):\n",
" patched = patch_resid(model, corrupt_seq, l, src, clean_resid[l + 1][:, src, :])\n",
" recovery = (logit_diff(patched) - kd_lp) / (cd_lp - kd_lp) # 0 = no recovery, 1 = full recovery\n",
" print(f\" patch after layer {l} at position {src}: recovery {recovery:+.0%}\")"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "6354d547",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.918435Z",
"iopub.status.busy": "2026-06-10T20:56:10.918367Z",
"iopub.status.idle": "2026-06-10T20:56:10.922175Z",
"shell.execute_reply": "2026-06-10T20:56:10.921912Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 5.3 patching localizes the information\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def _patching_localizes():\n",
" cd2, kd2 = logit_diff(clean_logits), logit_diff(corrupt_logits)\n",
" assert cd2 > kd2 + 5, \"clean and corrupt logit diffs should be well separated for a clean experiment\"\n",
" rec0 = (logit_diff(patch_resid(model, corrupt_seq, 0, src, clean_resid[1][:, src, :])) - kd2) / (cd2 - kd2)\n",
" rec1 = (logit_diff(patch_resid(model, corrupt_seq, 1, src, clean_resid[2][:, src, :])) - kd2) / (cd2 - kd2)\n",
" assert rec0 > rec1, \\\n",
" f\"patching the EARLY residual at the source recovers more (it is where the prev-token info \" \\\n",
" f\"enters): got layer0 {rec0:+.0%} vs layer1 {rec1:+.0%}\"\n",
"check(\"5.3 patching localizes the information\", _patching_localizes)"
]
},
{
"cell_type": "markdown",
"id": "501b7854",
"metadata": {},
"source": [
"> **Interpretation.** Patching the clean residual at the source position *after layer 0* recovers a large chunk of the clean answer, while patching after layer 1 at that same position recovers little. That localizes the relevant information to the early stream at the source token: layer 0 has written the \"what token am I\" signal there, the layer-1 induction head reads it, and restoring it to clean rescues the prediction. We just traced one wire of the circuit with a causal experiment.\n",
"\n",
"> **Key takeaways.** Ablation answers \"does it matter\", patching answers \"where is the information\". Redundancy (backup heads) means single-component ablations under-read importance. OOD generalization plus a non-repeat negative control is the proof a circuit is an algorithm. Every causal claim is an intervention, never a correlation.\n"
]
},
{
"cell_type": "markdown",
"id": "5e52d05a",
"metadata": {},
"source": [
"## Part 6 — Read the circuit: the OV decomposition\n",
"\n",
"> **Objectives.** Decompose the induction head's OV circuit and show, in the vocabulary basis, that the head copies whatever token it attends to. This is the \"what does it write\" half of a head (the QK circuit is the \"where does it look\" half).\n",
"\n",
"A single head factors into two independent circuits. The **QK circuit** $W_Q W_K^\\top$ decides *which* position to attend to. The **OV circuit** $W_V W_O$ decides *what to write* when a position is attended to. For an induction head, the QK circuit implements \"find the position after my previous occurrence\" (the stripe we already saw), and the OV circuit should implement *copy*: project it through the embedding and unembedding and it should map each token to itself.\n"
]
},
{
"cell_type": "markdown",
"id": "4e62f9b6",
"metadata": {},
"source": [
"### The OV circuit in the vocabulary basis\n",
"\n",
"The head reads value vectors from the residual stream via $W_V$ and writes via $W_O$. Composed with the token embedding $W_E$ (input side) and unembedding $W_U$ (output side), the full token-to-token map is:\n",
"\n",
"$$\\text{OV}_{\\text{vocab}} = W_U \\, W_O^{(h)} W_V^{(h)} \\, W_E^\\top \\in \\mathbb{R}^{\\text{vocab} \\times \\text{vocab}}$$\n",
"\n",
"If the head copies, this matrix is close to the identity up to scale: row `i` (source token `i`) has its largest entry in column `i` (it writes \"predict token `i`\"). We measure the fraction of rows whose argmax is on the diagonal, the *copying fraction*.\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "0e9cd054",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.922829Z",
"iopub.status.busy": "2026-06-10T20:56:10.922762Z",
"iopub.status.idle": "2026-06-10T20:56:10.925820Z",
"shell.execute_reply": "2026-06-10T20:56:10.925576Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OV copying fraction for the induction head: 96%\n",
"(fraction of source tokens whose top-predicted next token is themselves)\n"
]
}
],
"source": [
"def ov_vocab_matrix(model, layer, head):\n",
" \"\"\"Token-to-token map of one head's OV circuit: (vocab, vocab).\"\"\"\n",
" dh = model.d_head\n",
" W_V = model.layers[layer].W_V.weight.detach() # (d_model, d_model): v = W_V @ x (rows = out dims)\n",
" W_O = model.layers[layer].W_O.weight.detach() # (d_model, d_model)\n",
" W_V_h = W_V[head * dh:(head + 1) * dh, :] # (d_head, d_model): this head's value read\n",
" W_O_h = W_O[:, head * dh:(head + 1) * dh] # (d_model, d_head): this head's output write\n",
" W_OV = W_O_h @ W_V_h # (d_model, d_model)\n",
" W_E = model.tok_emb.weight.detach() # (vocab, d_model)\n",
" W_U = model.unembed.weight.detach() # (vocab, d_model)\n",
" return W_U @ W_OV @ W_E.T # (vocab, vocab)\n",
"\n",
"ov = ov_vocab_matrix(model, best_layer, best_head)\n",
"copying_fraction = (ov.argmax(dim=-1) == torch.arange(ov.shape[0])).float().mean().item()\n",
"print(f\"OV copying fraction for the induction head: {copying_fraction:.0%}\")\n",
"print(\"(fraction of source tokens whose top-predicted next token is themselves)\")"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "74981f2c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:10.926578Z",
"iopub.status.busy": "2026-06-10T20:56:10.926510Z",
"iopub.status.idle": "2026-06-10T20:56:11.002462Z",
"shell.execute_reply": "2026-06-10T20:56:11.002101Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAG4CAYAAABhHB/FAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAYOBJREFUeJzt3XlYVOXbB/DvgDAMy6AICCiyae67iWtqkrivr6VZqZm541YuleJSkZqllbmVYuVCVtqqaaa4pOaS+64ouOCGgICs87x/GPNrZD3PDMyMfD9e57qcM+c+5z5nznDPc7ZHJYQQICIiIotjY+4EiIiIKH8s0kRERBaKRZqIiMhCsUgTERFZKBZpIiIiC8UiTUREZKFYpImIiCwUizQREZGFYpEmIiKyUCzSTwh/f38MHjy41Jd75coVqFQqREZGmnS+kZGRUKlUuHLliknnaw0GDx4MZ2dnc6dhoF27dmjXrp250yAqc1ikJZw6dQovvfQSKleuDLVaDR8fHwwcOBCnTp3ST5OVlQV3d3e0bt26wPkIIeDr64vGjRuXRtql5rfffsPMmTPNnYbZpKSkIDw8HJ06dYKbm1uJ/IixRGlpaVi8eDE6duwIb29vuLi4oFGjRliyZAlycnLMnR6RVWKRVuiHH35A48aNsX37dgwZMgSff/45hg4dih07dqBx48bYuHEjAMDOzg79+vXDX3/9hatXr+Y7r127duHatWt46aWXjM7r3LlzWLFihdHzUcrPzw8PHz7Eyy+/rB/322+/YdasWaWei6W4e/cuZs+ejTNnzqBBgwbmTqfUXL58GWPHjoUQAhMnTsSHH36IgIAAjBo1Cq+++qq50yOySuXMnYA1uXTpEl5++WUEBgZi165d8PDw0L83btw4tGnTBi+//DKOHz+OwMBADBw4EEuXLsW6deswderUPPNbu3YtbGxs0L9/f6NzU6vVRU6TmpoKJycno5f1XyqVCg4ODiadpzUobFt6e3vj5s2b8PLywqFDh/D000+Xcnbm4eXlhRMnTqBOnTr6ccOHD8err76KVatWYfr06ahWrZoZMySyPmxJKzB//nykpaVh+fLlBgUaANzd3bFs2TKkpqZi3rx5AIBWrVrB398fa9euzTOvrKwsfPfdd2jfvj18fHwKXa5Op8OiRYtQr149ODg4wMPDA506dcKhQ4f00zx+Tjr3nG50dDRGjRoFT09PVKlSRf/+5s2b0bZtW7i4uECr1eLpp582yLOgc9yPn5t8/Jz04MGDsXjxYgCPCnjuYAo//vgjunbtCh8fH6jVagQFBWHOnDkGh1LDw8NhZ2eHO3fu5Il//fXXUb58eaSnp+vHbd68GW3atIGTkxNcXFzQtWtXg9MWuevk7OyMS5cuoUuXLnBxccHAgQMLzFOtVsPLy8vo9b1+/Tp69eoFZ2dneHh44I033shz2Fin02HhwoWoU6cOHBwcUKlSJQwfPhz37983mK442y7X8uXLERQUBI1Gg2bNmmH37t3Fytfd3d2gQOfq3bs3AODMmTMG4y9duoRLly4VOd+EhAS88cYbqFevHpydnaHVatG5c2ccO3asWHkBwDfffINmzZrB0dERFSpUwDPPPIOtW7caTPP555+jTp06+lNYo0ePRmJiosE07dq1Q926dXH48GG0bNkSGo0GAQEBWLp0qX6alJQUODk5Ydy4cXnyuHbtGmxtbREREVFgrrnfqQ8//BCLFy9GYGAgHB0d0bFjR8TFxUEIgTlz5qBKlSrQaDTo2bMnEhISDOZR3M+7OOtD5sUircDPP/8Mf39/tGnTJt/3n3nmGfj7++PXX38F8KhIvfjiizhx4kSeP/xbtmxBQkJCoX/scw0dOhTjx4+Hr68v5s6di6lTp8LBwQH79+8vMnbUqFE4ffo0ZsyYoW/NR0ZGomvXrkhISMC0adPwwQcfoGHDhtiyZUuR8yvK8OHD8dxzzwEAvv76a/1gCpGRkXB2dsbEiROxaNEiNGnSxGC9AODll19GdnY2oqKiDGIzMzPx3XffoW/fvvqW/9dff42uXbvC2dkZc+fOxfTp03H69Gm0bt06zwVr2dnZCA0NhaenJz788EP07dvXJOtUkJycHISGhqJixYr48MMP0bZtWyxYsADLly83mG748OF488030apVKyxatAhDhgzBmjVrEBoaiqysLP10xdl2APDll19i+PDh8PLywrx589CqVSv06NEDcXFx0usSHx8P4FER/68OHTqgQ4cORcZfvnwZmzZtQrdu3fDRRx/hzTffxIkTJ9C2bVvcuHGjyPhZs2bh5Zdfhp2dHWbPno1Zs2bB19cXf/75p36amTNnYvTo0fDx8cGCBQvQt29fLFu2DB07djTYjgBw//59dOnSBU2aNMG8efNQpUoVjBw5EitXrgQAODs7o3fv3oiKispTFNetWwchRLG+92vWrMHnn3+OsWPHYtKkSYiOjsbzzz+Pd955B1u2bMGUKVPw+uuv4+eff8Ybb7xhEFvcz7s460NmJqhYEhMTBQDRs2fPQqfr0aOHACCSk5OFEEKcOnVKABDTpk0zmK5///7CwcFBJCUlFTq/P//8UwAQYWFhed7T6XT6//v5+YlBgwbpX69atUoAEK1btxbZ2dkG6+Hi4iKCg4PFw4cPiz2/XG3bthVt27bVv46JiREAxKpVq/TjRo8eLYzdtXLzj4mJ0Y9LS0vLM93w4cOFo6OjSE9P149r0aKFCA4ONpjuhx9+EADEjh07hBBCPHjwQJQvX14MGzbMYLr4+Hjh6upqMH7QoEECgJg6dari9Th48GCe7VOU3OXNnj3bYHyjRo1EkyZN9K93794tAIg1a9YYTLdly5Y844uz7TIzM4Wnp6do2LChyMjI0E+3fPlyAcDgcy+ujIwMUbt2bREQECCysrIM3vPz8xN+fn5FziM9PV3k5OQYjIuJiRFqtTrPNnrchQsXhI2Njejdu3eeeeTu77dv3xb29vaiY8eOBtN89tlnAoBYuXKlflzbtm0FALFgwQKDdWzYsKHw9PQUmZmZQgghfv/9dwFAbN682WCZ9evXL3I75n6nPDw8RGJion78tGnTBADRoEEDg205YMAAYW9vb/AdKO53pbjrQ+bDlnQxPXjwAADg4uJS6HS57ycnJwMAateujUaNGmH9+vX6aVJTU/HTTz+hW7du0Gq1hc7v+++/h0qlQnh4eJ73inMYediwYbC1tdW/3rZtGx48eKBvjSudnzlpNBr9/x88eIC7d++iTZs2SEtLw9mzZ/XvvfLKKzhw4IDBodQ1a9bA19cXbdu2BfBoOyQmJmLAgAG4e/eufrC1tUVwcDB27NiRZ/kjR44swbXLa8SIEQav27Rpg8uXL+tfb9iwAa6urnjuuecM1qFJkyZwdnY2WIfibLtDhw7h9u3bGDFiBOzt7fXTDx48GK6urlLrMGbMGJw+fRqfffYZypUzvATmypUrxbrFTq1Ww8bm0Z+qnJwc3Lt3D87OzqhRowaOHDlSaOymTZug0+kwY8YM/Txy5e7vf/zxBzIzMzF+/HiDaYYNGwatVqs/MparXLlyGD58uP61vb09hg8fjtu3b+Pw4cMAgJCQEPj4+GDNmjX66U6ePInjx48X+0LRfv36GWz34OBgAMBLL71ksC2Dg4ORmZmJ69ev68cV97tS3PUh82GRLqbc4ptbrAuSXzEfOHAgYmJi8NdffwF49IcjLS2tWIe8Ll26BB8fH7i5uUnlHRAQkGd+AFC3bl2p+ZnTqVOn0Lt3b7i6ukKr1cLDw0P/By8pKUk/3QsvvAC1Wq3/A5mUlIRffvkFAwcO1P9hvnDhAgDg2WefhYeHh8GwdetW3L5922DZ5cqVMzinX9Jyrz34rwoVKhica75w4QKSkpLg6emZZx1SUlIM1qE42y73LoTq1asbLNfOzg6BgYGK12H+/PlYsWIF5syZgy5duiiOz6XT6fDxxx+jevXqUKvVcHd3h4eHB44fP27wuefn0qVLsLGxQe3atQucJne9a9SoYTDe3t4egYGBee7O8PHxyXPR4FNPPQUA+h8dNjY2GDhwoP67Djz6oejg4IB+/foVvdIAqlatavA6t2D7+vrmO/6/+0ZxvyvFXR8yH17dXUyurq7w9vbG8ePHC53u+PHjqFy5skELecCAAZg8eTLWrl2Lli1bYu3atahQoYJRf7iK67+/qJUoqFWdk5Nj0DIvLYmJiWjbti20Wi1mz56NoKAgODg44MiRI5gyZQp0Op1+2goVKqBbt25Ys2YNZsyYge+++w4ZGRkGLZjc6b/++ut8L/J6vNX339ZcaSjONtbpdPD09DRorf1XbpFXsu1MJTIyElOmTMGIESPwzjvvGDWv999/H9OnT8err76KOXPmwM3NDTY2Nhg/fnyJ5G4qr7zyCubPn49NmzZhwIABWLt2Lbp161bsoxIF7QMFjRdCADDP500lh0VagW7dumHFihXYs2dPvg8p2b17N65cuWJw6Ah49Eu1ffv22LBhA6ZPn45t27Zh8ODBBocUCxIUFITff/8dCQkJ0q3px+cHPDr0VtjtMBUqVMhzZSvwqNVRVKuqJA6b79y5E/fu3cMPP/yAZ555Rj8+JiYm3+lfeeUV9OzZEwcPHsSaNWvQqFEjgyuPc7eDp6cnQkJCTJ5vaQgKCsIff/yBVq1aFfpjrLjbzs/PD8CjFvqzzz6rH5+VlYWYmJhi3/P9448/4rXXXkOfPn30V/obI/cuiC+//NJgfGJiYp6L0R4XFBQEnU6H06dPo2HDhvlOk7ve586dM9i3MzMzERMTk2f/uHHjRp5b8M6fPw/g0V0RuerWrYtGjRphzZo1qFKlCmJjY/Hpp58Wub7GUvpdKe76kHnwcLcCb775JjQaDYYPH4579+4ZvJeQkIARI0bA0dERb775Zp7YgQMH4vbt2xg+fDiysrKKdagbAPr27QshRL4PB8n95axEx44d4eLigoiICINbkR6fX1BQEPbv34/MzEz9uF9++aVYV/nmftnzK/KyclsP/80xMzMTn3/+eb7Td+7cGe7u7pg7dy6io6PznAcMDQ2FVqvF+++/n+fqXQD53sJlaZ5//nnk5ORgzpw5ed7Lzs7Wb//ibrumTZvCw8MDS5cuNfjcIyMji/1Z7tq1C/3798czzzyDNWvWFHr0obi3YNna2ubZ1zds2GBwDrYgvXr1go2NDWbPnp2nBZk7z5CQENjb2+OTTz4xWM6XX36JpKQkdO3a1SAuOzsby5Yt07/OzMzEsmXL4OHhgSZNmhhM+/LLL2Pr1q1YuHAhKlasiM6dOxeZs7GUfleUrA+VPrakFahevTpWr16NgQMHol69ehg6dCgCAgJw5coVfPnll7h79y7WrVunb6X9V9++fTFq1Cj8+OOP8PX1NfiFW5j27dvj5ZdfxieffIILFy6gU6dO0Ol02L17N9q3b48xY8YoWgetVouPP/4Yr732Gp5++mm8+OKLqFChAo4dO4a0tDSsXr0aAPDaa6/hu+++Q6dOnfD888/j0qVL+Oabb/Jdt8flfrHDwsIQGhoKW1tb/QNbBg8ejNWrVyMmJkbRr/SWLVuiQoUKGDRoEMLCwqBSqfD1118X+EPFzs4O/fv3x2effQZbW1sMGDAgz3ZYsmQJXn75ZTRu3Bj9+/eHh4cHYmNj8euvv6JVq1b47LPPip3f4z777DMkJibqbxH6+eefce3aNQDA2LFjpS/E+q+2bdti+PDhiIiIwNGjR9GxY0fY2dnhwoUL2LBhAxYtWoT/+7//K/a2s7Ozw7vvvovhw4fj2WefxQsvvICYmBisWrWqWOekr169ih49ekClUuH//u//sGHDBoP369evj/r16+tf595+VdR5z27dumH27NkYMmQIWrZsiRMnTmDNmjXFyqlatWp4++23MWfOHLRp0wZ9+vSBWq3GwYMH4ePjg4iICHh4eGDatGmYNWsWOnXqhB49euDcuXP4/PPP8fTTT+f5gefj44O5c+fiypUreOqppxAVFYWjR49i+fLlsLOzM5j2xRdfxOTJk7Fx40aMHDkyz/slQel3Rcn6kBmY5ZpyK3f8+HExYMAA4e3tLezs7ISXl5cYMGCAOHHiRKFx/fr1EwDE5MmTFS0vOztbzJ8/X9SsWVPY29sLDw8P0blzZ3H48GH9NAXdgnXw4MF85/nTTz+Jli1bCo1GI7RarWjWrJlYt26dwTQLFiwQlStXFmq1WrRq1UocOnSoWLdgZWdni7FjxwoPDw+hUqkMbsfq27ev0Gg04v79+4Wuc363YO3du1c0b95caDQa4ePjIyZPnqy/1SX31qr/+vvvvwUA0bFjxwKXs2PHDhEaGipcXV2Fg4ODCAoKEoMHDxaHDh3STzNo0CDh5ORUaL6P8/PzEwDyHf67TvkpaHnh4eH53tq2fPly0aRJE6HRaISLi4uoV6+emDx5srhx44Z+GiXb7vPPPxcBAQFCrVaLpk2bil27duX53POzY8eOAtcZgAgPD8+zjYp7C9akSZOEt7e30Gg0olWrVmLfvn3FyinXypUrRaNGjYRarRYVKlQQbdu2Fdu2bTOY5rPPPhM1a9YUdnZ2olKlSmLkyJF59tO2bduKOnXqiEOHDokWLVoIBwcH4efnJz777LMCl92lSxcBQPz111/FyjX3OzV//nyD8bnbd8OGDQbj8/uuF/fzllkfKl0qISSOmRJJqlSpkv6CmpJ27NgxNGzYEF999ZXBs8WJZLVr1w53797FyZMnix3Tu3dvnDhxAhcvXizBzOTIrA+VLp6TplJz6tQpPHz4EFOmTCmV5a1YsQLOzs7o06dPqSyP6HE3b97Er7/+yh+JJI3npKnU1KlTR/+Ql5L0888/4/Tp01i+fDnGjBlj8k5FiIoSExODvXv34osvvoCdnV2eOz6IiotFmp44Y8eOxa1bt9ClS5cy3WUmmU90dDSGDBmCqlWrYvXq1SbpcIXKJp6TJiIislA8J01ERGShWKSJiIgs1BN/Tlqn0+HGjRtwcXGx+F6eiIieREIIPHjwAD4+PiZ7Bn56errBk/Fk2dvb5+kR0JI88UX6xo0beXqNISKi0hcXF2eS3uTS09MREFAZ8fEJRs/Ly8sLMTExFluon/gindtl5Hj/iVDbqBXFXn0gd01do4pSYQCAw3flltnSU+4owfH7RU+TH3tbueVVcZS/TjH+odwyq7vILfOI5Pc/1DtbKi4xS753sX8S5LbN9TS5XPv5le5Rqdvp8tvmYuG9yxaoipPcOiZINu78JL8bW27ItybrVlD2NzGX0q9/hi4Dn1z9yKALX2NkZmYiPj4BV65+C63WUXo+yclp8Pd7HpmZmSzS5pJ7iFtto4baRtmHYG8j16WbRrKAAYBdKS/T3kYuTi0Z52ArX6Rll6mxldumstvG0TZvhx3FkaGTL0T2kocQ7VRyuToasY/LcLCV/1NlL3l01UFyHdWSy5PdT+2MOI2n9G9iLtmP39SnHLXODtA6y3XHCwCwgm47reLCscWLF8Pf3x8ODg4IDg7G33//be6UiIjI3HQ64wcLZ/FFOioqChMnTkR4eDiOHDmCBg0aIDQ0FLdv3zZ3akREZE4s0ub30UcfYdiwYRgyZAhq166NpUuXwtHREStXrjR3akRERCXKoot0ZmYmDh8+jJCQEP04GxsbhISEYN++ffnGZGRkIDk52WAgIqInkBDGDxbOoov03bt3kZOTg0qVKhmMr1SpEuLj4/ONiYiIgKurq37g7VdERE8onTDycDeLdKmbNm0akpKS9ENcXJy5UyIiIpJi0bdgubu7w9bWFrdu3TIYf+vWrQJ7lVGr1VCr5e79IyIiK2LsxV+8cMw49vb2aNKkCbZv364fp9PpsH37drRo0cKMmRERkdmVgau7LbolDQATJ07EoEGD0LRpUzRr1gwLFy5EamoqhgwZYu7UiIjInMpAS9rii/QLL7yAO3fuYMaMGYiPj0fDhg2xZcuWPBeTERERPWksvkgDwJgxYzBmzBhzp0FERJZEGNmSFmxJWww3ewGNwudGvz3gstSyrh+Wf+C7o23+F8QVJSlL7pm4p1Pk7iPvW1krFWfMs7tXJ12RikvP8ZeK83CQ26YjLuyRigu2aS0VBwBDgnKk4uZekOt94qdr5aXi0nPk/ij295frCAQAHmTbScXdSZdbnpdGbh//86bctmnhKd8xREilJKm4P265Si/TlFRCB5URhdaY2NJi0ReOERERlWVlpiVNRERPGF44RkREZKF0wrinhlnBE8dYpImIyDqVgZY0z0kTERFZKLakiYjIOpWBljSLNBERWSchjLvX2Qq6qmSRJiIi61QGWtI8J01ERGSh2JImIiLrxFuwiIiILBQPdxMREZG5sCVNRETWib1gPTkcbAAHG2XnH+6dlOs9p2pn+fMcke/I9UrVw7u8VFwjV7nebHw0WVJxhxLktikAJOCaVNyL/u5ScfNPq6XiXnB9ViqujUemVBwAZEvucrdsYqXipvo5S8WdSJLbps28r0vFAYAKHlJxcy7dlIprmV1VKu5WdqpUnL2NfI9UJ5NcpOJ8Ncp6XXso2ftZUVQ6HVRGFGljYktLmSnSRET0hBHCuHudreA+aZ6TJiIislBsSRMRkXUqA1d3s0gTEZF1KgNFmoe7iYiILBRb0kREZJ3KwBPH2JImIiLrlHu425hBgZycHEyfPh0BAQHQaDQICgrCnDlzIErwKnG2pImIyDrphJHnpJUV17lz52LJkiVYvXo16tSpg0OHDmHIkCFwdXVFWFiYfB6FYJEmIiIqhr/++gs9e/ZE165dAQD+/v5Yt24d/v777xJbJg93ExGRdcp9mIkxgwItW7bE9u3bcf78eQDAsWPHsGfPHnTu3Lkk1g4AW9JERGStTHQLVnKy4eOY1Wo11Oq8j7CdOnUqkpOTUbNmTdja2iInJwfvvfceBg4cKJ9DEdiSJiIi6yTE/67wlhn+bUn7+vrC1dVVP0REROS7uG+//RZr1qzB2rVrceTIEaxevRoffvghVq9eXWKryJY0ERGVaXFxcdBqtfrX+bWiAeDNN9/E1KlT0b9/fwBAvXr1cPXqVURERGDQoEElkhuLNBERWScTHe7WarUGRbogaWlpsLExPABta2sLXQk+uazMFOmETBUcbJUd3f/6jJ/Usrb+mSIVBwBrn0mUivv2olx3dc0rKutyLldKttyZEl9H+fsJZ7m0kopbd1UlFfeCn9y2+fGaXNz+e/ZScQBQWyu3zAPPlZeKe+F3uW41q2rkuip9bXd5qTgAaFBBbrtODagsFVer/H2puM033KTiOvvclYoDgB/i5LpxzVC4u2XoSujMaik/FrR79+547733ULVqVdSpUwf//PMPPvroI7z66qvyORShzBRpIiIiY3z66aeYPn06Ro0ahdu3b8PHxwfDhw/HjBkzSmyZLNJERGSdSvmxoC4uLli4cCEWLlwov0yFWKSJiMg6Cd2jwZh4C8ciTURE1okdbBAREZG5sCVNRETWqZSv7jYHFmkiIrJOPNxNRERE5sKWNBERWadS7k/aHFikiYjIOpWBw90s0kREZKWMvE8aln/hGM9JExERWSi2pImIyDrxcPeTQ+aztJHrPAkv+WvkAgF8csJZKm7eR6lSce9OcZKKu/lQbuee1/WCVBwAvBBVVSpuQk25Q1rP7f9QKu4Nf7mH7Qc5yx96a1AhWSrOc8MGqbi1DSdKxXVrekkqrte33lJxALD7/h2puOYV5XqWu57qKBUX80DuO9X5yDmpOACY41dBKm7LdWW5ZunkemkrEos0ERGRhSoDDzPhOWkiIiILxZY0ERFZJx7uJiIislBloEjzcDcREZGFYkuaiIisUxm4cIxFmoiIrJMQjwZj4i0cizQREVknnpMmIiIic2FLmoiIrFMZaEmzSBMRkXUSRl44ZlQPWqWDh7uJiIgsFFvSRERknXi4+8nh5aCDxlZZTywrriZKLau2o1zPMgCgtZeL++E9O6m4acOuSsVFra0sFTdn61NScQCwpPV1qbi7D+R6JRrq/Y5UXJ8qcj1SpWbLfx0PJ8j12LSn1UipuPScTKm4jAe2UnFVNXKfIQD095Nb5v4EuS9jarZUGLwkV3F5lSZygQAWn8+SihtRXVkXgWk5OfghQWpRhdPByCJtskxKTJkp0kRE9IQpAy1pnpMmIiKyUBZdpGfOnAmVSmUw1KxZ09xpERGRBRA6YfRg6Sz+cHedOnXwxx9/6F+XK2fxKRMRUWngY0HNr1y5cvDy8jJ3GkREZGl4Ttr8Lly4AB8fHwQGBmLgwIGIjY0tdPqMjAwkJycbDERERNbIoot0cHAwIiMjsWXLFixZsgQxMTFo06YNHjx4UGBMREQEXF1d9YOvr28pZkxERKUmtyVtzGDhLLpId+7cGf369UP9+vURGhqK3377DYmJifj2228LjJk2bRqSkpL0Q1xcXClmTEREpcYMRfr69et46aWXULFiRWg0GtSrVw+HDh0qgZV7xOLPSf9X+fLl8dRTT+HixYsFTqNWq6FWq0sxKyIiKgvu37+PVq1aoX379ti8eTM8PDxw4cIFVKgg/wCrolhVkU5JScGlS5fw8ssvmzsVIiIyt1K+cGzu3Lnw9fXFqlWr9OMCAgLkl18MFn24+4033kB0dDSuXLmCv/76C71794atrS0GDBhg7tSIiMjMhDDyPmmFt2D99NNPaNq0Kfr16wdPT080atQIK1asKKG1e8SiW9LXrl3DgAEDcO/ePXh4eKB169bYv38/PDw8zJ0aERGZm4la0o/fBVTQadPLly9jyZIlmDhxIt566y0cPHgQYWFhsLe3x6BBg+TzKIRFF+n169ebOwUiInrCPX4XUHh4OGbOnJlnOp1Oh6ZNm+L9998HADRq1AgnT57E0qVLy2aRNiUXuxw4KuwFa0hVuYsBHG3lu1a5l6msd5lc22/J9YIlonyk4ro1vCIV991PVaXiAOBwvNwRlCyd3DZ9xlPuF/rv8VqpuCP3MqTiAKCKk9w6ejvI9fT01z25izNvPPSTimtSUW79ACAhU+674VxO7vN/kCWXq63kKiZkyv8ZzxRy+9w/iU6Kpk/PKaFbnUzUko6Li4NW+7/vbUEXH3t7e6N27doG42rVqoXvv/9ePocilJkiTURETxgTFWmtVmtQpAvSqlUrnDt3zmDc+fPn4ecn9+OzOCz6wjEiIiJLMWHCBOzfvx/vv/8+Ll68iLVr12L58uUYPXp0iS2TRZqIiKxTbgcbxgwKPP3009i4cSPWrVuHunXrYs6cOVi4cCEGDhxYQivIw91ERGSlhO7RYEy8Ut26dUO3bt3kF6oQizQREVkn9oJFRERE5sKWNBERWacy0JJmkSYiIqtkjnPSpY1FmoiIrJMwsiWt8Opuc5Aq0hcuXMCOHTtw+/Zt6HSGP0VmzJhhksSIiIjKOsVFesWKFRg5ciTc3d3h5eUFlep/z7JTqVQs0kREVDp0/w7GxFs4xUX63XffxXvvvYcpU6aURD5ERETFktvlpDHxlk7xLVj3799Hv379SiIXIiIi+g/FLel+/fph69atGDFiREnkU2JOJdtBbaOsx58AJ2W9ZuXKEcb02CMX+177i1Jx47cGSsWti5F7oPyGqTek4gDg7SVyy2zjLtfTT8TlW1Jx9dTeUnHVtHI9SwHAsOp3pOIaRsv13vN9wyFScXcy5K5VPX5fKgwAYGdjKxUn27PYs57pUnE/35D7/L+5e1YqDgC+ayK3r446rOw7lS3Z21aReLg7r2rVqmH69OnYv38/6tWrBzs7w27gwsLCTJYcERFRgcS/gzHxFk5xkV6+fDmcnZ0RHR2N6Ohog/dUKhWLNBERlYqycE5acZGOiYkpiTyIiIjoMdLP7s7MzMS5c+eQnZ1tynyIiIiKR2eCwcIpLtJpaWkYOnQoHB0dUadOHcTGxgIAxo4diw8++MDkCRIREeUn97GgxgyWTnGRnjZtGo4dO4adO3fCwcFBPz4kJARRUVEmTY6IiKgsU3xOetOmTYiKikLz5s0NnjZWp04dXLp0yaTJERERFYi3YOV1584deHp65hmfmppqULSJiIhKUlnoBUvx4e6mTZvi119/1b/OLcxffPEFWrRoYbrMiIiICiNg3EVjln8HlvKW9Pvvv4/OnTvj9OnTyM7OxqJFi3D69Gn89ddfee6bJiIiInmKW9KtW7fG0aNHkZ2djXr16mHr1q3w9PTEvn370KRJk5LIkYiIKA8hjB8sneKW9MmTJ1G3bl2sWLEiz3ubNm1Cr169TJEXERFRoXhOOh+hoaH5PnXs+++/x8CBA02SFBERUZH4MJO8XnvtNYSEhCA+Pl4/LioqCq+88goiIyNNmRsREVGZpvhw96xZs5CQkICQkBDs2rULW7ZswWuvvYavv/4affv2LYkcTSLIKQeOtsoeYbr5htwtZTZG3InWuKLcSZKF+6pJxS0KlevicvupqlJxH34p190kALy/t4ZUXOLEzVJxk7Lkci1vJ/eo3A2x8jvOlxc8pOKOtpX9ziZLRT1lI9d02Xc3722fxfX+c3L7+D/n5Lpx/OSsXJeTY2qkScXZXa8rFQcAX1+WixsZoGwfT8vJwIETcssqTFk43C3Vueunn36KgQMHonnz5rh+/TrWrVuHnj17mjo3IiKiAhl78dcTc+HYTz/9lGdcnz59sHv3bgwYMAAqlUo/TY8ePUybIRERURlVrCJd2BXbK1euxMqVKwE8erBJTk6OSRIjIiIqlE71aDAm3sIVq0jrdFZw4J6IiMoUnpMmIiKyUEKoIIR8a9iY2NKi+BYsAIiOjkb37t1RrVo1VKtWDT169MDu3btNnRsREVGZprhIf/PNNwgJCYGjoyPCwsIQFhYGjUaDDh06YO3atSWRIxERUR65h7uNGSyd4sPd7733HubNm4cJEybox4WFheGjjz7CnDlz8OKLL5o0QSIiovwIYeQ5aSu4BUtxS/ry5cvo3r17nvE9evTI93GhREREJEdxkfb19cX27dvzjP/jjz/g6+trkqSIiIiKknvhmDGDrA8++AAqlQrjx4833QrlQ/Hh7kmTJiEsLAxHjx5Fy5YtAQB79+5FZGQkFi1aZPIEiYiI8qVTQZjhPumDBw9i2bJlqF+/vvyyi0lxkR45ciS8vLywYMECfPvttwCAWrVqISoqio8GJSKiUmOOx4KmpKRg4MCBWLFiBd599135hReT1H3SvXv3Ru/evU2dCxERUalLTjbsMEatVkOtzr+jlNGjR6Nr164ICQmxzCIdGBiIgwcPomLFigbjExMT0bhxY1y+LNmtSgmLT7eFg62y1W0k2SNVa3e5HoIAICrWVSqullbuEsdfj/tLxeVInsup5pwlFQcAt0dvlYrzerO2VNyiLnIXQvb2lOs9aXBgqlQcAJxIcpSKW3ZervesV6vdk4o7myi3f9tLPdHhkW3H5Xoz2xhnKxXXxksu2e/j5J4tlZYtf3mzh0Yu1/MpynLN0Mn1DFcUUz3M5PHrqcLDwzFz5sw8069fvx5HjhzBwYMHpZeplOK94sqVK/k+nzsjIwPXr183SVJERERFEUaek86NjYuLg1ar1Y/PrxUdFxeHcePGYdu2bXBwcJBeplLFLtL/7Qnr999/h6vr/34R5+TkYPv27fD39zdpckRERAUx1TlprVZrUKTzc/jwYdy+fRuNGzfWj8vJycGuXbvw2WefISMjA7a2ckdfClPsIp3bE5ZKpcKgQYMM3rOzs4O/vz8WLFhg0uSIiIgsQYcOHXDixAmDcUOGDEHNmjUxZcqUEinQgIIindsTVkBAAA4ePAh3d/cSSYiIiKg4SrODDRcXF9StW9dgnJOTEypWrJhnvCkpPifNp4oREZEl0OlU0BlxTtqY2NLCriqJiIgk7Ny5s8SXwSJNRERWyRwPMyltLNJERGSVSvOctLmwSBMRkVVikS5AVlYW4uPjkZaWBg8PD7i5uZk6LyIiojKv2M+Ee/DgAZYsWYK2bdtCq9XC398ftWrVgoeHB/z8/DBs2LBSfVQaERGVbTqhMnqwdMUq0h999BH8/f2xatUqhISEYNOmTTh69CjOnz+Pffv2ITw8HNnZ2ejYsSM6deqECxculHTeRERUxuU+FtSYwdIV63D3wYMHsWvXLtSpUyff95s1a4ZXX30VS5cuxapVq7B7925Ur17dpIkSERGVNcUq0uvWrSvWzNRqNUaMGGFUQiUlUwfYKPzRlJgp9ytr9on8uzgrjgH+eTsvKY5xl36Vi/PqIRVnK/kD9Mhd+R577G28pOJGddkmFRc7V+6H5urFcp/horN2UnEA4Ociua8+e1EqLvzPalJxG5P3S8WtqlVfKg4A7FRy+9wzleR6FruSKvdZ9KqSJhVX0SFDKg4ADt2T65XsYY6ydUzPKZl7ncrCLVjSHcBdvHgRv//+Ox4+fAgAENawtkRE9MTQwchz0rD8w92Ki/S9e/cQEhKCp556Cl26dMHNmzcBAEOHDsWkSZNMniAREVF+cm/BMmawdIqL9IQJE1CuXDnExsbC0fF/h4NeeOEFbNmyRdG8du3ahe7du8PHxwcqlQqbNm0yeF8IgRkzZsDb2xsajQYhISG8KI2IiMoMxUV669atmDt3LqpUqWIwvnr16rh69aqieaWmpqJBgwZYvHhxvu/PmzcPn3zyCZYuXYoDBw7AyckJoaGhSE9PV5o2ERE9YYSRt19ZQ0ta8cNMUlNTDVrQuRISEqBWK7tgqnPnzujcuXO+7wkhsHDhQrzzzjvo2bMnAOCrr75CpUqVsGnTJvTv319p6kRE9AQpC08cU9ySbtOmDb766iv9a5VKBZ1Oh3nz5qF9+/YmSywmJgbx8fEICQnRj3N1dUVwcDD27dtXYFxGRgaSk5MNBiIievLoTDBYOsUt6Xnz5qFDhw44dOgQMjMzMXnyZJw6dQoJCQnYu3evyRKLj48HAFSqVMlgfKVKlfTv5SciIgKzZs0yWR5ERETmorglXbduXZw/fx6tW7dGz549kZqaij59+uCff/5BUFBQSeSoyLRp05CUlKQf4uLizJ0SERGVgLJwdbdUBxuurq54++23TZ2LAS+vRw+uuHXrFry9vfXjb926hYYNGxYYp1arFZ8bJyIi66MTMOr52zoreLyH4pb0qlWrsGHDhjzjN2zYgNWrV5skKQAICAiAl5cXtm/frh+XnJyMAwcOoEWLFiZbDhERkaVSXKQjIiLg7u6eZ7ynpyfef/99RfNKSUnB0aNHcfToUQCPLhY7evQoYmNjoVKpMH78eLz77rv46aefcOLECbzyyivw8fFBr169lKZNRERPGB7uzkdsbCwCAgLyjPfz80NsbKyieR06dMjgivCJEycCAAYNGoTIyEhMnjwZqampeP3115GYmIjWrVtjy5YtcHBwUJo2ERE9YR4d7jYu3tIpLtKenp44fvw4/P39DcYfO3YMFStWVDSvdu3aFfrMb5VKhdmzZ2P27NlK0yQioidcWbhPWnGRHjBgAMLCwuDi4oJnnnkGABAdHY1x48ZZ9ANGbqcL2Nso+9n0S8phqWV1dGwiFQcAS2PuS8X1du4mFeeplrtT8MBdqTBUUNvKBQJIy5b7Qr3i1kYqbvEncj+zRw2/LhXXeadc71kA0GO73NGldYcCpeI6ez+UjGsgFXfjob1UHKC8x6Zcl1Lk+h/anXBPKi7YzUUqbu1Z+SOLadly3/82lZR9j5X2QEj/o7hIz5kzB1euXEGHDh1QrtyjcJ1Oh1deeUXxOWkiIiJZOhjXk5U19IKlqEgLIRAfH4/IyEi8++67OHr0KDQaDerVqwc/P7+SypGIiCiPstCftOIiXa1aNZw6dQrVq1dH9erVSyovIiKiMk/RSRcbGxtUr14d9+7JnXMhIiIyFWN6wModLJ3iKyM++OADvPnmmzh58mRJ5ENERFQs4t9z0rKDeNLOSQPAK6+8grS0NDRo0AD29vbQaDQG7yckJJgsOSIiooLwnHQ+Fi5cWAJpEBER0eMUF+lBgwaVRB5ERESKGHte2RrOSUs9FrQwVatWlU6GiIiouISR55WfyHPS/v7+UKkKXrGcHPmnJhERERUXn92dj3/++cfgdVZWFv755x989NFHeO+990yWGBERUVmnuEg3aJD32btNmzaFj48P5s+fjz59+pgkMSIiosKU9jnpiIgI/PDDDzh79iw0Gg1atmyJuXPnokaNGtI5FEXuCfL5qFGjBg4ePGiq2RERERUq95y0MYMS0dHRGD16NPbv349t27YhKysLHTt2RGpqagmtoURLOjk52eC1EAI3b97EzJkzLfoxobfTdLCzUXa+/FX3plLLqu+aKRUHAOlx5aXinO3kfm8lZ0uFwVYl13tOBbX878K5MTek4t4J8pSKW31Z7oRVp18Vf60AAEGLm0vFAUDnTtek4lzKye0A0Xc0RU+Uj7vpctu0hqtUGADgZppcS2vlvU1ScTOr9pSKW3oxXSrurdryf2+iYp2k4uwU9iiYbQXnfotjy5YtBq8jIyPh6emJw4cP63uFNDXFf03Kly+f58IxIQR8fX2xfv16kyVGRERUGHNfOJaUlAQAcHNzM25GhVBcpHfs2GHw2sbGBh4eHqhWrZq+60oiIqKSZqpbsB4/QqxWq6FWqwuN1el0GD9+PFq1aoW6detK51AUxVW1bdu2JZEHERGRIqZqSfv6+hqMDw8Px8yZMwuNHT16NE6ePIk9e/bIJ1AMUk3fS5cuYeHChThz5gwAoHbt2hg3bhyCgoJMmhwREVFJi4uLg1ar1b8uqhU9ZswY/PLLL9i1axeqVKlSorkpvorn999/R+3atfH333+jfv36qF+/Pg4cOIA6depg27ZtJZEjERFRHqbqqlKr1RoMBRVpIQTGjBmDjRs34s8//0RAQECJr6PilvTUqVMxYcIEfPDBB3nGT5kyBc8995zJkiMiIiqI+HcwJl6J0aNHY+3atfjxxx/h4uKC+Ph4AICrq2ueHiFNRXFL+syZMxg6dGie8a+++ipOnz5tkqSIiIgszZIlS5CUlIR27drB29tbP0RFRZXYMhW3pD08PHD06NE890QfPXoUnp5y96MSEREpJWDcE8eUXhkuzNABteIiPWzYMLz++uu4fPkyWrZsCQDYu3cv5s6di4kTJ5o8QSIiovzo/h2Mibd0iov09OnT4eLiggULFmDatGkAAB8fH8ycORNhYWEmT5CIiCg/QqggjGlJP4n9SatUKkyYMAETJkzAgwcPAAAuLi4mT4yIiKisU3zh2MOHD5GWlgbgUXFOSEjAwoULsXXrVpMnR0REVBCdCQZLp7hI9+zZE1999RUAIDExEc2aNcOCBQvQs2dPLFmyxOQJEhER5Sf3iWPGDJZOcZE+cuQI2rRpAwD47rvv4OXlhatXr+Krr77CJ598YvIEiYiI8lPaXVWag+Jz0mlpafpz0Fu3bkWfPn1gY2OD5s2b4+rVqyZP0FSCPW3gYGurKKa1e6LUsu6kO0jFAUCt8nJdOYZUSpSK+ztBrg/Ayk5yeXbzSZKKA4BzSR5ycQ/speIqO8r9zJ5woLxUXM0Oct1NAsDMjhek4vpv8C16onz0rSr3x00nlH0Hc9kr7Brxv7wlP8cRtr2k4mpr06TiDt+T+7txKrnoaQoyssY9qbgNV90VTZ+hs/xiaKkU/6WtVq0aNm3ahLi4OPz+++/o2LEjAOD27dsGzz4lIiIqSTzcnY8ZM2bgjTfegL+/P4KDg9GiRQsAj1rVjRo1MnmCRERE+eHh7nz83//9H1q3bo2bN2+iQYMG+vEdOnRA7969TZocERFRWSbVVaWXlxe8vLwMxjVr1swkCRERERWHqfqTtmTFOtw9YsQIXLtWvItaoqKisGbNGqOSIiIiKkpZOCddrJa0h4cH6tSpg1atWqF79+5o2rQpfHx84ODggPv37+P06dPYs2cP1q9fDx8fHyxfvryk8yYiojLO2PPKT8w56Tlz5mDMmDH44osv8Pnnn+fpktLFxQUhISFYvnw5OnXqVCKJEhERlTXFPiddqVIlvP3223j77bdx//59xMbG4uHDh3B3d0dQUBBUKsv/RUJERE8OYeQhazP0PKmY1IVjFSpUQIUKFUydCxERUbGVha4q5R4bRURERCVOqiVNRERkbuxPmoiIyEKVhcPdLNJERGSVysLDTMpMkb6VroLaRtmhjXdPOEot6yiOSsUBwLZmVaXiQg5ckYr7qdFTUnHnklyk4oYdT5SKA4Ad3bOk4r4+HCQV994zMVJxR2K8ip4oH27q+1JxAPDid3L7zaa7z0nF9fP8QyouUyf3GV5Mlus9CwCGBuVIxSU6yP15vJUu1+taH99sqbjQtvK9D/b9Uq4XtNeqZSqaPi1H7nOnYhbpRo0aFfsWqyNHjhiVEBERUXGIfwdj4i1dsYp0r1699P9PT0/H559/jtq1a+t7wNq/fz9OnTqFUaNGlUiSREREj3t0uFv+4q8n5nB3eHi4/v+vvfYawsLCMGfOnDzTxMXFmTY7IiKiApSFlrTi+6Q3bNiAV155Jc/4l156Cd9//71JkiIiIiKJIq3RaLB379484/fu3QsHBweTJEVERFQU9oKVj/Hjx2PkyJE4cuSIvg/pAwcOYOXKlZg+fbrJEyQiIsoP75POx9SpUxEYGIhFixbhm2++AQDUqlULq1atwvPPP2/yBImIiMoqqRsBn3/+eRZkIiIyKyGM68nKGnrBkupgIzExEV988QXeeustJCQkAHh0f/T169dNmhwREVFBBFTQGTEIPIHP7j5+/DhCQkLg6uqKK1eu4LXXXoObmxt++OEHxMbG4quvviqJPImIiAywJZ2PiRMnYvDgwbhw4YLB1dxdunTBrl27TJocERFRWaa4JX3w4EEsW7Ysz/jKlSsjPj7eJEkREREVhVd350OtViM5OTnP+PPnz8PDw8MkSRERERWFvWDlo0ePHpg9eza+/fZbAIBKpUJsbCymTJmCvn37mjxBU6nurIPGVllvOBeS5C4q+OKpulJxANB6326puKVPtZeKO50ode0gdt6S65XoTf/KUnEAcOlqilRcpyq3peLe3hUgFTc4MO+P2OL4KqaCVBwAdK8i1ybwd39PKi52vmTvaVFy+9vDLPkO+5Iy5Xqlupok93Cm3+PlPv8FDeV66/JbcVkqDgBW1ZTrBUun8GSuncoKqmExLV68GPPnz0d8fDwaNGiATz/9VP/MkJKg+BuzYMECpKSkwNPTEw8fPkTbtm1RrVo1uLi44L335L7wRERESgkTDEpERUVh4sSJCA8Px5EjR9CgQQOEhobi9m25hkBxKP556urqim3btmHPnj04fvw4UlJS0LhxY4SEhJREfkRERPkq7cPdH330EYYNG4YhQ4YAAJYuXYpff/0VK1euxNSpU+UTKYTcsScArVu3xqhRozB58mTpAr1r1y50794dPj4+UKlU2LRpk8H7gwcPhkqlMhg6deokmzIRET1Bcm/BMmYAgOTkZIMhIyMjz7IyMzNx+PBhg3pnY2ODkJAQ7Nu3r8TWsVgt6U8++aTYMwwLCyv2tKmpqWjQoAFeffVV9OnTJ99pOnXqhFWrVulfq9XqYs+fiIioKL6+hufmw8PDMXPmTINxd+/eRU5ODipVqmQwvlKlSjh79myJ5VasIv3xxx8bvL5z5w7S0tJQvnx5AI+eQObo6AhPT09FRbpz587o3LlzodOo1Wp4eXkVe55ERFQ2mOoWrLi4OGi1Wv14S2oMFutwd0xMjH5477330LBhQ5w5cwYJCQlISEjAmTNn0LhxY8yZM8fkCe7cuROenp6oUaMGRo4ciXv37hU6fUZGRp5DF0RE9OQxVVeVWq3WYMivSLu7u8PW1ha3bt0yGH/r1q0SbUgqPic9ffp0fPrpp6hRo4Z+XI0aNfDxxx/jnXfeMWlynTp1wldffYXt27dj7ty5iI6ORufOnZGTU/CtChEREXB1ddUPjx/GICKiJ0NpXt1tb2+PJk2aYPv27fpxOp0O27dvR4sWLYxfmQIovrr75s2byM7OzjM+Jycnzy8MY/Xv31///3r16qF+/foICgrCzp070aFDh3xjpk2bhokTJ+pfJycns1ATEZHRJk6ciEGDBqFp06Zo1qwZFi5ciNTUVP3V3iVBcZHu0KEDhg8fji+++AKNGzcGABw+fBgjR44s8duwAgMD4e7ujosXLxZYpNVqtUWdTyAiopJR2rdgvfDCC7hz5w5mzJiB+Ph4NGzYEFu2bMlzMZkpKT7cvXLlSnh5eaFp06b6gtisWTNUqlQJX3zxRUnkqHft2jXcu3cP3t7eJbocIiKyfOLf7iaNGZQaM2YMrl69ioyMDBw4cADBwcElsGb/o7gl7eHhgd9++w3nz5/XX3Zes2ZNPPWU8scEpqSk4OLFi/rXMTExOHr0KNzc3ODm5oZZs2ahb9++8PLywqVLlzB58mRUq1YNoaGhipdFRERkbaQfiPvUU09JFeb/OnToENq3/98zp3PPJQ8aNAhLlizB8ePHsXr1aiQmJsLHxwcdO3bEnDlzeDibiIggYNzhbmt4orjiIp2Tk4PIyEhs374dt2/fhk5neJfan3/+Wex5tWvXDqKQB7X//vvvStMjIqIygr1g5WPcuHGIjIxE165dUbduXahUcj1FlTYdVNApPP+gk/yddTlVrvccAGhs84xUXJvA61Jxey/L9UrV3F3uc7c1Yne5kuokFVffIe8j/opjY8ofUnHPZ7eUivPSSIUBAM4lyz3hVyX5ZODDX8vt403HyC1vxHhnqTgA8HKU2+nqu2ZKxZ2wl9tP112V61lOYyvfe9ri83JxVZ2U9SyWqSuZnptlOsl4PN7SKS7S69evx7fffosuXbqURD5ERET0L8VF2t7eHtWqVSuJXIiIiIqtLBzuVnzsadKkSVi0aFGh55KJiIhKmjDBP0unuCW9Z88e7NixA5s3b0adOnVgZ2dn8P4PP/xgsuSIiIjKMsVFunz58ujdu3dJ5EJERFRsZeFwt+Ii/d++nYmIiMylLFzdLXU/RHZ2Nv744w8sW7YMDx48AADcuHEDKSkpJk2OiIioIKbqqtKSKW5JX716FZ06dUJsbCwyMjLw3HPPwcXFBXPnzkVGRgaWLl1aEnkSERGVOYpb0uPGjUPTpk1x//59aDT/e/pC7969DfrZJCIiKklCGD9YOsUt6d27d+Ovv/6Cvb3hE2f8/f1x/brcU6+IiIiU0v07GBNv6RQXaZ1Oh5ycnDzjr127BhcXF5MkRUREVJSycHW34sPdHTt2xMKFC/WvVSoVUlJSEB4ezkeFEhERmZDilvSCBQsQGhqK2rVrIz09HS+++CIuXLgAd3d3rFu3riRyJCIiysvY88pW0JJWXKSrVKmCY8eOISoqCseOHUNKSgqGDh2KgQMHGlxIZmnScgChsBesnr5yvdLUcX0gFQcAUbFyvQS9vNVLKs5Jsluq+m5y2+ZMYt5TJcUVl5ksFedwUa5XorW1W0nFXUyR6yEqNrX0/2I0tXlaKu5Uktzn/9lYuf1tcY+LUnEA8OPBALm4a3LrGJ+ZJhX3rLfcfvqiTTOpOACo6SL3fXQql6Vo+rScbKy8KbWoQvGcdD527dqFli1bYuDAgRg4cKB+fHZ2Nnbt2oVnnpHrapGIiIgMKW62tW/fHgkJCXnGJyUloX379iZJioiIqCi8BSsfQgioVHkPWd27dw9OTnKHa4iIiJTi4e7/6NOnD4BHV3MPHjwYarVa/15OTg6OHz+Oli1bmj5DIiKifAghjOo22Rq6XC52kXZ1dQXwaKVcXFwMLhKzt7dH8+bNMWzYMNNnSEREVEYVu0jn9n7l7++PN998E46OjiWWFBERUVH4MJN8REdHIzMzM8/45ORkPPvssyZJioiIqCjCBIOlM1mRTk9Px+7du02SFBERESk43H38+HEAj85Jnz59GvHx8fr3cnJysGXLFlSuXNn0GRIREeWjLBzuLnaRbtiwIVQqFVQqVb6HtTUaDT799FOTJkdERFQQFun/iImJgRACgYGB+Pvvv+Hh4aF/z97eHp6enrC1lXuMHhERkVKPzisbcQuW6VIpMcUu0n5+fgAedVVJREREJU/xE8dynT59GrGxsXkuIuvRo4fRSRERERWFh7vzcfnyZfTu3RsnTpyASqXSP7El91GhOTnyvRwREREVl7HP37aCB44pL9Ljxo1DQEAAtm/fjoCAAPz999+4d+8eJk2ahA8//LAkcjQNiQ/T0Vbu0P61NPkuO9t42EvF3UiT29tae8it451MueWFeMt1VQgAAU5yXUDa28j9cIxLk/ssrj2U6260ihHPB6qlzXtbZHFce2gnFedmL7c8rb3cNv35kFx3kwDQsXqcVNz+e/5ScdW1LlJxm68r6/4xV39/+YZRRo7c9/FoorLPMaOETpMKCOiMOidt+VVa8V+Tffv2Yfbs2XB3d4eNjQ1sbGzQunVrREREICwsrCRyJCIishpXrlzB0KFDERAQAI1Gg6CgIISHh+f7jJGiKG5J5+TkwMXl0S9Fd3d33LhxAzVq1ICfnx/OnTunOAEiIiIZlnq4++zZs9DpdFi2bBmqVauGkydPYtiwYUhNTVV8xFlxka5bty6OHTuGgIAABAcHY968ebC3t8fy5csRGBiodHZERERSLLWryk6dOqFTp07614GBgTh37hyWLFlS8kX6nXfeQWpqKgBg9uzZ6NatG9q0aYOKFSsiKipK6eyIiIieeElJSXBzc1Mcp7hIh4aG6v9frVo1nD17FgkJCahQoYL+Cm8iIqKSZqr+pJOTkw3Gq9VqqNVqo3L7r4sXL+LTTz+Vurha7jLUx7i5ubFAExFRqcq9T9qYAQB8fX3h6uqqHyIiIvJd3tSpU/WPxy5oOHv2rEHM9evX0alTJ/Tr1w/Dhg1TvI7SDzMhIiIyJ52Rt2DlxsbFxUGr1erHF9SKnjRpEgYPHlzoPP97bdaNGzfQvn17tGzZEsuXL5fKkUWaiIjKNK1Wa1CkC+Lh4WHQb0Vhrl+/jvbt26NJkyZYtWoVbGzkDlyzSBMRkVUSMPIWLJNlYuj69eto164d/Pz88OGHH+LOnTv697y8vBTNi0WaiIiskqkOd5vatm3bcPHiRVy8eBFVqlQxeE/phW4muXCMiIiIHhk8eLD+yvPHB6XYkiYiIqskhHGHrJ/IDjaIiIgsgaUe7jalMlOkU7JVyBbK7uX+I15u81xJTZOKA4DXguTuN+/qI7fM7+JcpeJSs+R27qYV5b8Ut9PlelA6mij3OZaXWxwq2sut44E78g8pTM2WS9ZVch3PPZB70MMrAUlScXNOyPWABgBHE+V60Fr45jWpuJUrfKTi6gbI7TfLLmRLxQFA1ypyPfbdy1CWa2YJddysE0YWaStoSvOcNBERkYUqMy1pIiJ6soh//xkTb+lYpImIyCoJGNeTleWXaBZpIiKyUmXhwjGekyYiIrJQbEkTEZFVEsLIc9JWcHU3izQREVklHu4mIiIis2FLmoiIrFJZaEmzSBMRkVUS/5ZpY+ItHYs0ERFZpbLQkuY5aSIiIgvFljQREVmlstCSLjNF+n4moFZ43MDfWa5HKl8nJ6k4ANAJuR5tVJI7W6LC3mxyuTvIbZtqzvI9hI04fVsqbrhPoFTc9htZUnFO5Wyl4rpUlv+DEeicLBW39qqLVFyOZKoXkrVScY62OXILBPB94jGpOIdPGkjFjW9xUSpuVnQ1qbgule2k4gDAVu5rjPsZys7lZpVUL1j//jMm3tLxcDcREZGFMmuRjoiIwNNPPw0XFxd4enqiV69eOHfunME06enpGD16NCpWrAhnZ2f07dsXt27dMlPGRERkKYRKQKh0RgyWf7jbrEU6Ojoao0ePxv79+7Ft2zZkZWWhY8eOSE1N1U8zYcIE/Pzzz9iwYQOio6Nx48YN9OnTx4xZExGRJRD6m7DkBnZVWYQtW7YYvI6MjISnpycOHz6MZ555BklJSfjyyy+xdu1aPPvsswCAVatWoVatWti/fz+aN29ujrSJiMgC6KCDiuekS09SUhIAwM3NDQBw+PBhZGVlISQkRD9NzZo1UbVqVezbt88sORIREZUWi7m6W6fTYfz48WjVqhXq1q0LAIiPj4e9vT3Kly9vMG2lSpUQHx+f73wyMjKQkZGhf52cLHfVKxERWbay8MQxi2lJjx49GidPnsT69euNmk9ERARcXV31g6+vr4kyJCIiS6JT6YweLJ1FFOkxY8bgl19+wY4dO1ClShX9eC8vL2RmZiIxMdFg+lu3bsHLyyvfeU2bNg1JSUn6IS4uriRTJyIiM9GZ4J+lM2uRFkJgzJgx2LhxI/78808EBAQYvN+kSRPY2dlh+/bt+nHnzp1DbGwsWrRoke881Wo1tFqtwUBERGSNzHpOevTo0Vi7di1+/PFHuLi46M8zu7q6QqPRwNXVFUOHDsXEiRPh5uYGrVaLsWPHokWLFryym4iojCsLV3ebtUgvWbIEANCuXTuD8atWrcLgwYMBAB9//DFsbGzQt29fZGRkIDQ0FJ9//nkpZ0pERJamLFw4ZtYiLUTRN5I7ODhg8eLFWLx4cSlkREREZDks5hYsIiIiJXTIgQryna/ojIgtLWWmSPs6CjjYKnsE3PfX5e6x9ijnKBUHACnZcj3aPMiWu0Cui49cT0/77tlLxe25K9frEgD88LRcD1pfX5Lr6uelQLm4yprUoifKx6rL8tvm8H25z7+dZ6ZUXFWnh1JxsakaqbirafI9PbWyqS8V19rjvlTc1O1BUnGLBlyQiuv7pfxtpi/4yW3X16ulK5o+NTsD39+TWlShxL8PBjUm3tKVmSJNRERPFp1KB5UR9zpbw4VjFnGfNBEREeXFIk1ERFZJhxyjh5KWkZGBhg0bQqVS4ejRo4rjWaSJiMhKGdNRpQ4ohcPdkydPho+Pj3Q8izQREVEJ2Lx5M7Zu3YoPP/xQeh68cIyIiKySTuTAmLbmo/iScevWLQwbNgybNm2Co6P8HT8s0kREZJVM9cSxx7s0VqvVUKvV8vMVAoMHD8aIESPQtGlTXLlyRXpePNxNRERWSSDH6AEAfH19Dbo4joiIyHd5U6dOhUqlKnQ4e/YsPv30Uzx48ADTpk0zeh3ZkiYiojItLi7OoMfEglrRkyZN0vcrUZDAwED8+eef2LdvX575NG3aFAMHDsTq1auLnRuLNBERWSWdkVdo5z7MpLjdGnt4eMDDw6PI6T755BO8++67+tc3btxAaGgooqKiEBwcrChHFmkiIrJKlvpY0KpVqxq8dnZ2BgAEBQWhSpUqiubFIk1ERFZJiBwIyD1jPzfe0rFIExERlSB/f/9idc2cnzJTpG+kq6C2UfaLa0Sgk9SyLqfKb9a6WrleiT68INdD1LMe5aXiKkrenfDXbbn1A4BvbqdIxdW1rSgV16C8VBi+vOQsFTfqqUS5BQLYfquCVFwdN7ll9j2UJBWXAbn9dKSXXE9WABD/UK6lNfYfuT+qU5+S28fbfVFeKm73hOtScQAw7otAqbiELGX3/abnlMyNRKY6J23JykyRJiKiJ8uj26iMONxtBf1J8z5pIiIiC8WWNBERWSUhjHzimODhbiIiohLBc9JEREQWqizcgsVz0kRERBaKLWkiIrJKlvrEMVNikSYiIqv06MIxYw5385w0ERFRCckxsi3Mc9JEREQkiS1pIiKySo8OV/NwNxERkcUpC0Wah7uJiIgsFFvSRERklXTQQWVUBxuW35IuM0XawRZQKzxuUEWTLrUsOxvJfhwBRN+xl4p7q4bczrbsgly3epeFXPd4v7SWWz8AmLzfTyouW7If15hUO6m45Mxsqbj3T8p1cQkAn7SJlYprsuOiVNza2sFScQvO2ErF1dPKfRcB4OB9jVRcD283qbjYh3L728Sg8lJxXT9W1m3kf/32ttx+88mqqoqm15XQ7chl4XB3mSnSRET0ZDH2sZ58LCgRERFJY0uaiIis0qPHevKxoERERBbH2HPK1nBOmoe7iYiILBRb0kREZJXKQkuaRZqIiKySsfc58z5pIiKiElIWWtI8J01ERGSh2JImIiKrVBZa0izSRERkpYwtsizSREREJaIstKR5TpqIiMhClZmWtJPto56wlNh4Xa5XIq1c50kAACfJT+ST83KPt3u9mtzy9t4JkIoL3XtWboEAxvj4SMXdzpDrJaeas1wPYd5Kd7R/VdZkSMUBwMQ9cttmf2u5XH+Mket5KUgr91k8yJb7LADAVyPXicLpZLltc+iO3PKm1UuRiptQQ66XLwD4aKVcz3Ljul9QNH1yRibePi+1qELxFiwiIiILJYSRz+6W7Ma2NPFwNxERkYViS5qIiKxUDgC5UyiPWH5LmkWaiIis0qOrs+WLNA93ExERlRidCYaS8+uvvyI4OBgajQYVKlRAr169FM+DLWkiIiIT+/777zFs2DC8//77ePbZZ5GdnY2TJ08qng+LNBERWScjD3ejhA53Z2dnY9y4cZg/fz6GDh2qH1+7dm3F8+LhbiIiskrCBP8AIDk52WDIyJB/ZgEAHDlyBNevX4eNjQ0aNWoEb29vdO7cWaolzSJNRERlmq+vL1xdXfVDRESEUfO7fPkyAGDmzJl455138Msvv6BChQpo164dEhISFM2LRZqIiKyUaS4ci4uLQ1JSkn6YNm1avkubOnUqVCpVocPZs2eh0z2a79tvv42+ffuiSZMmWLVqFVQqFTZs2KBoDXlOmoiIrJQw8lbnR8FarRZarbbIqSdNmoTBgwcXOk1gYCBu3rwJwPActFqtRmBgIGJjYxVlyCJNRERWShhXoxVGe3h4wMPDo8jpmjRpArVajXPnzqF169YAgKysLFy5cgV+fsqel/7EF+ncm9XTc5RfCJAheQudbBwA5EjucdlC7kKHtBy5TgQydHIrmSOypOIA4GFOulRchk7u6s+0HLlcH0pu09Rs+YtVsiT3uQdZch1XpFvJZwHIfx4ZOrmzgVk6uQ42UiQ//7Qc+bOW6XKpIjlD2X7zIOPR51cyDw+xvAeSaLVajBgxAuHh4fD19YWfnx/mz58PAOjXr5+ymYknXFxcnMCjT5EDBw4cOJhxiIuLM8nf9YcPHwovLy+T5OTl5SUePnxokrz+KzMzU0yaNEl4enoKFxcXERISIk6ePKl4PiohrOC5aEbQ6XS4ceMGXFxcoFIZ/opPTk6Gr68v4uLiinU+oizhtikYt03BuG0KVpa3jRACDx48gI+PD2xsTHO9cnp6OjIz5bswzWVvbw8HBwcTZFQynvjD3TY2NqhSpUqh0xT3ooGyiNumYNw2BeO2KVhZ3Taurq4mnZ+Dg4NFF1dT4S1YREREFopFmoiIyEKV6SKtVqsRHh4OtVpt7lQsDrdNwbhtCsZtUzBuG5LxxF84RkREZK3KdEuaiIjIkrFIExERWSgWaSIiIgtVZov04sWL4e/vDwcHBwQHB+Pvv/82d0oWYebMmXl6dalZs6a50yp1u3btQvfu3eHj4wOVSoVNmzYZvC+EwIwZM+Dt7Q2NRoOQkBBcuHDBPMmWsqK2zeDBg/PsQ506dTJPsqUsIiICTz/9NFxcXODp6YlevXrh3LlzBtOkp6dj9OjRqFixIpydndG3b1/cunXLTBmTpSuTRToqKgoTJ05EeHg4jhw5ggYNGiA0NBS3b982d2oWoU6dOrh586Z+2LNnj7lTKnWpqalo0KABFi9enO/78+bNwyeffIKlS5fiwIEDcHJyQmhoKNLT5Z5pbU2K2jYA0KlTJ4N9aN26daWYoflER0dj9OjR2L9/P7Zt24asrCx07NgRqamp+mkmTJiAn3/+GRs2bEB0dDRu3LiBPn36mDFrsmgmfViplWjWrJkYPXq0/nVOTo7w8fERERERZszKMoSHh4sGDRqYOw2LAkBs3LhR/1qn0wkvLy8xf/58/bjExEShVqvFunXrzJCh+Ty+bYQQYtCgQaJnz55mycfS3L59WwAQ0dHRQohH+4mdnZ3YsGGDfpozZ84IAGLfvn3mSpMsWJlrSWdmZuLw4cMICQnRj7OxsUFISAj27dtnxswsx4ULF+Dj44PAwEAMHDhQcf+nT7qYmBjEx8cb7EOurq4IDg7mPvSvnTt3wtPTEzVq1MDIkSNx7949c6dkFklJSQAANzc3AMDhw4eRlZVlsO/UrFkTVatW5b5D+SpzRfru3bvIyclBpUqVDMZXqlQJ8fHxZsrKcgQHByMyMhJbtmzBkiVLEBMTgzZt2uDBgwfmTs1i5O4n3Ify16lTJ3z11VfYvn075s6di+joaHTu3Bk5OZL9IlopnU6H8ePHo1WrVqhbty6AR/uOvb09ypcvbzAt9x0qyBPfwQYp07lzZ/3/69evj+DgYPj5+eHbb7/F0KFDzZgZWYv+/fvr/1+vXj3Ur18fQUFB2LlzJzp06GDGzErX6NGjcfLkyTJ5TQeZTplrSbu7u8PW1jbP1ZS3bt2Cl5eXmbKyXOXLl8dTTz2FixcvmjsVi5G7n3AfKp7AwEC4u7uXqX1ozJgx+OWXX7Bjxw6DXvi8vLyQmZmJxMREg+m571BBylyRtre3R5MmTbB9+3b9OJ1Oh+3bt6NFixZmzMwypaSk4NKlS/D29jZ3KhYjICAAXl5eBvtQcnIyDhw4wH0oH9euXcO9e/fKxD4khMCYMWOwceNG/PnnnwgICDB4v0mTJrCzszPYd86dO4fY2FjuO5SvMnm4e+LEiRg0aBCaNm2KZs2aYeHChUhNTcWQIUPMnZrZvfHGG+jevTv8/Pxw48YNhIeHw9bWFgMGDDB3aqUqJSXFoOUXExODo0ePws3NDVWrVsX48ePx7rvvonr16ggICMD06dPh4+ODXr16mS/pUlLYtnFzc8OsWbPQt29feHl54dKlS5g8eTKqVauG0NBQM2ZdOkaPHo21a9fixx9/hIuLi/48s6urKzQaDVxdXTF06FBMnDgRbm5u0Gq1GDt2LFq0aIHmzZubOXuySOa+vNxcPv30U1G1alVhb28vmjVrJvbv32/ulCzCCy+8ILy9vYW9vb2oXLmyeOGFF8TFixfNnVap27FjhwCQZxg0aJAQ4tFtWNOnTxeVKlUSarVadOjQQZw7d868SZeSwrZNWlqa6Nixo/Dw8BB2dnbCz89PDBs2TMTHx5s77VKR33YBIFatWqWf5uHDh2LUqFGiQoUKwtHRUfTu3VvcvHnTfEmTRWMvWERERBaqzJ2TJiIishYs0kRERBaKRZqIiMhCsUgTERFZKBZpIiIiC8UiTUREZKFYpImIiCwUizQREZGFYpGmJ4K/vz8WLlyof61SqbBp06ZSz2PmzJlo2LBhkdNNnz4dr7/+esknVMKaN2+O77//vkTmfeXKFahUKhw9erRE5k9kDVik6Yl08+ZNg243C1Pcwmoq8fHxWLRoEd5++22Tzrck1yMyMjJPH8gA8M4772Dq1KnQ6XQFxrLYEsljkSaLkZmZabJ5eXl5Qa1Wm2x+pvTFF1+gZcuW8PPzM3cqRuvcuTMePHiAzZs3mzsVoicSizSViHbt2mHMmDEYM2YMXF1d4e7ujunTp+O/j4r39/fHnDlz8Morr0Cr1eoP/+7Zswdt2rSBRqOBr68vwsLCkJqaqo+7ffs2unfvDo1Gg4CAAKxZsybP8h8/3H3t2jUMGDAAbm5ucHJyQtOmTXHgwAFERkZi1qxZOHbsGFQqFVQqFSIjIwEAiYmJeO211+Dh4QGtVotnn30Wx44dM1jOBx98gEqVKsHFxQVDhw5Fenp6kdtm/fr16N69u8G4jIwMhIWFwdPTEw4ODmjdujUOHjyofz+/luymTZugUqn07xe0HiqVCkuWLEHnzp2h0WgQGBiI7777Tj+fnTt3QqVSGfRxfPToUahUKly5cgU7d+7EkCFDkJSUpJ/3zJkzAQC2trbo0qUL1q9fX+D65nbX2KhRI6hUKrRr1w7Aoy5iZ8+ejSpVqkCtVqNhw4bYsmVLgfPJycnBq6++ipo1ayI2NhYA8OOPP6Jx48ZwcHBAYGAgZs2ahezsbH2MSqXCF198gd69e8PR0RHVq1fHTz/9VOAyiCyOmTv4oCdU27ZthbOzsxg3bpw4e/as+Oabb4Sjo6NYvny5fho/Pz+h1WrFhx9+KC5evKgfnJycxMcffyzOnz8v9u7dKxo1aiQGDx6sj+vcubNo0KCB2Ldvnzh06JBo2bKl0Gg04uOPP9ZPA0Bs3LhRCCHEgwcPRGBgoGjTpo3YvXu3uHDhgoiKihJ//fWXSEtLE5MmTRJ16tQRN2/eFDdv3hRpaWlCCCFCQkJE9+7dxcGDB8X58+fFpEmTRMWKFcW9e/eEEEJERUUJtVotvvjiC3H27Fnx9ttvCxcXF9GgQYMCt8u9e/eESqXK0+taWFiY8PHxEb/99ps4deqUGDRokKhQoYJ+WatWrRKurq4GMRs3bhS5X+HC1gOAqFixolixYoU4d+6ceOedd4Stra04ffq0EOJ/vVrdv39fP+9//vlHABAxMTEiIyNDLFy4UGi1Wv28Hzx4oJ92yZIlws/Pr8B1/vvvvwUA8ccff4ibN2/q1+mjjz4SWq1WrFu3Tpw9e1ZMnjxZ2NnZifPnzwshhIiJiREAxD///CPS09NF7969RaNGjcTt27eFEELs2rVLaLVaERkZKS5duiS2bt0q/P39xcyZMw32gypVqoi1a9eKCxcuiLCwMOHs7KzPgcjSsUhTiWjbtq2oVauW0Ol0+nFTpkwRtWrV0r/28/MTvXr1MogbOnSoeP311w3G7d69W9jY2IiHDx+Kc+fOCQDi77//1r9/5swZAaDAIr1s2TLh4uJS4B/m8PDwPIV19+7dQqvVivT0dIPxQUFBYtmyZUIIIVq0aCFGjRpl8H5wcHChRTq3+MXGxurHpaSkCDs7O7FmzRr9uMzMTOHj4yPmzZsnhCi6SBe0HkI82hYjRozIk+fIkSOFEEUX6YKWn+vHH38UNjY2IicnJ9/3/1ts/8vHx0e89957BuOefvpp/TbNjdu9e7fo0KGDaN26tUhMTNRP26FDB/H+++8bxH/99dfC29vbYN3feecd/euUlBQBQGzevDnfXIksDQ93U4lp3ry5/nAsALRo0QIXLlxATk6OflzTpk0NYo4dO4bIyEg4Ozvrh9DQUOh0OsTExODMmTMoV64cmjRpoo+pWbNmvhc15Tp69CgaNWoENze3Yud+7NgxpKSkoGLFiga5xMTE4NKlSwCAM2fOIDg42CCuRYsWhc734cOHAAAHBwf9uEuXLiErKwutWrXSj7Ozs0OzZs1w5syZYudcmMfzatGihcnmrdFooNPpkJGRUeyY5ORk3Lhxw2CdAaBVq1Z58howYABSU1OxdetWuLq66scfO3YMs2fPNvh8hg0bhps3byItLU0/Xf369fX/d3Jyglarxe3bt5WuJpFZlDN3AlS2OTk5GbxOSUnB8OHDERYWlmfaqlWr4vz584qXodFoFMekpKTA29sbO3fuzPNeYT8IiuLu7g4AuH//Pjw8PIodZ2NjY3A+HwCysrKk83h83gAM5q9k3gkJCXBycpLazsXRpUsXfPPNN9i3bx+effZZ/fiUlBTMmjULffr0yRPz3x9BdnZ2Bu+pVKpCr0YnsiRsSVOJOXDggMHr/fv3o3r16rC1tS0wpnHjxjh9+jSqVauWZ7C3t0fNmjWRnZ2Nw4cP62POnTtncNHT4+rXr4+jR48iISEh3/ft7e0NWve5ecTHx6NcuXJ58sgttLVq1cp3HQsTFBQErVaL06dPG4yzt7fH3r179eOysrJw8OBB1K5dGwDg4eGBBw8eGFxA9/gtTfmtR0F57d+/H7Vq1dLPG3h025rMvE+ePIlGjRrl+15uLACDeK1WCx8fH4N1BoC9e/fq1znXyJEj8cEHH6BHjx6Ijo7Wj2/cuDHOnTuX776S+8ODyOqZ+3g7PZlyLxybMGGCOHv2rFi7dq1wcnISS5cu1U/j5+dncB5ZCCGOHTsmNBqNGD16tPjnn3/E+fPnxaZNm8To0aP103Tq1Ek0atRI7N+/Xxw6dEi0bt260AvHMjIyxFNPPSXatGkj9uzZIy5duiS+++478ddffwkhhFizZo1wcnIS//zzj7hz545IT08XOp1OtG7dWjRo0ED8/vvvIiYmRuzdu1e89dZb4uDBg0IIIdavXy8cHBzEypUrxblz58SMGTOKvHBMCCH69OkjJk2aZDBu3LhxwsfHR2zevNngwrGEhAQhxKMLzpycnERYWJi4ePGiWLNmjfDx8TE4J53feuRuC3d3d/Hll1/q87SxsRGnTp0SQjw6/+3r6yv69esnzp8/L3755RdRo0YNg3PSe/fu1V/8defOHZGammrwWc+ePbvA9c3KyhIajUa8++67Ij4+Xn9e+eOPPxZarVasX79enD17VkyZMqXAC8dyp3d2dha7d+8WQgixZcsWUa5cOTFz5kxx8uRJcfr0abFu3Trx9ttv57sf5HJ1dRWrVq0q9DMishQs0lQi2rZtK0aNGiVGjBghtFqtqFChgnjrrbcMLiTLr0gL8ehq4Oeee044OzsLJycnUb9+fYMLjG7evCm6du0q1Gq1qFq1qvjqq6/yzOvxP85XrlwRffv2FVqtVjg6OoqmTZuKAwcOCCGESE9PF3379hXly5cXAPR/wJOTk8XYsWOFj4+PsLOzE76+vmLgwIEGF3299957wt3dXTg7O4tBgwaJyZMnF1mkf/vtN1G5cmWDC60ePnwoxo4dK9zd3YVarRatWrUyuDhOiEcXilWrVk1oNBrRrVs3sXz5coMiXdB6ABCLFy8Wzz33nFCr1cLf319ERUUZzHvPnj2iXr16wsHBQbRp00Zs2LDBoEgLIcSIESNExYoVBQARHh4uhBDi2rVrws7OTsTFxRW6zitWrBC+vr7CxsZGtG3bVgghRE5Ojpg5c6aoXLmysLOzEw0aNDC4oCu/C84WLFggXFxcxN69e4UQjwp17tX9Wq1WNGvWzOAOAhZpsnYqIR470UVkAu3atUPDhg0NHtVJjwghEBwcjAkTJmDAgAElvjyVSoWNGzeiV69eJp/3lClTcP/+fSxfvtzk8yYinpMmKnUqlQrLly83eOiGtfL09MScOXPMnQbRE4tXdxOZQcOGDUv1eeElZdKkSeZOgeiJxsPdREREFoqHu4mIiCwUizQREZGFYpEmIiKyUCzSREREFopFmoiIyEKxSBMREVkoFmkiIiILxSJNRERkoVikiYiILNT/A/INfVJWeGKGAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: a corner of the OV vocab matrix. A copy head shows a bright diagonal.\n",
"fig, ax = plt.subplots(figsize=(5, 4.5))\n",
"im = ax.imshow(ov[:25, :25].numpy(), cmap=\"magma\")\n",
"ax.set_xlabel(\"predicted (output) token\"); ax.set_ylabel(\"attended (source) token\")\n",
"ax.set_title(f\"OV circuit, layer {best_layer} head {best_head}: a copy map\")\n",
"plt.colorbar(im, ax=ax, fraction=0.046); plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "79fc199a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:11.003556Z",
"iopub.status.busy": "2026-06-10T20:56:11.003474Z",
"iopub.status.idle": "2026-06-10T20:56:11.006641Z",
"shell.execute_reply": "2026-06-10T20:56:11.006387Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 6.1 OV circuit copies tokens\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def _ov_copies():\n",
" m = ov_vocab_matrix(model, best_layer, best_head)\n",
" frac = (m.argmax(dim=-1) == torch.arange(m.shape[0])).float().mean().item()\n",
" assert frac > 0.5, \\\n",
" f\"an induction head's OV circuit should copy: most source tokens map to themselves. \" \\\n",
" f\"copying fraction {frac:.0%} is too low (expect well above 50%).\"\n",
"check(\"6.1 OV circuit copies tokens\", _ov_copies)"
]
},
{
"cell_type": "markdown",
"id": "92812ebf",
"metadata": {},
"source": [
"> **Interpretation.** A bright diagonal: source token `i` most strongly predicts token `i`. The induction head, having decided *where* to look (QK: the stripe), writes *copy that token* (OV: the identity-like map). We have now read both halves of the circuit. This is what \"reverse-engineered a circuit\" means in the smallest honest case: we can state the head's algorithm in one sentence and back every clause with a measurement.\n",
"\n",
"> **Key takeaways.** A head splits into an independent QK circuit (where to look) and OV circuit (what to write). The induction OV circuit is a copy map, legible in the vocabulary basis as a diagonal. The QK circuit is the stripe from Part 4. Together they are the whole head.\n"
]
},
{
"cell_type": "markdown",
"id": "ba306b0a",
"metadata": {},
"source": [
"### Experiment log\n",
"\n",
"The record of what we built and measured, the expected-value reference for your own run. If your numbers are far from these, something upstream broke; re-run from Setup.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "9c123c95",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:11.007514Z",
"iopub.status.busy": "2026-06-10T20:56:11.007441Z",
"iopub.status.idle": "2026-06-10T20:56:11.009739Z",
"shell.execute_reply": "2026-06-10T20:56:11.009448Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"metric value\n",
"------------------------------------------------------\n",
"parameters 26,304\n",
"training steps (FAST=False) 320\n",
"final training loss 0.0003\n",
"held-out next-token accuracy 100.0%\n",
"strongest induction score (L1 H2) 0.79\n",
"accuracy after ablating induction layer 77.4%\n",
"OOD repeat accuracy 100.0%\n",
"OV copying fraction 96%\n"
]
}
],
"source": [
"# The chapter artifact, summarized. (Values are from the full run; FAST shifts them slightly.)\n",
"print(f\"{'metric':<42} {'value':>10}\")\n",
"print(\"-\" * 54)\n",
"print(f\"{'parameters':<42} {n_params:>10,}\")\n",
"print(f\"{'training steps (FAST=%s)' % FAST:<42} {STEPS:>10}\")\n",
"print(f\"{'final training loss':<42} {losses[-1]:>10.4f}\")\n",
"print(f\"{'held-out next-token accuracy':<42} {clean_acc:>9.1%}\")\n",
"print(f\"{'strongest induction score (L%d H%d)' % (best_layer, best_head):<42} {scores.max():>10.2f}\")\n",
"print(f\"{'accuracy after ablating induction layer':<42} {all_acc:>9.1%}\")\n",
"print(f\"{'OOD repeat accuracy':<42} {ood_acc:>9.1%}\")\n",
"print(f\"{'OV copying fraction':<42} {copying_fraction:>9.0%}\")"
]
},
{
"cell_type": "markdown",
"id": "8ff8a4ae",
"metadata": {},
"source": [
"> **Interpretation.** Eight numbers reproduce the entire chapter claim: a small model learns the copy task to near-perfect accuracy, the behavior is carried by a layer-1 induction head with a high stripe score, ablating that layer breaks it, the circuit generalizes OOD, and the head's OV circuit is a copy map. That is a reverse-engineered circuit, start to finish.\n"
]
},
{
"cell_type": "markdown",
"id": "3fa9c914",
"metadata": {},
"source": [
"## Safety lens\n",
"\n",
"This is the chapter where the safety case *is* the chapter. The bet behind mech interp is that, for powerful systems, behavioral evaluation alone will not be enough (evaluations can be gamed, capabilities can be hidden), and we will need to look inside and check. The induction head you just reverse-engineered is the smallest demonstration that \"looking inside and checking\" is a real, repeatable thing, not a hope.\n",
"\n",
"Three concrete safety applications, with their honest current state:\n",
"\n",
"- **Refusal-direction monitoring.** Arditi et al. 2024 showed chat models encode refusal in a low-dimensional, often nearly one-dimensional subspace of the residual stream. You find it by contrasting activations on harmful versus harmless prompts. You can ablate it (project the stream onto its orthogonal complement) and the model stops refusing, or amplify it and it refuses more. Double-edged: defenders get a deployment-time tripwire (log each prompt's projection onto the refusal direction, flag compliance while the direction is elevated, roughly 50 lines), and attackers get a recipe for cheap fine-tuning jailbreaks. The arms race is live.\n",
"- **Deception and sycophancy features.** \"Scaling Monosemanticity\" (Anthropic 2024) found SAE features for \"deception in role-play\", \"flattery without basis\", and \"withholding information\". Real and reproducible on Claude 3 Sonnet. Whether they generalize to a future model's deliberate deception under deployment is the open question; current methods catch simple cases and miss subtle ones.\n",
"- **Backdoor detection.** \"Sleeper Agents\" (Hubinger et al. 2024) trained models with triggered backdoors that survived standard safety training. Follow-up interp work asks whether probes or SAEs can detect the \"I am in deployment\" state. Detection works on the specific backdoors tested; generalization to novel triggers is unreliable.\n",
"\n",
"The discipline the induction head taught us is the discipline these applications demand. We found a head whose *pattern* looked like induction (correlation), then *ablated* it and measured the damage (causation), then verified *out of distribution* (it is an algorithm, not a memory). The cardinal sin in a safety claim is to stop at the first step. \"The model has a refusal feature\" is a correlational claim until you ablate the direction and watch refusal change, and even then it holds only on the distribution you tested. Do not write \"the model thinks X\" without an intervention experiment, and state the distribution your claim covers. The cost of a wrong interpretation in a safety argument is hard to walk back.\n"
]
},
{
"cell_type": "markdown",
"id": "30803297",
"metadata": {},
"source": [
"### Optional capstone — induction heads in real GPT-2 (requires `transformer_lens`)\n",
"\n",
"Everything above used a transformer we built and trained ourselves, so it runs anywhere with torch. The same induction-stripe detector applied to a real pretrained GPT-2 small finds genuine induction heads in a 124M-parameter model. That requires the `transformer_lens` library and a model download, neither of which is on the canonical path. The cell below runs it *if and only if* `transformer_lens` is already installed, and otherwise prints exactly what it would have shown. It never errors and never blocks run-all.\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "4abe94b8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:11.010442Z",
"iopub.status.busy": "2026-06-10T20:56:11.010373Z",
"iopub.status.idle": "2026-06-10T20:56:11.013849Z",
"shell.execute_reply": "2026-06-10T20:56:11.013532Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"skipped: `transformer_lens` not installed.\n",
"If installed, this would load gpt2-small, run it on a random-token repeat sequence,\n",
"cache attention patterns, and apply our induction_score detector to all 144 heads.\n",
"Expected result: a handful of heads (e.g. layer 5 head 1, layer 6 head 9) score high,\n",
"reproducing the same stripe signature in a real model that we found in our toy one.\n"
]
}
],
"source": [
"# deeper: real GPT-2 induction heads. Optional, network + extra dependency. Print-and-skip if absent.\n",
"import importlib.util\n",
"USE_TRANSFORMER_LENS = importlib.util.find_spec(\"transformer_lens\") is not None\n",
"\n",
"if not USE_TRANSFORMER_LENS:\n",
" print(\"skipped: `transformer_lens` not installed.\")\n",
" print(\"If installed, this would load gpt2-small, run it on a random-token repeat sequence,\")\n",
" print(\"cache attention patterns, and apply our induction_score detector to all 144 heads.\")\n",
" print(\"Expected result: a handful of heads (e.g. layer 5 head 1, layer 6 head 9) score high,\")\n",
" print(\"reproducing the same stripe signature in a real model that we found in our toy one.\")\n",
"else:\n",
" from transformer_lens import HookedTransformer\n",
" gpt2 = HookedTransformer.from_pretrained(\"gpt2\", device=\"cpu\")\n",
" torch.manual_seed(SEED)\n",
" seq = torch.randint(1000, 5000, (1, 50))\n",
" rep = torch.cat([seq, seq], dim=-1)\n",
" _, cache = gpt2.run_with_cache(rep)\n",
" seq_len = rep.shape[1]\n",
" best = []\n",
" for layer in range(gpt2.cfg.n_layers):\n",
" pattern = cache[\"pattern\", layer] # (1, n_heads, seq, seq)\n",
" s = induction_score(pattern, seq_len) # our detector, unchanged\n",
" for h in range(gpt2.cfg.n_heads):\n",
" best.append((float(s[h]), layer, h))\n",
" best.sort(reverse=True)\n",
" print(\"top induction heads in real gpt2-small (our detector):\")\n",
" for sc, layer, h in best[:5]:\n",
" print(f\" layer {layer:>2} head {h:>2}: induction score {sc:.2f}\")"
]
},
{
"cell_type": "markdown",
"id": "f2254e6e",
"metadata": {},
"source": [
"> **Interpretation.** If you have `transformer_lens`, the very same `induction_score` function you wrote for the toy model finds induction heads in real GPT-2. The mechanism we reverse-engineered in seconds on a 2-layer model is genuinely present in a model trained on the open internet. That is the existence proof the field is built on: a real, useful behavior implemented by a specific, findable circuit.\n"
]
},
{
"cell_type": "markdown",
"id": "18a46893",
"metadata": {},
"source": [
"## Test yourself\n",
"\n",
"Three parts: concept self-checks with folded answers, two auto-checked problems, and a capstone with a rubric and a folded reference. Every answer is in this notebook; if unsure, re-run that section. Try before you peek.\n"
]
},
{
"cell_type": "markdown",
"id": "3f0d3776",
"metadata": {},
"source": [
"### Part A — Concepts\n",
"\n",
"1. What does mech interp try to produce, and how is it different from behavioral evaluation? AnswerA *program*: a description of the computation the model runs, in human-auditable variables and operations. Behavioral evaluation measures what the model does on average from the outside; mech interp explains *how*, from the inside.\n",
"2. Why does looking at a single MLP neuron usually fail to tell you what it \"means\"? AnswerPolysemanticity: one neuron responds to many unrelated features, because the model is in superposition (more features than neurons, packed as sparse near-orthogonal directions). The right basis is not the neuron basis. We saw this in the Part 1 toy.\n",
"3. Why is the final-logit decomposition into per-component contributions *exact* in our model but only approximate in a real one? AnswerOur model has no LayerNorm. With LayerNorm, the final stream is rescaled per-position before the unembed, so the decomposition is linear only after folding in that per-position scale (the logit-lens correction).\n",
"4. On a length-`2H` repeat sequence, which key position does an induction head attend to from query position `t`, and why? AnswerPosition `t - H + 1`. The current token at `t` equals the token at `t - H`; its successor (what to copy) is at `t - H + 1`. That is the induction stripe.\n",
"5. You ablate the single strongest induction head and accuracy barely moves. What does this tell you, and what should you do? AnswerThe circuit is redundant (backup heads). Single-component ablation under-reads importance. Ablate the whole set of induction heads (or the layer) to see the real effect, as in Part 5.\n",
"6. Why is the out-of-distribution test (fresh random tokens, same repeat structure) stronger evidence than high accuracy on the training distribution? AnswerHigh in-distribution accuracy is consistent with memorization. Generalizing to never-seen tokens shows the model learned the structure-dependent *algorithm*. The non-repeat negative control (chance accuracy) completes the dissociation.\n",
"7. A linear probe decodes a concept from layer 6 at 95%. Name the one experiment that converts \"the model represents this\" into \"the model uses this\". AnswerA causal intervention: ablate or steer the probed direction and measure whether the behavior changes. Probes are correlational; only intervention establishes use.\n",
"8. In the activation-patch experiment, why did patching the *early* residual at the source position recover more than the late residual? AnswerThe prev-token / \"what token am I\" information enters the stream at the source position early (layer 0). The layer-1 induction head reads it from there. Restoring the early stream rescues the downstream computation; restoring it late, after the head has already read corrupt values, does less.\n",
"9. What are the QK and OV circuits of a head, in one phrase each? AnswerQK = where to look (which positions attend to which). OV = what to write (how an attended token is transformed into the output it contributes). For induction: QK = the stripe, OV = copy.\n",
"10. Compute, in one line, the mean of each row of a `(seq, seq)` attention matrix `A` (you will use this kind of reduction constantly). Answer`A.mean(dim=1)` (average over the key axis), giving one number per query position.\n"
]
},
{
"cell_type": "markdown",
"id": "86038f5c",
"metadata": {},
"source": [
"### Part B1 — The patching recovery metric\n",
"`Difficulty 1/5 · ~5 min`\n",
"\n",
"The recovery score normalizes a patched logit difference between the corrupt baseline (0) and the clean target (1). Implement it: `recovery(clean, corrupt, patched) = (patched - corrupt) / (clean - corrupt)`. Return `0.0` if clean equals corrupt (degenerate, nothing to recover).\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "17b5648d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:11.014742Z",
"iopub.status.busy": "2026-06-10T20:56:11.014653Z",
"iopub.status.idle": "2026-06-10T20:56:11.018084Z",
"shell.execute_reply": "2026-06-10T20:56:11.017844Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] B1 recovery metric: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def recovery(clean, corrupt, patched):\n",
" # TODO: normalized recovery; 0 = no recovery (== corrupt), 1 = full recovery (== clean)\n",
" result = None\n",
" attempted(result)\n",
" return result\n",
"\n",
"def _rec():\n",
" check_close(recovery(10.0, -10.0, -10.0), 0.0, msg=\"patched==corrupt -> 0 recovery\")\n",
" check_close(recovery(10.0, -10.0, 10.0), 1.0, msg=\"patched==clean -> full recovery\")\n",
" check_close(recovery(10.0, -10.0, 0.0), 0.5, msg=\"halfway -> 0.5\")\n",
" assert recovery(5.0, 5.0, 5.0) == 0.0, \"degenerate clean==corrupt must return 0.0, not divide by zero\"\n",
"check(\"B1 recovery metric\", _rec)"
]
},
{
"cell_type": "markdown",
"id": "1227330f",
"metadata": {},
"source": [
"Hint 1Translate the formula directly. Guard the denominator: if `clean == corrupt`, return `0.0` before dividing.\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"if clean == corrupt:\n",
" return 0.0\n",
"return (patched - corrupt) / (clean - corrupt)\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "6e858298",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:11.018935Z",
"iopub.status.busy": "2026-06-10T20:56:11.018863Z",
"iopub.status.idle": "2026-06-10T20:56:11.021569Z",
"shell.execute_reply": "2026-06-10T20:56:11.021174Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] B1 recovery metric\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines recovery; the check re-verifies the reference.\n",
"def recovery(clean, corrupt, patched):\n",
" if clean == corrupt:\n",
" return 0.0\n",
" return (patched - corrupt) / (clean - corrupt)\n",
"\n",
"check(\"B1 recovery metric\", _rec, required=True)"
]
},
{
"cell_type": "markdown",
"id": "f19c8a09",
"metadata": {},
"source": [
"### Part B2 — Direct logit attribution of one component\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"DLA projects a component's residual-stream output onto the logit-difference direction. Given a component output vector and the unembedding, the component's contribution to `logit(correct) - logit(incorrect)` is `component @ (W_U[correct] - W_U[incorrect])`. Implement `dla(component, W_U, correct, incorrect)`. This is the scalar that says \"how much did this component push toward the correct token over the incorrect one\".\n"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "3f1add84",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:11.022183Z",
"iopub.status.busy": "2026-06-10T20:56:11.022115Z",
"iopub.status.idle": "2026-06-10T20:56:11.026054Z",
"shell.execute_reply": "2026-06-10T20:56:11.025762Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] B2 direct logit attribution: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def dla(component, W_U, correct, incorrect):\n",
" \"\"\"component: (d_model,). W_U: (vocab, d_model) unembedding (rows are token directions).\n",
" Return the scalar contribution to logit(correct) - logit(incorrect).\"\"\"\n",
" # TODO 1: the logit-difference direction u = W_U[correct] - W_U[incorrect] (shape d_model)\n",
" u = None\n",
" attempted(u)\n",
" # TODO 2: project the component onto u (a dot product) and return a Python float\n",
" return float(component @ u)\n",
"\n",
"def _dla():\n",
" torch.manual_seed(SEED)\n",
" W_U = torch.randn(7, 5) # (vocab=7, d_model=5)\n",
" comp = torch.randn(5)\n",
" got = dla(comp, W_U, correct=2, incorrect=4)\n",
" want = float(comp @ (W_U[2] - W_U[4])) # independent reference\n",
" check_close(got, want, msg=\"projection onto W_U[correct]-W_U[incorrect]\")\n",
" # a component proportional to +u should give a positive contribution:\n",
" u = W_U[2] - W_U[4]\n",
" assert dla(u, W_U, 2, 4) > 0, \"a component along +u must push toward the correct token\"\n",
"check(\"B2 direct logit attribution\", _dla)"
]
},
{
"cell_type": "markdown",
"id": "0c02e720",
"metadata": {},
"source": [
"Hint 1 (conceptual)The unembedding row `W_U[token]` is the direction in residual space that increases that token's logit. The difference of two rows is the \"prefer correct over incorrect\" direction. Dot the component with it.\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"u = W_U[correct] - W_U[incorrect] # (d_model,)\n",
"return float(component @ u)\n",
"```\n",
"\n",
"Help — \"shape mismatch in the matmul\"`W_U` here has shape `(vocab, d_model)`, so `W_U[correct]` is already a `(d_model,)` vector. If your `W_U` is transposed `(d_model, vocab)`, index columns instead: `W_U[:, correct]`.\n"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "9e51e8eb",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:56:11.026913Z",
"iopub.status.busy": "2026-06-10T20:56:11.026851Z",
"iopub.status.idle": "2026-06-10T20:56:11.029844Z",
"shell.execute_reply": "2026-06-10T20:56:11.029418Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] B2 direct logit attribution\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines dla; the check re-verifies the reference.\n",
"def dla(component, W_U, correct, incorrect):\n",
" u = W_U[correct] - W_U[incorrect] # (d_model,) logit-diff direction\n",
" return float(component @ u)\n",
"\n",
"check(\"B2 direct logit attribution\", _dla, required=True)"
]
},
{
"cell_type": "markdown",
"id": "e0fed868",
"metadata": {},
"source": [
"### Part C — Capstone: a mini circuit report\n",
"\n",
"Reverse-engineer the induction circuit one more time, but as a *report* you could hand to another researcher, and push one finding further than the notebook did. Deliverables:\n",
"\n",
"1. Retrain the model with a **different config** (try `D_MODEL = 32` or `N_HEADS = 8` or `STEPS` doubled) and confirm an induction head still emerges. Report the score and which head.\n",
"2. Produce the induction-stripe attention plot and the OV copying fraction for the new run.\n",
"3. Run the ablation experiment and the OOD test on the new model; report whether the conclusions hold.\n",
"4. Pick one open question and investigate it in code: *How many heads in layer 1 are induction heads at this config, and does ablating them one-at-a-time vs all-at-once reveal the redundancy?* Report a table.\n",
"5. A 150-word writeup that states the circuit's algorithm in one sentence, names one thing you verified causally, and names one thing you did *not* establish (be honest about the limit).\n",
"\n",
"Self-assessment (pass / partial / fail): (a) an induction head emerges at the new config and you show its score; (b) at least one *causal* result (ablation or patch), not just the attention plot; (c) the OOD test is present and you state whether it generalized; (d) the redundancy investigation produces a table, not a sentence; (e) the writeup names a genuine limit (\"I did not establish X because...\").\n",
"\n",
"My solution (reference, ~30 s on CPU)\n",
"\n",
"```python\n",
"# Retrain wider, confirm the circuit survives.\n",
"torch.manual_seed(SEED)\n",
"cap = TinyTransformer(VOCAB, d_model=32, n_heads=8, n_layers=2, max_seq_len=SEQ_LEN)\n",
"cap_losses = train(cap, STEPS, seed=SEED)\n",
"gcap = torch.Generator().manual_seed(SEED + 100)\n",
"cap_batch = make_batch(BATCH, SEQ_LEN, VOCAB, gcap)\n",
"cap_scores = all_head_scores(cap, cap_batch)\n",
"bl, bh = np.unravel_index(int(cap_scores.argmax()), cap_scores.shape)\n",
"print(f\"d_model=32 n_heads=8 -> best induction head L{int(bl)}H{int(bh)}, score {cap_scores.max():.2f}\")\n",
"\n",
"# Redundancy table: leave-one-out vs leave-all-out for layer-1 induction heads.\n",
"ind = [h for h in range(cap.n_heads) if cap_scores[1, h] > 0.5]\n",
"_, base_acc = eval_metrics(cap, cap_batch)\n",
"print(f\"clean acc {base_acc:.1%} · {len(ind)} induction heads in layer 1\")\n",
"for h in ind:\n",
" _, a = eval_metrics(cap, cap_batch, ablate={1: [h]})\n",
" print(f\" ablate only head {h}: acc {a:.1%} (drop {base_acc - a:.1%})\")\n",
"_, a_all = eval_metrics(cap, cap_batch, ablate={1: ind})\n",
"print(f\" ablate all {len(ind)}: acc {a_all:.1%} (drop {base_acc - a_all:.1%})\")\n",
"\n",
"# OOD check holds:\n",
"_, cap_ood = eval_metrics(cap, make_batch(BATCH, SEQ_LEN, VOCAB, torch.Generator().manual_seed(99999)))\n",
"print(f\"OOD accuracy {cap_ood:.1%} -> algorithm generalizes at this config too\")\n",
"```\n",
"\n",
"The leave-one-out drops are each small while the leave-all-out drop is large: that *is* the redundancy, quantified. The one-sentence algorithm: \"at each second-half position, attend back to the token after the previous copy of the current token, and copy it.\" Verified causally by the leave-all-out ablation. Not established: *which* layer-0 computation supplies the matching key (we localized it to the early stream with patching, but did not decompose the layer-0 QK circuit). That is the next experiment.\n"
]
},
{
"cell_type": "markdown",
"id": "c3b411b3",
"metadata": {},
"source": [
"## Reflection\n",
"\n",
"Write ~150 words on the dumbest bug you hit in this notebook and how you found it. The off-by-one in the induction stripe (`t - half` vs `t - half + 1`) and the missing `zero_grad` are the usual suspects; maybe yours was a transpose in the OV matrix or a generator you forgot to seed. What was the symptom? What did you print to localize it? How long did you stare before you saw it? Nobody grades this. Writing it is the point: the muscle you are building is *debugging interpretability code*, where a silent index error produces a plot that looks plausible and tells you the opposite of the truth. That failure mode, a confident wrong interpretation, is the one the whole field is most afraid of. The habit of distrusting a beautiful plot until a causal check confirms it is the most transferable thing in this chapter.\n"
]
},
{
"cell_type": "markdown",
"id": "959fae1c",
"metadata": {},
"source": [
"## Going further\n",
"\n",
"- Olsson et al. 2022, *In-context Learning and Induction Heads* — the paper behind everything in this notebook. Read it now that you have built the result.\n",
"- Elhage et al. 2021, *A Mathematical Framework for Transformer Circuits* — the residual-stream and QK/OV decomposition, formalized.\n",
"- ARENA 3.0, *Chapter 1.2 Intro to Mech Interp* — the best hands-on curriculum for this material; the `transformer_lens` version of what you just did by hand.\n",
"- Neel Nanda's blog, *Induction Heads Illustrated* and *An Opinionated List of Favourite Mech Interp Papers* — the field's reading map.\n",
"- Wang et al. 2023, *Interpretability in the Wild (IOI)* — the first real-model circuit; activation and path patching at scale.\n",
"- Anthropic 2024, *Scaling Monosemanticity*, and 2025, *On the Biology of a Large Language Model* — SAEs and attribution graphs at frontier scale.\n",
"- `neuronpedia.org` — click through SAE features for an hour; the dashboards teach better than any paper.\n",
"- `transformer_lens` (the `pip install transformer-lens` library) — run the optional capstone above on real GPT-2.\n",
"\n",
"## What this enables\n",
"\n",
"- **Ch 23 — Evaluation Science**: once you can locate a direction (refusal, a feature), you can write evals that check it directly instead of relying on behavioral red-teams that can be gamed.\n",
"- **Ch 24 — Safety and Red-Team**: refusal monitoring, persona vectors, and backdoor probes are the working substrate of safety arguments; this chapter is their vocabulary.\n",
"- **Ch 25 — MLOps**: monitoring a deployed model's interp signals (refusal-direction projection, feature firing rates) is a frontier operational pattern with no standard tooling yet.\n",
"- **Ch 26 — Reading Papers**: half the papers worth reading in 2026 are mech-interp papers. You now have the vocabulary to read them, not skim them.\n",
"\n",
"The gap this notebook leaves: we reverse-engineered a 2-layer toy. The same `induction_score` detector finds real induction heads in GPT-2 (the optional capstone), but reverse-engineering a *single behavior* in a 124M model (the IOI circuit) took the field months, and reverse-engineering a frontier model end-to-end is not yet possible. SAEs and attribution graphs are the field's bet on scaling past the toy. You have done the toy honestly; that is where everyone real started.\n"
]
},
{
"cell_type": "markdown",
"id": "58ddefda",
"metadata": {},
"source": [
"---\n",
"*Built top-to-bottom. If every check above printed `[ ok ]`, you have reverse-engineered an induction circuit from scratch: trained the model, found the head, ablated it, verified it OOD, and read its OV circuit. Total running time and verified-on stamp written by CI.*\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "obvix-nb",
"language": "python",
"name": "obvix-nb"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"obvix": {
"title": "Ch 22 — Mechanistic Interpretability"
}
},
"nbformat": 4,
"nbformat_minor": 5
}