{
"cells": [
{
"cell_type": "markdown",
"id": "6d973b62",
"metadata": {},
"source": [
"# Ch 17 — Efficient Inference (notebook)\n",
"\n",
"`[<- 16 multimodal-transformers]` · **this notebook** · `[18 generative-models ->]`\n",
"\n",
"Runs top-to-bottom in ~4 min on free Colab CPU. Last verified 2026-06-11.\n",
"\n",
"**What you'll build**\n",
"- A tiny GPT (same architecture as Ch 15), then a KV cache bolted onto its attention, with a `torch.equal` correctness proof and a wall-clock **timing** proof that caching is faster.\n",
"- The arithmetic-intensity calculator: feed it a layer's shapes and it tells you compute-bound vs memory-bound, and where the memory-bandwidth wall sits on real hardware.\n",
"- An int8 round-trip quantizer you write from scratch, with a measured error bound, then a per-tensor-vs-per-channel comparison that shows why granularity matters.\n",
"- A deliberate bug: a \"cached\" decode that returns the wrong tokens because the position index is off by the cache length. You see it fail, then fix it.\n",
"\n",
"**How this notebook works.** Code cells with a `# TODO` are yours to fill in. Run the cell to grade yourself: `[ ok ]` passed, `[FAIL]` shows what went wrong, `[ -- ]` means not attempted yet. Every exercise has a hint ladder (open only as many as you need) and a folded solution below it. The notebook runs top-to-bottom even if you fill in nothing: the folded solutions redefine the functions so the later cells work. See Ch 00 for the full protocol.\n"
]
},
{
"cell_type": "markdown",
"id": "ae4c70e6",
"metadata": {},
"source": [
"## Before you start\n",
"\n",
"1. You generate token number 1000 from a transformer. With no cache, attention at that step costs work proportional to what? AnswerThe context length, ~1000, because the new query attends to all 1000 keys. Generating the whole 1000-token sequence that way is the sum 1+2+...+1000, which is O(T^2) attention work per layer across the run, and re-deriving every old K and V each step is the wasteful part. The KV cache removes the re-derivation.\n",
"2. An H100 does ~989 TFLOP/s of compute but only moves ~3.35 TB/s from memory. If a kernel reads 140 GB of weights and does one matrix-vector product (a few hundred MFLOP), is it waiting on the math or on the bytes? AnswerOn the bytes. 140 GB / 3.35 TB/s is ~42 ms of transfer; the FLOPs finish in microseconds. The chip is starved. This is \"memory-bound\", and single-token generation is the canonical example.\n",
"3. Predict before you run: you cast a tensor of FP32 weights to int8 and back. Roughly how large is the worst-case round-trip error, in units of the quantization step `s`? AnswerAbout `s/2`. Rounding to the nearest integer level can be off by at most half a step. The mean squared error of uniform quantization is about `s^2/12`. You will measure both in Part 4.\n"
]
},
{
"cell_type": "markdown",
"id": "86cbd4c2",
"metadata": {},
"source": [
"## Setup\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "91ceeb82",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:44.912784Z",
"iopub.status.busy": "2026-06-10T19:24:44.912712Z",
"iopub.status.idle": "2026-06-10T19:24:49.141618Z",
"shell.execute_reply": "2026-06-10T19:24:49.141196Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"numpy 2.2.6 · torch 2.12.0+cpu\n",
"device cpu (this notebook is CPU-canonical; GPU sections print-and-skip)\n"
]
}
],
"source": [
"import numpy as np\n",
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import matplotlib.pyplot as plt\n",
"print(f\"numpy {np.__version__} · torch {torch.__version__}\")\n",
"if np.__version__ < \"2.0\":\n",
" print(\"WARN: written for NumPy 2.x; older versions may differ slightly\")\n",
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
"print(f\"device {device} (this notebook is CPU-canonical; GPU sections print-and-skip)\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "595837e6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.143215Z",
"iopub.status.busy": "2026-06-10T19:24:49.143055Z",
"iopub.status.idle": "2026-06-10T19:24:49.173986Z",
"shell.execute_reply": "2026-06-10T19:24:49.173188Z"
}
},
"outputs": [],
"source": [
"import os, random, math, time\n",
"SEED = 0\n",
"FAST = bool(os.environ.get(\"NB_FAST\")) # CI smoke mode: smaller model + fewer steps, same code paths\n",
"TRAIN_STEPS = 60 if FAST else 400 # tiny-GPT fit steps; loss log documents both settings below\n",
"GEN_TOKENS = 24 if FAST else 64 # tokens to generate in the correctness demos\n",
"TIME_GEN = 48 if FAST else 80 # tokens for the TIMING demos: long enough that recompute's\n",
" # O(T^2) reliably dominates the cache's per-step overhead (see Part 3)\n",
"rng = np.random.default_rng(SEED)\n",
"torch.manual_seed(SEED)\n",
"random.seed(SEED)\n",
"torch.set_num_threads(2) # steadier wall-clock timings on a shared CPU; not a correctness knob\n",
"\n",
"# ── house self-check harness (identical across all chapter notebooks) ──\n",
"import numpy as _np\n",
"\n",
"def check(label, test_fn, required=False):\n",
" \"\"\"Run one self-check. test_fn raises AssertionError (with a teaching\n",
" message) on failure, NotImplementedError if the stub is unfilled.\n",
" required=True is used only in solution cells; it is what CI grades.\"\"\"\n",
" try:\n",
" test_fn()\n",
" except NotImplementedError:\n",
" if required:\n",
" raise AssertionError(f\"{label}: reference solution incomplete\")\n",
" print(f\"[ -- ] {label}: not attempted yet — fill in the TODO above, then re-run.\")\n",
" return False\n",
" except AssertionError as e:\n",
" if required:\n",
" raise\n",
" print(f\"[FAIL] {label}: {e}\")\n",
" return False\n",
" print(f\"[ ok ] {label}\")\n",
" return True\n",
"\n",
"def attempted(*vals):\n",
" \"\"\"Treat None placeholders as 'not attempted'.\"\"\"\n",
" if any(v is None for v in vals):\n",
" raise NotImplementedError\n",
"\n",
"def check_shape(x, want):\n",
" assert tuple(x.shape) == tuple(want), \\\n",
" f\"shape {tuple(x.shape)}, expected {tuple(want)} — check your reshape/transpose order\"\n",
"\n",
"def check_close(got, want, atol=1e-5, rtol=1e-4, msg=\"\"):\n",
" g, w = _np.asarray(got, dtype=float), _np.asarray(want, dtype=float)\n",
" assert g.shape == w.shape, f\"shape {g.shape} vs expected {w.shape}. {msg}\"\n",
" bad = ~_np.isclose(g, w, atol=atol, rtol=rtol)\n",
" assert not bad.any(), \\\n",
" f\"{bad.mean():.2%} of values wrong (max diff {abs(g - w).max():.3g}). {msg}\"\n",
"\n",
"# ── small house helpers used across this notebook (defined once, never imported) ──\n",
"def param_count(module):\n",
" \"Total number of parameters in an nn.Module.\"\n",
" return sum(p.numel() for p in module.parameters())\n",
"\n",
"def time_call(fn, *args, runs=3, **kwargs):\n",
" # Median wall-clock seconds over `runs` timed calls, after one warmup call.\n",
" # Timing is noisy on CPU; the median over a few runs is the stable summary.\n",
" fn(*args, **kwargs) # warmup: triggers any lazy allocation / cache fill\n",
" times = []\n",
" for _ in range(runs):\n",
" t0 = time.perf_counter()\n",
" fn(*args, **kwargs)\n",
" times.append(time.perf_counter() - t0)\n",
" times.sort()\n",
" return times[len(times) // 2]"
]
},
{
"cell_type": "markdown",
"id": "9e3f6f81",
"metadata": {},
"source": [
"> **Note:** seeds make this notebook's printed numbers reproduce on CPU. Wall-clock timings are different: they depend on your machine's load, thread count, and BLAS, so the absolute milliseconds will not match the page. What is stable, and what every timing claim below asserts, is the *ratio* (cached generation is faster than recompute) and the *direction* of every trend. If your speedup is 4.1x and the page says 3.7x, you did nothing wrong.\n"
]
},
{
"cell_type": "markdown",
"id": "6e71b647",
"metadata": {},
"source": [
"## The map\n",
"\n",
"> **Part 1 — The memory wall.** Compute arithmetic intensity for a matmul, derive the break-even ridge from hardware specs, and watch single-token generation fall far below it. This inequality is why the rest of the chapter exists.\n",
"> **Part 2 — A tiny GPT to make fast.** Build and briefly fit the same small transformer as Ch 15. It is the model every later part optimizes; nothing here is a black box.\n",
"> **Part 3 — The KV cache.** Add a cache to attention, prove it returns *identical* tokens to the recompute baseline (`torch.equal`), then prove with a stopwatch that it is faster. Includes the last-position trick and a deliberate off-by-cache-length bug.\n",
"> **Part 4 — Quantization on CPU.** Write int8 linear quantization from scratch, measure the `s/2` error bound, compare per-tensor vs per-channel granularity, and run a real int8 matmul through `torch.ao` to halve the bytes.\n",
"> **Part 5 — Accounting that scales.** KV-cache memory in bytes, MQA/GQA reductions, and a back-of-the-envelope MFU number for our own model. The capstone assembles a cached generator and a roofline read.\n"
]
},
{
"cell_type": "markdown",
"id": "4f9a6433",
"metadata": {},
"source": [
"## Part 1 — The memory wall: arithmetic intensity\n",
"\n",
"Every GPU operation has two costs: the math (FLOPs) and the bytes it has to move to and from memory. Their ratio is the *arithmetic intensity*:\n",
"\n",
"$$\\text{intensity} = \\frac{\\text{FLOPs}}{\\text{bytes read} + \\text{bytes written}}$$\n",
"\n",
"A chip has a peak compute rate (FLOP/s) and a peak memory bandwidth (bytes/s). Divide them and you get a single number, the *ridge point*: the intensity at which a kernel stops waiting on memory and starts being limited by the math.\n",
"\n",
"$$\\text{ridge} = \\frac{\\text{peak FLOP/s}}{\\text{peak bytes/s}}$$\n",
"\n",
"Below the ridge a kernel is **memory-bound** (the silicon idles, waiting for bytes). Above it the kernel is **compute-bound** (you are using the chip). The whole chapter lives below this line.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "ebd916b2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.175355Z",
"iopub.status.busy": "2026-06-10T19:24:49.174925Z",
"iopub.status.idle": "2026-06-10T19:24:49.180794Z",
"shell.execute_reply": "2026-06-10T19:24:49.179931Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"A100-80GB ridge = 312.0 TFLOP/s / 2.039 TB/s = 153.0 FLOP/byte\n",
"H100-80GB ridge = 989.5 TFLOP/s / 3.350 TB/s = 295.4 FLOP/byte\n",
"H200 ridge = 989.5 TFLOP/s / 4.800 TB/s = 206.1 FLOP/byte\n"
]
}
],
"source": [
"# Hardware spec sheet (published peak numbers; these are the magic constants, and here is the source).\n",
"# H100 SXM BF16: 989.5 TFLOP/s dense matmul; 3.35 TB/s HBM3 bandwidth (NVIDIA H100 datasheet).\n",
"HW = {\n",
" # name : (peak_flops, peak_bytes_per_sec)\n",
" \"A100-80GB\": (312e12, 2.039e12), # A100 BF16 312 TFLOP/s, 2.039 TB/s HBM2e\n",
" \"H100-80GB\": (989.5e12, 3.35e12), # H100 BF16 989.5 TFLOP/s, 3.35 TB/s HBM3\n",
" \"H200\": (989.5e12, 4.8e12), # H200: same compute as H100, 4.8 TB/s HBM3e\n",
"}\n",
"for name, (flops, bw) in HW.items():\n",
" ridge = flops / bw\n",
" print(f\"{name:11s} ridge = {flops/1e12:6.1f} TFLOP/s / {bw/1e12:.3f} TB/s = {ridge:6.1f} FLOP/byte\")"
]
},
{
"cell_type": "markdown",
"id": "25e999e0",
"metadata": {},
"source": [
"> **Interpretation.** The H100's ridge is ~295 FLOP/byte. Any kernel that does fewer than 295 floating-point operations per byte it touches will leave the compute units idle. Notice the H200 has a *lower* ridge than the H100: same compute, more bandwidth, so it tolerates lower-intensity (more memory-bound) work better. That is exactly the regime inference lives in.\n"
]
},
{
"cell_type": "markdown",
"id": "13f3d26a",
"metadata": {},
"source": [
"### Intensity of a dense matmul\n",
"\n",
"Take $X \\in \\mathbb{R}^{B \\times D}$ times $W \\in \\mathbb{R}^{D \\times F}$. The multiply does $2 B D F$ FLOPs (one multiply and one add per output element, summed over $D$). At 2 bytes per element (BF16) it reads $X$ and $W$ and writes the result, so the bytes are $2(BD + DF + BF)$. We compute both directly rather than trusting the simplified formula.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "d66b1ea7",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.182231Z",
"iopub.status.busy": "2026-06-10T19:24:49.181610Z",
"iopub.status.idle": "2026-06-10T19:24:49.188499Z",
"shell.execute_reply": "2026-06-10T19:24:49.188236Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"batch 1: intensity 1.00 FLOP/byte -> memory-bound on H100\n",
"batch 8: intensity 7.98 FLOP/byte -> memory-bound on H100\n",
"batch 32: intensity 31.69 FLOP/byte -> memory-bound on H100\n",
"batch 256: intensity 237.45 FLOP/byte -> memory-bound on H100\n"
]
}
],
"source": [
"def matmul_intensity(B, D, Fdim, dtype_bytes=2):\n",
" flops = 2 * B * D * Fdim\n",
" bytes_moved = dtype_bytes * (B * D + D * Fdim + B * Fdim)\n",
" return flops / bytes_moved\n",
"\n",
"# A Llama-ish MLP up-projection: D=4096, F=16384.\n",
"for B in [1, 8, 32, 256]:\n",
" inten = matmul_intensity(B, 4096, 16384)\n",
" bound = \"compute-bound\" if inten > 295 else \"memory-bound\"\n",
" print(f\"batch {B:4d}: intensity {inten:8.2f} FLOP/byte -> {bound} on H100\")"
]
},
{
"cell_type": "markdown",
"id": "f5f0c35d",
"metadata": {},
"source": [
"> **Predict:** in the limit $D, F \\gg B$ the intensity simplifies. To what? AnswerTo $B$, the batch size. The $DF$ weight-read term dominates the bytes ($2DF$), the FLOPs are $2BDF$, so the ratio approaches $2BDF / 2DF = B$. This is why the only way to make an MLP compute-bound is to batch many requests: a single token ($B=1$) sits at intensity ~1, deep in the memory-bound regime.\n"
]
},
{
"cell_type": "markdown",
"id": "704078f0",
"metadata": {},
"source": [
"### Exercise 17.1 — Why prefill is fast and generation is slow\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"Prefill processes the whole prompt at once: the MLP sees $B \\cdot T$ rows. Generation processes one new token per sequence: the MLP sees $B \\cdot 1$ rows. Same weights, wildly different intensity.\n",
"\n",
"Fill in `mlp_intensity(B, T, D, Fdim)` for the MLP up-projection during a forward pass that processes `T` tokens per sequence (so the row count is `B*T`). Reuse the byte accounting from `matmul_intensity`: the weight `W` is read once regardless of how many rows pass through it.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a4b5094e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.189576Z",
"iopub.status.busy": "2026-06-10T19:24:49.189502Z",
"iopub.status.idle": "2026-06-10T19:24:49.198484Z",
"shell.execute_reply": "2026-06-10T19:24:49.198175Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 17.1 prefill >> decode: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def mlp_intensity(B, T, D, Fdim, dtype_bytes=2):\n",
" \"\"\"Intensity of (B*T, D) @ (D, Fdim). Prefill: T = prompt length. Decode: T = 1.\"\"\"\n",
" rows = B * T\n",
" # TODO 1: flops = 2 * rows * D * Fdim\n",
" flops = None\n",
" # TODO 2: bytes_moved reads X (rows*D), reads W (D*Fdim), writes Y (rows*Fdim); times dtype_bytes\n",
" bytes_moved = None\n",
" attempted(flops, bytes_moved)\n",
" return flops / bytes_moved\n",
"\n",
"def _prefill_beats_decode():\n",
" decode = mlp_intensity(B=1, T=1, D=4096, Fdim=16384) # one token, one sequence\n",
" prefill = mlp_intensity(B=1, T=512, D=4096, Fdim=16384) # a 512-token prompt at once\n",
" assert prefill > decode * 100, \\\n",
" f\"prefill intensity {prefill:.1f} should dwarf decode {decode:.1f}: T=512 rows reuse the same weight read\"\n",
" assert abs(decode - 1.0) < 0.1, \\\n",
" f\"decode intensity is ~1 (one matrix-vector product); got {decode:.2f} — did you forget rows=B*T with T=1?\"\n",
"\n",
"check(\"17.1 prefill >> decode\", _prefill_beats_decode)"
]
},
{
"cell_type": "markdown",
"id": "6a54aca6",
"metadata": {},
"source": [
"Hint 1 (conceptual)The only change from `matmul_intensity` is that the row count is `B*T` instead of `B`. The weight matrix `W` of shape `(D, Fdim)` is read exactly once no matter how many rows you push through it. That is the whole reason prefill is efficient: you amortize the weight read over `T` tokens.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"rows = B * T\n",
"flops = 2 * rows * D * Fdim\n",
"bytes_moved = dtype_bytes * (rows * D + D * Fdim + rows * Fdim)\n",
"```\n",
"\n",
"\n",
"Help — \"decode intensity is not ~1\"At `B=1, T=1` the rows term is 1, so bytes are `2*(D + D*Fdim + Fdim)` and FLOPs are `2*D*Fdim`. The ratio is `2*D*Fdim / (2*(D + D*Fdim + Fdim))`, which is just under 1 because the `D*Fdim` weight-read term dominates the denominator. If you got a huge number you probably used `rows = B` (forgot to multiply by T) and then divided wrong.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "9369c008",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.199374Z",
"iopub.status.busy": "2026-06-10T19:24:49.199301Z",
"iopub.status.idle": "2026-06-10T19:24:49.205329Z",
"shell.execute_reply": "2026-06-10T19:24:49.204819Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 17.1 prefill >> decode\n",
"decode (T=1): 1.00 FLOP/byte (memory-bound)\n",
"prefill (T=512): 442.81 FLOP/byte (compute-bound on H100)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines mlp_intensity; the check below re-verifies the reference.\n",
"def mlp_intensity(B, T, D, Fdim, dtype_bytes=2):\n",
" rows = B * T\n",
" flops = 2 * rows * D * Fdim\n",
" bytes_moved = dtype_bytes * (rows * D + D * Fdim + rows * Fdim)\n",
" return flops / bytes_moved\n",
"\n",
"check(\"17.1 prefill >> decode\", _prefill_beats_decode, required=True)\n",
"print(f\"decode (T=1): {mlp_intensity(1,1,4096,16384):8.2f} FLOP/byte (memory-bound)\")\n",
"print(f\"prefill (T=512):{mlp_intensity(1,512,4096,16384):8.2f} FLOP/byte (compute-bound on H100)\")"
]
},
{
"cell_type": "markdown",
"id": "ba096882",
"metadata": {},
"source": [
"> **Interpretation.** Same matrix, same model, two regimes. Prefill reads the weight once and runs hundreds of tokens through it, so it is compute-bound and fast. Decode reads the entire weight to produce one token, so it is memory-bound and slow. This single gap is why \"time to first token\" and \"time per output token\" are reported as separate numbers in every serving benchmark.\n"
]
},
{
"cell_type": "markdown",
"id": "bfa5bff3",
"metadata": {},
"source": [
"### The roofline, drawn\n",
"\n",
"A roofline plot puts intensity on the x-axis and achievable throughput on the y-axis. The \"roof\" has two pieces: a sloped memory-bound line (throughput = intensity x bandwidth) and a flat compute-bound ceiling (peak FLOP/s). A kernel's achievable speed is the lower of the two at its intensity. Where they meet is the ridge.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "baa2a121",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.206123Z",
"iopub.status.busy": "2026-06-10T19:24:49.206053Z",
"iopub.status.idle": "2026-06-10T19:24:49.862552Z",
"shell.execute_reply": "2026-06-10T19:24:49.862282Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAGGCAYAAACHemKmAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAkG9JREFUeJzs3Xd4FNXXwPHvZtMTkgAh1BB6CdKbCEgXUJoUEZTepXcQ6SCgVCE0eQVEehH4AQqIRIoISIdIlU5CJyEBUnbv+8eaDUsKKZtskj2f58nDzp07M2d2Zpazs3fu1SilFEIIIYQQQmQwNpYOQAghhBBCiOSQRFYIIYQQQmRIksgKIYQQQogMSRJZIYQQQgiRIUkiK4QQQgghMiRJZIUQQgghRIYkiawQQgghhMiQJJEVQgghhBAZkiSyQgghhBAiQ5JEVpjdlStX+OCDD3B3d0ej0bB161ZWrFiBRqPhxo0bxnq1a9emdu3aFoszsSZMmIBGo7F0GPEyd3wZ+Vilphs3bqDRaFixYoVF46hduzbvvPOORWOIVqBAATp37mzpMBKU1Ovjww8/pEePHqkYUeYVfY3MnDnT0qEky5vnSmRkJN7e3ixcuNCCUcXv+PHjvPfee7i4uKDRaDh9+rSlQ7IISWQzoehEJPrP1taWvHnz0rlzZ+7evZvq2+/UqRPnzp1j6tSprFq1ikqVKqX6NoXITO7du8eECROs9j8mSzl8+DB79uxh5MiRlg5FpAN2dnYMGTKEqVOn8urVK0uHYyIyMpI2bdrw5MkT5syZw6pVq/Dx8bF0WBZha+kAROqZNGkSBQsW5NWrV/z111+sWLGCQ4cOcf78eRwdHVNlmy9fvuTIkSOMGTOGfv36JVh3z549qRKDMD85Vmnr3r17TJw4kQIFClCuXDlLh2M1vv32W+rVq0eRIkUsHYpIJ7p06cKoUaNYs2YNXbt2tXQ4RteuXePmzZt8//33dO/e3dLhWJTckc3EGjduzOeff0737t1ZtmwZw4YN49q1a2zfvj3Vtvnw4UMAPDw83lrX3t4ee3v7VItFmI8cK/MLCwuzdAjiNQ8ePGDnzp188sknlg7FIuR8jJuHhwcffPBBspsUvXr1Cr1eb96gMJyvkLj/azM7SWStSM2aNQHDN7nX/f7779SsWRMXFxc8PDxo3rw5//zzT6zlT506RePGjXFzc8PV1ZV69erx119/GedPmDDB+NPG8OHD0Wg0FChQIN543mx36e/vj0ajYcOGDUydOpV8+fLh6OhIvXr1uHr1aqzljx49SqNGjXB3d8fZ2ZlatWpx+PDhWPUuXrzIrVu3Enxvoh06dIjKlSvj6OhI4cKFWbJkSbx1f/rpJypWrIiTkxPZsmXj008/5fbt23HG+eGHH5I1a1ZcXFwoU6YM8+bNM6mT2GOQGvElxuvH6v79+9ja2jJx4sRY9S5duoRGo2HBggXGsmfPnjFo0CC8vb1xcHCgSJEizJgxI9aH+7p166hYsSJZsmTBzc2N0qVLx3qf3hR9zvj7+5uUx9WetXPnzri6unL37l1atGiBq6srOXLkYNiwYeh0OpPlnz17RufOnXF3d8fDw4NOnTrx7NmzOGO4ePEirVu3Jlu2bDg6OlKpUqVYXxajm/v88ccffPHFF3h5eZEvX75496ly5cqA4W5QdBOhN/8jDQgIoE6dOjg7O5M3b16++eabWOsKDw9n/PjxFClSBAcHB7y9vRkxYgTh4eFxbvt1V65coVWrVuTKlQtHR0fy5cvHp59+SnBwcILL/fvvv7Rp04Zs2bLh7OzMu+++y86dO2Pto0ajYf369Xz55ZfkypULFxcXmjVrFu81lJhrPSnXx5t27txJVFQU9evXNymPPnaHDh1iwIAB5MiRAw8PD3r16kVERATPnj2jY8eOZM2alaxZszJixAiUUibr0Ov1zJ07l1KlSuHo6EjOnDnp1asXT58+NalXoEABmjRpgr+/P5UqVcLJyYnSpUsbz+8tW7ZQunRpHB0dqVixIqdOnYq1H4n5LIluCxoQEED79u3JmjUrNWrUYPny5Wg0mjjX+/XXX6PVahPdPG3OnDn4+Pjg5ORErVq1OH/+fLJi7dy5c5z/j8TV9lmj0dCvXz+2bt3KO++8g4ODA6VKleLXX3+NtXxSzpUGDRpw6NAhnjx5kuA+R5/X69at46uvviJv3rw4OzsTEhICwMaNG42fyZ6ennz++edxvp9ve186d+5MrVq1AGjTpg0ajca6n2FQItNZvny5AtTx48dNyhcsWKAAtWjRImPZ3r17la2trSpWrJj65ptv1MSJE5Wnp6fKmjWrun79urHe+fPnlYuLi8qdO7eaPHmymj59uipYsKBycHBQf/31l1JKqTNnzqg5c+YoQLVr106tWrVK/fzzzyYxvb7OWrVqqVq1ahmn9+/frwBVvnx5VbFiRTVnzhw1YcIE5ezsrKpUqWKyL/v27VP29vaqWrVqatasWWrOnDmqTJkyyt7eXh09etSkLmCynficPXtWOTk5qfz586tp06apyZMnq5w5c6oyZcqoNy+VKVOmKI1Go9q2basWLlxofN8KFCignj59aqy3Z88eZW9vr3x8fNT48ePVokWL1IABA1T9+vWTfAxSI764JOZY1a1bV/n6+sZaduLEiUqr1aqgoCCllFJhYWGqTJkyKnv27OrLL79UixcvVh07dlQajUYNHDjQ5H0CVL169ZSfn5/y8/NT/fr1U23atEkw1uhzZv/+/Sbl169fV4Bavny5saxTp07K0dFRlSpVSnXt2lUtWrRItWrVSgFq4cKFxnp6vV69//77ysbGRn3xxRdq/vz5qm7dusb3+fV1nj9/Xrm7uytfX181Y8YMtWDBAvX+++8rjUajtmzZEus99fX1VbVq1VLz589X06dPj3OfgoKC1KRJkxSgevbsqVatWqVWrVqlrl27ZjwWefLkUd7e3mrgwIFq4cKFqm7dugpQu3btMq5Hp9OpDz74QDk7O6tBgwapJUuWqH79+ilbW1vVvHnzBN/X8PBwVbBgQZUnTx41ZcoUtWzZMjVx4kRVuXJldePGDWM9Hx8f1alTJ5PYc+bMqbJkyaLGjBmjZs+ercqWLatsbGxM3o/o41a6dGlVpkwZNXv2bDVq1Cjl6OioihUrpl68eGGsm9hrPSnXR1y6d++usmfPHqs8+tiVK1dONWrUSPn5+akOHTooQI0YMULVqFFDtW/fXi1cuFA1adJEAWrlypWx1m1ra6t69OihFi9erEaOHKlcXFxU5cqVVUREhMn7Wbx4cZU7d241YcIENWfOHJU3b17l6uqqfvrpJ5U/f341ffp0NX36dOXu7q6KFCmidDqdcfnEfpaMHz/eeD42b95cLVy4UPn5+amQkBDl5OSkhg4dGut98PX1VXXr1k3wPYy+7kqXLq0KFCigZsyYoSZOnKiyZcumcuTIYfxcSEqsnTp1Uj4+PrG2Fb0PrwNU2bJljf9PzZ07VxUqVEg5OzurR48eGesl9Vw5dOiQAtT//ve/BPc/+rz29fVV5cqVU7Nnz1bTpk1TYWFhxvOocuXKas6cOWrUqFHKyckp1mdyYt6XP//8U3355ZcKUAMGDFCrVq1Se/bsSTC2zEwS2Uwo+oL57bff1MOHD9Xt27fVpk2bVI4cOZSDg4O6ffu2sW65cuWUl5eXevz4sbHszJkzysbGRnXs2NFY1qJFC2Vvb2/8z1Qppe7du6eyZMmi3n//fWNZ9AfZt99+G2dMiUlkS5YsqcLDw43l8+bNU4A6d+6cUsqQaBQtWlQ1bNhQ6fV6Y70XL16oggULqgYNGphsO7GJbIsWLZSjo6O6efOmsSwgIEBptVqTD7cbN24orVarpk6darL8uXPnlK2trbE8KipKFSxYUPn4+MRKHl+POynHwJzxxScxx2rJkiUmxyTam//ZTZ48Wbm4uKjLly+b1Bs1apTSarXq1q1bSimlBg4cqNzc3FRUVFSCsb0pqYksoCZNmmRSN/qLU7StW7cqQH3zzTfGsqioKFWzZs1Y66xXr54qXbq0evXqlbFMr9er9957TxUtWtRYFv2e1qhRI1H7ePz48VjbilarVi0FqB9//NFYFh4ernLlyqVatWplLFu1apWysbFRBw8eNFl+8eLFClCHDx+Od/unTp1SgNq4cWOCcb6ZyA4aNEgBJtt8/vy5KliwoCpQoIAx6Yo+bnnz5lUhISHGuhs2bFCAmjdvnlIqadd6Yq+P+NSoUcPkPIgWfezejKFatWpKo9Go3r17G8uioqJUvnz5TK6VgwcPKkCtXr3aZL2//vprrHIfHx8FqD///NNYtnv3bgUoJycnk32LvgZfP/cT+1kSnQS2a9cu1v62a9dO5cmTxyRBPnnyZLzn4+uirzsnJyd1584dY/nRo0cVoAYPHpzkWJOayNrb26urV6+arBNQ8+fPN5Yl9Vy5d++eAtSMGTMS3P/o87pQoUImX8YiIiKUl5eXeuedd9TLly+N5Tt27FCAGjduXJLfl+htve0atQbStCATq1+/Pjly5MDb25vWrVvj4uLC9u3bjT9pBgYGcvr0aTp37ky2bNmMy5UpU4YGDRqwa9cuAHQ6HXv27KFFixYUKlTIWC937ty0b9+eQ4cOGX86MYcuXbqYtMeMbhLx77//AnD69GmuXLlC+/btefz4MY8ePeLRo0eEhYVRr149Dhw4YPKztVIq1k/Pb9LpdOzevZsWLVqQP39+Y3nJkiVp2LChSd0tW7ag1+v55JNPjNt+9OgRuXLlomjRouzfvx8wNMW4fv06gwYNitWOKfonsaQcA3PHlxItW7bE1taW9evXG8vOnz9PQEAAbdu2NZZt3LiRmjVrkjVrVpNY6tevj06n48CBA4ChnVdYWBh79+5NcWxv07t3b5PpmjVrGs8tgF27dmFra0ufPn2MZVqtlv79+5ss9+TJE37//Xc++eQTnj9/bty3x48f07BhQ65cuRLrZ8MePXqg1WpTvA+urq58/vnnxml7e3uqVKlish8bN26kZMmSlChRwuS9r1u3LkCC54G7uzsAu3fv5sWLF4mOa9euXVSpUoUaNWqYxNqzZ09u3LhBQECASf2OHTuSJUsW43Tr1q3JnTu38bxP7LWelOsjPo8fPyZr1qzxzu/WrZvJT9lVq1ZFKUW3bt2MZVqtlkqVKsU6Du7u7jRo0MDkOFSsWBFXV9dYx8HX15dq1aqZbAegbt26JvsWXR69rcR+lrzuzWsBDMfk3r17JnGtXr0aJycnWrVqFe/787oWLVqQN29e43SVKlWoWrWqMYbkxJpY9evXp3DhwibrdHNzM75PyTlXos+LR48eJSqGTp064eTkZJz++++/efDgAV988YXJg9YfffQRJUqUMDa9Sc33JTOTRDYT8/PzY+/evWzatIkPP/yQR48e4eDgYJx/8+ZNAIoXLx5r2ZIlSxr/w3j48CEvXryIt55er09228u4vP7hAjEfItHtya5cuQIYPixy5Mhh8rds2TLCw8Pf2o7vTQ8fPuTly5cULVo01rw39/vKlSsopShatGis7f/zzz/GRvjRbZET6vMzKcfA3PGlhKenJ/Xq1WPDhg3GsvXr12Nra0vLli1NYvn1119jxRHdDjE6li+++IJixYrRuHFj8uXLR9euXeNs15ZSjo6O5MiRw6Qsa9asJm0Vb968Se7cuXF1dTWp9+b7fPXqVZRSjB07Ntb+jR8/3mT/ohUsWNAs+5EvX75Y7QPf3I8rV65w4cKFWLEVK1YsztjejHPIkCEsW7YMT09PGjZsiJ+f31uvq5s3b8Z7LkfPf92b57NGo6FIkSLGPowTe60n5fpIiHqjbevr3vxcik72vb29Y5W/eRyCg4Px8vKKtQ+hoaGxjkNStgMxn4uJ/Sx5XVznY4MGDcidOzerV68GDO17165dS/PmzY1fOp48eUJQUJDx783zIq7jUKxYMeNxTU6sifXm+wem10ZyzpXo8yKx/RG/+b4mtL8lSpQwzk/N9yUzk+63MrEqVaoY+3Bt0aIFNWrUoH379ly6dCnWf9LpSXx3rKI/TKLvtn777bfxdk2Umvun1+vRaDT88ssvccZq6fc2reL79NNP6dKlC6dPn6ZcuXJs2LCBevXq4enpaRJLgwYNGDFiRJzriE6qvLy8OH36NLt37+aXX37hl19+Yfny5XTs2JGVK1fGG0N8/7G8+fBWNHPcDY0WfR4OGzYs3js5b3bj9PpdmpR42zUSHV/p0qWZPXt2nHXfTIzeNGvWLDp37sy2bdvYs2cPAwYMYNq0afz111/xPqhmbom91hPz8NrbZM+ePdbDV6+L7z2Pq/zN4+Dl5WVMDN/05herpGznzW0lVVzno1arpX379nz//fcsXLiQw4cPc+/ePZNfAFq2bMkff/xhnO7UqVOqDRRirms8Je9T9Hnx+mdbQsx1nYvEkUTWSmi1WqZNm0adOnVYsGABo0aNMvYwcOnSpVj1L168iKenJy4uLjg6OuLs7BxvPRsbm7f+p2hO0T8bubm5xXrCOLly5MiBk5OT8Q7Q697c78KFC6OUomDBgsZELKE4z58/H2+cSTkG5o4vpVq0aEGvXr2MzQsuX77M6NGjY8USGhqaqONkb29P06ZNadq0KXq9ni+++IIlS5YwduzYePv1jL5b/2aPAm/e+UsKHx8f9u3bR2hoqEnS/+b7HN3Mxs7OzmznYTRzjNRWuHBhzpw5Q7169ZK9vtKlS1O6dGm++uor/vzzT6pXr87ixYuZMmVKnPV9fHziPZej57/uzfNZKcXVq1cpU6aMcR/g7dd6Uq7f+JQoUYLNmzcnqm5SFC5cmN9++43q1aunaoKT2M+SxOjYsSOzZs3if//7H7/88gs5cuQw+bI2a9Ysk6Q/T548JsvHdRwuX75s7H0gKbFmzZo1zh5DknuNJ+dcuX79OhDzy0JSvb6/0U17Xt9m9HxzHkNrIk0LrEjt2rWpUqUKc+fO5dWrV+TOnZty5cqxcuVKkw+K8+fPs2fPHj788EPAkAR/8MEHbNu2zWTY0vv377NmzRpq1KiBm5tbmu1HxYoVKVy4MDNnziQ0NDTW/Oi+bKMlpvstrVZLw4YN2bp1q0ndf/75h927d5vUbdmyJVqtlokTJ8b6lq+U4vHjxwBUqFCBggULMnfu3FgfxNHLJeUYmDu+lPLw8KBhw4Zs2LCBdevWYW9vT4sWLUzqfPLJJxw5ciRWjGBIPqOiogBixWRjY2NMZhK62+bj44NWqzW2tY2WkiElP/zwQ6Kioli0aJGxTKfTMX/+fJN6Xl5e1K5dmyVLlhAYGBhrPW+eh0kR/Z9VfF1+JcYnn3zC3bt3+f7772PNe/nyZYI/UYaEhBiPTbTSpUtjY2OT4PH48MMPOXbsGEeOHDGWhYWFsXTpUgoUKICvr69J/R9//JHnz58bpzdt2kRgYCCNGzcGEn+tJ+X6iE+1atV4+vSpSftWc/jkk0/Q6XRMnjw51ryoqKgUHePXJfazJDHKlClDmTJlWLZsGZs3b+bTTz/F1jbmvlfFihWpX7++8e/N47p161aT9uHHjh3j6NGjxuOalFgLFy5McHAwZ8+eNZYFBgby888/J3p/Xpecc+XEiRNoNBqTtstJUalSJby8vFi8eLHJ9fPLL7/wzz//8NFHHwHmPYbWRO7IWpnhw4fTpk0bVqxYQe/evfn2229p3Lgx1apVo1u3brx8+ZL58+fj7u7OhAkTjMtNmTKFvXv3UqNGDb744gtsbW1ZsmQJ4eHhcfZfmZpsbGxYtmwZjRs3plSpUnTp0oW8efNy9+5d9u/fj5ubG//73/+M9UuWLEmtWrXe+sDXxIkT+fXXX6lZsyZffPEFUVFRzJ8/n1KlSpl8iBYuXJgpU6YwevRobty4QYsWLciSJQvXr1/n559/pmfPngwbNgwbGxsWLVpE06ZNKVeuHF26dCF37txcvHiRCxcuGD80E3sMzB2fObRt25bPP/+chQsX0rBhw1gPtQ0fPpzt27fTpEkTOnfuTMWKFQkLC+PcuXNs2rSJGzdu4OnpSffu3Xny5Al169YlX7583Lx5k/nz51OuXLkE74K4u7vTpk0b5s+fj0ajoXDhwuzYsSNF7YCbNm1K9erVGTVqFDdu3MDX15ctW7bE2T7Uz8+PGjVqULp0aXr06EGhQoW4f/8+R44c4c6dO5w5cyZZMRQuXBgPDw8WL15MlixZcHFxoWrVqklqY9uhQwc2bNhA79692b9/P9WrV0en03Hx4kU2bNjA7t274x0++vfff6dfv360adOGYsWKERUVxapVq9BqtQk+8DNq1CjWrl1L48aNGTBgANmyZWPlypVcv36dzZs3Y2Njeu8kW7Zs1KhRgy5dunD//n3mzp1LkSJF6NGjB5C0az2x10d8PvroI2xtbfntt9/o2bNnYt/mt6pVqxa9evVi2rRpnD59mg8++AA7OzuuXLnCxo0bmTdvHq1btzbLthL7WZIYHTt2NH5OvN6sIDGKFClCjRo16NOnD+Hh4cydO5fs2bObNDFKbKyffvopI0eO5OOPP2bAgAG8ePGCRYsWUaxYMU6ePJmkuKIl9VzZu3cv1atXJ3v27Mnanp2dHTNmzKBLly7UqlWLdu3acf/+febNm0eBAgUYPHiwsa45j6HVSNtOEkRaiK8fWaUMfUsWLlxYFS5c2NgN0G+//aaqV6+unJyclJubm2ratKkKCAiItezJkydVw4YNlaurq3J2dlZ16tQx6SZGKfN0v/VmdyJxdaWklKGLoJYtW6rs2bMrBwcH5ePjoz755BO1b98+k3oksvstpZT6448/VMWKFZW9vb0qVKiQWrx4cZzdvCil1ObNm1WNGjWUi4uLcnFxUSVKlFB9+/ZVly5dMql36NAh1aBBA5UlSxbl4uKiypQpY9IVjFKJPwapEd+bEnOsokX3Owmon376Kc71PX/+XI0ePVoVKVJE2dvbK09PT/Xee++pmTNnGvvQ3LRpk/rggw+Ul5eXsre3V/nz51e9evVSgYGBCcaqlFIPHz5UrVq1Us7Ozipr1qyqV69e6vz583F2v+Xi4hJr+bjev8ePH6sOHTooNzc35e7urjp06GDskurN8/DatWuqY8eOKleuXMrOzk7lzZtXNWnSRG3atMlYJ6FrMj7btm1Tvr6+ytbW1mS7tWrVUqVKlYpVP65uiiIiItSMGTNUqVKllIODg8qaNauqWLGimjhxogoODo532//++6/q2rWrKly4sHJ0dFTZsmVTderUUb/99ptJvTe734p+P1q3bq08PDyUo6OjqlKlitqxY4dJnehrfe3atWr06NHKy8tLOTk5qY8++sikS6Roib3Wk3J9xKVZs2aqXr16JmXxHbvo9T58+NCkPL7zbOnSpapixYrKyclJZcmSRZUuXVqNGDFC3bt3z1jHx8dHffTRR7GWBVTfvn1NyuL7rE3MZ0l8sb8uMDBQabVaVaxYsXjrvOn1mGbNmqW8vb2Vg4ODqlmzpjpz5kys+on93NuzZ4965513lL29vSpevLj66aef4u1+6833Sam4z9PEnivPnj1T9vb2atmyZW/d/7d1ibV+/XpVvnx55eDgoLJly6Y+++wzk27KoiXmfZHut2JolEpBC2ghhBAiifz9/alTpw4bN240291Iczh48CC1a9fm4sWLcT7Vbk0ePXpE7ty5GTduHGPHjrV0OBYzd+5cvvnmG65duyYPcaVT0kZWCCGEwNCn8AcffJDmzaXSoxUrVqDT6ejQoYOlQ7GYyMhIZs+ezVdffSVJbDombWSFEEKI//zyyy+WDsGifv/9dwICApg6dSotWrQw9jRgjezs7N76oLCwPElkhRBCCAHApEmTjF2tvdlThxDpkbSRFUIIIYQQGZK0kRVCCCGEEBmSJLJCCCGEECJDsuo2snq9nnv37pElSxazDAkphBBCCCFSRinF8+fPyZMnT6yBVN5k1YnsvXv38Pb2tnQYQgghhBDiDbdv3yZfvnwJ1rHqRDZLliyA4Y1yc3OzcDRCCCFSIjIykpUrVwLQqVMn7OzsLByRECI5QkJC8Pb2NuZpCbHqXgtCQkJwd3cnODhYElkhhBBCiHQgKfmZPOwlhBBCCCEyJElkhRBCCCFEhmTVbWSFEEJkHlFRUWzfvh2AZs2aYWsr/8UJkdnJVS6EECJT0Ov1XLlyxfhaCJH5ZfhE9tmzZ9SvX5+oqCiioqIYOHAgPXr0sHRYQggh0phWq6Vhw4bG16ni2W24/geEh4KDKxSsBR7SjaMQlpLhey3Q6XSEh4fj7OxMWFgY77zzDn///TfZs2d/67LSa4EQQohEuXMCDnwDl3cDCjQ2oPSABoo1glrDIW9FS0cpRKZgVb0WaLVanJ2dAQgPD0cpRQbPzYUQQqQnAdvhhw/gyl7gv/9fVHTTBQVX9sD/fWCoJ4RIUxZPZA8cOEDTpk3JkycPGo2GrVu3xqrj5+dHgQIFcHR0pGrVqhw7dsxk/rNnzyhbtiz58uVj+PDheHp6plH0Qggh0gulFA8fPuThw4fmu6Fx5wRs6gJ6HShdPBvWGeZv6gJ3T5hnu0KIRLF4IhsWFkbZsmXx8/OLc/769esZMmQI48eP5+TJk5QtW5aGDRvy4MEDYx0PDw/OnDnD9evXWbNmDffv30+r8IUQQqQTkZGR/PDDD/zwww9ERkaaZ6UHvgWlMN6JjZcy1Dsw0zzbFUIkSrpqI6vRaPj5559p0aKFsaxq1apUrlyZBQsWAIYnUb29venfvz+jRo2KtY4vvviCunXr0rp161jzwsPDCQ8PN05HD4EmbWSFECLji4iIwG/hYiKjwNOnNxob+xStzyXiNm0ulUbz1iQ2hkLDxuLnCLOXB8BE5vNuOShZOPW3k5Q2sum614KIiAhOnDjB6NGjjWU2NjbUr1+fI0eOAHD//n2cnZ3JkiULwcHBHDhwgD59+sS5vmnTpjFx4sQ0iV0IIUTa0evhhy32+O0cQGSUedbZJucffFIiafd6NCiObjvApvufmScIIdKRKYPSJpFNCos3LUjIo0eP0Ol05MyZ06Q8Z86cBAUFAXDz5k1q1qxJ2bJlqVmzJv3796d06dJxrm/06NEEBwcb/27fvp3q+yCEECJ1BT2Ez4bBtCWYLYkFcLENRaeS9t+kTtngavvcfEEIIRKUru/IJkaVKlU4ffp0ouo6ODjg4OCQugEJIYRIM3+egn6T4PEzw7RGA91aQ5niKV93/iBXtFeSNrCCVqOn+YdZqJDz7XWFyGjMcV2ZW7pOZD09PdFqtbEe3rp//z65cuWyUFRCCCEsTSlYttFwF1b3X66Z1yuK7h/9Qo6s0LhW45QPUfusFszV8PYHvV6nocKH71PBI2WbFkIkTrpuWmBvb0/FihXZt2+fsUyv17Nv3z6qVatmwciEEEJYSthL6D8ZpiyKSWJrVYatfnoeBgYQEBBgniFqPbwNgx1oEjlKmEYLxRvLSF9CpCGL35ENDQ3l6tWrxunr169z+vRpsmXLRv78+RkyZAidOnWiUqVKVKlShblz5xIWFkaXLl0sGLUQQghLuH4Heo6Fyzdiyvp9DkM6A2ipW7cuYMYhamsNh6t7Qb3tzqzG0K7h/WHm2a4QIlEs3v2Wv78/derUiVXeqVMnVqxYAcCCBQv49ttvCQoKoly5cnz33XdUrVo12dv08/PDz88PnU7H5cuXpfstIYTIAH77EwZ9Dc/DDNOuzjBnNHxQI5U3HLDdMNiBUnEPiqDRGpLYNiugZNNUDkaIzC9DDVFbu3Zt47Cyr/9FJ7EA/fr14+bNm4SHh3P06NEUJbEAffv2JSAggOPHj6cweiGEEKlNr4fZy6HbmJgktogPbF+UBkksgG8z6LYHin4AaAxlmuj/PjVQrKFhfhomsS9evKBVq1a4ubmh0Wh49uwZBQoUYO7cucY6r4+WeePGDTQaTaIfjs4M9u3bR8mSJdHp4hmRLZ1YvHgxTZvKF6DksngiK4QQQsQn+Dl0/RLm/RhT9mEt2LYQCuc3rauUMnavaPYfG/NWhPbrYNA5aL4QGk4z/DvoHLRba5ifhlauXMnBgwf5888/CQwMxN3dnePHj9OzZ88krys6yU3o7/WbS4k1YcKEWOspUaKESZ2lS5dSu3Ztk4T8zdi6detGwYIFcXJyonDhwowfP56IiIi3bn/EiBF89dVXyW5m0rlz51jxN2rUyKTO1KlTee+993B2dsbDwyPWOs6cOUO7du3w9vbGycmJkiVLMm/ePJM6Xbt25eTJkxw8eDBZcVo7i7eRFUIIIeLyzzXoOQ5u3TNM29jAyB7Qq63hl/w3RUZGsnjxYgAGDx6MvX3KRvaKk4c3lE+9wQ4iIiISFfe1a9coWbIk77zzjrEsR44cydqmt7c3gYGBxumZM2fy66+/8ttvvxnL3N3dk7XuUqVKmaznzZ4kXrx4QaNGjWjUqJHJ4EfRLl68iF6vZ8mSJRQpUoTz58/To0cPwsLCmDkz/uGADx06xLVr12jVqlWy4o7WqFEjli9fbpx+swvPiIgI2rRpQ7Vq1fi///u/WMufOHECLy8vfvrpJ7y9vfnzzz/p2bMnWq2Wfv36AYYH29u3b893331HzZo1UxSvNZJEVgghRLqz9TcYORNe/TeqeFY3WDAOarzlxqednV3qB5dItWvXNiaaq1atws7Ojj59+jBp0iQ0/2XiBQoUoFu3bly5coWtW7fSsmVLVqxYwaFDhxg9ejR///03np6efPzxx0ybNg0XFxdq167NH3/8ARiaD9SqVQt/f38KFCjAoEGDGDRoUJLi1Gq1Jl1aurq6Ymtra5ZuLt+2nuhY/f3945wfneRGK1SoEJcuXWLRokUJJrLr1q2jQYMGODo6JivuaA4ODgnGHz1aaHx3rLt27WoyXahQIY4cOcKWLVuMiSxA06ZNadCgAS9fvsTJySlFMVsbq2xa4Ofnh6+vL5UrV7Z0KEIIIV4TGQUTF8DAqTFJbOlisGPJ25NYe3t7hgwZwpAhQ1LnbmwyrFy5EltbW44dO8a8efOYPXs2y5YtM6kzc+ZMypYty6lTpxg7dizXrl2jUaNGtGrVirNnz7J+/XoOHTpkTHy2bNlCjx49qFatGoGBgWzZsiVN9uXWrVu4urom+Pf111+bLHPlyhXy5MlDoUKF+Oyzz7h161aK4wgODiZbtmwJ1jl48CCVKlVK8rrf7LbN398fLy8vihcvTp8+fXj8+HGS1/mmuOKvVKkSUVFRHD16NMXrtzZWeUe2b9++9O3b1/hUnBBCCMt78AT6TYSjZ2PK2jSCKYPBMX3kpUnm7e3NnDlz0Gg0FC9enHPnzjFnzhx69OhhrFO3bl2GDh1qnO7evTufffaZ8W5l0aJF+e6776hVqxaLFi0iW7ZsODs7Y29vn6aDA+XJk+etD4u9nqBVrVqVFStWULx4cQIDA5k4cSI1a9bk/PnzZMmSJVkxXL16lfnz5yd4NxYMw9fnyZPHpEwpxdKlS/Hz8yMoKIiKFSvSqVMnmjZtio2NDUuXLsXR0ZFevXoBhrvBLVu2pGDBgly7do0vv/ySxo0bc+TIkWS3u/3zzz9Zv349O3fuNCl3dnbG3d2dmzdvJmu91swqE1khhBDpy4kL0GcC3H9kmLazhYn9oX3TuNvDZhTvvvuusRkBQLVq1Zg1axY6nc6YDL155/DMmTOcPXuW1atXG8uUUuj1eq5fv07JkiXTJvg32NraUqRIkUTXb9y4sfF1mTJlqFq1Kj4+PmzYsIFu3boleft3796lUaNGtGnTxuSLQFxevnwZq1nBwYMHWbduHbNnzyZbtmzs3r2byZMn0759e+zs7GjQoAGLFi0y1v/000+Nr0uXLk2ZMmUoXLgw/v7+1KtXL8nxnz9/nubNmzN+/Hg++OCDWPOdnJx48eJFktdr7SSRFUIIYTFKwer/wYT5hmYFALk8YdFEqOCbtHVFRUWxd+9eABo0aJDyIWrTiIuLi8l0aGgovXr1YsCAAbHq5s+fP1ZZWrl16xa+vgkflC+//JIvv/wyznkeHh4UK1bMZBCkxLp37x516tThvffeY+nSpW+t7+npydOnT03KypUrx759+7CxMbSqrFChAqNHj+bBgwdkyZLlrW1TCxUqhKenJ1evXk1yIhsQEEC9evXo2bMnX331VZx1njx5kuwH9qxZxrjKhRBCZDqvwuGrubDx15iyqmXAbzzkSLgJZJz0ej1nzxraJSTnjllqeLPN419//UXRokUT/Gm6QoUKBAQEJOnuZ1pIatOCN4WGhnLt2jU6dOiQpO3evXuXOnXqULFiRZYvX25MRBNSvnx5AgICTMri61jfy8srUXHcuXOHx48fkzt37kTVj3bhwgXq1q1Lp06dmDp1apx1rl27xqtXryhfvnyS1i0kkRVCCGEBd4Kg1zg4fyWmrFtrGN3L0KwgObRarbH7IrMNUZtCt27dYsiQIfTq1YuTJ08yf/58Zs2aleAyI0eO5N1336Vfv350794dFxcXAgIC2Lt3LwsWLEijyGNLatOCYcOG0bRpU3x8fLh37x7jx49Hq9XSrl07Y52goCCCgoKMd2nPnTtHlixZyJ8/P9myZePu3bvUrl0bHx8fZs6cycOHD43LJtQ+uGHDhqxcuTIZe2kQGhrKxIkTadWqFbly5eLatWuMGDGCIkWK0LBhQ2O9W7du8eTJE27duoVOpzMm+kWKFMHV1ZXz589Tt25dGjZsyJAhQwgKCgIM5+frd18PHjxIoUKFKFy4cLJjtlaSyAohhEhTh05Av0nwNMQw7eQIM4ZB8xTeRNVqtbz33nspD9CMOnbsyMuXL6lSpQparZaBAwe+ddCCMmXK8McffzBmzBhq1qyJUorChQvTtm3bNIraPO7cuUO7du14/PgxOXLkoEaNGvz1118mCdzixYuNXVgBvP/++wAsX76czp07s3fvXq5evcrVq1fJly+fyfoTGvTis88+Y8SIEVy6dInixYsnOXatVsvZs2dZuXIlz549I0+ePHzwwQdMnjzZpC/ZcePGmSTM0XdU9+/fT+3atdm0aRMPHz7kp59+4qeffjLW8/Hx4caNG8bptWvXvrXdr4ibRpl9+JP0z8/PDz8/P3Q6HZcvX07UWL5CCCFSRilYvA6+WWYYdhbAJw8smQQlM+GNqNq1a1OuXDmTYWNF2hk+fDghISEsWbLE0qEkKLrpweXLl6Unpf9E9yqVmPzMKvuR7du3LwEBARw/ftzSoQghhFUIfWHolWD60pgktk5V+N9i8yWxSilevHjBixcvzD9ErchwxowZg4+PT6y+YdObwMBAfvzxR0lik0maFgghhEhVV28Z2sNefa2LzMGdYUAHw7Cz5hIZGcn8+fMN60+tIWpFhuHh4RFvDwrpSf369S0dQoYmiawQQohU8+tBGDrdcEcWwM0F5o6BetUsG1daiG/YVSGE+VhlG9loSWmDIYQQIvF0Opi1HPxi+vSneEFYOhkK5LVcXEKI9C8p+ZnckRVCCGFWT4NhwBQ48HdMWbO6hp4JnBPuc14IIZJEElkhhBBmc+4y9B5v6CcWQGsDX/Y29BGbkYeaFUKkT1aZyL7e/ZYQQgjz2LwbRs+G8AjDtGdWWDAOqpVLm+1HRUUZ26XWrl07wwxRK4RIPul+SwghRIpERMLYeTBkekwSW74k7FiSdkksGIaoPXHiBCdOnEj3XS4JIcxDvq4KIYRItvuPDP3DnrgQU/ZZUxjfDxzSuPcrrVZLtWrVjK+FEJmfJLJCCCGS5fg5QxL78Ilh2sEOJg2ETz+yTDxardY4xKkQwjpIIiuEECJJlIIVP8OUhRD136MGebxg8UQoW8KysQkhrIskskIIIRLt5SvDA10/740pe6+84aGu7B4WCwswDFEbGRkJgJ2dHRrpJkGITE8SWSGEEIly655hqNmAazFlvdrCiB5gmw6apEZGRjJnzhxAhqgVwlpIIiuEEOKt/I8ZBjkIfm6YdnaEmSPho9oWDUsIYeUkkRVCCBEvvR781sCsHwxtYwEKecOSiVCsoGVje5OdnR2DBw82vhZCZH5WmcjKgAhCCPF2IaEwdDrsORxT9kF1mDUK3FwtF1d8NBqNNCcQwspolIr+jm19QkJCcHd3Jzg4GDc3N0uHI4QQ6cblG4b2sP/eNkxrNDC0K/RtDzZWOZSOECKtJCU/s8o7skIIIeK30x+GzYAXrwzT7lngu6+gdhWLhvVWOp2Ow4cNt4+rV68ugyIIYQUkkRVCCAEY+oT95ntYsj6mzLeIoT1s/jyWiyuxdDodR44cAeDdd9+VRFYIKyCJrBBCCB4/g36T4M9TMWUfN4BpQ8DJ0WJhJYmNjQ0VK1Y0vhZCZH6SyAohhJU7e8nQHvbeA8O0rRbG9oVOLQxtYzMKW1tb6tevb+kwhBBpSBJZIYSwYut3wdi5EG4YEIsc2WDRBKhc2pJRCSFE4kgiK4QQVig8AibMhzU7YsoqljIksTk9LRaWEEIkiSSyQghhZQIfQu9xcPpiTFnHFjD2C7DPwOMIREREyBC1QlgZSWSFEMKKHDlteKjr0VPDtIM9fD0YWjeyaFhCCJEsVpnIysheQghroxT83yb4ejHo9IayfLlgySR4p6hlYzMXOzs7+vfvb3wthMj8ZGQvGdlLCJHJvXgJI2fC9t9jyt6vZBjkIKu75eISQoi4yMheQgghALhxF3qOhUvXY8r6fgZDu4CMFyCEyOgkkRVCiExq3xEYNBVCwgzTrs4waxQ0qmnZuFKLTqfj6NGjAFStWlVG9hLCCkgiK4QQmYxeD9+tgjkrYsoK54elk6FIfouFlep0Oh0HDx4EoFKlSpLICmEFJJEVQohMJPg5DPoafv8rpqzx+zBzpOGObGZmY2NDmTJljK+FEJmfJLJCCJFJXPzX0B725j3DtI0NjOgOvT/NWEPNJpetrS2NGze2dBhCiDQkiawQQmQC2/YZeiZ4+cowndUNFoyDGhUtG5cQQqQmSWSFECIDi4yCaUsMfcRGe6eooX/YfLksF5cQQqQFSWSFECKDevjEMErXX2diyto0gimDwNHBYmFZTEREBAsWLACgX79+MkStEFZAElkhhMiATgZAn/EQ9MgwbWcLE/rDZ02toz1sfCIjIy0dghAiDUkiK4QQGYhSsOZ/MH6+oVkBQE5PWDQBKpayaGgWZ2dnR+/evY2vhRCZnySyQgiRQbyKgLFzYcMvMWVVy8CC8eCVzWJhpRsajQZ3dxlzVwhrIomsEEJkAHfvQ+/xcPZSTFm31jC6l6FZgRBCWCOr/Pjz8/PDz88PnU5n6VCEEOKtDp2A/pPhSbBh2tEBZgyDFvUtG1d6o9PpOHnyJAAVKlSQkb2EsAIapZSydBCWEhISgru7O8HBwbi5uVk6HCGEMKEULFkPM743DDsLkD8PLJ0EJQtbNrb0KCIigjlz5gAwePBg6bVAiAwqKfmZVd6RFUKI9C70BQz/Bnb9EVNWuyp8Nwbcs1gurvTMxsYGX19f42shROYniawQQqQz125Bz3Fw9WZM2cCOMKiTYdhZETdbW1uaNm1q6TCEEGlIElkhhEhHdh+CIdMMd2QB3FxgzpdQ/z3LxiWEEOmRJLJCCJEO6HQwewUs+CmmrFgBWDoZCuazVFRCCJG+SSIrhBAW9iwEBkyBP47HlDWtAzOGg4uT5eLKaCIiIli8eDEAvXv3loe9hLACksgKIYQFXbgKvcbB7UDDtNbG0Dds9zbWPdRscr18+dLSIQgh0pAkskIIYSFb9sCoWRAeYZjO7gELxsF75S0aVoZlZ2dH165dja+FEJmfJLJCCJHGIiJhyiJY+XNMWbkSsGgi5PGyXFwZnUajIUeOHJYOQwiRhiSRFUKINHT/MXwxAf4+H1PW7iOYOAAcpEmnEEIkiSSyQgiRRo6fgz4T4OETw7S9HUwaaEhkRcrpdDrOnTsHQOnSpWWIWiGsgCSyQgiRypSCH7fCJD+I0hnKcueAxROhXEmLhpap6HQ6du/eDYCvr68kskJYAUlkhRAiFb0Khy9nw+Y9MWXVysOCseCZ1XJxZUY2NjYULVrU+FoIkfkl60pfuXIlO3fuNE6PGDECDw8P3nvvPW7evJnAkkIIYT1uBcLH/UyT2J5t4advJYlNDba2trRs2ZKWLVtiayv3aYSwBslKZL/++mucnAy9dB85cgQ/Pz+++eYbPD09GTx4sFkDFEKIjOjAcWjaGwKuGqadHcFvHIzpDbbyi7cQQphFsr6y3r59myJFigCwdetWWrVqRc+ePalevTq1a9c2Z3xCCJGh6PWwcA3M/MHQNhYMQ8wunQTFClo2NiGEyGySdUfW1dWVx48fA7Bnzx4aNGgAgKOjo4yqIoSwWs/DoPd4+Pb/YpLYBu/B9kWSxKaFyMhIFi1axKJFi4iMjLR0OEKINJCsO7INGjSge/fulC9fnsuXL/Phhx8CcOHCBQoUKGDO+IQQIkO4fAN6j4Nrtw3TGg0M6QL9PgN57ihtKKUICQkxvhZCZH7JSmT9/Pz46quvuH37Nps3byZ79uwAnDhxgnbt2pk1QCGESO92/QHDZkDYfz9IubnCd19BnaqWjcva2Nra0rFjR+NrIUTmp1FJ+Nr6ww8/0KxZMzw9PVMzplTn5+eHn58fOp2Oy5cvExwcjJubm6XDEkJkMFE6+HYZLF4XU1ayECyZBD55LReXEEJkZCEhIbi7uycqP0tSIlu3bl3+/PNPKlSoQPPmzWnevDklSpRIccCWkpQ3SgghXvckGPpNgsMnY8pa1IfpQ8HJ0XJxCSFERpeU/CxJLbd+//13AgMD+eKLLzhx4gRVqlShaNGiDB06lAMHDqDX61MUuBBCZARnL0GTXjFJrK0WJvSHuV9KEmtJer2eCxcucOHCBfn/SAgrkeRHELJmzcrnn3/Ohg0bePToEfPnz+fly5d89tlneHl50bFjRzZt2kRYWFhqxCuEEBa14Rdo3R/u3jdM58gKa2dDl5aGB7yE5URFRbFjxw527NhBVFSUpcMRQqSBFD1La29vT6NGjVi4cCG3b9/m119/pUCBAkyePJnZs2ebK0YhhLC48AgYMweGfwPh//XsVKEU7FwKVcpYNjZhoNFo8PHxwcfHB418qxDCKiSpjSwY2i0cPXqUiIgIqlSpQo4cOeKsFxkZiZ2dnVmCTC3SRlYIkRhBD6H3BDgVEFPWoTmM6wv26ftjTgghMpyk5GdJ6p/k9OnTfPjhh9y/fx+lFFmyZGHDhg00bNgwVt30nsQKIURiHD0DX0yER08N0w52MHUItGlk2biEEEIksWnByJEjKViwIIcOHeLEiRPUq1ePfv36pVZsQghhMUrB/22CdkNikth8OWHzAklihRAivUjSHdkTJ06wZ88eKlSoABj6lc2WLRshISHy07wQItN48RJGzYJt+2LKalSE+WMhm7vl4hIJi4yMZOXKlQB06tRJfhkUwgokKZF98uQJ+fLlM057eHjg4uLC48ePJZEVQmQKN+9Cz3Fw8d+Ysr7tYWhX0GotF5d4O6UUjx8/Nr4WQmR+SR7DLyAggKCgIOO0Uop//vmH58+fG8vKlJFHeIUQGc/vf8HAqRASaph2cYJZo6Dx+5aNSySOra2tcZh0GaJWCOuQpF4LbGxs0Gg0cX7TjS7XaDTodDqzBplapNcCIQSAXg/zV8GclYa2sQCF8xuGmi3qY9nYhBDC2qRarwXXr19PUWBCCJHeBIfC4K9h35GYsoY1YdZIyOJiubiEEEK8XZISWR8fuTUhhMg8Lv4LvcbBjbuGaRsbGN4N+rSTUboyIr1ez9WrVwEoUqQINjYpGvNHCJEBJKsR0fHjx1m7di2XL18GoFixYrRv355KlSqZNTghhEgt23+HEd/Cy1eGaQ83WDAWasrHWIYVFRXFzz//DMDgwYOxt7e3cERCiNSW5K+rI0aMoGrVqixbtow7d+5w584dvv/+e6pWrcrIkSNTI0YhhDCbKB1MXgj9J8ckse8UhR1LJInN6DQaDXnz5iVv3rwyRK0QViJJd2RXrlzJ/Pnz+e677+jVq5exj77IyEgWLVrEyJEjKVWqFB07dkyVYIUQIiUePYW+k+Cv0zFlrRvC1MHg6GCxsISZ2NnZ8fnnn1s6DCFEGkpSrwVVqlShXbt2DB48OM75s2fPZt26dRw7dsxsAaYm6bVACOtxKgD6TIDAh4ZpO1sY3w8+bybtYYUQIj1JSn6WpKYFFy5coHnz5vHOb9GiBRcuXEjKKoUQItWt3QGfDIpJYr2yw7o50KG5JLFCCJGRJalpgVarJSIiIt75kZGRaGXoGyFEOvEqAsZ/B+t2xpRVLg0LJ4BXNouFJVJJZGQka9asAaB9+/YyRK0QViBJd2QrVKjA6tWr452/atUqKlSokOKghBAipe49gE8GmiaxXVrC2tmSxGZWSimCgoIICgqSIWqFsBJJuiM7bNgwWrRoQXh4OEOHDiVnzpwABAUFMWvWLObOnWvs+kQIISzl8EnoNwmeBBumHR1g2lBo2cCycYnUZWtrS+vWrY2vhRCZX5Ie9gKYP38+w4YNIyoqCnd3dwCCg4OxtbXlm2++YeDAgakSaGqQh72EyFyUgqUbYPpSw7CzAN65Yekk8C1i2diEEEIkTlLysyQnsgB37txh48aNXLlyBTAMiNCqVSu8vb2TF7GFSCIrROYR9hKGz4Cdf8SU1aoC340xDHYghBAiY0j1RDY+Z8+epVKlSgk+EJaeSCIrRObw723oORau3IwpG9ABBnUCef7Ueuj1em7eNJwEPj4+MkStEBlUqnW/9TZKKXQ6nTlXKYQQCdp7GJr1iUlis7jAsikwtKsksdYmKiqKDRs2sGHDBqKioiwdjhAiDUhreCFEhqTTwZyVMH9VTFmxArBkEhTKWK2chJloNBq8vLyMr4UQmV+GT2Rv375Nhw4dePDgAba2towdO5Y2bdpYOiwhRCp6FgIDpsIfrw0i2KQ2fDMCXJwsFpawMDs7O7p06WLpMIQQaShJiWxISEiC858/f56iYJLD1taWuXPnUq5cOYKCgqhYsSIffvghLi4uaR6LECL1BVyFnuPgdqBh2sYGRveCHm1klC4hhLA2SUpkPTw8Evy5RimV5j/n5M6dm9y5cwOQK1cuPD09efLkiSSyQmRCW/bC6FnwKtwwnc0dFoyD6jIOixBCWKUkJbK///672RPVAwcO8O2333LixAkCAwP5+eefadGihUkdPz8/vv32W4KCgihbtizz58+nSpUqsdZ14sQJdDpdhusGTAiRsMgomLoIlm+JKStbAhZPhDxelotLpC+RkZFs3LgRgDZt2sgQtUJYgSQlsrVr1zZ7AGFhYZQtW5auXbvSsmXLWPPXr1/PkCFDWLx4MVWrVmXu3Lk0bNiQS5cuGRv1Azx58oSOHTvy/fffmz1GIYTlPHgCfSfCsbMxZZ9+BBMHgKO95eIS6Y9Situ3bxtfCyEyvyT1Iztu3DhGjRqFs7MzAE+fPiVr1qzmC0ajiXVHtmrVqlSuXJkFCxYAhn4Cvb296d+/P6NGjQIgPDycBg0a0KNHDzp06BDv+sPDwwkPDzdOh4SE4O3tLf3ICpFO/X0e+kyAB48N0/Z2MGkAtGti0bBEOqXX67l8+TJgGKhH+pEVImNKtX5kp06dSmhoqHHax8eHf//9N3lRJkJERAQnTpygfv36xjIbGxvq16/PkSNHAMO37s6dO1O3bt0Ek1iAadOm4e7ubvyTJghCpE9KwY9b4dPBMUls7hywcZ4ksSJ+NjY2lChRghIlSkgSK4SVSNKV/ubN29T+6ebRo0fodDpy5sxpUp4zZ06CgoIAOHz4MOvXr2fr1q2UK1eOcuXKce7cuTjXN3r0aIKDg41/0T9BCSHSj1fhMGwGjJ1naBsL8G452LEEypW0aGhCCCHSmQzfj2yNGjXQ6/WJquvg4ICDg0MqRySESK7bQdBrHFy4ElPW4xMY1RNsZZQu8RZ6vZ579+4BkCdPHrkrK4QVSFIiq9FoeP78OY6OjsautkJDQ2P1L2uu9qaenp5otVru379vUn7//n1y5cpllm0IIdKHg39Dv8mGwQ4AnBzh2+HQtK5l4xIZR1RUFKtXrwZg8ODB2NvL04BCZHZJSmSVUhQrVsxkunz58ibTGo0GnU5nluDs7e2pWLEi+/btMz4Aptfr2bdvH/369TPLNoQQlqUULFoL3/4fRP+4UiAvLJ0MxQtaNjaRsWg0GuMDyDJErRDWIUmJ7P79+80eQGhoKFevXjVOX79+ndOnT5MtWzby58/PkCFD6NSpE5UqVaJKlSrMnTuXsLCwFA1D6Ofnh5+fn9kSbiFE8jwPg6EzYPfBmLJ61WDOl+Duarm4RMZkZ2dHz549LR2GECINJan7rR9//JG2bduatZ2pv78/derUiVXeqVMnVqxYAcCCBQuMAyKUK1eO7777jqpVq6Z420np3kEIYV5XbkKvsXDtv2cuNRoY3An6dzAMOyuEEMI6JSU/S1Iiq9VqCQwMNBmIICOTRFYIy/jlAAydDmEvDdNurjBvDNR917JxCSGEsLyk5GdJbiMrhBDJpdPBzB9g4ZqYshKFYOkk8MlrubhE5hAVFcXPP/8MwMcff4ytbYbvmEcI8RZJvsqlAb0QIjmeBMOAKYbeCaI1rwfTh4Kzk+XiEpmHXq83DtKT2G4ZhRAZW5IT2Xr16r31W+7JkyeTHVBakIe9hEhb5y5D73Fw57+e9LQ28FUf6NLK0DZWCHPQarV8+OGHxtdCiMwvSW1kbWxsGDp0KK6uCT9OPH78+BQHlhakjawQqW/Tr/DlbAiPNEx7ZoWF46FqWcvGJYQQIn1KtTayAMOHD880D3sJIVJPRCRMXAA/bY8pK+8LiydArhwWC0sIIUQmkuSRvYQQ4m2CHkKfiXDyQkzZ581gXF9wkMGWRCrR6/U8fPgQgBw5csgQtUJYAem1QAhhVsfOwhcT4OFTw7SDHUwZDJ80tmhYwgpERUUZ+x+XIWqFsA5JSmSvX79Ojhzym6AQIjalYMUWmLIIov57jjJvTlgyEUoXt2xswjpoNBrjMxzyC6IQ1iFJD3tlNvKwlxDm8fIVjJwJ2/bFlFWvAAvGQTZ3y8UlhBAi40nVh70yA+l+SwjzuXUPeo6Ff/6NKevTDoZ1A1vpAUkIIUQqkjuyckdWiGTbf9QwyEFIqGHaxQlmjoQPa1k2LiGEEBmX3JEVQqQqvR7m/wRzVhjaxgIU9oYlk6Goj0VDE1YsKiqKHTt2ANCkSRMZolYIK5DsvkkOHjzI559/TrVq1bh79y4Aq1at4tChQ2YLTgiR/oSEQo+xMHt5TBLbsAZsWyRJrLAsvV7PpUuXuHTpkgxRK4SVSFYiu3nzZho2bIiTkxOnTp0iPDwcgODgYL7++muzBiiESD8uXYdmfeC3Pw3TGg2M6A6LJ0IWF8vGJoRWq6VBgwY0aNBAhqgVwkokK5GdMmUKixcv5vvvv8fOzs5YXr16dU6ePGm24IQQ6ceO/dDiC7h+xzDt4QY/zoC+n4H0Oy/SA61WS4UKFahQoYIkskJYiWQ1ILp06RLvv/9+rHJ3d3eePXuW0piEEOlIlA5mfA9L18eU+RaBJZMgf27LxSWEEEIk6z5Krly5uHr1aqzyQ4cOUahQoRQHldr8/Pzw9fWlcuXKlg5FiHTt0VP4fLhpEtvqA/h5gSSxIv1RSvHkyROePHkiI1EKYSWSlcj26NGDgQMHcvToUTQaDffu3WP16tUMGzaMPn36mDtGs+vbty8BAQEcP37c0qEIkW6d/gea9IIjpwzTtlqYPBBmjQJHB8vGJkRcIiMj+f777/n++++JjIy0dDhCiDSQrKYFo0aNQq/XU69ePV68eMH777+Pg4MDw4YNo3///uaOUQiRxtbuhHHzIOK/XMArOywcD5VLWzYuId7GwUG+ZQlhTVI0IEJERARXr14lNDQUX19f4xjXGYUMiCCEqfAIGP+dIZGNVrk0+I2HnNktF5cQQgjrkWYDItjb2+Pr65uSVQgh0ol7D6DPeDh9Maas88cwpg/Y28W/nBBCCGEpiU5kW7ZsmeiVbtmyJVnBCCEs489T0G8SPH5mmHZ0gGlDoWUDi4YlhBBCJCjRiay7u3tqxiGEsACl4PuNMH0J6P4bCClfLlg6GUoVsWxsQiRVVFQUu3fvBqBhw4YyRK0QViDRV/ny5ctTMw4hRBoLewkjvoEd/jFltSrDd18ZBjsQIqPR6/WcP38egAYN5OcEIaxBir6uPnjwgEuXLgFQvHhxvLy8zBKUECJ1Xb8DPcfC5RsxZf07wOBOIAMiiYxKq9VSu3Zt42shROaXrEQ2JCSEvn37sm7dOnQ6HWD40Gjbti1+fn7pvhmCn58ffn5+xtiFsCa//QmDvobnYYZpV2eYMxo+qGHZuIRIKa1WS9WqVS0dhhAiDSV7QISjR4+yY8cOnj17xrNnz9ixYwd///03vXr1MneMZicDIghrpNfD7OXQbUxMElvUB7YvkiRWCCFExpSsfmRdXFzYvXs3NWqY/u938OBBGjVqRFhYmNkCTE3Sj6ywFsHPYeBU2H80puzDWvDtCMMdWSEyA6UUoaGhALi6uqLRaCwckRAiOVK9H9ns2bPH2XzA3d2drFmzJmeVQohU8s816DkObt0zTNvYwMge0KstyP/zIjOJjIxk4cKFAAwePBh7e3sLRySESG3Jalrw1VdfMWTIEIKCgoxlQUFBDB8+nLFjx5otOCFEymz9DVr0jUlis7nDqm+g96eSxIrMycbGBhubZP3XJoTIgBLdtKB8+fImP9NcuXKF8PBw8ufPD8CtW7dwcHCgaNGinDx5MnWiNTNpWiAyq8go+Hox/LA5pqxMcVg8EfLmtFxcQgghxNukStOCFi1apDQuIUQaePAE+k6EY2djyj5pDJMHgaP80iqEECITSdbDXpmF3JEVmc2JC9BnAtx/ZJi2s4VJA6BdE2lKIIQQImNI9Ye9hBDpi1Lw03aYuMDQrAAgl6ehKUF5X8vGJkRaiYqK4vfffwegbt26MkStEFYgWVe5Tqdjzpw5bNiwgVu3bhEREWEy/8mTJ2YJTgjxdq/C4au5sPHXmLJ3y8KCcZAjm8XCEiLN6fV6Tp06BWAc4UsIkbkl69HOiRMnMnv2bNq2bUtwcDBDhgyhZcuW2NjYMGHCBDOHKISIz50gaNXfNInt3gZ+milJrLA+Wq2W6tWrU716dRmiVggrkaw2soULF+a7777jo48+IkuWLJw+fdpY9tdff7FmzZrUiNVsXh+i9vLly9JGVmRIh05Av0nwNMQw7eQI3wyHZnUtG5cQQgiREklpI5vskb3++ecf8ufPT+7cudm5cycVKlTg33//pXz58gQHByc7+LQkD3uJjEgpWLwOvllmGHYWwCcPLJ0MJQpZNjYhhBAipZKSnyWraUG+fPkIDAwEDHdn9+zZA8Dx48dxcHBIziqFEIkQ+sLQK8H0pTFJbN134X9LJIkVQinFq1evePXqFVbcIY8QViVZiezHH3/Mvn37AOjfvz9jx46laNGidOzYka5du5o1QCGEwdVb0KwP/HIgpmxwZ/i/qeDuarGwhEg3IiMjmTdvHvPmzSMyMtLS4Qgh0kCyei2YPn268XXbtm3x8fHhzz//pGjRojRt2tRswQkhDH49CEOnG+7IAri5wNwxUK+aZeMSQgghLClZbWRfvXqFo6NjasSTpqSNrEjvdDqY9QP4vfb8ZIlCsGQSFMhrubiESI+UUuj/a3NjY2NjMqy6ECLjSPU2sl5eXnTq1Im9e/caPzSEEOb1NBg6jTJNYpvVhZ8XSBIrRFw0Gg1arRatVitJrBBWIlmJ7MqVK3nx4gXNmzcnb968DBo0iL///tvcsQlhtc5dhia94eB/l5XWBsb1he++Amcny8YmhBBCpBfJfthr48aN3L9/n6+//pqAgADeffddihUrxqRJk8wdoxBWZdOvhkEO7gQZpj2zwppZ0K01yE0mIeKn0+nYv38/+/fvR6fTWTocIUQaSFYiGy1Llix06dKFPXv2cPbsWVxcXJg4caK5YhPCqkREwth5MHQGhP836nP5krBjCbxbzqKhCZEh6HQ6jh07xrFjxySRFcJKJKvXgmivXr1i+/btrFmzhl9//ZWcOXMyfPhwc8UmhNW4/8jQP+yJCzFlnzWF8f3Awd5iYQmRoWi1WqpUqWJ8LYTI/JKVyO7evZs1a9awdetWbG1tad26NXv27OH99983d3xCZHrHzsIXE+HhE8O0gx1MHgRtP7RoWEJkOFqtljp16lg6DCFEGkpWIvvxxx/TpEkTfvzxRz788EPs7OzMHZcQmZ5SsOJnmLIQov77FTSPl6FrrTLFLRubEEIIkREkK5G9f/8+WbJkMXcsacbPzw8/Pz9pQyUs5uUrGD0bft4bU1a9AswfC9k9LBaWEBma9CMrhPVJ1oAIANeuXWP58uVcu3aNefPm4eXlxS+//EL+/PkpVaqUueNMFTIggrCEW/eg1zgIuBZT1vtTGN4dbKVZnxDJFhERwZw5cwAYPHgw9vbSwFyIjCjVB0T4448/KF26NEePHmXLli2EhoYCcObMGcaPH5+cVQphFfyPGfqHjU5inR1h4XgY3UuSWCGEECKpknVHtlq1arRp04YhQ4aQJUsWzpw5Q6FChTh27BgtW7bkzp07qRGr2ckdWZFW9HpYsBpmLze0jQUo5G1oD1usgEVDEyLTUEoRHh4OgIODgzQtECKDSkp+lqw2sufOnWPNmjWxyr28vHj06FFyVilEphUSCkOnw57DMWUfVIdZo8DN1XJxCZHZaDQaHB0dLR2GECINJatpgYeHB4GBgbHKT506Rd68Mgi8ENEu34DmX8QksRoNDOtmuBMrSawQQgiRMslKZD/99FNGjhxJUFAQGo0GvV7P4cOHGTZsGB07djR3jEJkSDv9oXkf+Pe2Ydo9C6ycDv0/B5sUjaknhIiLTqfj0KFDHDp0SHqlEcJKJKtpwddff03fvn3x9vZGp9Ph6+uLTqejffv2fPXVV+aOUYgMJUoH33wPS9bHlPkWMdyFzZ/bcnEJkdnpdDoOHzb8/FGlShUZ3UsIK5CsRNbe3p7vv/+esWPHcv78eUJDQylfvjxFixY1d3xCZCiPn0G/SfDnqZiylg3g6yHgJE33hEhVNjY2lC9f3vhaCJH5Jbsf2cxAei0Q5nTmIvQeD/ceGKZttTC2L3RqYWgbK4QQQoi3S5VeC4YMGcLkyZNxcXFhyJAhCdadPXt2YlcrRKawbieMnQcRkYbpHNlg0QSoXNqiYQkhhBCZWqIT2VOnThEZGWl8HR/pt09Yk/AImDAf1uyIKav0DiycADmzWywsIYQQwipI0wJpWiCSKfAh9B4Hpy/GlHX6GL7qA/Z2lotLCGsVERHBvHnzABg4cKAMUStEBpXqAyIIYe2OnDY81PXoqWHawR6mDYFWDS0alhBWT6/XWzoEIUQaSlYiGxYWxvTp09m3bx8PHjyI9cHx77//miU4IdIbpeD/NsHXi0H332mfL5eha613pNMOISzKzs6OL774wvhaCJH5JSuR7d69O3/88QcdOnQgd+7c0i5WWIUXL2HkTNj+e0zZ+5Xgu68gq7vl4hJCGGg0GrJkyWLpMIQQaShZiewvv/zCzp07qV69urnjESJdunEXeo6FS9djyvp+BkO7gPS5LoQQQlhGshLZrFmzki1bNnPHIkS6tO8IDJoKIWGGaVdnmD0aGtawbFxCCFM6nY6///4bgEqVKsnIXkJYgWQNfTJ58mTGjRvHixcvzB2PEOmGXg9zVkDXL2OS2CI+sH2RJLFCpEc6nQ5/f3/8/f3R6XSWDkcIkQYSfUe2fPnyJm1hr169Ss6cOSlQoECsRvUnT540X4RCWEDwcxj0Nfz+V0xZ4/dh5kjDHVkhRPpjY2PDO++8Y3wthMj8Ep3ItmjRIhXDSFt+fn74+fnJN3YRp3+uQa9xcPOeYdrGBkZ2h16fylCzQqRntra2fPTRR5YOQwiRhmRABBkQQbxm2z5DzwQvXxmms7rBgnFQo6Jl4xJCCCGsRaoPiHD8+HH0ej1Vq1Y1KT969CharZZKlSolZ7VCWExkFExbYugjNlrpYrB4oqGfWCGEEEKkP8lqRNS3b19u374dq/zu3bv07ds3xUEJkZYePoHPhpomsW0awab5ksQKkZFEREQwd+5c5s6dS0REhKXDEUKkgWTdkQ0ICKBChQqxysuXL09AQECKgxIirZwMgN7j4f4jw7SdLUzsD+2bSntYITKi8PBwS4cghEhDyUpkHRwcuH//PoUKFTIpDwwMxNY2WasUIk0pBWv+B+PnG5oVAOT0NDQlqOBr2diEEMljZ2dHjx49jK+FEJlfspoWfPDBB4wePZrg4GBj2bNnz/jyyy9p0KCB2YITIjW8ioAR38KXc2KS2KplYOcSSWKFyMg0Gg3ZsmUjW7ZsMnS6EFYiWbdPZ86cyfvvv4+Pjw/ly5cH4PTp0+TMmZNVq1aZNUAhzOnufUNTgrOXYsq6tYbRvQzNCoQQQgiRcSTrv+68efNy9uxZVq9ezZkzZ3BycqJLly60a9dOfs4R6dahE9B/Mjz574cERwf4Zjg0r2fZuIQQ5qHT6Thz5gwAZcuWlSFqhbACyb4H5eLiQs+ePc0ZixCpQilYsg5mLDMMOwuQPw8snQQlC1s2NiGE+eh0Ovbu3QvAO++8I4msEFYgRT+mBgQEcOvWrVjdnDRr1ixFQQlhLqEvYPg3sOuPmLI6VWHeGHDPYrm4hBDmZ2NjQ/HixY2vhRCZX7IS2X///ZePP/6Yc+fOodFoiB4cLLpxvQz9KtKDa7eg5zi4ejOmbFAnGNjRMOysECJzsbW1zVTDqQsh3i5Z/50PHDiQggUL8uDBA5ydnblw4QIHDhygUqVK+Pv7mzlEIZJu9yFo1icmiXVzgR++hsGdJYkVQgghMotk3ZE9cuQIv//+O56entjY2GBjY0ONGjWYNm0aAwYM4NSpU+aOU4hE0elg9gpY8FNMWfGCsGQSFMxnsbCEEEIIkQqSdW9Kp9ORJYuhgaGnpyf37t0DwMfHh0uXLiW0qBCp5mkwdBltmsQ2qwtb/SSJFcIaREZG4ufnh5+fH5GRkZYORwiRBpJ1R/add97hzJkzFCxYkKpVq/LNN99gb2/P0qVLY432JURaOH8Feo2DO0GGaa0NfNnb0Ees9IsuhHVQShEaGmp8LYTI/JKVyH711VeEhYUBMGnSJJo0aULNmjXJnj0769evN2uAQrzNlj0wahaE/9d5RnYP8BsP1cpZMiohRFqztbWlc+fOxtdCiMxPo8z0tfXJkydkzZo1Qw0LGBISgru7O8HBwbi5uVk6HJFEEZEwZRGs/DmmrFwJWDwJcuewXFxCCCGESL6k5Gdm+8qaLVs2c61KiLe6/xi+mAB/n48pa98EJvQHB3uLhSWEEEKINCS/vYgM5/g56DMBHj4xTNvbwaSB0O4ji4YlhLAwnU5HQEAAAL6+vjKylxBWQBJZkWEoBT9uhUl+EPXfmBt5vGDxRChbwqKhCSHSAZ1Ox65duwAoXry4JLJCWAFJZEWG8CocvpwNm/fElL1XHhaMMzzcJYQQNjY2xp5zZIhaIayDJLIi3bsVaOhaK+BqTFmvtjCiB9jKDRchxH9sbW1p06aNpcMQQqQhSWRFuvbHMeg/BYKfG6adHWHmSPiotkXDEkIIIUQ6IImsSJf0eli4Bmb+YGgbC4bRuZZOgmIFLRubEEIIIdIHSWRFuvM8DIZOh92HYsoavAezR4Obq+XiEkKkb5GRkSxfvhyALl26YGdnZ+GIhBCpTRJZka5cvmFoD/vvbcO0RgNDukC/z0Ce3RBCJEQpxdOnT42vhRCZX6ZIDT7++GOyZs1K69atLR2KSIFdf0CLL2KSWPcssHwaDOggSawQ4u1sbW357LPP+Oyzz2SIWiGsRKZIDwYOHMiPP/5o6TBEMkXpYNoSwyAHYS8NZb6FYcdiqFPVoqEJITIQGxsb8uXLR758+aT7LSGsRKa40mvXrk2WLFksHYZIhifB0HEELF4XU9aiPmxZAPnzWC4uIYQQQqR/Fk9kDxw4QNOmTcmTJw8ajYatW7fGquPn50eBAgVwdHSkatWqHDt2LO0DFWZ39hI06QWHTxqmbbUwoT/M/RKcHC0bmxAi49Hr9Vy8eJGLFy+i1+stHY4QIg1YPJENCwujbNmy+Pn5xTl//fr1DBkyhPHjx3Py5EnKli1Lw4YNefDgQRpHKsxpwy/Quj/cvW+YzpEN1s2BLi0ND3gJIURSRUVFsW3bNrZt20ZUVJSlwxFCpAGLt4Zv3LgxjRs3jnf+7Nmz6dGjB126dAFg8eLF7Ny5kx9++IFRo0YlaVvh4eGEh4cbp0NCQpIXtEi28AiY5Ac/bY8pq1gKFk2AnJ4WC0sIkQloNBq8vb2Nr4UQmZ/FE9mEREREcOLECUaPHm0ss7GxoX79+hw5ciTJ65s2bRoTJ040Z4giCYIeQu8JcCogpqxDcxjXF+ylu0chRArZ2dnRvn17S4chhEhDFm9akJBHjx6h0+nImTOnSXnOnDkJCgoyTtevX582bdqwa9cu8uXLF2+SO3r0aIKDg41/t2/fTtX4RYy/TsNHvWKSWAd7mDUSpgySJFYIIYQQyZOu78gm1m+//Zaoeg4ODjg4OKRyNOJ1SsEPm2HqItD99+xFvpyweBKULmbZ2IQQQgiRsaXrRNbT0xOtVsv9+/dNyu/fv0+uXLksFJVIrBcvYeRM2P57TFnNSjD/K8jqbrm4hBCZU2RkJD/99BMAn3/+uQxRK4QVSNdNC+zt7alYsSL79u0zlun1evbt20e1atUsGJl4m5t34eN+pkls3/awcroksUKI1KGU4sGDBzx48ECGqBXCSlj8jmxoaChXr141Tl+/fp3Tp0+TLVs28ufPz5AhQ+jUqROVKlWiSpUqzJ07l7CwMGMvBsnh5+eHn58fOp3OHLsg3vD7XzBwKoSEGqZdnGDWKGj8vmXjEkJkbra2tnzyySfG10KIzE+jLPy11d/fnzp16sQq79SpEytWrABgwYIFfPvttwQFBVGuXDm+++47qlZN+dilISEhuLu7ExwcjJubW4rXZ+30evhuFcxdaWgbC1A4PyydDEXyWzY2IYQQQmQMScnPLJ7IWpIksuYTHAqDv4Z9r3UY0aim4U6sq7Pl4hJCCCFExpKU/Ex+exEpdvFf6DUObtw1TNvYwIju0PtTGaVLCJF29Ho9169fB6BgwYLY2KTrx0CEEGYgV7lIke2/Q4u+MUlsVjf4cQb0aSdJrBAibUVFRbFp0yY2bdokQ9QKYSXkjqxIligdTFsCyzbGlL1T1NA/rLf0jCaEsACNRmPsmlGGqBXCOlhlIiu9FqTMo6fQd5JhtK5obRoZRulylPEmhBAWYmdnR6dOnSwdhhAiDcnDXvKwV5KcCoDe4yHokWHazhYm9IfPmkpTAiGEddPpdERGRlo6DCHSPTs7O7Rabbzz5WEvkSrW7oBx30HEf5/TOT1h0QSoWMqiYQkhhEUppQgKCuLZs2eWDkWIDMPDw4NcuXKluBmQJLLirV5FwLh5sH5XTFmVMuA3HryyWS4uIYR4XWRkJOvXrwegbdu2aTZEbXQS6+XlhbOzs7TPFSIBSilevHjBgwcPAMidO3eK1ieJrEjQ3fuGpgRnL8WUdW0FX/Y2NCsQQoj0QinF3bt3ja/Tgk6nMyax2bNnT5NtCpHROTk5AfDgwQO8vLwSbGbwNpKKiHgdPgn9JsGTYMO0owNMHwofN7BsXEIIERdbW1s+/vhj4+u0EN0m1tlZRn4RIimir5nIyEhJZJNKei1ImFKwdANMX2oYdhYgfx5YMhF8i1g2NiGEiI+NjQ3FihWzyLalOYEQSWOua8YqB0To27cvAQEBHD9+3NKhpDthL6HvRPh6cUwSW7sq7FgsSawQQggh0herTGRF3P69Dc37wM4/YsoGdIAfpoJ7FsvFJYQQiaHX67l16xa3bt1CH/1NXIg4XLx4kXfffRdHR0fKlSvHjRs30Gg0nD59GgB/f380Go30RJEBSCIrANhzCJr1gSs3DdNZXOD/psLQrpCCpitCCJFmoqKiWLt2LWvXrpUhat+ic+fOtGjRIlb5mwncq1ev6Ny5M6VLl8bW1jbOZaKXq1ChAg4ODhQpUoQVK1bEquPn50eBAgVwdHSkatWqHDt2zHw7lETjx4/HxcWFS5cusW/fvljz33vvPQIDA3F3d7dAdCIpJJG1cjodzPwBeoyF52GGsmIF4H+Lof57Fg1NCCGSRKPRkD17drJnzy5tVs1Ep9Ph5OTEgAEDqF+/fpx1rl+/zkcffUSdOnU4ffo0gwYNonv37uzevdtYZ/369QwZMoTx48dz8uRJypYtS8OGDY1dMCVGREREivcn2rVr16hRowY+Pj5x9jZhb29vlj5OReqTRNaKPQuBLl/C/FUxZU1qw9aFUDCfxcISQohksbOzo3v37nTv3j3N+pDN7FxcXFi0aBE9evQgV65ccdZZvHgxBQsWZNasWZQsWZJ+/frRunVr5syZY6wze/ZsevToQZcuXfD19WXx4sU4Ozvzww8/xLvt6LvGU6dOJU+ePBQvXhyAc+fOUbduXZycnMiePTs9e/YkNDTUuJxer2fSpEnky5cPBwcHypUrx6+//mqcr9FoOHHiBJMmTUKj0TBhwoRY237zzvSKFSvw8PBg9+7dlCxZEldXVxo1akRgYKDJcsuWLaNkyZI4OjpSokQJFi5c+Nb3WKSMJLJWKuAqNOkNf/z3y47WBsb0gQXjwMXJsrEJIYTIOI4cORLrbm3Dhg05cuQIYLiTeuLECZM6NjY21K9f31gnPvv27ePSpUvs3buXHTt2EBYWRsOGDcmaNSvHjx9n48aN/Pbbb/Tr18+4zLx585g1axYzZ87k7NmzNGzYkGbNmnHlyhUAAgMDKVWqFEOHDiUwMJBhw4Ylaj9fvHjBzJkzWbVqFQcOHODWrVsmy65evZpx48YxdepU/vnnH77++mvGjh3LypUrE7V+kTxW2f2WtduyF0bPglfhhunsHoYE9r3yFg1LCCEyhSa94OGTtN9ujmywY0ni6+/YsQNXV1eTsuR0SxkUFETOnDlNynLmzElISAgvX77k6dOn6HS6OOtcvHgxwXW7uLiwbNky7O3tAfj+++959eoVP/74Iy4uLgAsWLCApk2bMmPGDHLmzMnMmTMZOXIkn376KQAzZsxg//79zJ07Fz8/P3LlyoWtrS2urq7Gu8yPHj16635GRkayePFiChcuDEC/fv2YNGmScf748eOZNWsWLVu2BKBgwYIEBASwZMkSOnXq9Nb1i+SxykTWWvuRjYyCKQthxc8xZWVLwOKJkMfLcnEJIYQ5REZGsnnzZgBatWplseYFD59A0NvzIourU6cOixYtMik7evQon3/+uYUiiq106dLGJBbgn3/+oWzZssYkFqB69ero9XouXbqEk5MT9+7do3r16ibrqV69OmfOnElRLM7OzsYkFgxDq0a38Q0LC+PatWt069aNHj16GOtERUXJA2OpzCoT2b59+9K3b19CQkKs5gR78AS+mADHz8WUtfsIJgwAR/t4FxNCiAxDKcXNmzeNry0lR7aMsV0XFxeKFDHtIPzOnTtJ3m6uXLm4f/++Sdn9+/dxc3PDyckJrVaLVquNs0587W5fjzG9ePOLkUajMZ5n0W10v//+e6pWrWpSLyWjVom3s8pE1tr8fR76TIAHjw3T9nYwaQC0a2LRsIQQwqxsbW1p0qSJ8bWlJOXn/cygWrVq7Nq1y6Rs7969VKtWDTD0AFCxYkX27dtn7L5Lr9ezb98+k7atiVGyZElWrFhBWFiYMck9fPgwNjY2FC9eHDc3N/LkycPhw4epVauWcbnDhw9TpUqVFOxlwnLmzEmePHn4999/+eyzz1JtOyI2SWQzMaVg1TaY5GdoVgCQO4ehKUG5kpaNTQghzM3GxoZSpUpZOoxMJyAggIiICJ48ecLz58+NgwaUK1cOgN69e7NgwQJGjBhB165d+f3339mwYQM7d+40rmPIkCF06tSJSpUqUaVKFebOnUtYWBhdunRJUiyfffYZ48ePp1OnTkyYMIGHDx/Sv39/OnToYGyDO3z4cMaPH0/hwoUpV64cy5cv5/Tp06xevdos70d8Jk6cyIABA3B3d6dRo0aEh4fz999/8/TpU4YMGZKq27ZmkshmUq/CYcwc2BTTjR/VysOCseCZ1XJxCSGEyFg+/PBDY5MNgPLlDU8GR/+sXrBgQXbu3MngwYOZN28e+fLlY9myZTRs2NC4TNu2bXn48CHjxo0jKCjI2CXWmw+AvY2zszO7d+9m4MCBVK5cGWdnZ1q1asXs2bONdQYMGEBwcDBDhw7lwYMH+Pr6sn37dooWLZqSt+GtunfvjrOzM99++y3Dhw/HxcWF0qVLM2jQoFTdrrXTKEs2JLKw6DaywcHBuLm5WTocs7kdBL3GwYUrMWU928LIHmArTXWEEJmUXq83tsPMmTMnNjap38Pkq1evuH79OgULFsTR0THVtydEZpHQtZOU/EzuyGYyB45D/ymGwQ4AnB3hm+HQtK5l4xJCiNQWFRXFjz/+CMDgwYNNnnYXQmROkshmEkrBwjXw7f8ZXgMUyAtLJ0PxgpaNTQgh0oJGozHevZGhRYWwDlaZyGa2fmSfh8HQGbD7YExZ/fdg9mhwd41/OSGEyEzs7Ozo06ePpcMQQqQhqxyitm/fvgQEBHD8+HFLh5JiV25C8z4xSaxGA0O6wPeTJYkVQgghROZmlXdkM4tfDsDQ6RD20jDt5grzxkDddy0blxBCCCFEWpBENgPS6WDmD4Y2sdFKFoIlk8Anr+XiEkIIS4qKimL79u0ANGvWzKKDIggh0oZc5RnMk2DoPxkOnYgpa14PZgwDJ+n5RQhhxfR6PVeuXDG+FkJkfpLIZiDnLkPvcXDnv+GqtTYw9gvo3NLQNlYIIayZVqs1dsIv49sLYR0kkc0gNv4KY2ZDeKRhOkdW8BsPVctaNi4hhEgvtFqtcdhUIYR1sMpeCzKSiEjDULPDZsQkseV9YccSSWKFEEKYl0ajYevWrfHOv3HjBhqNhtOnT6dZTMJ8OnfuTIsWLSwdhllJIpuOBT2EtoPhp+0xZZ83gw1zIVcOi4UlhBDpklKKhw8f8vDhQ6x49PUUCQwMpHHjxpYOI8W2bNlCgwYNyJEjB25ublSrVo3du3eb1Hn+/DmDBg3Cx8cHJycn3nvvvVjdcnbu3BmNRmPy16hRowS3HdcyGo2Gq1evGucnlEy+fPmS8ePHU6xYMRwcHPD09KRNmzZcuHDBpN6ECROM67a1taVAgQIMHjyY0NBQk3o3b97EyckpVnlyTZgwIV398iGJbDp19Aw06QUn/ztvHexg5kiYOhjs7SwbmxBCpEeRkZH88MMP/PDDD0RGRlo6nAwlIiICgFy5cuHg4GDhaFLuwIEDNGjQgF27dnHixAnq1KlD06ZNOXXqlLFO9+7d2bt3L6tWreLcuXN88MEH1K9fn7t375qsq1GjRgQGBhr/1q5d+9btv7lMYGAgBQu+fZjN8PBw6tevzw8//MCUKVO4fPkyu3btIioqiqpVq/LXX3+Z1C9VqhSBgYHcuHGDGTNmsHTpUoYOHWpSZ9u2bdSpUwdX18zZubwksumMUrB8M7QfCg+fGsry5YTN86FNwl8ChRDC6jk5OeHk5GTpMNK92rVr069fPwYNGoSnp6fxIbk3mxYcO3aM8uXL4+joSKVKlUwSwWjbt2+naNGiODo6UqdOHVauXIlGo+HZs2fGOocOHaJmzZo4OTnh7e3NgAEDCAsLS7X9mzt3LiNGjKBy5coULVqUr7/+mqJFi/K///0PMNz13Lx5M9988w3vv/8+RYoUYcKECRQpUoRFixaZrMvBwYFcuXIZ/7JmzfrW7b+5TK5cuRL1AOLcuXM5cuQIO3bs4JNPPsHHx4cqVaqwefNmSpYsSbdu3Ux+bbC1tSVXrlzky5ePtm3b8tlnnxm7oIu2bds2mjVrZlI2ceJE493q3r17G7/I/Pjjj2TPnp3w8HCT+i1atKBDhw6sWLGCiRMncubMGePd4BUrVgDw7Nkzunfvblxv3bp1OXPmzFv3OaWsMpH18/PD19eXypUrWzoUEy9fwcCpMGEBRP03em6NivC/JVC6uGVjE0KI9M7e3p4BAwYwYMAA7O3tLRpLREQEERERJkmHTqcjIiKCqKgos9dNjpUrV2Jvb8/hw4dZvHhxrPmhoaE0adIEX19fTpw4wYQJExg2bJhJnevXr9O6dWtatGjBmTNn6NWrF2PGjDGpc+3aNRo1akSrVq04e/Ys69ev59ChQ/Tr1y/e2A4ePIirq2uCf6tXr070vur1ep4/f062bNkAQ5/DOp0OR0fTfiudnJw4dOiQSZm/vz9eXl4UL16cPn368Pjx40RvN6nWrFlDgwYNKFvW9CEYGxsbBg8eTEBAQILJoZOTkzEpBUNyeejQIZNEdt++ffzzzz/4+/uzdu1atmzZwsSJEwFo06YNOp3OJBl+8OABO3fupGvXrrRt25ahQ4ca7wQHBgbStm1b47IPHjzgl19+4cSJE1SoUIF69erx5MkTs7w38VJWLDg4WAEqODg41bd15+kLtf74LfXDoX/V+uO31J2nL0zm37ijVMOuSuWvHfM3falSUVGpHpoQQohkevnypQoICFAvX740KZ8+fbqaPn26CgsLM5YdPnxYTZ8+Xe3atcuk7qxZs9T06dPVs2fPjGXHjh1T06dPV9u3bzepO2/ePDV9+nT14MEDY9mpU6eSHHetWrVU+fLlY5UD6ueff1ZKKbVkyRKVPXt2k31btGiRAozbHDlypHrnnXdM1jFmzBgFqKdPnyqllOrWrZvq2bOnSZ2DBw8qGxubWO9btBcvXqgrV64k+BcSEpLo/Z0xY4bKmjWrun//vrGsWrVqqlatWuru3bsqKipKrVq1StnY2KhixYoZ66xdu1Zt27ZNnT17Vv3888+qZMmSqnLlyioqgf+cO3XqpLRarXJxcTH+tW7d2mR+8+bN41zW0dFRDRw4MM55J0+eVIBav369Ukqp8ePHq7Jlyxrn//3338rT09NkW6tXr1aVKlUy2Xa2bNlMzstFixYpV1dXpdPplFJK9enTRzVu3Ng4f9asWapQoUJKr9fHuV2lDMfTzc1NvXr1yqS8cOHCasmSJXHuT3zXjlJJy8+k+61Udvr2M+bvu8Lvlx6gFNhoQK8M/b7WLeHFgLpFeXLPgwFTIOS/dtguTjBrFDR+37KxCyGEyLwqVqyY4Px//vmHMmXKmNy1rFatmkmdS5cuxfp1s0qVKibTZ86c4ezZsyZ3UJVS6PV6rl+/TsmSJWNt28nJiSJFiiR6XxKyZs0aJk6cyLZt2/Dy8jKWr1q1iq5du5I3b160Wi0VKlSgXbt2nDgRM+LQp59+anxdunRpypQpQ+HChfH396devXrxbrNOnTomTRRcXFwSHa9KwoOK586dw9XV1XgH/6OPPmLBggXG+XE1KyhbtizOzs7G6WrVqhEaGsrt27fx8fGhR48eVK5cmbt375I3b15WrFhhfIAtPmfOnCE0NJTs2bOblL98+ZJr164len+SQxLZVPTr+UD6rTmFwtD2FQxJLP9N+198yP5/HqK/Uh4VmhuAwt6wZDIU9bFMzEIIkVFFRUXxyy+/ANC4cWOLDlE7ePBgAOzsYp7OrVq1KpUqVcLGxrRVX/RP7K/XrVChAmXLlo1Vt3fv3rHqli5dOlkxJiW5SonQ0FB69erFgAEDYs3Lnz9/nMscPHjwrb0nLFmyhM8++yzBOuvWraN79+5s3LiR+vXrm8wrXLgwf/zxB2FhYYSEhJA7d27atm1LoUKF4l1foUKF8PT05OrVqwkmsi4uLslKxIsVK8Y///wT57zo8mLFihnLihcvzvbt27G1tSVPnjwmTWoiIiL49ddf+fLLL5MUQ/ny5Slbtiw//vgjH3zwARcuXGDnzp0JLhMaGkru3Lnx9/ePNc/DwyNJ208qSWRTyenbz+i35hQ6vSK+71Y6pQwJbuFTEO5EwwoezBoJWdLms0UIITIVvV5PQEAAgPHhJUuJq42uVquN84Efc9RNDSVLlmTVqlW8evXKeFf2zafmixcvzq5du0zK3uzCqkKFCgQEBCQpsatUqdJb+6rNmTNngvPXrl1L165dWbduHR999FG89VxcXHBxceHp06fs3r2bb775Jt66d+7c4fHjx+TOnTvBbSfXp59+ypgxYzhz5oxJO1m9Xs+cOXPw9fU1Kbe3t4/3ffX39ydr1qyx2tueOXOGly9fGh+K/Ouvv3B1dcXb29tYp3v37sydO5e7d+9Sv359k3n29vax2mVXqFCBoKAgYzdgackqH/ZKC/N/v2K4E/uWehqN4a9ItassmShJrBBCJJdWq6Vu3brUrVtXhqg1g/bt26PRaOjRowcBAQHs2rWLmTNnmtTp1asXFy9eZOTIkVy+fJkNGzYYn2KP/il65MiR/Pnnn/Tr14/Tp09z5coVtm3bluDDXtFNCxL6y5IlS7zLr1mzho4dOzJr1iyqVq1KUFAQQUFBBAcHG+vs3r2bX3/9levXr7N3717q1KlDiRIl6NKlC2C4yzh8+HD++usvbty4wb59+2jevDlFihRJ8Rel4OBgTp8+bfJ3+/ZtBg8eTJUqVWjatCkbN27k1q1bHD9+nFatWvHPP//wf//3fwn+xP+67du3x2pWAIY7td26dTMe0/Hjx9OvXz+Tu//t27fnzp07fP/993Tt2tVk+QIFCnD9+nVOnz7No0ePjF2GVatWjRYtWrBnzx5u3LjBn3/+yZgxY/j7779T9F69jSSyqeDus5f8fvEBOn0i27loFP+G3ude8MvUDUwIITIxrVZL5cqVqVy5siSyZuDq6sr//vc/zp07R/ny5RkzZgwzZswwqVOwYEE2bdrEli1bKFOmDIsWLTL2WhDdH22ZMmX4448/uHz5MjVr1qR8+fKMGzeOPHnypFrsS5cuJSoqir59+5I7d27j38CBA411goOD6du3LyVKlKBjx47UqFGD3bt3G5ttaLVazp49S7NmzShWrBjdunWjYsWKHDx4MMV97fr7+1O+fHmTv4kTJ+Lo6Mjvv/9Ox44d+fLLLylSpAiNGjVCq9Xy119/8e677yZ6G/ElsvXq1aNo0aK8//77tG3blmbNmjFhwgSTOu7u7rRq1QpXV9dYgze0atWKRo0aUadOHXLkyMHatWvRaDTs2rWL999/ny5dulCsWDE+/fRTbt68+dY75ymlUUlpVZzJhISE4O7uTnBwMG5ubmZb74a/bzNi09kkL/dt6zK0qeT99opCCCHShVevXnH9+nUKFiwYqysnazV16lQWL17M7du3LR2K1Tp58iR169bl4cOHJu2pk6JevXqUKlWK7777zszRGSR07SQlP5M2sqkgLDzK2DtBYtloIDQ86u0VhRBCxEkpRUhICABubm6J/glWpMzChQupXLky2bNn5/Dhw3z77bcJNhsQqS8qKor58+cnK4l9+vQp/v7++Pv7s3DhwlSIzrwkkU0FLg62SUpiwZD0ujrI4RBCiOSKjIw0duw/ePBgiw+KYC2uXLnClClTePLkCfnz52fo0KGMHj3a0mFZtSpVqsTqBi2xypcvz9OnT5kxYwbFi6f/0Zgkc0oF1Yt4otHEdLmVGBoNvFfEM/WCEkIIK5Dcn1FF8s2ZM4c5c+ZYOgxhJjdu3LB0CEkiiWwqyOvhRN0SXvhfepioB760NhrqFPcir4eMDy6EEMllb2/PkCFDLB2GECINSa8FqWRA3aJogLe10Iqu07+ueUYwEUIIkfas+LlpIZLFXNeMJLKppKy3Bwval0dro0FrE3c6Gz1vQfsKlPX2SNsAhRBCpFh0U4YXL15YOBIhMpboayalzYGssmmBn58ffn5+sUamMLdG7+Rmcx8nvvv9Cr9ffIBSGHsz0GigTnEv+tctIkmsEEKYQVRUFHv37gWgQYMGaTJErVarxcPDgwcPHgDg7OwsvSUIkQClFC9evODBgwd4eHikuM9n6Uc2FfqRjcvdZy/58+ojQsOjcHWw5b0inkluE1u7dm3KlSvH3LlzUyfIRPD396dOnTo8ffo0xeMnv//++/Tu3Zv27dubJ7g3LF68mJ07d/K///0vVdYvhEhfIiIijA8dpWWvBUopgoKCePbsWZpsT4jMwMPDg1y5csX5xU/6kU2H8no4yWAHr9m+fTv379/n008/Tdbyr169onfv3pw4cYJ//vmHJk2asHXrVpM6Xbt2ZfLkyRw8eJCaNWuaIWohRHqm1WqN13pajuyl0WjInTs3Xl5eREZGptl2hcio7OzszHaNSiIrLOK7776jS5cuJmM7J4VOp8PJyYkBAwawefPmOOvY29vTvn17vvvuO0lkhbACWq2W9957z6Lbl6FxhUhb8rBXOhUWFkbHjh1xdXUld+7czJo1K1ad8PBwhg0bRt68eXFxcaFq1ar4+/ub1Dl8+DC1a9fG2dmZrFmz0rBhQ54+fWpcfsCAAXh5eeHo6EiNGjU4fvy4yfK7du2iWLFiODk5UadOnTj7lzt06BA1a9bEyckJb29vBgwYQFhYWLz79vDhQ37//XeaNm2a9DfmPy4uLixatIgePXqQK1eueOs1bdqU7du38/Lly2RvSwghhBDpkySy6dTw4cP5448/2LZtG3v27MHf35+TJ0+a1OnXrx9Hjhxh3bp1nD17ljZt2tCoUSOuXLkCwOnTp6lXrx6+vr4cOXKEQ4cO0bRpU+NDbiNGjGDz5s2sXLmSkydPUqRIERo2bMiTJ08AuH37Ni1btqRp06acPn2a7t27M2rUKJMYrl27RqNGjWjVqhVnz55l/fr1HDp0KMHhCQ8dOoSzszMlS5ZM0nui1+uTVB+gUqVKREVFcfTo0SQvK4TIWKIfInnx4oV0hyWEtVBWLDg4WAEqODjY0qGYeP78ubK3t1cbNmwwlj1+/Fg5OTmpgQMHKqWUunnzptJqteru3bsmy9arV0+NHj1aKaVUu3btVPXq1ePcRmhoqLKzs1OrV682lkVERKg8efKob775Riml1OjRo5Wvr6/JciNHjlSAevr0qVJKqW7duqmePXua1Dl48KCysbFRL1++jHPbc+bMUYUKFYpVfvXqVdWuXTuVI0cOVaxYMTVy5Eh1/vx5pZRSZ8+eVU2aNIlzfZ06dVLNmzePc55SSmXNmlWtWLEi3vlCiMwhPDxcTZ8+XU2fPl2Fh4dbOhwhRDIlJT+z6jay6r9v7CEhIRaOxNS5c+eIiIjA19fXGJutrS1FihQhIiKCkJAQjh49ik6no1ixYibLhoeH4+bmRkhICCdPnqRFixZx7t/58+eJjIykTJkyJvMrVKjA2bNnCQkJ4dy5c1SoUMFkftmyZQHDe2ZjY8PJkye5cOECq1evNtZRSqHX6zl37lyc4zQ/ffoUe3v7WHH179+fqlWr0r9/f27cuMGmTZuoXLkykZGRZM+endGjR8e5L5GRkURFRcV7HB0dHXn8+HG6O85CCPOKiIjg1atXgOEzKq16LRBCmFf0/9cqEb+sWHX3W3fu3MHbW3oSEEIIIYRIb27fvk2+fPkSrGPViaxer+fevXtkyZIlzn7MKleuHOvhp8TOj2teSEgI3t7e3L59O8F+0UJDQylQoADff/89H3/8MWC4i+nr60unTp2YPn06V69epWLFivzyyy/xPqX7xRdfcO3aNXbv3h1rXlhYGAUKFGDhwoW0adMGwHiHtk+fPgwYMICCBQuSM2dO/vrrL+NyEyZMYM6cOdy8eRMPDw+6d+/OgwcP2L59e7z786aTJ09St25dChUqFKvdb3J4e3tTs2ZN1qxZE2vev//+S/ny5Tl16hSFChUylr/t2L4usXUtdb6kB0l5Py29jeSsR84X88oo50ty12Hu8yWldeR8SZttZJTzJTnz0/p8UUrx/Plz8uTJ89bejay6aYGNjU2Cmb5Wq03wgCU0P6F5bm5uCa7Xzc2Nbt26MX78eLy9vfHy8mLMmDHY2Nhgb2+Pm5sbFSpU4LPPPqNPnz7MmjWL8uXL8/DhQ/bt20eZMmX46KOPGDduHKVLl2bUqFH07t0be3t79u/fT5s2bcidOzd9+vRh3Lhx5M2bl/z58/PNN9/w8uVL+vbti5ubG9myZePatWtMnjyZ7t27c+LECdauXWuyD1999RXvvvsuX375Jd27d8fFxYWAgAD27t3LggUL4ty/mjVr4unpaWwGkVwBAQFERESg1+sJCwvj33//BaBcuXLGOqdPn6ZQoUImZfD2Y5ucupY6X9KDpLyflt5GctYj54t5ZZTzJbnrMPf5ktI6cr6kzTYyyvmSkvlpeb64u7snqp5VJ7Jv07dv32TPf9uyb/Ptt98SGhpK06ZNyZIlC0OHDiU4ONikzvLly5kyZQpDhw7l7t27eHp68u6779KkSRMAihUrxp49e/jyyy+pUqUKTk5OVK1alXbt2gEwffp09Ho9HTp04Pnz51SqVIndu3eTNWtWAAYNGoSPjw+DBw9m/vz5VKlSha+//pquXbsaYyhTpgx//PEHY8aMoWbNmiilKFy4MG3bto1337RaLV26dGHfvn0peo8+/PBDbt68CRhGHCtfvjxg2qZm7dq19OjRI9aySTk+ia1ryfPF0tIifnNtIznrkfPFvDLK+ZLcdZj7fElpHTlf0mYbGeV8Sen89MaqmxaktbQcEje9CwoKolSpUpw8eRIfH59U2caFCxeoW7culy9fTvQ3u/REzheRFHK+iKSQ80UkRXo+X6Qf2TTk4ODA+PHjcXBwsHQoFpcrVy7+7//+j1u3bqXaNgIDA/nxxx8zZBILcr6IpJHzRSSFnC8iKdLz+SJ3ZIUQQgghRIYkd2SFEEIIIUSGJImsEEIIIYTIkCSRFUIIIYQQGZIkskIIIYQQIkOSRDYd2bFjB8WLF6do0aIsW7bM0uGIdO7jjz8ma9astG7d2tKhiHTu9u3b1K5dG19fX8qUKcPGjRstHZJIx549e0alSpUoV64c77zzDt9//72lQxLp3IsXL/Dx8WHYsGFpvm3ptSCdiIqKwtfXl/379+Pu7k7FihX5888/yZ49u6VDE+mUv78/z58/Z+XKlWzatMnS4Yh0LDAwkPv371OuXDmCgoKoWLEily9fxsXFxdKhiXRIp9MRHh6Os7MzYWFhvPPOO/z999/y/5GI15gxY7h69Sre3t7MnDkzTbctd2TTiWPHjlGqVCny5s2Lq6srjRs3Zs+ePZYOS6RjtWvXJkuWLJYOQ2QAuXPnNg7TnCtXLjw9PXny5IllgxLpllarxdnZGYDw8HCUUsg9LxGfK1eucPHiRRo3bmyR7UsiayYHDhygadOm5MmTB41Gw9atW2PV8fPzo0CBAjg6OlK1alWOHTtmnHfv3j3y5s1rnM6bNy93795Ni9CFBaT0fBHWxZzny4kTJ9DpdHh7e6dy1MJSzHG+PHv2jLJly5IvXz6GDx+Op6dnGkUv0pI5zpVhw4Yxbdq0NIo4NklkzSQsLIyyZcvi5+cX5/z169czZMgQxo8fz8mTJylbtiwNGzbkwYMHaRypSA/kfBFJYa7z5cmTJ3Ts2JGlS5emRdjCQsxxvnh4eHDmzBmuX7/OmjVruH//flqFL9JQSs+Vbdu2UaxYMYoVK5aWYZtSwuwA9fPPP5uUValSRfXt29c4rdPpVJ48edS0adOUUkodPnxYtWjRwjh/4MCBavXq1WkSr7Cs5Jwv0fbv369atWqVFmGKdCK558urV69UzZo11Y8//phWoYp0ICWfL9H69OmjNm7cmJphinQgOefKqFGjVL58+ZSPj4/Knj27cnNzUxMnTkzLsJXckU0DERERnDhxgvr16xvLbGxsqF+/PkeOHAGgSpUqnD9/nrt37xIaGsovv/xCw4YNLRWysKDEnC9CREvM+aKUonPnztStW5cOHTpYKlSRDiTmfLl//z7Pnz8HIDg4mAMHDlC8eHGLxCssJzHnyrRp07h9+zY3btxg5syZ9OjRg3HjxqVpnLZpujUr9ejRI3Q6HTlz5jQpz5kzJxcvXgTA1taWWbNmUadOHfR6PSNGjJAnRK1UYs4XgPr163PmzBnCwsLIly8fGzdupFq1amkdrrCwxJwvhw8fZv369ZQpU8bYBm7VqlWULl06rcMVFpaY8+XmzZv07NnT+JBX//795VyxQon9v8jSJJFNR5o1a0azZs0sHYbIIH777TdLhyAyiBo1aqDX6y0dhsggqlSpwunTpy0dhshgOnfubJHtStOCNODp6YlWq43VWP7+/fvkypXLQlGJ9ErOF5EUcr6IpJDzRSRWRjlXJJFNA/b29lSsWJF9+/YZy/R6Pfv27ZOfgkUscr6IpJDzRSSFnC8isTLKuSJNC8wkNDSUq1evGqevX7/O6dOnyZYtG/nz52fIkCF06tSJSpUqUaVKFebOnUtYWBhdunSxYNTCUuR8EUkh54tICjlfRGJlinMlTftIyMT279+vgFh/nTp1MtaZP3++yp8/v7K3t1dVqlRRf/31l+UCFhYl54tICjlfRFLI+SISKzOcKxqlZNw5IYQQQgiR8UgbWSGEEEIIkSFJIiuEEEIIITIkSWSFEEIIIUSGJImsEEIIIYTIkCSRFUIIIYQQGZIkskIIIYQQIkOSRFYIIYQQQmRIksgKIYQQQogMSRJZIYQQQgiRIUkiK4QwmxUrVuDh4fHWehqNhq1bt6Z6PIlVu3ZtBg0alOzlCxQowNy5c80WT2pJ7PFJrg4dOvD111+n2vrNKaXHPCVGjRpF//79LbJtITIbSWSFEGbTtm1bLl++bJyeMGEC5cqVs1xAb/D390ej0fDs2TOT8i1btjB58uRkr/f48eP07NkzxXGkttQ8PmfOnGHXrl0MGDDAWFa7dm00Gk2sv6ioKOP8hJLJJ0+eMGjQIHx8fLC3tydPnjx07dqVW7dumdTr3Lmzcd329vYUKVKESZMmGbcT7Y8//sDb29ss+xu93RYtWiR5uWHDhrFy5Ur+/fdfs8UihLWSRFYIYRaRkZE4OTnh5eVl6VCSLFu2bGTJkiXZy+fIkQNnZ2czRpQ6UvP4zJ8/nzZt2uDq6mpS3qNHDwIDA03+bG1t37q+J0+e8O677/Lbb7+xePFirl69yrp167h69SqVK1eOlQQ2atSIwMBArly5wtChQ5kwYQLffvutSZ1t27bRtGnTlO9sCnl6etKwYUMWLVpk6VCEyPAkkRVCxPLrr79So0YNPDw8yJ49O02aNOHatWvG+Tdu3ECj0bB+/Xpq1aqFo6Mjq1evNvnpesWKFUycOJEzZ84Y75atWLHCuI5Hjx7x8ccf4+zsTNGiRdm+fbtxXvQdy927d1O+fHmcnJyoW7cuDx484JdffqFkyZK4ubnRvn17Xrx4YVxOr9czbdo0ChYsiJOTE2XLlmXTpk3GmOvUqQNA1qxZ0Wg0dO7cGYh9ZzA8PJyRI0fi7e2Ng4MDRYoU4f/+7//ifb/ebFqg0WhYtmxZnPuXUBwJxf/6+7Jv3z4qVaqEs7Mz7733HpcuXTLWOXPmDHXq1CFLliy4ublRsWJF/v77b+Mxedvx6dq1K02aNDHZv8jISLy8vOJ9D3Q6HZs2bYozSXR2diZXrlwmf4kxZswY7t27x2+//Ubjxo3Jnz8/77//Prt378bOzo6+ffua1HdwcCBXrlz4+PjQp08f6tevb3JOAWzfvp1mzZoZp6OioujXrx/u7u54enoyduxYlFIATJo0iXfeeSdWXOXKlWPs2LFMmDCBlStXsm3bNuP75+/vD8Dt27f55JNP8PDwIFu2bDRv3pwbN26YrKdp06asW7cuUe+FECIBSggh3rBp0ya1efNmdeXKFXXq1CnVtGlTVbp0aaXT6ZRSSl2/fl0BqkCBAmrz5s3q33//Vffu3VPLly9X7u7uSimlXrx4oYYOHapKlSqlAgMDVWBgoHrx4oVSSilA5cuXT61Zs0ZduXJFDRgwQLm6uqrHjx8rpZTav3+/AtS7776rDh06pE6ePKmKFCmiatWqpT744AN18uRJdeDAAZU9e3Y1ffp0Y9xTpkxRJUqUUL/++qu6du2aWr58uXJwcFD+/v4qKipKbd68WQHq0qVLKjAwUD179kwppVStWrXUwIEDjev55JNPlLe3t9qyZYu6du2a+u2339S6devifb98fHzUnDlzjNMJ7V9CcSQU/+vvS9WqVZW/v7+6cOGCqlmzpnrvvfeM2y5VqpT6/PPP1T///KMuX76sNmzYoE6fPq2UUok6PocPH1ZarVbdu3fPuM4tW7YoFxcX9fz58zj3/+TJkwpQQUFBJuVvvq9vim++TqdTHh4eqmfPnnEuN3XqVKXRaIznS6dOnVTz5s1N6jRr1kxVqFDBOH3+/HmVJUsWFR4ebty2q6urGjhwoLp48aL66aeflLOzs1q6dKlSSqnbt28rGxsbdezYMZP91Gg06tq1a+r58+fqk08+UY0aNTK+f+Hh4SoiIkKVLFlSde3aVZ09e1YFBASo9u3bq+LFixu3rZT6//buNSSKqI0D+H/d3M1r5iVTo7zkZc2KTCoFMUlZJaQysVTQ0hXJLNGEDAy1KPNDUEgJ+WHVUNc+aJGyhSWmeUMqsVBExbIgTS0Cdb3QnvdDr5OTu+7qa29Izw8GnJkz5zznjAvPzpyZZT09PQwAGxwc1Do+hBDdKJElhOg0OjrKALC3b98yxn4lsrdu3eKVW5goMcZYdnY2271796L6ALCsrCxufWJiggFgSqWSMfYrYXv27BlXJi8vjwFgAwMD3LakpCQmlUoZY4xNT08zY2Nj1tLSwmsrISGBRUVF8er99u0br8zChKq3t5cBYHV1dXqMzE+aEll9+rcwjuXEv3BcamtrGQCmUqkYY4yZmZmx4uJijXHqe348PT1Zfn4+tx4WFsZOnTqltf/V1dVMKBQytVrN2x4QEMAMDQ2ZiYkJt6Snp/P2a0pkh4eHGQDemC5UVVXFALD29nbGGD+RVavVrK6ujonFYpaRkcEdc+3aNRYREcFrWyKR8GK+ePEik0gk3HpoaCg7c+YMt37u3Dl28OBBbl1TAn3//n3m7u7Oq3dmZoYZGRmxp0+fctu+f//OAHBfUgghK0NTCwghi/T19SEqKgrOzs4wNzeHo6MjACx6yMbHx2fFbezatYv728TEBObm5vjy5YvWMra2tjA2NoazszNv2/wx/f39mJqaQnBwMExNTbmltLSUNy1Cl87OTgiFQgQEBKy0a4ti19a/hZYT/8K67ezsAICrOz09HTKZDEFBQbhx48ay+j5PJpNBLpcDAEZGRqBUKhEfH6+1vEqlglgshkAgWLQvJiYGnZ2d3HLp0iW942D/vc2vj5qaGpiammL9+vUIDQ3FiRMnkJOTw+1/9OgRb1oBABw4cIAXs6+vL/r6+vDjxw8AP+f3VlRUYHp6GrOzsygvL19yHICfUzv6+/thZmbGnUNLS0tMT0/zzoWRkREA8KbGEEKWT/eMe0LIPycsLAzbtm1DUVER7O3toVar4eXlhdnZWV45ExOTFbdhaGjIWxcIBFCr1VrLCASCJY+ZmJgAANTW1sLBwYFXTiwW6x3XfILxv9KnfwstJ/7fxwUAV3dOTg6io6NRW1sLpVKJ7OxsKBQKHDt2TO/YY2NjkZmZidbWVrS0tMDJyQn+/v5ay1tbW2Nqagqzs7MQiUS8fRs2bMD27dv1bhv4+fCchYUFenp6NO7v6emBQCDg1RsYGIjCwkLu7QYLHyj7/Pkz3rx5g8OHDy8rjrCwMIjFYlRXV0MkEmFubg4RERFLHjMxMYG9e/eirKxMY7/mff36ddE2QsjyUSJLCOEZHx9Hb28vioqKuOTl5cuXK6pLJBJxV7f+NE9PT4jFYgwNDWm9mjqfZC0V086dO6FWq/HixQsEBQX9kVg1xaFP/Ppyc3ODm5sb0tLSEBUVBblcrjGR1XZ+rKyscPToUcjlcrS2tuL06dNLtjf/Cq/u7u5VeZ2XgYEBIiMjUVZWhitXrvAeEFOpVLh79y6kUiksLS257SYmJloT5sePH8PPz49XHgDa29t5621tbXB1dYVQKAQArFu3DnFxcZDL5RCJRDh58iTvi46m8fP29kZlZSU2bdoEc3NzrX189+4dDA0NsWPHDh2jQQhZCk0tIITwbNy4EVZWVrh37x76+/tRX1+P9PT0FdXl6OiIwcFBdHZ2YmxsDDMzM6sc7S9mZmbIyMhAWloaSkpKMDAwgNevX6OgoAAlJSUAgG3btkEgEKCmpgajo6PcVdDfY46Li0N8fDwePnyIwcFBNDQ04MGDB6sWq6Y49IlfF5VKhZSUFDQ0NODDhw9obm5GR0cHJBKJxvJLnR+ZTIaSkhL09PQgLi5uyXZtbGzg7e29oi88o6OjvKkHnZ2dGBkZwfXr17F582YEBwdDqVTi48ePaGxshFQqxdzcHO7cuaN3G7+/rWDe0NAQ0tPT0dvbi4qKChQUFCA1NZVXRiaTob6+Hk+ePFk0rcDR0RFdXV3o7e3F2NgY5ubmEBMTA2traxw5cgRNTU3c/8/58+fx6dMn7timpib4+/uv2h0AQv5VlMgSQngMDAygUCjw6tUreHl5IS0tbdH7OPV1/PhxhISEIDAwEDY2NqioqFjlaPmuXr2Ky5cvIy8vDxKJBCEhIaitrYWTkxMAwMHBAbm5ucjMzIStrS1SUlI01lNYWIiIiAgkJyfDw8MDiYmJmJycXLU4tcWhK35dhEIhxsfHERsbCzc3N0RGRiI0NBS5ubkayy91foKCgmBnZwepVAp7e3udbctkMo2303UpLy/Hnj17eEtRURGsrKzQ1taGwMBAJCUlwcXFBZGRkXBxcUFHRwdvrvRSJicn8fz5c42JbGxsLFQqFfbt24ezZ88iNTV10Q9buLq6ws/PDx4eHti/fz9vX2JiItzd3eHj4wMbGxs0NzfD2NgYjY2N2Lp1K8LDwyGRSJCQkIDp6WneFVqFQoHExMRljxchhE/AljObnhBCyD9hYmICDg4OkMvlCA8P11lepVLB3d0dlZWV8PX1/T9EqJ+qqipkZWWhu7t7RcczxuDq6ork5OQV35n4nVKpxIULF9DV1aXXj0MQQrSjTxAhhBCOWq3G2NgYbt68CQsLC41XMjUxMjJCaWkpxsbG/nCEy2Nqaor8/PwVHTs6OgqFQoHh4WGd84SXY3JyEnK5nJJYQlYBXZElhBDCef/+PZycnLBlyxYUFxfj0KFDfzukv0YgEMDa2hq3b99GdHT03w6HEKIBJbKEEEIIIWRNooe9CCGEEELImkSJLCGEEEIIWZMokSWEEEIIIWsSJbKEEEIIIWRNokSWEEIIIYSsSZTIEkIIIYSQNYkSWUIIIYQQsiZRIksIIYQQQtYkSmQJIYQQQsia9B+aGG0QuyponQAAAABJRU5ErkJggg==",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: roofline for the H100 with our two operating points\n",
"peak_flops, peak_bw = HW[\"H100-80GB\"]\n",
"ridge = peak_flops / peak_bw\n",
"inten = np.logspace(0, 4, 200) # 1 .. 10000 FLOP/byte\n",
"achievable = np.minimum(inten * peak_bw, peak_flops) # roofline: min(memory roof, compute roof)\n",
"\n",
"fig, ax = plt.subplots(figsize=(7, 4))\n",
"ax.loglog(inten, achievable / 1e12, color=\"#1E40FF\", lw=2, label=\"H100 roofline\")\n",
"ax.axvline(ridge, ls=\":\", c=\"#888\", label=f\"ridge = {ridge:.0f} FLOP/byte\")\n",
"for label, x in [(\"decode (~1)\", mlp_intensity(1, 1, 4096, 16384)), (\"prefill T=512 (~512)\", mlp_intensity(1, 512, 4096, 16384))]:\n",
" y = min(x * peak_bw, peak_flops) / 1e12\n",
" ax.plot(x, y, \"o\", ms=9); ax.annotate(label, (x, y), textcoords=\"offset points\", xytext=(6, -12))\n",
"ax.set_xlabel(\"arithmetic intensity (FLOP/byte)\"); ax.set_ylabel(\"achievable TFLOP/s\")\n",
"ax.set_title(\"Roofline: decode lives under the sloped (memory-bound) roof\")\n",
"ax.legend(loc=\"lower right\"); plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "2054a199",
"metadata": {},
"source": [
"> **What is the interpretation of this plot?** AnswerThe decode point sits on the steep sloped part of the roof: its achievable throughput is capped at `intensity x bandwidth`, a tiny fraction of the chip's peak. Prefill sits near the corner, close to the flat compute ceiling. To make decode faster you can only move it right (raise intensity, e.g. by batching) or lift the sloped roof (more bandwidth). You cannot help it by buying more FLOPs, which is the counterintuitive heart of inference economics.\n"
]
},
{
"cell_type": "markdown",
"id": "ec47be4f",
"metadata": {},
"source": [
"### Exercise 17.1b — The memory-bandwidth wall\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"The roofline implies a hard floor on per-token latency. To generate one token, a memory-bound decode step must *at minimum* read every weight byte from memory once. So the time per token cannot beat:\n",
"\n",
"$$t_\\text{token} \\ge \\frac{\\text{bytes read}}{\\text{bandwidth}}$$\n",
"\n",
"This is the memory-bandwidth wall, stated as an inequality. Implement `min_decode_latency_ms(n_params, bytes_per_param, bandwidth_bytes_per_s)`: the lower bound on milliseconds per token, assuming you must read all the weights once. Then `max_tokens_per_s` is its reciprocal. The check uses a 7B model in BF16 on an H100 and confirms the bound lands in the right ballpark (a few hundred tokens/s ceiling, which matches real single-stream serving).\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "36452753",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.864276Z",
"iopub.status.busy": "2026-06-10T19:24:49.863881Z",
"iopub.status.idle": "2026-06-10T19:24:49.872313Z",
"shell.execute_reply": "2026-06-10T19:24:49.871672Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 17.1b memory-bandwidth wall: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def min_decode_latency_ms(n_params, bytes_per_param, bandwidth_bytes_per_s):\n",
" \"\"\"Lower bound on ms-per-token: time to read every weight once from memory.\"\"\"\n",
" # TODO 1: bytes_to_read = n_params * bytes_per_param\n",
" bytes_to_read = None\n",
" # TODO 2: seconds = bytes_to_read / bandwidth; return milliseconds (x1000)\n",
" ms = None\n",
" attempted(bytes_to_read, ms)\n",
" return ms\n",
"\n",
"def _bandwidth_wall():\n",
" # 7B params, BF16 (2 bytes), H100 (3.35 TB/s)\n",
" ms = min_decode_latency_ms(7e9, 2, 3.35e12)\n",
" # 14e9 bytes / 3.35e12 = 4.18e-3 s = 4.18 ms per token -> ~239 tokens/s ceiling\n",
" assert 3.5 < ms < 5.0, f\"expected ~4.2 ms/token for 7B BF16 on H100, got {ms:.2f} ms\"\n",
" tok_per_s = 1000.0 / ms\n",
" assert 200 < tok_per_s < 290, f\"single-stream ceiling ~239 tok/s; got {tok_per_s:.0f}\"\n",
" # Halving the bytes (int8) should halve the latency / double the ceiling\n",
" ms_int8 = min_decode_latency_ms(7e9, 1, 3.35e12)\n",
" assert abs(ms / ms_int8 - 2.0) < 1e-6, \"int8 (1 byte) should be exactly 2x faster than BF16 (2 bytes)\"\n",
"\n",
"check(\"17.1b memory-bandwidth wall\", _bandwidth_wall)"
]
},
{
"cell_type": "markdown",
"id": "88dd715c",
"metadata": {},
"source": [
"Hint 1 (conceptual)The wall is just \"you cannot read 14 GB of weights faster than your bandwidth allows\". Bytes to read = params x bytes-per-param. Divide by bandwidth to get seconds, multiply by 1000 for ms. This ignores everything except the weight read, which is exactly why it is a *lower* bound: real latency is higher, never lower.\n",
"\n",
"Hint 2 (the lines)\n",
"\n",
"```python\n",
"bytes_to_read = n_params * bytes_per_param\n",
"ms = (bytes_to_read / bandwidth_bytes_per_s) * 1000.0\n",
"```\n",
"\n",
"\n",
"Help — \"got ~2 ms or ~8 ms, not ~4.2\"7e9 params x 2 bytes = 14e9 bytes. 14e9 / 3.35e12 = 4.18e-3 s = 4.18 ms. If you got 2.1 ms you used 1 byte (int8) by mistake; if 8.4 ms you used 4 (fp32). The exercise's first call is BF16, so 2 bytes.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "a53e2e13",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.873417Z",
"iopub.status.busy": "2026-06-10T19:24:49.873341Z",
"iopub.status.idle": "2026-06-10T19:24:49.878343Z",
"shell.execute_reply": "2026-06-10T19:24:49.877529Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 17.1b memory-bandwidth wall\n",
"7B in BF16: >= 4.18 ms/token -> <= 239 tokens/s single-stream on H100\n",
"7B in int8: >= 2.09 ms/token -> <= 479 tokens/s single-stream on H100\n",
"7B in int4: >= 1.04 ms/token -> <= 957 tokens/s single-stream on H100\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines min_decode_latency_ms; the check below re-verifies the reference.\n",
"def min_decode_latency_ms(n_params, bytes_per_param, bandwidth_bytes_per_s):\n",
" bytes_to_read = n_params * bytes_per_param\n",
" return (bytes_to_read / bandwidth_bytes_per_s) * 1000.0\n",
"\n",
"check(\"17.1b memory-bandwidth wall\", _bandwidth_wall, required=True)\n",
"for bpp, name in [(2, \"BF16\"), (1, \"int8\"), (0.5, \"int4\")]:\n",
" ms = min_decode_latency_ms(7e9, bpp, 3.35e12)\n",
" print(f\"7B in {name:4s}: >= {ms:5.2f} ms/token -> <= {1000/ms:5.0f} tokens/s single-stream on H100\")"
]
},
{
"cell_type": "markdown",
"id": "5e8e4b2e",
"metadata": {},
"source": [
"> **Interpretation.** A 7B BF16 model cannot exceed ~239 tokens/s on a single H100 stream, no matter how good your code is, because reading 14 GB of weights at 3.35 TB/s takes 4.18 ms. Quantizing to int8 halves the bytes and doubles the ceiling; int4 doubles it again. This is the cleanest statement of why quantization is the highest-leverage single-stream speedup: it attacks the bytes term in the wall directly. Batching attacks it differently, by amortizing one weight read across many tokens.\n",
"\n",
"> **Key takeaways**\n",
"> - The memory-bandwidth wall: `t_token >= bytes_read / bandwidth`. A hard floor no kernel can beat.\n",
"> - Arithmetic intensity = FLOPs / bytes; the hardware ridge = peak FLOP/s / peak bandwidth (~295 for H100).\n",
"> - A dense matmul's intensity scales with batch size; a single token sits near intensity 1, far below the ridge.\n",
"> - Prefill (many tokens, one weight read) is compute-bound; decode (one token, full weight read) is memory-bound. Same model, opposite regime.\n"
]
},
{
"cell_type": "markdown",
"id": "114381d7",
"metadata": {},
"source": [
"## Part 2 — A tiny GPT to make fast\n",
"\n",
"We need a concrete model to optimize. It is the same decoder-only transformer as Ch 15: token + position embeddings, a stack of pre-norm blocks (causal self-attention then an MLP), a final norm, and a tied-weight head. We build the no-cache version first so the cache in Part 3 has a ground-truth baseline to match exactly.\n",
"\n",
"We train it on a tiny synthetic copy task: the model must echo a short random prefix. The data is generated with a known rule, so \"did it learn\" is checkable, and it is small enough to fit in seconds on CPU.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "83cf0e4a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.879164Z",
"iopub.status.busy": "2026-06-10T19:24:49.879088Z",
"iopub.status.idle": "2026-06-10T19:24:49.888298Z",
"shell.execute_reply": "2026-06-10T19:24:49.887773Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"one sequence (note the repeated prefix): [ 1 20 16 13 7 1 20 16 13]\n",
"x shape (4, 9) y shape (4, 9)\n"
]
}
],
"source": [
"# Synthetic anchor dataset: a copy/echo task with known structure.\n",
"# Vocab: tokens 2..V are \"content\"; 0 is PAD, 1 is a SEP marker.\n",
"# A sample is [SEP, a, b, c, SEP, a, b, c] -> the model learns to copy the prefix after the second SEP.\n",
"VOCAB = 24\n",
"SEP, PAD = 1, 0\n",
"PREFIX = 4 # length of the chunk to copy\n",
"BLOCK = 1 + PREFIX + 1 + PREFIX # full sequence length = 10\n",
"data_rng = np.random.default_rng(SEED)\n",
"\n",
"def make_batch(n):\n",
" chunks = data_rng.integers(2, VOCAB, size=(n, PREFIX))\n",
" seq = np.concatenate([np.full((n, 1), SEP), chunks,\n",
" np.full((n, 1), SEP), chunks], axis=1) # (n, BLOCK)\n",
" x = torch.tensor(seq[:, :-1], dtype=torch.long) # inputs (n, BLOCK-1)\n",
" y = torch.tensor(seq[:, 1:], dtype=torch.long) # targets (n, BLOCK-1), next-token\n",
" return x, y\n",
"\n",
"xb, yb = make_batch(4)\n",
"print(\"one sequence (note the repeated prefix):\", np.concatenate([xb[0, :1].numpy(), xb[0].numpy()[1:]]))\n",
"print(\"x shape\", tuple(xb.shape), \"y shape\", tuple(yb.shape))"
]
},
{
"cell_type": "markdown",
"id": "3868c4bc",
"metadata": {},
"source": [
"> **Interpretation.** The first half is `[SEP, a, b, c, d]`; after the second `SEP` the target is the same `a, b, c, d`. A model that solves this has to attend back to the prefix, which is exactly the attention behavior the KV cache will accelerate.\n"
]
},
{
"cell_type": "markdown",
"id": "ed6fcdef",
"metadata": {},
"source": [
"### Exercise 17.A — The causal mask, isolated\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"Before the attention class uses it, build the mask on its own. A causal mask stops position `q` from attending to any key position `k > q` (the future). We apply it by setting those score entries to `-inf` *before* the softmax, so they become exactly 0 probability.\n",
"\n",
"Implement `causal_fill(scores)` where `scores` has shape `(T, T)`: return a copy with every strictly-upper-triangular entry (k > q) replaced by `-inf`. The check verifies the masked positions, that the diagonal survives (a token may attend to itself), and that softmax of the result has zeros exactly where the mask was.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "64514dfd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.890856Z",
"iopub.status.busy": "2026-06-10T19:24:49.890511Z",
"iopub.status.idle": "2026-06-10T19:24:49.895834Z",
"shell.execute_reply": "2026-06-10T19:24:49.895385Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 17.A causal mask: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def causal_fill(scores):\n",
" \"\"\"scores: (T, T). Return scores with positions k > q set to -inf (no peeking at the future).\"\"\"\n",
" T = scores.shape[-1]\n",
" # TODO 1: build a boolean (T, T) mask that is True where column index k > row index q\n",
" # (hint: torch.triu(ones, diagonal=1) is exactly the strict upper triangle)\n",
" mask = None\n",
" # TODO 2: return scores.masked_fill(mask, float(\"-inf\"))\n",
" out = None\n",
" attempted(mask, out)\n",
" return out\n",
"\n",
"def _causal():\n",
" s = torch.zeros(3, 3)\n",
" m = causal_fill(s)\n",
" # row 0 may see only key 0; row 1 keys 0,1; row 2 all. Upper triangle is -inf.\n",
" assert torch.isinf(m[0, 1]) and torch.isinf(m[0, 2]) and torch.isinf(m[1, 2]), \\\n",
" \"positions k>q must be -inf (the future is masked)\"\n",
" assert not torch.isinf(m[0, 0]) and not torch.isinf(m[1, 1]) and not torch.isinf(m[2, 0]), \\\n",
" \"the diagonal and the past must survive (a token attends to itself and earlier tokens)\"\n",
" probs = torch.softmax(m, dim=-1)\n",
" assert torch.allclose(probs[0, 1:], torch.zeros(2)), \\\n",
" \"after softmax the masked (future) positions must be exactly 0 probability\"\n",
" assert torch.allclose(probs.sum(-1), torch.ones(3)), \"each row of probabilities must still sum to 1\"\n",
"\n",
"check(\"17.A causal mask\", _causal)"
]
},
{
"cell_type": "markdown",
"id": "e2532e4e",
"metadata": {},
"source": [
"Hint 1 (conceptual)`torch.triu(M, diagonal=1)` keeps the strictly-upper triangle (everything above the main diagonal). Build it on a `torch.ones(T, T, dtype=torch.bool)` and you have the \"future\" mask directly. `diagonal=1` excludes the diagonal, so a token can still see itself.\n",
"\n",
"Hint 2 (the lines)\n",
"\n",
"```python\n",
"mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)\n",
"out = scores.masked_fill(mask, float(\"-inf\"))\n",
"```\n",
"\n",
"\n",
"Help — \"the diagonal got masked too\"You used `diagonal=0` (or no diagonal arg). `triu` with `diagonal=0` includes the main diagonal, which would forbid a token from attending to itself. Use `diagonal=1` so only strictly-future positions (k > q) are masked.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "6bea7222",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.897055Z",
"iopub.status.busy": "2026-06-10T19:24:49.896956Z",
"iopub.status.idle": "2026-06-10T19:24:49.902452Z",
"shell.execute_reply": "2026-06-10T19:24:49.902171Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 17.A causal mask\n",
"causal mask: future positions -> -inf -> 0 probability after softmax, rows still sum to 1.\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines causal_fill; the check below re-verifies the reference.\n",
"def causal_fill(scores):\n",
" T = scores.shape[-1]\n",
" mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)\n",
" return scores.masked_fill(mask, float(\"-inf\"))\n",
"\n",
"check(\"17.A causal mask\", _causal, required=True)\n",
"print(\"causal mask: future positions -> -inf -> 0 probability after softmax, rows still sum to 1.\")"
]
},
{
"cell_type": "markdown",
"id": "6a675a96",
"metadata": {},
"source": [
"> **Interpretation.** This is the exact line both `CausalSelfAttention` (below) and `CachedAttention` (Part 3) use, only generalized to non-square score matrices when the cache makes the query and key lengths differ. Isolating it here means that when the cache misbehaves, you already know the mask is not the culprit.\n"
]
},
{
"cell_type": "markdown",
"id": "ebfd2387",
"metadata": {},
"source": [
"### The model, no cache (the Ch 15 baseline)\n",
"\n",
"One attention class, one block, one GPT. This is the recompute baseline: every forward pass re-derives K and V for the entire context. Per-line shape comments on every reshape, because that is where attention bugs hide.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "b73b90d1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.903546Z",
"iopub.status.busy": "2026-06-10T19:24:49.903470Z",
"iopub.status.idle": "2026-06-10T19:24:49.925207Z",
"shell.execute_reply": "2026-06-10T19:24:49.909190Z"
}
},
"outputs": [],
"source": [
"class CausalSelfAttention(nn.Module):\n",
" def __init__(self, d_model, n_heads):\n",
" super().__init__()\n",
" assert d_model % n_heads == 0, \"d_model must divide evenly into n_heads\"\n",
" self.n_heads, self.d_k = n_heads, d_model // n_heads\n",
" self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)\n",
" self.proj = nn.Linear(d_model, d_model, bias=False)\n",
"\n",
" def forward(self, x):\n",
" B, T, C = x.shape # (B, T, d_model)\n",
" q, k, v = self.qkv(x).split(C, dim=-1) # each (B, T, d_model)\n",
" q = q.view(B, T, self.n_heads, self.d_k).transpose(1, 2) # (B, n_heads, T, d_k)\n",
" k = k.view(B, T, self.n_heads, self.d_k).transpose(1, 2) # (B, n_heads, T, d_k)\n",
" v = v.view(B, T, self.n_heads, self.d_k).transpose(1, 2) # (B, n_heads, T, d_k)\n",
" scores = q @ k.transpose(-2, -1) / math.sqrt(self.d_k) # (B, n_heads, T, T)\n",
" mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)\n",
" scores = scores.masked_fill(mask, float(\"-inf\")) # causal: no peeking ahead\n",
" attn = F.softmax(scores, dim=-1)\n",
" out = (attn @ v).transpose(1, 2).contiguous().view(B, T, C) # (B, T, d_model)\n",
" return self.proj(out)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "d52d1024",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.937582Z",
"iopub.status.busy": "2026-06-10T19:24:49.937445Z",
"iopub.status.idle": "2026-06-10T19:24:49.964253Z",
"shell.execute_reply": "2026-06-10T19:24:49.954095Z"
}
},
"outputs": [],
"source": [
"class Block(nn.Module):\n",
" def __init__(self, d_model, n_heads):\n",
" super().__init__()\n",
" self.ln1, self.ln2 = nn.LayerNorm(d_model), nn.LayerNorm(d_model)\n",
" self.attn = CausalSelfAttention(d_model, n_heads)\n",
" self.mlp = nn.Sequential(nn.Linear(d_model, 4 * d_model), nn.GELU(),\n",
" nn.Linear(4 * d_model, d_model))\n",
"\n",
" def forward(self, x):\n",
" x = x + self.attn(self.ln1(x)) # pre-norm residual\n",
" x = x + self.mlp(self.ln2(x))\n",
" return x\n",
"\n",
"class TinyGPT(nn.Module):\n",
" def __init__(self, vocab=VOCAB, d_model=64, n_heads=4, n_layers=3, max_len=128):\n",
" super().__init__()\n",
" self.max_len = max_len\n",
" self.tok = nn.Embedding(vocab, d_model)\n",
" self.pos = nn.Embedding(max_len, d_model)\n",
" self.blocks = nn.ModuleList([Block(d_model, n_heads) for _ in range(n_layers)])\n",
" self.ln_f = nn.LayerNorm(d_model)\n",
" self.head = nn.Linear(d_model, vocab, bias=False)\n",
" self.head.weight = self.tok.weight # weight tying (Ch 15): head and embedding share storage\n",
"\n",
" def forward(self, idx):\n",
" B, T = idx.shape\n",
" pos = torch.arange(T, device=idx.device)\n",
" x = self.tok(idx) + self.pos(pos) # (B, T, d_model)\n",
" for blk in self.blocks:\n",
" x = blk(x)\n",
" return self.head(self.ln_f(x)) # (B, T, vocab)"
]
},
{
"cell_type": "markdown",
"id": "b1ca545c",
"metadata": {},
"source": [
"> **Predict:** weight tying shares the embedding and head storage. We can prove it with pointer identity. What should `head.weight.data_ptr() == tok.weight.data_ptr()` return? Answer`True`. The two refer to the same underlying tensor, so they have the same data pointer. Tying halves the parameter count of those two layers and is standard in small LMs.\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "0a0730c2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:49.972238Z",
"iopub.status.busy": "2026-06-10T19:24:49.972111Z",
"iopub.status.idle": "2026-06-10T19:24:50.010699Z",
"shell.execute_reply": "2026-06-10T19:24:50.010332Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"forward ok · logits (8, 9, 24) · params 159,040\n"
]
}
],
"source": [
"# Shape smoke test BEFORE training (lucidrains discipline): a forward pass must produce (B, T, vocab).\n",
"torch.manual_seed(SEED)\n",
"model = TinyGPT()\n",
"xb, yb = make_batch(8)\n",
"logits = model(xb)\n",
"check_shape(logits, (8, BLOCK - 1, VOCAB))\n",
"assert model.head.weight.data_ptr() == model.tok.weight.data_ptr(), \\\n",
" \"weight tying broken: head and embedding should share storage (same data_ptr)\"\n",
"print(f\"forward ok · logits {tuple(logits.shape)} · params {param_count(model):,}\")"
]
},
{
"cell_type": "markdown",
"id": "c5cdc076",
"metadata": {},
"source": [
"> **Interpretation.** The forward pass is shape-correct and the head is tied to the embedding before we spend a single training step. Catching shape and tying bugs here, on random weights, is far cheaper than discovering them after a training run produces garbage.\n"
]
},
{
"cell_type": "markdown",
"id": "43da1593",
"metadata": {},
"source": [
"### Fit it briefly\n",
"\n",
"A short Adam loop on the copy task. The four-comment skeleton (forward / backward / update / track) is identical in every notebook in this series. We log the loss so you have an expected value: if yours is wildly different, something is wrong.\n",
"\n",
"> **Runtime:** ~10-40 s on CPU depending on `FAST`.\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "3b587420",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:24:50.018449Z",
"iopub.status.busy": "2026-06-10T19:24:50.018331Z",
"iopub.status.idle": "2026-06-10T19:26:28.427197Z",
"shell.execute_reply": "2026-06-10T19:26:28.426900Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"step 0 loss 43.918\n",
"step 50 loss 2.838\n",
"step 100 loss 2.817\n",
"step 150 loss 2.815\n",
"step 200 loss 2.761\n",
"step 250 loss 2.625\n",
"step 300 loss 1.450\n",
"step 350 loss 1.416\n",
"step 399 loss 1.385\n"
]
}
],
"source": [
"torch.manual_seed(SEED)\n",
"model = TinyGPT()\n",
"opt = torch.optim.Adam(model.parameters(), lr=3e-3)\n",
"model.train()\n",
"loss_log = []\n",
"for step in range(TRAIN_STEPS):\n",
" xb, yb = make_batch(64)\n",
" logits = model(xb) # forward\n",
" loss = F.cross_entropy(logits.reshape(-1, VOCAB), yb.reshape(-1))\n",
" opt.zero_grad(set_to_none=True) # backward\n",
" loss.backward()\n",
" opt.step() # update\n",
" if step % max(1, TRAIN_STEPS // 8) == 0 or step == TRAIN_STEPS - 1:\n",
" loss_log.append((step, loss.item())) # track stats\n",
"for step, lv in loss_log:\n",
" print(f\"step {step:4d} loss {lv:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "decd53fa",
"metadata": {},
"source": [
"> **Note:** the copy task is easy, so the loss falls fast. Expected final loss: roughly 0.1-0.5 at full settings, a little higher under `FAST` with fewer steps. We do not assert an exact threshold (seed-fragile across BLAS); the qualitative check below is the real test.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "a26b14b8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:26:28.429034Z",
"iopub.status.busy": "2026-06-10T19:26:28.428866Z",
"iopub.status.idle": "2026-06-10T19:26:28.781650Z",
"shell.execute_reply": "2026-06-10T19:26:28.781022Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"prompt : [1, 9, 11, 14, 10, 1]\n",
"cont. : [9, 11, 14, 10] (should echo the 4 content tokens of the prompt)\n"
]
}
],
"source": [
"# Qualitative payoff: does greedy decoding actually copy the prefix? (the emotional payoff, Karpathy-style)\n",
"model.eval()\n",
"@torch.no_grad()\n",
"def greedy_baseline(idx, n_new):\n",
" # Recompute generation: re-run the whole growing context every step. The slow, correct baseline.\n",
" # No context windowing: positions stay absolute (0..T-1), so this matches the cached generator\n",
" # token-for-token. (A real deployment would cap context; here the point is the cache equivalence.)\n",
" for _ in range(n_new):\n",
" logits = model(idx)\n",
" nxt = logits[:, -1, :].argmax(-1, keepdim=True)\n",
" idx = torch.cat([idx, nxt], dim=1)\n",
" return idx\n",
"\n",
"probe = make_batch(1)[0][:, :PREFIX + 2] # [SEP, a, b, c, d, SEP] -> model should now emit a,b,c,d\n",
"out = greedy_baseline(probe, PREFIX)\n",
"print(\"prompt :\", probe[0].tolist())\n",
"print(\"cont. :\", out[0, probe.shape[1]:].tolist(), \" (should echo the 4 content tokens of the prompt)\")"
]
},
{
"cell_type": "markdown",
"id": "45bf44ac",
"metadata": {},
"source": [
"> **Interpretation.** The continuation echoes the prefix, so the model learned the task and its attention genuinely looks back. This `greedy_baseline` is the recompute reference: every later optimization must reproduce its output token-for-token.\n",
"\n",
"> **Key takeaways**\n",
"> - The model is the Ch 15 decoder-only transformer: tied head, pre-norm blocks, causal attention.\n",
"> - A randn shape smoke test and a `data_ptr` tying check run before training, on purpose.\n",
"> - `greedy_baseline` recomputes the full context every step. It is correct but wasteful, and it is the oracle the KV cache must match.\n"
]
},
{
"cell_type": "markdown",
"id": "d9b016de",
"metadata": {},
"source": [
"## Part 3 — The KV cache: the most important inference optimization\n",
"\n",
"`greedy_baseline` re-derives K and V for every past token at every step. But those K and V depend only on past tokens' embeddings, which never change. So compute each position's K and V *once*, store them, and at the next step only compute K and V for the single new token, append, and attend against the stored cache.\n",
"\n",
"There are two phases:\n",
"- **Prefill**: run the whole prompt at once, filling the cache for positions `0..T-1`. Needs a causal mask.\n",
"- **Decode**: feed one new token, compute its K and V, append, attend against everything. The new query is the most recent position, so it is allowed to see all keys: **no mask needed.** This is the \"last-position trick.\"\n",
"\n",
"We implement a cached attention layer, then prove two things: it returns the *same tokens* as the baseline, and it is *faster*.\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "25fc61d9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:26:28.782931Z",
"iopub.status.busy": "2026-06-10T19:26:28.782565Z",
"iopub.status.idle": "2026-06-10T19:26:28.814197Z",
"shell.execute_reply": "2026-06-10T19:26:28.813753Z"
}
},
"outputs": [],
"source": [
"class CachedAttention(nn.Module):\n",
" # Causal self-attention with a pre-allocated KV cache (production-style: write into a slice).\n",
" def __init__(self, d_model, n_heads, max_len):\n",
" super().__init__()\n",
" self.n_heads, self.d_k, self.max_len = n_heads, d_model // n_heads, max_len\n",
" self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)\n",
" self.proj = nn.Linear(d_model, d_model, bias=False)\n",
" # Buffers move with the model (.to(device)) but carry no gradient. Pre-allocated, not torch.cat'd.\n",
" self.register_buffer(\"k_cache\", torch.zeros(1, n_heads, max_len, self.d_k))\n",
" self.register_buffer(\"v_cache\", torch.zeros(1, n_heads, max_len, self.d_k))\n",
" self.register_buffer(\"cache_len\", torch.zeros(1, dtype=torch.long))\n",
"\n",
" def reset_cache(self):\n",
" self.k_cache.zero_(); self.v_cache.zero_(); self.cache_len.zero_()\n",
"\n",
" def forward(self, x, use_cache=False):\n",
" B, T_new, C = x.shape\n",
" q, k_new, v_new = self.qkv(x).split(C, dim=-1)\n",
" q = q.view(B, T_new, self.n_heads, self.d_k).transpose(1, 2) # (B, h, T_new, d_k)\n",
" k_new = k_new.view(B, T_new, self.n_heads, self.d_k).transpose(1, 2)\n",
" v_new = v_new.view(B, T_new, self.n_heads, self.d_k).transpose(1, 2)\n",
" if use_cache:\n",
" cl = int(self.cache_len.item())\n",
" assert cl + T_new <= self.max_len, \"cache overflow: increase max_len\"\n",
" self.k_cache[:, :, cl:cl + T_new] = k_new # write into the slice (no realloc)\n",
" self.v_cache[:, :, cl:cl + T_new] = v_new\n",
" k = self.k_cache[:, :, :cl + T_new] # active range = all stored keys\n",
" v = self.v_cache[:, :, :cl + T_new]\n",
" self.cache_len += T_new\n",
" q_offset = cl # absolute position of the first new query\n",
" else:\n",
" k, v, q_offset = k_new, v_new, 0\n",
" scores = q @ k.transpose(-2, -1) / math.sqrt(self.d_k) # (B, h, T_new, T_total)\n",
" if T_new > 1: # prefill: mask; decode (T_new==1): no mask\n",
" T_total = k.size(-2)\n",
" q_pos = torch.arange(T_new) + q_offset # absolute query positions\n",
" k_pos = torch.arange(T_total) # absolute key positions\n",
" scores = scores.masked_fill(k_pos[None, :] > q_pos[:, None], float(\"-inf\"))\n",
" attn = F.softmax(scores, dim=-1)\n",
" out = (attn @ v).transpose(1, 2).contiguous().view(B, T_new, C)\n",
" return self.proj(out)"
]
},
{
"cell_type": "markdown",
"id": "de522747",
"metadata": {},
"source": [
"> **Stop and think:** why is no mask needed when `T_new == 1`? AnswerThe single new query is the most recent position. Every key in the cache is at a position less than or equal to it, so causality is automatically satisfied. The mask only matters during prefill, when you process many queries at once and an early query must not see later keys.\n",
"\n",
"> **Common confusion:** the cached cache writes into `self.k_cache[:, :, cl:cl+T_new]`, an in-place slice assignment, not a `torch.cat`. `torch.cat` allocates a brand-new larger tensor every step, which is what vLLM and TensorRT-LLM avoid by pre-allocating. The cost difference is exactly what Part 3's timing demo measures.\n"
]
},
{
"cell_type": "markdown",
"id": "bff4c757",
"metadata": {},
"source": [
"### Build a cached twin of the trained model\n",
"\n",
"We rebuild the model with `CachedAttention` and copy the trained weights across. Same parameters, different attention plumbing. Then generation has two modes: prefill the prompt, then decode one token at a time.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "b9e803df",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:26:28.815643Z",
"iopub.status.busy": "2026-06-10T19:26:28.815546Z",
"iopub.status.idle": "2026-06-10T19:26:28.833687Z",
"shell.execute_reply": "2026-06-10T19:26:28.832891Z"
}
},
"outputs": [],
"source": [
"class CachedBlock(nn.Module):\n",
" def __init__(self, d_model, n_heads, max_len):\n",
" super().__init__()\n",
" self.ln1, self.ln2 = nn.LayerNorm(d_model), nn.LayerNorm(d_model)\n",
" self.attn = CachedAttention(d_model, n_heads, max_len)\n",
" self.mlp = nn.Sequential(nn.Linear(d_model, 4 * d_model), nn.GELU(),\n",
" nn.Linear(4 * d_model, d_model))\n",
" def forward(self, x, use_cache=False):\n",
" x = x + self.attn(self.ln1(x), use_cache=use_cache)\n",
" x = x + self.mlp(self.ln2(x))\n",
" return x\n",
"\n",
"class CachedGPT(nn.Module):\n",
" def __init__(self, vocab=VOCAB, d_model=64, n_heads=4, n_layers=3, max_len=128):\n",
" super().__init__()\n",
" self.max_len = max_len\n",
" self.tok = nn.Embedding(vocab, d_model)\n",
" self.pos = nn.Embedding(max_len, d_model)\n",
" self.blocks = nn.ModuleList([CachedBlock(d_model, n_heads, max_len) for _ in range(n_layers)])\n",
" self.ln_f = nn.LayerNorm(d_model)\n",
" self.head = nn.Linear(d_model, vocab, bias=False)\n",
" self.head.weight = self.tok.weight\n",
" def reset_caches(self):\n",
" for blk in self.blocks:\n",
" blk.attn.reset_cache()\n",
" def forward(self, idx, use_cache=False, positions=None):\n",
" B, T = idx.shape\n",
" if positions is None:\n",
" positions = torch.arange(T, device=idx.device)\n",
" x = self.tok(idx) + self.pos(positions)\n",
" for blk in self.blocks:\n",
" x = blk(x, use_cache=use_cache)\n",
" return self.head(self.ln_f(x))"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "dd5ed56f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:26:28.834684Z",
"iopub.status.busy": "2026-06-10T19:26:28.834600Z",
"iopub.status.idle": "2026-06-10T19:26:28.865475Z",
"shell.execute_reply": "2026-06-10T19:26:28.865178Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"copied 35 weight tensors from the trained TinyGPT into CachedGPT\n",
"cached params 159,040 (matches trained model: 159,040)\n"
]
}
],
"source": [
"# Copy the trained weights into the cached architecture (same names, attention differs only by qkv/proj).\n",
"torch.manual_seed(SEED)\n",
"cached = CachedGPT(max_len=128)\n",
"sd_src = model.state_dict()\n",
"sd_dst = cached.state_dict()\n",
"copied = 0\n",
"for name, tensor in sd_src.items():\n",
" if name in sd_dst and sd_dst[name].shape == tensor.shape:\n",
" sd_dst[name].copy_(tensor); copied += 1\n",
"cached.load_state_dict(sd_dst)\n",
"cached.eval()\n",
"print(f\"copied {copied} weight tensors from the trained TinyGPT into CachedGPT\")\n",
"print(f\"cached params {param_count(cached):,} (matches trained model: {param_count(model):,})\")"
]
},
{
"cell_type": "markdown",
"id": "6d66ac40",
"metadata": {},
"source": [
"### Exercise 17.2 — Cached greedy generation\n",
"`Difficulty 3/5 · ~20 min`\n",
"\n",
"Implement `generate_cached(net, idx, n_new)`. This is the heart of the chapter.\n",
"\n",
"1. Reset the caches.\n",
"2. **Prefill**: run the whole prompt with `use_cache=True` and `positions = arange(T)`. Take the logits at the *last* position to pick the first new token.\n",
"3. **Decode loop** `n_new - 1` more times: feed only the single newest token with `use_cache=True`, and pass its **absolute** position (the off-by-one that the deliberate-failure cell below gets wrong). Append each argmax token.\n",
"\n",
"The check compares against `greedy_baseline` with `torch.equal`: greedy decoding is deterministic, so cached and recompute must be bit-identical, not merely close.\n",
"\n",
"*Harder:* make it work for a batch of prompts (the cache buffers would need a batch dimension `B`).\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "da2cfa10",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:26:28.872639Z",
"iopub.status.busy": "2026-06-10T19:26:28.872527Z",
"iopub.status.idle": "2026-06-10T19:26:30.147649Z",
"shell.execute_reply": "2026-06-10T19:26:30.147225Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 17.2 cached == recompute: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"@torch.no_grad()\n",
"def generate_cached(net, idx, n_new):\n",
" net.reset_caches()\n",
" B, T = idx.shape\n",
" # TODO 1: prefill. logits = net(idx, use_cache=True, positions=arange(T)); take last-position argmax.\n",
" logits = None\n",
" nxt = None\n",
" attempted(logits, nxt)\n",
" idx = torch.cat([idx, nxt], dim=1)\n",
" for _ in range(n_new - 1):\n",
" # TODO 2: the new token's ABSOLUTE position is (current length - 1). Build a 1-element long tensor.\n",
" pos = None\n",
" # TODO 3: feed ONLY the last token: net(idx[:, -1:], use_cache=True, positions=pos)\n",
" step_logits = None\n",
" attempted(pos, step_logits)\n",
" nxt = step_logits[:, -1, :].argmax(-1, keepdim=True)\n",
" idx = torch.cat([idx, nxt], dim=1)\n",
" return idx\n",
"\n",
"def _cached_matches_baseline():\n",
" prompt = make_batch(1)[0][:, :PREFIX + 2] # a fresh prompt\n",
" ref = greedy_baseline(prompt.clone(), GEN_TOKENS)\n",
" got = generate_cached(cached, prompt.clone(), GEN_TOKENS)\n",
" assert torch.equal(got, ref), (\n",
" \"cached generation must be token-identical to recompute under greedy decoding.\\n\"\n",
" f\"baseline: {ref[0].tolist()}\\ncached: {got[0].tolist()}\\n\"\n",
" \"If they diverge after the first token, your decode positions are wrong (see the deliberate-failure cell).\")\n",
"\n",
"check(\"17.2 cached == recompute\", _cached_matches_baseline)"
]
},
{
"cell_type": "markdown",
"id": "0df95caf",
"metadata": {},
"source": [
"Hint 1 (conceptual)Prefill is one call on the whole prompt. After it, `cache_len` is `T` in every layer. Each decode call feeds a `(B, 1)` tensor (just the last token) and must tell the model what absolute position that token sits at, so the position embedding `pos(positions)` is correct.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"logits = net(idx, use_cache=True, positions=torch.arange(T))\n",
"nxt = logits[:, -1, :].argmax(-1, keepdim=True)\n",
"idx = torch.cat([idx, nxt], dim=1)\n",
"for _ in range(n_new - 1):\n",
" pos = torch.tensor([idx.shape[1] - 1]) # absolute position of the newest token\n",
" step_logits = net(idx[:, -1:], use_cache=True, positions=pos)\n",
" nxt = step_logits[:, -1, :].argmax(-1, keepdim=True)\n",
" idx = torch.cat([idx, nxt], dim=1)\n",
"```\n",
"\n",
"\n",
"Help — \"diverges from the baseline after the first token\"The usual cause is the position. If you pass `positions=torch.tensor([0])` on every decode step, the model thinks every new token is at position 0 and the position embedding is wrong, so the logits drift. Print `idx.shape[1] - 1` each step; it should climb. The cache itself can be correct while the *position* fed to `pos_emb` is wrong, which is the subtle version of this bug.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "6253fe67",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:26:30.148663Z",
"iopub.status.busy": "2026-06-10T19:26:30.148579Z",
"iopub.status.idle": "2026-06-10T19:26:32.784461Z",
"shell.execute_reply": "2026-06-10T19:26:32.784178Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 17.2 cached == recompute\n",
"cached generation reproduces the recompute baseline exactly.\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines generate_cached; the check below re-verifies the reference.\n",
"@torch.no_grad()\n",
"def generate_cached(net, idx, n_new):\n",
" net.reset_caches()\n",
" B, T = idx.shape\n",
" logits = net(idx, use_cache=True, positions=torch.arange(T, device=idx.device))\n",
" nxt = logits[:, -1, :].argmax(-1, keepdim=True)\n",
" idx = torch.cat([idx, nxt], dim=1)\n",
" for _ in range(n_new - 1):\n",
" pos = torch.tensor([idx.shape[1] - 1], device=idx.device) # absolute position\n",
" step_logits = net(idx[:, -1:], use_cache=True, positions=pos)\n",
" nxt = step_logits[:, -1, :].argmax(-1, keepdim=True)\n",
" idx = torch.cat([idx, nxt], dim=1)\n",
" return idx\n",
"\n",
"check(\"17.2 cached == recompute\", _cached_matches_baseline, required=True)\n",
"print(\"cached generation reproduces the recompute baseline exactly.\")"
]
},
{
"cell_type": "markdown",
"id": "8de31f5b",
"metadata": {},
"source": [
"> **Interpretation.** `torch.equal` (not `allclose`) passes: the cache changes *how much you compute*, never *what you compute*. This is the property that lets serving systems ship it without changing model outputs.\n"
]
},
{
"cell_type": "markdown",
"id": "568a4b5c",
"metadata": {},
"source": [
"### A deliberate failure: the off-by-cache-length position bug\n",
"\n",
"The single most common KV-cache bug: during decode, feeding the wrong position to the position embedding. The cache holds the right keys, but if you tell the model the new token is at position 0 (instead of its true absolute position), the position embedding is wrong and the output drifts. We write the broken version, watch it disagree with the baseline, then point at the one-line fix.\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "4e3f7277",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:26:32.789259Z",
"iopub.status.busy": "2026-06-10T19:26:32.785641Z",
"iopub.status.idle": "2026-06-10T19:26:34.207280Z",
"shell.execute_reply": "2026-06-10T19:26:34.206900Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"baseline: [1, 12, 17, 14, 20, 1, 12, 17, 14, 20, 20, 20, 21, 7, 21, 7, 21, 14, 7, 21, 7, 14, 21, 18, 18, 1, 3, 3, 3, 3, 3, 3, 3, 2, 14, 18, 18, 21, 7, 11, 14, 7, 21, 7, 21, 18, 3, 17, 14, 18, 11, 7, 4, 14, 7, 11, 14, 7, 8, 18, 1, 12, 18, 18, 11, 7, 22, 14, 7, 14]\n",
"broken: [1, 12, 17, 14, 20, 1, 12, 18, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 11, 7, 11, 7, 11, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17]\n",
"tokens matching: 8/70 (they diverge once decode positions go wrong)\n",
"\n",
"The fix is one line: pos = torch.tensor([idx.shape[1] - 1]) instead of torch.tensor([0]).\n"
]
}
],
"source": [
"@torch.no_grad()\n",
"def generate_cached_BROKEN(net, idx, n_new):\n",
" net.reset_caches()\n",
" B, T = idx.shape\n",
" logits = net(idx, use_cache=True, positions=torch.arange(T))\n",
" nxt = logits[:, -1, :].argmax(-1, keepdim=True)\n",
" idx = torch.cat([idx, nxt], dim=1)\n",
" for _ in range(n_new - 1):\n",
" pos = torch.tensor([0]) # BUG: always position 0, not the absolute position\n",
" step_logits = net(idx[:, -1:], use_cache=True, positions=pos)\n",
" nxt = step_logits[:, -1, :].argmax(-1, keepdim=True)\n",
" idx = torch.cat([idx, nxt], dim=1)\n",
" return idx\n",
"\n",
"prompt = make_batch(1)[0][:, :PREFIX + 2]\n",
"ref = greedy_baseline(prompt.clone(), GEN_TOKENS)\n",
"bad = generate_cached_BROKEN(cached, prompt.clone(), GEN_TOKENS)\n",
"n_agree = int((ref[0] == bad[0]).sum())\n",
"print(f\"baseline: {ref[0].tolist()}\")\n",
"print(f\"broken: {bad[0].tolist()}\")\n",
"print(f\"tokens matching: {n_agree}/{ref.shape[1]} (they diverge once decode positions go wrong)\")\n",
"assert not torch.equal(ref, bad), \"the broken version is supposed to diverge; if it matches, the model ignores positions\"\n",
"print(\"\\nThe fix is one line: pos = torch.tensor([idx.shape[1] - 1]) instead of torch.tensor([0]).\")"
]
},
{
"cell_type": "markdown",
"id": "473f6961",
"metadata": {},
"source": [
"> **Interpretation.** The broken decoder agrees with the baseline up to and including the first generated token (prefill is fine) and then drifts. The cache stored correct keys and values; the *position* fed to `pos_emb` was wrong. This is why the cached attention tracks `cache_len` and the generator passes an absolute `pos`. A passing `torch.equal` check is the only reliable guard against this class of bug, which is exactly why Exercise 17.2 uses one.\n"
]
},
{
"cell_type": "markdown",
"id": "eb4e8432",
"metadata": {},
"source": [
"### The timing verification: cached vs recompute\n",
"\n",
"Correctness is proven. Now the payoff: speed. We time both generators on a longer run and assert the cache wins. Timing on CPU is noisy, so we use the median over a few runs (the `time_call` helper from Setup) and assert the *ratio*, never an absolute millisecond count.\n",
"\n",
"> **Runtime:** ~10-30 s on CPU.\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "578c8024",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:26:34.210470Z",
"iopub.status.busy": "2026-06-10T19:26:34.210353Z",
"iopub.status.idle": "2026-06-10T19:26:43.316118Z",
"shell.execute_reply": "2026-06-10T19:26:43.315837Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"recompute : 1341.0 ms for 80 tokens\n",
"cached : 966.1 ms for 80 tokens\n",
"speedup : 1.39x\n"
]
}
],
"source": [
"prompt = make_batch(1)[0][:, :PREFIX + 2]\n",
"n_gen = TIME_GEN # long enough that recompute's O(T^2) re-processing dominates the cache's per-step overhead\n",
"t_recompute = time_call(greedy_baseline, prompt.clone(), n_gen, runs=3)\n",
"t_cached = time_call(generate_cached, cached, prompt.clone(), n_gen, runs=3)\n",
"speedup = t_recompute / t_cached\n",
"print(f\"recompute : {t_recompute*1e3:7.1f} ms for {n_gen} tokens\")\n",
"print(f\"cached : {t_cached*1e3:7.1f} ms for {n_gen} tokens\")\n",
"print(f\"speedup : {speedup:.2f}x\")\n",
"assert t_cached < t_recompute, (\n",
" \"the KV cache should be faster: recompute re-derives K,V for the whole context every step, \"\n",
" f\"the cache reuses them. Got cached={t_cached*1e3:.1f}ms vs recompute={t_recompute*1e3:.1f}ms. \"\n",
" \"Timing is noisy on CPU; if this ever flips, it is the per-step overhead winning at short length, \"\n",
" \"not a broken cache (correctness is proven separately by torch.equal). Re-run, or raise TIME_GEN.\")"
]
},
{
"cell_type": "markdown",
"id": "bfb9bbe2",
"metadata": {},
"source": [
"> **Note:** the speedup grows with the generated length. Recompute does O(T^2) attention work across the run; the cache does O(T). On our tiny CPU model with a short sequence the constant factors are large and the speedup is modest (often 1.5-4x), but the *direction* is the law. On a real model at sequence length 1000+, the cache is the difference between usable and unusable.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "71c5fffd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:26:43.321422Z",
"iopub.status.busy": "2026-06-10T19:26:43.321315Z",
"iopub.status.idle": "2026-06-10T19:27:13.870167Z",
"shell.execute_reply": "2026-06-10T19:27:13.869814Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAGGCAYAAACNCg6xAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAApN5JREFUeJzs3XdYFFcXwOHf0juIUkRErNh7CXZFATsKYo8aYzTRGE2ixiRGTdOYGGOaJjHRxK6o2MWKvbdYQVGsICLS++58f0zYzxVQUGAo932efdydvTtzZkH27J17z1VJkiQhCIIgCIIgvJCe0gEIgiAIgiCUFCJxEgRBEARByCOROAmCIAiCIOSRSJwEQRAEQRDySCROgiAIgiAIeSQSJ0EQBEEQhDwSiZMgCIIgCEIeicRJEARBEAQhj0TiJAiCIAiCkEcicRKEQhYcHIxKpSIgIEDpUEqE8PBwVCoVS5cufWHbESNG4OrqWugxFaX8nNOIESOwsLB46WN17NiRjh07vvTri8rMmTNRqVRER0crHYogiMRJEF6GSqXK0y04OFjpUIUSLjk5mZkzZ5aJ36Wvv/6awMBApcMQhOcyUDoAQSiJli1bpvP4n3/+Yffu3dm216lTh6tXrxZlaCVelSpVSElJwdDQUOlQFPHHH3+g0Wi0j5OTk5k1axZAiegdehVff/01fn5++Pj4KB2KIORKJE6C8BKGDh2q8/j48ePs3r0723ZAJE75pFKpMDExUToMxZTVhFEQSgpxqU4QiohGo+Grr77C2dkZExMTPDw8uHHjRrZ2J06cwNvbG2tra8zMzOjQoQNHjhzJ0zFu375N7969MTc3x97enkmTJhEUFJTtsuGhQ4fo378/Li4uGBsbU7lyZSZNmkRKSorO/rLG0Ny8eRMvLy/Mzc1xcnLi888/R5Kk58by/vvvU758eZ127777LiqVih9//FG77eHDh6hUKhYuXAjkPsYpMDCQ+vXrY2JiQv369dm4cWOOx9VoNPzwww/Uq1cPExMTHBwcGDNmDE+ePNFp5+rqSs+ePTl8+DAtW7bExMSEatWq8c8//+i0y8jIYNasWdSsWRMTExPKly9P27Zt2b17d67nHhsbi76+vs55RkdHo6enl+09efvtt3F0dNQ+fnqMU3h4OHZ2dgDMmjVLewl45syZOse7f/8+Pj4+WFhYYGdnx4cffoharc41vudJS0tjxowZ1KhRQ/u7MWXKFNLS0nTaqVQqxo8fr/25GBsbU69ePXbu3Jltn8HBwTRv3hwTExOqV6/Ob7/9ph239PT+kpKS+Pvvv7XnOWLECJ39xMbGMmLECGxsbLC2tmbkyJEkJye/1HkKwssSiZMgFJE5c+awceNGPvzwQ6ZNm8bx48cZMmSITpt9+/bRvn174uPjmTFjBl9//TWxsbF07tyZkydPPnf/SUlJdO7cmT179jBhwgQ++eQTjh49ytSpU7O1XbduHcnJybz99tv89NNPeHl58dNPP/H6669na6tWq/H29sbBwYG5c+fSrFkzZsyYwYwZM54bT7t27YiJieHy5cvabYcOHUJPT49Dhw7pbANo3759rvvatWsXvr6+qFQqZs+ejY+PDyNHjuT06dPZ2o4ZM4bJkyfTpk0bFixYwMiRI1mxYgVeXl5kZGTotL1x4wZ+fn507dqVefPmUa5cOUaMGKET88yZM5k1axadOnXi559/5pNPPsHFxYWzZ8/mGq+NjQ3169fn4MGD2m2HDx9GpVIRExPDlStXdM6/Xbt2Oe7Hzs5Om1D27duXZcuWsWzZMvr166dto1ar8fLyonz58nz33Xd06NCBefPm8fvvv+caX240Gg29e/fmu+++o1evXvz000/4+Pgwf/58BgwYkK394cOHeeeddxg4cCBz584lNTUVX19fHj9+rG1z7tw5vL29efz4MbNmzWLUqFF8/vnn2cYyLVu2DGNjY9q1a6c9zzFjxui08ff3JyEhgdmzZ+Pv78/SpUu1lzEFochIgiC8snHjxkm5/Xfav3+/BEh16tSR0tLStNsXLFggAdLFixclSZIkjUYj1axZU/Ly8pI0Go22XXJyslS1alWpa9euz41h3rx5EiAFBgZqt6WkpEi1a9eWAGn//v06+3zW7NmzJZVKJd2+fVu7bfjw4RIgvfvuu9ptGo1G6tGjh2RkZCQ9evQo13iioqIkQPr1118lSZKk2NhYSU9PT+rfv7/k4OCgbTdhwgTJ1tZWe863bt2SAGnJkiXaNo0bN5YqVqwoxcbGarft2rVLAqQqVapotx06dEgCpBUrVujEsnPnzmzbq1SpIgHSwYMHdWI2NjaWPvjgA+22Ro0aST169Mj1PHMzbtw4nfN8//33pfbt20v29vbSwoULJUmSpMePH0sqlUpasGCBtt3w4cN1zunRo0cSIM2YMSPbMbJ+Pp9//rnO9iZNmkjNmjV7YYwdOnSQOnTooH28bNkySU9PTzp06JBOu0WLFkmAdOTIEe02QDIyMpJu3Lih3XbhwgUJkH766Sfttl69eklmZmbS/fv3tduuX78uGRgYZPs/Y25uLg0fPjxbnDNmzJAA6Y033tDZ3rdvX6l8+fIvPE9BKEiix0kQisjIkSMxMjLSPs7qZbh58yYA58+f5/r16wwePJjHjx8THR1NdHQ0SUlJeHh4cPDgQZ1Bw8/auXMnlSpVonfv3tptJiYmjB49OltbU1NT7f2kpCSio6Np3bo1kiRx7ty5bO3Hjx+vvZ91iSY9PZ09e/bkGo+dnR21a9fW9rocOXIEfX19Jk+ezMOHD7l+/Tog97i0bdtW57LN0yIiIjh//jzDhw/H2tpau71r167UrVtXp+26deuwtrama9eu2vcvOjqaZs2aYWFhwf79+3Xa161bV6e3x87ODjc3N+3PBOTeo8uXL2vjzat27drx8OFDQkJCtOfZvn172rVrp+1lO3z4MJIk5drjlFdjx47NduynzyGv1q1bR506dahdu7bO+9e5c2eAbO9fly5dqF69uvZxw4YNsbKy0h5brVazZ88efHx8cHJy0rarUaMG3bp1y3d8OZ3n48ePiY+Pz/e+BOFlicHhglBEXFxcdB6XK1cOQDv2JuuDefjw4bnuIy4uTvu6Z92+fZvq1atnS0Bq1KiRre2dO3f47LPP2Lx5c7axP3FxcTqP9fT0qFatms62WrVqAfIYnOdp164d27dvB+TEoXnz5jRv3hxbW1sOHTqEg4MDFy5cYPDgwbnu4/bt2wDUrFkz23Nubm46l8yuX79OXFwc9vb2Oe4rKipK5/GzPxOQfy5Pvyeff/45ffr0oVatWtSvXx9vb2+GDRtGw4YNn3Pm/0+MDx06hLOzM+fOnePLL7/Ezs6O7777TvuclZUVjRo1eu6+nsfExEQ7Diq3c8ir69evc/Xq1Wz7y5Lf9y8qKoqUlJQcfwdz2vYiz/s/ZGVlle/9CcLLEImTIBQRfX39HLdL/w0UzupN+vbbb2ncuHGObV+l2GEWtVpN165diYmJYerUqdSuXRtzc3Pu37/PiBEjnturlV9t27bljz/+4ObNm9qxPCqVirZt23Lo0CGcnJzQaDSv3OOSRaPRYG9vz4oVK3J8/tmE4EU/E5DHXoWFhbFp0yZ27drF4sWLmT9/PosWLeLNN9/MNRYnJyeqVq3KwYMHcXV1RZIk3N3dsbOz47333uP27dscOnSI1q1bo6f38p3/uZ3Dy9BoNDRo0IDvv/8+x+crV66cp2NLL5g48LKK+niCkBOROAlCMZF1ycPKyoouXbrk+/VVqlThypUrSJKk0+v07My9ixcvEhoayt9//60zGDy3WWIajYabN29qe5kAQkNDAV5Y4TorIdq9ezenTp3io48+AuRkZOHChTg5OWFubk6zZs2ee15AjpfKsi6DZalevTp79uyhTZs2OpcjX5WtrS0jR45k5MiRJCYm0r59e2bOnPncxAnk8z948CBVq1alcePGWFpa0qhRI6ytrdm5cydnz5594eDm3C5hFobq1atz4cIFPDw8CuS49vb2mJiY5Dh7NKdtRXmugvCyxBgnQSgmmjVrRvXq1fnuu+9ITEzM9vyjR4+e+3ovLy/u37/P5s2btdtSU1P5448/dNplfWt/+lu6JEksWLAg133//PPPOm1//vlnDA0N8fDweG5MVatWpVKlSsyfP5+MjAzatGkDyAlFWFgYAQEBvPbaaxgY5P4drmLFijRu3Ji///5b5zLi7t27dWangTzrSq1W88UXX2TbT2ZmJrGxsc+NNydPzxADudevRo0a2abn56Rdu3aEh4ezZs0abRKpp6dH69at+f7778nIyHhhb5uZmRnAS8WeX/7+/ty/fz/b7wxASkoKSUlJ+dqfvr4+Xbp0ITAwkAcPHmi337hxgx07dmRrb25uXiTnKQivQvQ4CUIxoaenx+LFi+nWrRv16tVj5MiRVKpUifv377N//36srKzYsmVLrq8fM2YMP//8M4MGDeK9996jYsWKrFixQltMMuvbfO3atalevToffvgh9+/fx8rKivXr1+c6JsbExISdO3cyfPhwWrVqxY4dO9i2bRsff/xxrmNhntauXTtWr15NgwYNtGNSmjZtirm5OaGhoc8d35Rl9uzZ9OjRg7Zt2/LGG28QExPDTz/9RL169XSSzA4dOjBmzBhmz57N+fPn8fT0xNDQkOvXr7Nu3ToWLFiAn5/fC4/3tLp169KxY0eaNWuGra0tp0+fJiAgQGfA/PPOHeSesa+//lq7vX379uzYsQNjY2NatGjx3H2YmppSt25d1qxZQ61atbC1taV+/frUr18/X+eRF8OGDWPt2rWMHTuW/fv306ZNG9RqNdeuXWPt2rUEBQXRvHnzfO1z5syZ7Nq1izZt2vD222+jVqv5+eefqV+/PufPn9dp26xZM/bs2cP333+vvdTZqlWrAjxDQXh1osdJEIqRjh07cuzYMZo3b87PP//Mu+++y9KlS3F0dGTSpEnPfa2FhQX79u2jc+fOLFiwgC+//JJ27doxffp0AG0CZWhoyJYtW2jcuDGzZ8/WFnd8tvBjFn19fXbu3ElkZCSTJ0/m1KlTzJgxI8denZxkJQ9t27bVbjMwMMDd3V3n+efx9vZm3bp1qNVqpk2bxoYNG1iyZEmOH+KLFi3i999/Jyoqio8//php06axb98+hg4dqu3xyo8JEyYQHh7O7NmzmTBhAgcOHODLL79k3rx5L3ytm5ubdqD60+efdc4tW7bE2Nj4hftZvHgxlSpVYtKkSQwaNKjQFozW09MjMDCQOXPmcPHiRT788ENmzZrFqVOneO+993Qu1+ZVs2bN2LFjB+XKlWP69On8+eeffP7553h4eGSrEP/999/TrFkzPv30UwYNGqStYSUIxYlKEqPqBKFU++GHH5g0aRL37t2jUqVK+XrtiBEjCAgIyPHSoSC8Ch8fn5cq8yAIShM9ToJQijy7ZEpqaiq//fYbNWvWzHfSJAgF5dnfy+vXr7N9+/ZSv2ixUDqJMU6CUIr069cPFxcXGjduTFxcHMuXL+fatWu5Ts8XhKJQrVo1RowYQbVq1bh9+zYLFy7EyMiIKVOmKB2aIOSbSJwEoRTx8vJi8eLFrFixArVaTd26dVm9enWO64wJQlHx9vZm1apVREZGYmxsjLu7O19//XWORU0FobgTY5wEQRAEQRDySIxxEgRBEARByCOROAmCIAiCIOSRGOOUBxqNhgcPHmBpaSmWBBAEQRCEYkySJBISEnBycnqldSBzIxKnPHjw4EG2xS0FQRAEQSi+7t69i7Ozc4HvVyROeWBpaQnIPwQrKyuFoxEEQRAEITfx8fFUrlxZ+9ld0ETilAdZl+esrKxE4iQIgiAIJUBhDa0Rg8MFQRAEQRDySCROgiAIgiAIeSQSJ0EQBEEQhDwSY5wEQSiR1Go1GRkZSochCEIRMzQ0RF9fX7Hji8RJEIQSRZIkIiMjiY2NVToUQRAUYmNjg6OjoyK1FUXiJAhCiZKVNNnb22NmZiaK0gpCGSJJEsnJyURFRQFQsWLFIo9BJE5CqabRaLh37x6JiYlYWFjg7OxcKJVkhaKhVqu1SVP58uWVDkcQBAWYmpoCEBUVhb29fZFfthOJk1BqhYSEsHfvXhISErTbLC0t8fDwwM3NTcHIhJeVNabJzMxM4UgEQVBS1t+AjIyMIk+cxFdvoVQKCQkhMDBQJ2kCSEhIIDAwkJCQEIUiEwqCuDwnCGWbkn8DROIklDoajYa9e/c+t83evXvRaDRFFJEgCIJQWojESSh17t27l62n6VkJCQncu3eviCISBOFV/fnnn3h6er706x8/foy9vT3h4eEv9fqNGzdiYGBArVq1tAOT82vnzp00bty41HxpCw4ORqVSFegM1+joaOzt7Yv132eROAmlTmJiYoG2E0ofjUbDnTt3uHLlCnfu3Ck1H2TFycyZM2ncuHGB7Cs1NZXp06czY8YMne0xMTFMnDiRKlWqYGRkhJOTE2+88QZ37tzJto+vvvqKPn364OrqysyZM1GpVM+9PW3//v0MHjyYmTNnYm9vj7e3N/Hx8TptwsPDGTVqFFWrVsXU1JTq1aszY8YM0tPTtW28vb0xNDRkxYoVr/yeqFQqAgMDdbYV5Hv+rI4dOzJx4kSdba1btyYiIgJra+sCO06FChV4/fXXs/2sixOROAmljoWFRYG2E0qXkJAQFi1axKpVq9iyZQurVq1i0aJFRTru7ekPU+HFAgICsLKyok2bNtptMTExvPbaa+zZs4dFixZx48YNVq9ezY0bN2jRogU3b97Utk1OTubPP/9k1KhRAHz44YdERERob87Oznz++ec627KcOXOGvn37Mn/+fD799FOCgoKwtbWlT58+pKWladtdu3YNjUbDb7/9xuXLl5k/fz6LFi3i448/1jmXESNG8OOPPxbWW1WkjIyMCqWW0siRI1mxYgUxMTEFut+CIhInodRxdnbG0tLyuW2yShMIZYtSkwY6duzI+PHjmThxIhUqVMDLywuAS5cu0a1bNywsLHBwcGDYsGFER0drX6fRaJg7dy41atTA2NgYFxcXvvrqK+3zFy9epHPnzpiamlK+fHneeustnZ7UESNG4OPjw9dff42DgwM2NjZ8/vnnZGZmMnnyZGxtbXF2dmbJkiXa14SHh6NSqVi9ejWtW7fGxMSE+vXrc+DAAW2bpUuXYmNjo3OOgYGB2g/QpUuXMmvWLC5cuKDtwVm6dCkAsbGxvPnmm9jZ2WFlZUXnzp25cOHCc9+/1atX06tXL51tn3zyCQ8ePGDPnj1069YNFxcX2rdvT1BQEIaGhowbN07bdvv27RgbG/Paa68B8v9/R0dH7U1fXx9LS0udbSD/vvTs2ZOffvqJsWPHAmBubs62bduwtLRk0KBBqNVqQO5NWrJkCZ6enlSrVo3evXvz4YcfsmHDBp24e/XqxenTpwkLC8v1fE+dOkXXrl2pUKEC1tbWdOjQgbNnz2qfd3V1BaBv376oVCpcXV1f6T3P6qlatmwZrq6uWFtbM3DgQO3/kxEjRnDgwAEWLFig3Xd4eHiOl+rWr19PvXr1MDY2xtXVlXnz5umcm6urK19//TVvvPEGlpaWuLi48Pvvv+u0qVevHk5OTmzcuDHX90hJInESSh09PT3atm373DYmJibi8kwpIEkS6enpebqlpqayZ8+e5+5vz549pKam5ml/kiTlK9a///4bIyMjjhw5wqJFi4iNjaVz5840adKE06dPs3PnTh4+fIi/v7/2NdOmTWPOnDlMnz6dK1eusHLlShwcHABISkrCy8uLcuXKcerUKdatW8eePXsYP368znH37dvHgwcPOHjwIN9//z0zZsygZ8+elCtXjhMnTjB27FjGjBmTbUzJ5MmT+eCDDzh37hzu7u706tWLx48f5+lcBwwYwAcffEC9evW0PTgDBgwAoH///kRFRbFjxw7OnDlD06ZN8fDweG7vwuHDh2nevLn2sUajYfXq1QwZMkSb5GQxNTXlnXfeISgoSLvPQ4cO0axZszzF/jQ3NzciIiIYNmyYznZjY2M2b97Mhg0bnjsVPi4uDltbW51tLi4uODg4cOjQoVxfl5CQwPDhwzl8+DDHjx+nZs2adO/eXZvInDp1CoAlS5YQERHBqVOnXvk9DwsLIzAwkK1bt7J161YOHDjAnDlzAFiwYAHu7u6MHj1au+/KlStni/vMmTP4+/szcOBALl68yMyZM5k+fbo2gcsyb948mjdvzrlz53jnnXd4++23s31padmy5XPfIyWJOk5CqZQ1AFRPT08nQTIzMyM9PZ3o6Gi2bNlCnz59REHMEiwjI4P58+cX2P4SExNZsGBBntpOmjQJIyOjPO+7Zs2azJ07V/v4yy+/pEmTJnz99dfabX/99ReVK1cmNDSUihUrsmDBAn7++WeGDx8OQPXq1bVfClauXElqair//PMP5ubmAPz888/06tWLb775Rptg2dra8uOPP6Knp4ebmxtz584lOTlZewkpKzk7fPgwAwcO1MYyfvx4fH19AVi4cCE7d+7kzz//ZMqUKS88V1NTUywsLDAwMNBJbA4fPszJkyeJiorC2NgYgO+++47AwEACAgJ46623su0rNjaWuLg4nJyctNsePXpEbGwsderUyfH4derUQZIkbty4QcuWLbl9+7bO64vCjRs3+Omnn/juu++yPefk5MTt27dzfW3nzp11Hv/+++/Y2Nhw4MABevbsiZ2dHfD/ZUeyvMp7rtFoWLp0qba3ftiwYezdu5evvvoKa2trjIyMMDMzy5aoPu3777/Hw8OD6dOnA1CrVi2uXLnCt99+y4gRI7TtunfvzjvvvAPA1KlTmT9/Pvv379epr+fk5MS5c+dyPZaSROIklDphYWFcvXoVlUrFkCFDyMzM1KkcfvfuXdatW0doaCi7du3Cy8tL1AUSCt2zPR4XLlxg//79OY61CwsLIzY2lrS0NDw8PHLc39WrV2nUqJE2aQJo06YNGo2GkJAQbeJUr149nS8HDg4O1K9fX/tYX1+f8uXLZ5sp5u7urr1vYGBA8+bNuXr1aj7OOLsLFy6QmJiYrep7SkpKrpeuUlJSALmX+Fl57fVLSUnJ8fWF5f79+3h7e9O/f39Gjx6d7XlTU1OSk5Nzff3Dhw/59NNPCQ4OJioqCrVaTXJyco6D3l8kr++5q6urzhCHihUr5nv24NWrV+nTp4/OtjZt2vDDDz+gVqu1vXMNGzbUPq9SqXB0dMx2rBe9R0oSiZNQqqSnp7Nr1y4AmjdvnuO3zCpVqtCrVy82bdrEhQsXMDMzo3379kUdqlAADA0NmTRpUp7a3r17l4CAgBe28/Pzy/EyRE7Hzo+nExyQe7eyeoeeVbFiRZ3Bza/i2ThVKlWO2/Jz6VpPTy9b0pJV1f15EhMTqVixIsHBwdmee3bMVJby5cujUql48uSJdpudnR02Nja5JnJZX5xq1KgByDO1nn59YXrw4AGdOnWidevW2cbuZImJidH2GuVk+PDhPH78mAULFlClShWMjY1xd3d/qUkFeX3PX/V3Ij/ycqwXvUdKEtcohFLl8OHDxMfHY2Vl9dxxTm5ubtqaMMeOHeP06dNFFaJQgFQqFUZGRnm6Va1a9YWTBiwtLalatWqe9veqvZRNmzbl8uXLuLq6UqNGDZ2bubk5NWvWxNTUNNdirnXq1OHChQskJSVptx05ckR7Se5VHT9+XHs/MzOTM2fOaC+N2dnZkZCQoHPs8+fP67zeyMhIO3A6S9OmTYmMjMTAwCDbOVeoUCHHOIyMjKhbty5XrlzRbtPT08Pf35+VK1cSGRmp0z4lJYVff/0VLy8v7fiiJk2a6Ly+sNy/f5+OHTvSrFkzlixZkuMwgNTUVMLCwmjSpEmu+zly5AgTJkyge/fu2oHWT08aADn5ePb9Laj3PCc57ftZderU4ciRI9nOpVatWvleFuXSpUvPfY+UJBInodSIjIzUJkCenp4vHH/SuHFj2rVrB8iVxC9fvlzoMQrK0dPTy/WyVxYPD48iG/M2btw4YmJiGDRoEKdOnSIsLIygoCBGjhyJWq3GxMSEqVOnMmXKFP755x/CwsI4fvw4f/75JwBDhgzBxMSE4cOHc+nSJfbv38+7777LsGHDtJfpXsUvv/zCxo0buXbtGuPGjePJkye88cYbALRq1QozMzM+/vhjwsLCWLlyZbYBwK6urty6dYvz588THR1NWloaXbp0wd3dHR8fH3bt2kV4eDhHjx7lk08+ee6XFy8vLw4fPqyz7euvv8bR0ZGuXbuyY8cO7t69y8GDB/Hy8iIjI4NffvlF5/WXL18u1F6nrKTJxcWF7777jkePHhEZGZktsTt+/Li2Byk3NWvWZNmyZVy9epUTJ04wZMgQ7cK2WVxdXdm7dy+RkZHa8yrI9/xZrq6unDhxgvDwcKKjo3Psjfrggw/Yu3cvX3zxBaGhofz999/8/PPPfPjhh3k+DsjlI86cOfNKBU8Lk0ichFJBo9EQFBSEJEnUrl2b6tWr5+l17u7u2rEn27dvf+4UYaHkc3Nzw8fHJ1vPk6WlJT4+PkW6+LOTkxNHjhxBrVbj6elJgwYNmDhxIjY2Ntrkbfr06XzwwQd89tln1KlThwEDBmjHgpiZmWlnjrVo0QI/Pz88PDz4+eefCyS+OXPmMGfOHBo1asThw4fZvHmztofC1taW5cuXs337dho0aMCqVauYOXOmzut9fX3x9vamU6dO2NnZsWrVKlQqFdu3b6d9+/aMHDmSWrVqMXDgQG7fvv3cZG/UqFFs376duLg47bby5ctz/PhxOnXqxJgxY6hevTr+/v5Ur16dU6dOUa1aNW3bBg0a0LRpU9auXVsg701Odu/ezY0bN9i7dy/Ozs5UrFhRe3vaqlWrGDJkyHMXqv7zzz958uQJTZs2ZdiwYUyYMAF7e3udNvPmzWP37t1UrlxZ2zNTkO/5sz788EP09fWpW7cudnZ2OY63ynqPV69eTf369fnss8/4/PPPdQaG58WmTZtwcXHRfrEtblRSfufUlkHx8fFYW1sTFxeHlZWV0uEIOTh16hT79u3D2NiYN998M1/FLSVJYuvWrVy5cgUDAwMGDhxIpUqVCjFa4WWlpqZy69Ytqlat+kqDfTUaDffu3dOZNCBmV8rCw8OpWrUq586dK7Qq1C+jf//+NG3alGnTpr3U67dt28bkyZO5dOmSYj/r6Oho3NzcOH36NFWrVlUkhpLgtddeY8KECQwePDjXNs/7W1DYn9niL4VQ4sXHx2vrfXTs2DHfFcFVKhXdu3enatWqZGZmEhAQwKNHjwojVKGY0NPTw8XFhbp16+Li4iKSphLg22+/faVq/z169OCtt97i/v37BRhV/oSHh/Prr7+KpOk5oqOj6devH4MGDVI6lFyJHqc8ED1OxZckSaxfv56wsDCcnZ0ZPHjwSw/aTU9PZ/Xq1URERGBhYcGwYcPEz7uYKageJyF3xbXHSRCeJnqcBOElhYSEEBYWhp6e3ivXYzIyMqJ///6UL1+exMRE1qxZU2zriAhCYXF1dUWSJJE0CUIuROIklFhPL6Hh7u6er6m1uTE1NcXf3x9LS0tiYmIICAjQWchTEARBKNtE4iSUWAcOHCApKQlbW1vt4p0FwcrKigEDBmBqakpERAQbN24kMzOzwPYvCIIglFyKJk4LFy6kYcOGWFlZYWVlhbu7Ozt27NA+n5qayrhx4yhfvjwWFhb4+vry8OFDnX3cuXOHHj16YGZmhr29PZMnT872IRccHEzTpk0xNjamRo0a2eqNCCXPvXv3tAX3vLy8MDAo2CL45cuXx8/PD0NDQ27fvs22bdvEosCCIAiCsomTs7Mzc+bM4cyZM5w+fZrOnTvTp08fbSHCSZMmsWXLFtatW8eBAwd48OAB/fr1075erVbTo0cP0tPTOXr0KH///TdLly7ls88+07a5desWPXr0oFOnTpw/f56JEyfy5ptvEhQUVOTnKxQMtVrNzp07AXnNIxcXl0I5jpOTE3379kVPT49r166xd+/ePK+NJQiCIJROxW5Wna2tLd9++y1+fn7Y2dmxcuVK/Pz8ALh27Rp16tTh2LFjvPbaa+zYsYOePXvy4MEDbSGvRYsWMXXqVB49eoSRkRFTp05l27ZtXLp0SXuMgQMHEhsbq/3wfRExq654OXr0KIcOHcLMzIw333wzW0Xdgnb16lU2b94MQNu2bWnTpk2hHk/InZhVJwgCiFl1gNyLsHr1apKSknB3d+fMmTNkZGTQpUsXbZvatWvj4uLCsWPHAHmNsQYNGuhUP/Xy8iI+Pl7ba3Xs2DGdfWS1ydpHTtLS0oiPj9e5CcVDTEwMR48eBeTlMQo7aQJ5/aWs36HDhw9z7ty5Qj+mIAiCUDwpnjhdvHgRCwsLjI2NGTt2LBs3bqRu3bpERkZiZGSUbcVsBwcH7do/kZGR2UrGZz1+UZv4+HhSUlJyjGn27NlYW1trb3lZKV0ofJIkERQUhFqtpmrVqtoFR4tCs2bNaN26NQC7du3KdVV2QShtVCoVgYGBhbLvYcOG8fXXX7/06/fu3UudOnVeuPhslvT0dFxdXUvVot6urq788MMPBbrPgQMHMm/evALdZ2mieOLk5ubG+fPnOXHiBG+//TbDhw8vklWsn2fatGnExcVpb3fv3lU0HkF28eJF7ty5g4GBAZ6enq+8On1+tW3bVlvbZuvWrYSHhxfp8YWCo1bDsfOwaa/8bx4/d1/aiBEj8PHx0dkWEBCAiYkJ8+bNo1evXnh7e+f42kOHDqFSqfj3338LN8giduHCBbZv386ECRN0tl++fBl/f3/s7OwwNjamVq1afPbZZznWVJsyZQqffvop+vr6dOzYEZVKleutY8eOGBkZ8eGHHzJ16tRXjj+nn2l4eDgqlUo7caUgLV26NFtHAsjLTb311lsFeqxPP/2Ur776SmdtQOH/FE+cjIyMqFGjBs2aNWP27Nk0atSIBQsW4OjoSHp6OrGxsTrtHz58iKOjIwCOjo7ZZtllPX5RGysrq1wv8xgbG2tn+mXdBGUlJSWxf/9+QE5gcvoDUthUKhVdu3bFzc0NjUbDhg0biIiIKPI4hFez4yC0GQQDJ8GEL+V/2wyStxeVxYsXM2TIEBYuXMgHH3zAqFGj2L17N/fu3cvWdsmSJTRv3pyGDRsWXYBF4KeffqJ///46y6gcP36cVq1akZ6ezrZt2wgNDeWrr75i6dKldO3alfT0dG3bw4cPExYWhq+vL4D2/2NERAQnT54EYM+ePdptGzZsAGDIkCEcPnxYO5yjpLOzs3vugsEvo379+lSvXp3ly5cX6H5LC8UTp2dpNBrS0tJo1qwZhoaG7N27V/tcSEgId+7cwd3dHZCLHl68eFG7WjjIK1RbWVlRt25dbZun95HVJmsfQsmwb98+UlNTsbe3p0WLForFoaenR8+ePalSpQoZGRmsW7eOx48fKxaPkD87DsLbMyDimaUIIx/J24sieZo7dy7vvvsuq1evZuTIkQD07NkTOzu7bKVSEhMTWbduHaNGjcp1f2lpaUydOpXKlStrS678+eefgDx2dNSoUVStWhVTU1Pc3NxYsGBBtn389ddf1KtXD2NjYypWrMj48eN1no+OjqZv376YmZlRs2ZN7WSJLJcuXaJbt25YWFjg4ODAsGHDiI6OzjVmtVpNQEAAvXr10m6TJIlRo0ZRp04dNmzYQMuWLalSpQr9+/dny5YtHDt2jPnz52vbr169mq5du2oHBtva2uLo6IijoyN2dnaAXFYka5utrS0A5cqVo02bNqxevfq58T3vfZs5cyZ///03mzZt0vZoBQcHa9ega9KkibaXK8vixYupU6cOJiYm1K5dm19//VX7XFZP1YYNG+jUqRNmZmY0atRIOxY3ODiYkSNHEhcXpz3ezJkzgeyX6u7cuUOfPn2wsLDAysoKf39/nc6DmTNn0rhxY5YtW4arqyvW1tYMHDiQhIQEnfegV69ez32PyjRJQR999JF04MAB6datW9K///4rffTRR5JKpZJ27dolSZIkjR07VnJxcZH27dsnnT59WnJ3d5fc3d21r8/MzJTq168veXp6SufPn5d27twp2dnZSdOmTdO2uXnzpmRmZiZNnjxZunr1qvTLL79I+vr60s6dO/McZ1xcnARIcXFxBXfyQp7dvHlTmjNnjvTNN99IDx48UDocSZIkKTU1VVq6dKk0Z84c6ddffxW/G0UkJSVFunLlipSSkiJJkiRpNJKUlJy3W3yCJLX0kySXjjnfqnSUn49PyNv+NJq8xz18+HCpT58+0pQpUyQLCwtpz5492dpMnjxZql69uqR5asd//fWXZGpqKsXGxua6b39/f6ly5crShg0bpLCwMGnPnj3S6tWrJUmSpPT0dOmzzz6TTp06Jd28eVNavny5ZGZmJq1Zs0b7+l9//VUyMTGRfvjhBykkJEQ6efKkNH/+fO3zgOTs7CytXLlSun79ujRhwgTJwsJCevz4sSRJkvTkyRPt392rV69KZ8+elbp27Sp16tQp15jPnj0rAVJkZGS2bStXrszxNV27dpUaNWqkfdywYUNpzpw5Oba9deuWBEjnzp3L8fmpU6dKHTp0yDW+F71vCQkJkr+/v+Tt7S1FRERIERERUlpamnTy5EkJkPbs2SNFRERo36Ply5dLFStWlNavXy/dvHlTWr9+vWRraystXbpUJ97atWtLW7dulUJCQiQ/Pz+pSpUqUkZGhpSWlib98MMPkpWVlfZ4CQkJkiRJUpUqVbQ/L7VaLTVu3Fhq27atdPr0aen48eNSs2bNdM51xowZkoWFhdSvXz/p4sWL0sGDByVHR0fp448/1nkPduzYIRkZGUmpqam5vk9KevZvwdMK+zNb0cTpjTfekKpUqSIZGRlJdnZ2koeHhzZpkiT5jXnnnXekcuXKSWZmZlLfvn2liIgInX2Eh4dL3bp1k0xNTaUKFSpIH3zwgZSRkaHTZv/+/VLjxo0lIyMjqVq1atKSJUvyFadInJSTnp4uLVy4UJozZ460e/dupcPRkZSUJP3+++/SnDlzpD/++ENKTk5WOqRS79k/lknJuSdChX1LysePe/jw4ZKRkZEESHv37s2xzdWrVyVA2r9/v3Zbu3btpKFDh+a635CQEAnI1/+NcePGSb6+vtrHTk5O0ieffJJre0D69NNPtY8TExMlQNqxY4ckSZL0xRdfSJ6enjqvuXv3rgRIISEhOe5z48aNkr6+vk6SuHr16ucmOxMmTJBMTU21j62traV//vknx7YvSpwWLFggubq65vhcbp5937KS4bwct3r16tkSwi+++ELbEZD1usWLF2ufv3z5sgRIV69elSRJkpYsWSJZW1tni+vpxGnXrl2Svr6+dOfOnWz7OXnypCRJcuJkZmYmxcfHa9tMnjxZatWqlc5+L1y4IAFSeHj4c94V5SiZOBVsueV8yupOzo2JiQm//PILv/zyS65tqlSpwvbt25+7n44dO4op5CXUkSNHiIuLw9LSknbt2ikdjg4zMzP8/f1Zvnw5jx8/JiAggAEDBmBkZKR0aEIx1LBhQ6Kjo5kxYwYtW7bUGdsDcrmV1q1b89dff9GxY0du3LjBoUOH+Pzzz3Pd5/nz59HX16dDhw65tvnll1/466+/uHPnDikpKaSnp2snOURFRfHgwQM8PDxeGHsWc3NzrKystEMkLly4wP79+7OdD0BYWBi1atXKtj0lJQVjY+McJ3hIeSwtmJKS8tK1vExNTV+4gPfz3rf8SEpKIiwsjFGjRjF69Gjt9szMTKytrXXaPv0+V6xYEZB/RrVr187Tsa5evUrlypV1ZoLXrVsXGxsbrl69qh3m4OrqiqWlpc6xnh7yAmjHAIuFzrNTNHEShOeJiorSDvL09PTE2NhY4Yiys7a2xt/fn5UrV/LgwQM2bdpEv3790NfXVzq0MsHUBK4+/3uT1sl/YfhHL2739xxomYdx2Kb5/MyuVKkSAQEBdOrUCW9vb3bs2KHz4QUwatQo3n33XX755ReWLFlC9erVn5sUvaiO2erVq/nwww+ZN28e7u7uWFpa8u2333LixIk8vT6LoaGhzmOVSqVdgigxMZFevXrxzTffZHtd1of/sypUqEBycjLp6enaLxpZCdbVq1dp0qRJttdcvXpVJwmrUKECT548yVP8z4qJidGOg8rJi963/EhMTATgjz/+oFWrVjrPPft34un3OSupLIylnp7388wSExMD8Nz3qawqdoPDBQHkPxY7d+5EkiTc3NyoUaOG0iHlys7ODl9fXwwMDLh58yY7duwQS7MUEZUKzEzzdmvXHCraQW5FLFTIz7drnrf9vUw1jCpVqnDgwAEiIyPx9vbONiDX398fPT09Vq5cyT///MMbb7zx3LIbDRo0QKPRcODAgRyfP3LkCK1bt+add96hSZMm1KhRg7CwMO3zlpaWuLq6ZptAkx9Nmzbl8uXLuLq6UqNGDZ2bubl5jq/J6rl5uvRM48aNqV27NvPnz8/2IX7hwgX27NnDoEGDtNuaNGny0qVrLl26lGNyluVF7xvIM8KfrR+VlQQ+vd3BwQEnJydu3ryZ7f3JGkyeFzkd71l16tTh7t27OiV0rly5QmxsrHbCVF5dunQJZ2dnKlSokK/XlQUicRKKpXPnzhEREYGRkdELLyMUB87Ozvj4+KBSqbh8+TL79u0TyVMxo68PM/6bLPZsKpL1eMZ4uV1hqly5MsHBwURFRWlXOshiYWHBgAEDmDZtGhEREYwYMeK5+3J1dWX48OG88cYbBAYGcuvWLYKDg1m7di0ANWvW5PTp0wQFBREaGsr06dM5deqUzj5mzpzJvHnz+PHHH7l+/Tpnz57lp59+yvP5jBs3jpiYGAYNGsSpU6cICwsjKCiIkSNH5vpBb2dnR9OmTTl8+LB2m0ql4s8//+TKlSv4+vpy8uRJ7ty5w7p16+jVqxfu7u5MnDhR297Ly0vn9flx6NAhPD09c30+L++bq6sr//77LyEhIURHR5ORkYG9vT2mpqbs3LmThw8fausgzZo1i9mzZ/Pjjz8SGhrKxYsXWbJkCd9//32eY3Z1dSUxMZG9e/cSHR2d4yW0Ll260KBBA4YMGcLZs2c5efIkr7/+Oh06dKB58+Z5Pha8+D0qy0TiJBQ78fHxHDwozwvv2LFjtssZxVX16tXp3r07AKdPn+b48eMKRyQ8q1t7WDgLHJ+5+uBoJ2/v1r5o4nB2diY4OJjo6OhsydOoUaN48uQJXl5eODk5vXBfCxcuxM/Pj3feeYfatWszevRokpKSABgzZgz9+vVjwIABtGrVisePH/POO+/ovH748OH88MMP/Prrr9SrV4+ePXty/fr1PJ+Lk5MTR44cQa1W4+npSYMGDZg4cSI2Njbo6eX+EfPmm2+yYsUKnW2tW7fm+PHj6Ovr061bN2rUqMG0adMYPnw4u3fv1rlcP2TIEC5fvkxISEieYwV5Ga64uDjtGqg5ycv7Nnr0aNzc3GjevDl2dnYcOXIEAwMDfvzxR3777TecnJzo06eP9lwXL17MkiVLaNCgAR06dGDp0qX56nFq3bo1Y8eOZcCAAdjZ2TF37txsbVQqFZs2baJcuXK0b9+eLl26UK1aNdasWZPn44C8DlxgYKDOmCzh/4rdIr/FkVjkt2ht2LCB69evU6lSJYYMGVLkFcJf1alTp9i3bx8A3t7eNGrUSOGISo+CWuRXrYaTFyHqMdiXh5YNCr+nSdCVkpKCm5sba9aseem6epMnTyY+Pp7ffvstz68ZMGAAjRo14uOPP36pY5YFCxcuZOPGjezatUvpUHIlFvkVhP+EhoZy/fp19PT08PLyKnFJE0CLFi147bXXALRd/ULxoq8P7o2hj4f8r0iaip6pqSn//PPPcwtlvsgnn3xClSpV8jyAOj09nQYNGjBp0qSXPmZZYGhomK/LtWWN6HHKA9HjVDTS0tJYvHgxiYmJuLu70759EV03KQSSJLFz507+/fdf9PX18ff3x8XFRemwSryC6nESBKFkEz1OggAcPHiQxMREypUrR+vWrZUO55WoVCq8vLyoWbMmarWa9evXZ1szURAEQSh5ROIkFAv379/n7NmzgDxbxsCg5JcY09PTo3fv3lSuXJn09HTWrl370nVnBEEQhOJBJE6C4tRqNTt37gTkVbmrVKmicEQFx8DAAF9fX+zt7UlOTmbNmjXagniCIAhCySMSJ0FxJ0+eJDo6GlNTUzp37qx0OAXO2NgYf39/bGxsiIuLY+3ataSmpiodVolWGNWUBUEoGpIkkZaWRkpKCmlpaS9V807JvwEl/3qIUKI9efKEo0ePAuDh4ZHnJSBKGnNzcwYMGMDy5ct59OgR69evx9/fP9vSB8LzGRkZoaenx4MHD7Czs8PIyKhEzrx8GZIEyalyKQV9fTAzebnq5YKgpLS0NBITE3WKo+rr62NhYZGnZbUkSSI9PZ1Hjx6hp6enyNqgInESFCNJEkFBQWRmZuLq6prvJQFKGhsbG/z9/VmxYgX37t1j8+bN9O3b97lFAgVdenp6VK1alYiICB48eKB0OEUmJRViE0D91JdsfT2wscz/mnmCoJSMjIznLhpsZmaW5y+TZmZmuLi4KPL3UyROgmIuX77M7du3MTAwwNPTs0z0HNjb2+Pn58eaNWu4ceMGO3fupFu3bmXi3AuKkZERLi4uZGZmvnDtrtLg4CmY9RM8ezFDu0zMu9C+RVFHJQj5o9FoWL16tbaqfU7Mzc0ZOHDgC5MhfX19DAwMFPu7KRInQRHJycna6tpt2rShXLlyCkdUdCpXrkyfPn3YuHEjFy9exMzMjI4dOyodVomiUqkwNDQs9Zc61Wr47CeIeJTz8yrk54+sEkU8heLtzp07PHqUyy/yf5KTk4mOji72Ne/ENQJBEfv37yclJQU7OztatCh7X5dr1qyJt7c3ACdOnODEiRMKRyQURycv5p40gdwLFfFIbicIxVleZxOXhFnHInESilx4eDiXLl0C5LXc9MvoV+WGDRtqe5qCg4O5eFF8+gm6oh4XbDtBUIqFhUWBtlOSSJyEIpWRkaFdOLJp06Z5Wv29NGvZsqW2x23Hjh3cuHFD4YiE4sS+fN7aRT7/CoggKE5fX/+FY5IsLS1xdnYuoohenkichCJ17Ngxnjx5goWFRYlei66gqFQqOnXqRP369ZEkiU2bNnHv3j2lwxKKiZYNwNb6xe2+/g3enw1P4go/JkHIr9DQUFavXv3Cek0eHh4lYpZx8Y9QKDUePXqkHcvTtWvXPNXsKAtUKhXe3t5Ur16dzMxMAgICiIqKUjosoRi4HwWp6Tk/p/rv1vk1uZ7T+l3gMQI275NrPgmC0iRJ4uTJk2zcuJHMzEyqVatGz549sbS01GlnaWmJj48Pbm5uCkWaPyrpZUp2ljGFvdJyWSBJEsuXL+fBgwfUrFmTfv36KR1SsZORkcGaNWu4f/8+FhYWDBkyBBsbG6XDEhSSnAL9xsPVm1ClEqSlQWT0/5+vaAczxkO39nD2Ckz9FkLD5ec83OHLieBkr0TkgiCXH9izZw/nzp0DoEmTJnTp0gU9PT00Gg337t0jMTERCwsLnJ2dC7SnqbA/s0XilAcicXp1586dY9euXRgZGTFq1CjxPuYiNTWVFStWEB0dTbly5RgyZAjm5uZKhyUUMUmC8Z/D1mCoUA62/gb2tvLsuajH8tinlg10SxCkZ8DCVfDzcvm+hRlMHQ1De0MJuPohlCJpaWls2rSJW7duAdCpUydatGhRZHWXCvszW/x3EgpdQkICwcHBALRv314kTc9hYmKCv78/VlZWPHnyhHXr1pGWlqZ0WEIRW7RaTpoM9GHRLLl3SV8f3BtDHw/532cnoxoZwnuvw7bfoVk9SEyG6Qug/3tw/bYCJyGUSfHx8axYsYJbt25hYGBA3759admyZakq8isSJ6HQ7dmzh/T0dCpWrEiTJk2UDqfYs7S0ZMCAAZiZmfHw4UM2bNhAZmam0mEJRST4JHzzh3x/1gRo0SB/r6/lCgE/wucTwNwUTl+C7qNhwT9yT5QgFJbIyEiWLVvGo0ePMDc3Z/DgwdSqVUvpsAqcSJyEQnX9+nVCQ0PR09PD29u7RMyYKA5sbW3p378/RkZG3Llzhy1btii6GrhQNG7dg3c/ly/VDe4pX2Z7GXp6MLwv7FkqDx5Pz4Dvl0DPMXDuSoGGLAgA3Lhxg5UrV5KYmEiFChUYNmwYFStWVDqsQiE+xYRCk5aWxu7duwG5XpG9vRipmh+Ojo7069cPfX19QkND2bVr1wun8wolV2IyvPkpxCfJl9pmvvvq+3Syh7++hh8/lcsahNyCvuNh1s+QlPLq+xcEgNOnT7NhwwYyMjJwdXVl6NChWFvnoY5GCSUSJ6HQHDp0iISEBGxsbGjdurXS4ZRIVapUoVevXgBcuHCBQ4cOKRyRUBg0GrkO043b4FABFs4CY6OC2bdKJY+L2vs3+HrKvVl/rQfPkXDgZMEcQyibsmbO7d27F0mSaNSoEX5+fqW+1IxInIRC8eDBA86cOQOAp6dnqV+MtTC5ubnh5eUFyAVET58+rXBEQkH7aRkEHZYHeP82CxzyWDE8P2yt4ftp8M834OwA9x7C61Nh0teicKaQf+np6WzcuFH7d75jx454eXmViSW0ROIkFDi1Wk1QUBAA9erVo2rVqgpHVPI1btyYdu3aAbB3716uXBEDVUqL3Ufg+6Xy/a8mQZO6hXu8Di1h1xIY5Sf3Rm3YLRfO3LRXFM4U8iYhIYGVK1dy48YNDAwM6NOnD61atSpVM+eeRyROQoE7ffo0UVFRmJiY0LlzZ6XDKTXc3d1p1qwZANu2bePmzZsKRyS8quu3YeLX8v0RfcG/W9Ec19wUPhsHG38Gt6rwOBYmfAlvfAz3HxZNDELJFBUVxbJly3j48CFmZmYMHDiQ2rVrKx1WkRKJk1CgYmNjOXz4MACdO3fGzMxM4YhKD5VKhYeHB3Xq1EGj0RAYGMj9+/eVDkt4SXGJMPpTeVD4a43g03eKPoYmdeXimu+PlC8T7jsOXUfC0o3yuCtBeFpYWBgrVqwgISEBW1tbhg0bRqVKlZQOq8iJxEkoMJIksWvXLjIzM3FxcaF+/fpKh1TqqFQqevToQdWqVcnIyCAgIIDo6OgXv1AoVtRquYfn1j2o5AC/zgRDA2ViySqcuf0PaF5fnm0340fwm/D/JVwE4dy5c6xfv5709HRcXFwYNmxYmV0SSiROQoG5evUqt27dQl9fHy8vrzJzvbuo6evr4+PjQ8WKFUlNTWXt2rXEx8crHZaQD/OWQPAJMDGG3z6H8jZKRwQ1q8C6BfDFe/KlvDOXocdbsOBvUTizLNNoNOzbt09bDqV+/fr4+/tjYmKidGiKEYmTUCBSUlLYu3cvAK1bt8bW1lbhiEo3IyMj+vfvT/ny5UlISGDNmjUkJycrHZaQB9uC4ZcV8v1vPoQGxaiwsp4evO4jF870cP+vcOZSOYE6K+YjlDnp6ekEBgZy6tQpQF4yq3v37mVi5tzzKJo4zZ49mxYtWmBpaYm9vT0+Pj6EhITotOnYsSMqlUrnNnbsWJ02d+7coUePHpiZmWFvb8/kyZOzLVERHBxM06ZNMTY2pkaNGixdurSwT69M2b9/P8nJyVSoUIFWrVopHU6ZYGpqir+/P5aWlsTExBAQEEB6errSYQnPcTUMPvhGvv/WAPDpomw8uXGyhz+/gp+ny71hoeHQTxTOLFMSExNZtWoV169fR19fn169euHu7i6uJKBw4nTgwAHGjRvH8ePH2b17NxkZGXh6epKUlKTTbvTo0URERGhvc+fO1T6nVqvp0aMH6enpHD16lL///pulS5fy2WefadvcunWLHj160KlTJ86fP8/EiRN58803tVPmhVdz584dLl68CFBm6ngUF1ZWVtpu84iICDZu3IharVY6LCEHT+Jg9HRISYV2zWHqaKUjej6VCnp1hr1Lwc/r/4Uzu46U19MTSq9Hjx6xbNkyIiMjMTU1ZeDAgdStW8h1MkoQlVSM1nB49OgR9vb2HDhwgPbt2wNyj1Pjxo354YcfcnzNjh076NmzJw8ePMDBwQGARYsWMXXqVB49eoSRkRFTp05l27ZtXLp0Sfu6gQMHEhsby86dO18YV3x8PNbW1sTFxWFlZfXqJ1qKZGZmsmTJEmJiYmjcuLG2UKNQtB48eMDq1avJyMigdu3a9O7dW3wzLEYy1TB8Khw+Ay5OsGUh2JSwPyUHT8G07+FepPy4b1e5pIFt6V1Zo0y6desWgYGBpKenU65cOfr370+5cuWUDitfCvszu1iNcYqLk8vXPjs+ZsWKFVSoUIH69eszbdo0nbEcx44do0GDBtqkCeRej/j4eC5fvqxt06WLbp+4l5cXx44dK6xTKTOOHTtGTEwMFhYWdOjQQelwyiwnJyf69u2Lnp4e165dY8+ePWJdu2Jkzu9y0mRmAn98UfKSJoD2LWD3X3LhTD092LgbPIZD4B5ROLO0OH/+POvWrSM9PZ3KlSszbNiwEpc0FYVikzhpNBomTpxImzZtdKaxDx48mOXLl7N//36mTZvGsmXLGDp0qPb5yMhInaQJ0D6OjIx8bpv4+HhSUrJfsE9LSyM+Pl7nJmQXHR3N8ePHAfDw8CjTsyyKg6pVq9KjRw8Azp49y9GjRxWOSAA5wfhjrXx/3kdQu5qy8bwKs6cKZ9auBjFx8N5XMHKaKJxZkkmSRHBwMEFBQUiSRL169fD398fU1FTp0IolhSqHZDdu3DguXbqkLZ6Y5a233tLeb9CgARUrVsTDw4OwsDCqV69eKLHMnj2bWbNmFcq+SwtJkggKCkKj0VCjRg3c3NyUDkkA6tatS0pKCnv27OHw4cOYmZnRpEkTpcMqsy6GwNTv5Pvjh0L3UtIp27gObFkEv62BH/+B/SfksU9TRsOw3iCGOZYcGRkZbNu2TTsxq02bNrRp00Zc6n+OYtHjNH78eLZu3cr+/ftxdnZ+btusGVs3btwAwNHRkYcPdb/qZD12dHR8bhsrK6scM+pp06YRFxenvd29e/flTqwUu3DhAvfu3cPQ0JCuXbuK/2TFSLNmzWjdujUAu3bt4tq1awpHVDZFP4G3PoO0dOj8Grw/QumICpaRIbw7FHYshhYNniqc+Z4onFlSJCUlsWrVKkJCQtDT06Nnz560bdtW/D1/AUUTJ0mSGD9+PBs3bmTfvn15Wgz2/PnzAFSsWBGQ1++6ePEiUVFR2ja7d+/GyspKOwvA3d1dW2Po6Tbu7u45HsPY2BgrKyudm/B/iYmJBAcHA3JdD/H+FD9t27alcePGAGzZsoXw8HBF4ylrMjLhnZnwIAqqVYYfPim9vTA1XGDtD/DlRLAwg7OXofto+OFvOWkUiqfo6GiWLVtGREQEJiYmDBgwgHr16ikdVomgaOI0btw4li9fzsqVK7G0tCQyMpLIyEjtuKOwsDC++OILzpw5Q3h4OJs3b+b111+nffv2NGzYEABPT0/q1q3LsGHDuHDhAkFBQXz66aeMGzcOY2NjAMaOHcvNmzeZMmUK165d49dff2Xt2rVMmjRJsXMvyfbu3UtaWhqOjo40bdpU6XCEHKhUKrp27YqbmxsajYaNGzcSERGhdFhlxhe/wol/5UTi9y/A2kLpiAqXnh4M6yMXzuzSWk4c5y+VC2eeuax0dMKzbt++zfLly4mLi8PGxoahQ4fi4uKidFglhqLlCHLrDlyyZAkjRozg7t27DB06lEuXLpGUlETlypXp27cvn376qU4vx+3bt3n77bcJDg7G3Nyc4cOHM2fOHAwM/j+EKzg4mEmTJnHlyhWcnZ2ZPn06I0aMyFOcohzB/4WFhREQEIBKpWL48OHZBt0LxUtmZiYBAQHcvn0bU1NThg4dKqq6F7I122HKt/L9P7+SE4myRJLk6ugzfpIvV6pUMNwHJr8pJ5KCsv7991/t+NRKlSrRr1+/UrcYe2F/ZherOk7FlUicZOnp6fz555/Ex8fTsmVLOnXqpHRIQh6kpaWxevVqIiMjsbKyYujQoVhaWiodVql09goMmCgvVfL+SHnx3LLqSRx8tQjW/Vcqr5IDfDUJOomFBRQhSRKHDh3SluGpU6cO3bt31+lgKC3KVB0noXg7fPiw9heyTZs2Socj5JGxsTF+fn6UK1eO+Ph41q5dm2MZDuHVPHwMYz6TkyavdvLA6bKsnDV8NxWWfwuVK8rlCkZ8JJcveByrdHRlS2ZmJlu2bNEmTe7u7vTq1atUJk1FQSROQp5ERkZy+vRpQB5XZmRkpHBEQn6Ym5szYMAALCwsiI6OJiAggIwMseR9QUlLh7EzIOox1KwC338kj/sR5OVldv0Jb/aX35PAPdBlhFzfSlzvKHzJycmsXr2aq1evoqenR7du3Wjfvr2YOfcKxH9t4YU0Gg07d+5EkiTq1KlDtWoluIJfGWZtbY2/vz/GxsY8ePCAwMBAsa5dAZAk+OxHeTaZlQUs/kqM5XmWmSlMfwcCf4E6/xXOnPg1jJj2/yVchIIXExPDsmXLuH//PsbGxvj7+2snVgkvTyROwgudPn2ahw8fYmxsjIeHh9LhCK/Azs4OPz8/DAwMuHnzJjt27BBLs7yiFVtg9Ta5N+Xn6eBaSemIiq9GtWHLbzB5FBgbQvB/hTOXrAeRwxesu3fvsmzZMmJjY7G2tmbo0KFUqVJF6bBKBZE4Cc8VFxenrebeqVMnzM3NFY5IeFXOzs74+PigUqm4fPky+/btE8nTSzr5r1z0EWDqm9ChpbLxlASGBnIV9R2LoWVDSE6FmT+D7wQIvaV0dKXDpUuXWL16NampqTg5OTFs2DAqVKigdFilhkichFxJksSuXbvIyMigcuXKoou3FKlevTrdu3cH5B7FEydOKBxRyfMgCt6eCZlq6NUJxgxUOqKSpboLrJkvz7SzMINzV6D7W3L9J1E48+VIksThw4fZtm0bGo0GNzc3Bg4cKL7wFjCROAm5unbtGjdv3kRfXx8vLy8xmLCUqV+/vrakxIEDB7hw4YLCEZUcqWnw1nS5TlHd6jB3slyvSMgfPT0Y2hv2LoWu/xXO/OFvUTjzZWRmZrJt2zaOHDkCyMuT9enTB0NDQ4UjK31E4iTkKDU1VbtMzWuvvUb58uUVjkgoDC1btuS1114DICgoiNDQUIUjKv4kCT7+Hi6GQjkr+P1LefCz8PIc7eCPL+HXGVChHFy/Db7vyoPuE5OVjq74S0lJYe3atVy+fBmVSoWXlxcdO3YUX3YLiUichBwFBweTlJSEra2t9oNVKJ3at29PgwYNkCSJzZs3c+fOHaVDKtaWrIf1u0BfD36ZAZUdlY6odFCpoEdHufepv7ecoP69UR48vu+40tEVX0+ePGH58uXcvXsXIyMj+vfvr12nUigcInESsrl79672so23t7coklbKqVQqvL29qVmzJmq1mvXr1/Pw4UOlwyqWDp+BLxfK9z99G9qIpRoLnI2VXDhzxXdy4cwHUTByGkz4UhTOfNa9e/dYtmwZMTEx2lUBqlatqnRYpZ5InAQdmZmZBAUFAdCoUSMqV66scERCUdDT06NXr15UrlyZ9PR01q1bx5MnT5QOq1i5EwHjPwe1Bnw9YaSv0hGVbm2byYUz3xogj4XatBc8hsMGUTgTgCtXrrB69WpSUlJwdHRk2LBh2NnZKR1WmSASJ0HHiRMnePz4Mebm5nTs2FHpcIQiZGhoiK+vL/b29iQlJbF27VoSExOVDqtYSE6BMdPhSTw0dIOv3xeDwYuCmSl8MhY2/SoXznwSD5O+huFTy27hTEmSOHr0KFu2bEGtVlOzZk0GDRqEhYWF0qGVGSJxErQeP36sXcvIw8MDExMThSMSipqxsTH9+/fHxsaG2NhY1q1bR2pqqtJhKUqSYMq3cCVMHrj82+dgYqx0VGVLQze5cOaUN+XCmQdOyWOf/gooW4Uz1Wo127dv59ChQwC0aNECHx8fsQRWEROJkwDI32KCgoJQq9VUq1aN2rVrKx2SoBALCwv8/f0xNzcnKiqKDRs2lOl17Rathi37wUAfFs0CJ3ulIyqbDA1g3BDY+Se0+q9w5qxf5Nl3IWWgcGZqaipr167l0qVLqFQqPD096dy5M3piUcQiJ95xAYB///2Xu3fvYmhoiKenp5jGWsaVK1eO/v37Y2RkxN27d9myZQsajUbpsIpc8En45g/5/qwJ0KKBsvEIUK0yrJ4PX08CS3M4d1Wu+/T9ktJbODM2NpZly5Zx584djIyM8PX1pUmTJkqHVWaJxEkgKSmJ4OBgANq2bYu1tbWyAQnFgoODA76+vujr63P9+nXtQs9lxa178O4X8qW6QT1gSC+lIxKy6OnBkN6wZwl4tpELZy74R648fvqS0tEVrPv372tnzllYWDBkyBCqV6+udFhlmkicBPbu3UtqaioODg40b95c6XCEYsTFxYXevXujUqm4ePEiBw4cUDqkIpGYDKM/hfhEaFpP7m0SnbDFj6Md/P4F/DoT7MrBjdvgNwGmLygdhTOvXbvG6tWrSU5OxsHBgddffx17e3GtWGkicSrjbt68ydWrV7W1fMT1cuFZtWrVwsvLC5BnXZ48eVLhiAqXRgPvz5arVztUkMc1GYuxt8WWSgU9OsCepeDfTe4h/CcQuoyAvccUDu4lSZLEiRMn2LRpE5mZmVSvXp3BgwdjaWmpdGgCInEq09LT09m1axcAzZo1w9FRlEAWctaoUSM6dOgAwP79+7l0qZRdD3nKT8sg6DAYGcJvs8BBrDZUIthYwbdT5MKZLk4Q8Qje+Fi+3BpdgkqSqdVqgoKCtMMnmjVrRr9+/cTMuWJEJE5l2JEjR4iLi8PKyop27dopHY5QzLVq1YoWLVoAsH37dsLCwhSOqODtPgLfL5XvfzUJmtRVNBzhJWQVzhzzX+HMzfvAYwRs2FX8C2empaUREBDAhQsXUKlUeHh40KVLF3EloJgRP40y6uHDh5w6dQoAT09P8W1GeCGVSkWnTp2oV68ekiQRGBjIvXv3lA6rwFy/DRO/lu8P7ytf9hFKJlMT+HgsbP4V6laH2HiYNBtenwJ3i2nhzLi4OJYvX054eDiGhob069dPjDktpvKdOJ09e5aLFy9qH2/atAkfHx8+/vhj0tNL6VzQUkaj0WhnSLm5uYkZGkKeqVQqunXrRrVq1cjMzCQgIIBHjx4pHdYri0uUB4MnJss1gqa/o3REQkFo4AabF8HU0XLhzIOn5cKZfxazwpkREREsW7aM6OhoLCwsGDx4MDVq1FA6LCEX+U6cxowZQ2hoKCAPLB44cCBmZmasW7eOKVOmFHiAQsE7e/YskZGRGBsb06VLF6XDEUoYfX19fHx8qFSpEmlpaaxdu5a4uDilw3ppajW896VcfsDJXp6hZSjWtS41DA3gncFy4czXGkFKKnz+C/QbD9duKh0dhIaGsnLlSpKSkrCzs2PYsGFivGkxl+/EKTQ0lMaNGwOwbt062rdvz8qVK1m6dCnr168v6PiEAhYfH8/BgwcB6Nixo1jfSHgphoaG+Pn5UaFCBRITE1mzZg1JSUlKh/VS5i2B/SfkmXO/fyEvqyKUPtUqw6rvYc4HcuHM89fkwpnf/aVM4UxJkjh58iQbN24kMzOTatWqMWTIEKysrIo+GCFf8p04SZKkrSC8Z88eunfvDkDlypWJjo4u2OiEAiVJErt37yYjI4NKlSrRqFEjpUMSSjATExP8/f2xsrLiyZMnrFu3jrS0NKXDypdtwfDLCvn+Nx9Cg1qKhiMUMj09GNQT9i4Fr7aQqZZnUXYbDacuvvDlBUaj0bB79272798PQJMmTfD19cXYWCyCWBLkO3Fq3rw5X375JcuWLePAgQP06NEDgFu3buHg4FDgAQoFJzQ0lBs3bqCnp4e3t7dYVkV4ZZaWlgwYMABTU1MePnyo/fZcElwNgw++ke+P9oe+XZWNRyg6DhXk3sWFM8HOFsLuyIUzP/0BEgq54zQtLY3169dz7tw5ADp16kTXrl3FzLkSJN8/qR9++IGzZ88yfvx4PvnkE+0AtoCAAFq3bl3gAQoFIy0tjT179gDw2muvUaFCBYUjEkoLW1tb7bp2t2/fLhHr2sXGw+jp8niXds3ho7eUjkhQQvcOcu/TAPnCCcs2yYPHC6twZnx8PCtWrODmzZsYGBjQt29fWrZsKb7EljAqqYAWn0pNTUVfXx9DQ8OC2F2xEh8fj7W1tbbmUUm0a9cuzp07h62tLSNHjsTAQIx+FQpWeHg4AQEBqNVqGjduXGwXi85Uw4iP4NBpuVDiloVy8UShbDtyFqbNg9sP5Me9O8OM8QU35i0yMpL169eTmJiIubk5vr6+VKxYsWB2Lugo7M/sV+obTExMJD4+nvj4eNLT00lJSSmouIQCdO/ePW23sJeXl0iahELh6upKz549ATh//jyHDx9WOKKczfldTppMTeCPL0TSJMjaNIWgP2HsQNB/qnBmwM5XL5x548YNVq5cSWJiIhUqVGDYsGEiaSrB8p043bp1ix49emBubo61tTXlypWjXLly2NjYUK6cmI5S3GSV7wdo0KABLi4uCkcklGa1a9fG09MTgKNHj3LmzBmFI9K1cTf8sVa+P+8jqF1N2XiE4sXUBKaNgU0LoW4N+ZLuB9/AsClwJ+Ll9nnmzBk2bNhARkYGrq6uDB06FGtr64INXChS+e56GDp0KJIk8ddff+Hg4FAsu+KF/ztx4gTR0dGYmZnRqVMnpcMRyoAmTZqQnJzM4cOH2bNnD6amptStq/zaJRdDYep38v1xQ+SFYQUhJw1qweaFsHgdzF8q91B6vgEfvgEj+4G+/ov3odFo2Ldvn/bLQ8OGDfH09EQ/Ly8WirV8j3GysLDgzJkzuLm5FVZMxU5JHeMUExPDX3/9hVqtpmfPntSrV0/pkIQyQpIk9uzZw9mzZ9HT08PX15dq1ZTr3ol+Ar3GwoMo6NQK/vwqbx9+gnDrHnw0D46flx83qi2XrqjznAUX0tPT2bJlCzdu3ADkmnliEHjRKXZjnFq0aMHdu3cLPBChYEmSRFBQEGq1GldX12LxjV8oO1QqFV26dKFOnTpoNBoCAwN58OCBIrFkZMK4WXLSVK0yLPhUJE1C3lV1hlXzYM6HYGUOF65BzzHw7Z+QmkPhzISEBFauXMmNGzcwMDCgT58+tGrVSiRNpUi+e5zCwsIYO3YsQ4cOpX79+tlm0TVs2LBAAywOSmKP08WLF9m+fTsGBgaMGjUKGxsbpUMSyiC1Wk1AQADh4eGYmJgwZMiQIi+F8dmP8PdGsDCDwF+hZpUiPbxQijyMhuk/QtAh+XH1ynJC1fK/j72oqCgCAgJISEjAzMyMfv36UalSJeUCLqMK+zM734nT8ePHGTx4MOHh4f/fiUqFJEmoVCrUxWnlxAJS0hKn5ORk/vjjD1JTU+nYsSOtWrVSOiShDEtPT2f16tVERERgaWnJ0KFDi+z/0ZrtMOVb+f7iL6FrmyI5rFDK7TgI0xfAoxj58dDeMNDzFnt2BZKenq6tbSa+sCqj2F2qe+ONN2jSpAnHjh3j5s2b3Lp1S+ff/Jg9ezYtWrTA0tISe3t7fHx8CAkJ0WmTmprKuHHjKF++PBYWFvj6+vLw4UOdNnfu3KFHjx6YmZlhb2/P5MmTs1UvDg4OpmnTphgbG1OjRg2WLl2a31MvMfbt20dqaip2dnY0b95c6XCEMs7IyAg/Pz9sbW1JSEhgzZo1RVK65NwVuRI0wPsjRNIkFJxu7WHv3zBQXjiD5ZthwAcVCL1XGRcXF4YNGyaSplIs34nT7du3+eabb2jVqhWurq5UqVJF55YfBw4cYNy4cRw/fly7hpqnp6fOYqGTJk1iy5YtrFu3jgMHDvDgwQP69eunfV6tVtOjRw/S09M5evQof//9N0uXLuWzzz7TtskqodCpUyfOnz/PxIkTefPNN7XT9EuTW7ducfnyZQC8vb3FDA6hWDAzM2PAgAFYWloSExPDunXrSE8vvJVVHz6GMTMgPUNek+zdYYV2KKGMsraA2e9rmDLsLNZmMSSlWbL9rB+HQgaQkGyidHhCIcr3pbpevXoxYsQIfH19CzyYR48eYW9vz4EDB2jfvj1xcXHY2dmxcuVK/Pz8ALh27Rp16tTh2LFjvPbaa+zYsYOePXvy4MED7Vp5ixYtYurUqTx69AgjIyOmTp3Ktm3buHTpkvZYAwcOJDY2lp07d74wrpJyqS4jI4O//vqL2NhYmjVrRpcuXZQOSRB0REdHs2LFClJTU3F1dcXPz6/Ak/u0dBj4Ppy9LI9nCvxVHt8kCAUpIyODrVu3EhoaSqbagMi0gWw77IRao8LaEqa/A35eIMaEF71id6muV69eTJo0iZkzZ7J+/Xo2b96sc3sVcXFxgLz2FciFwzIyMnQSgNq1a+Pi4sKxY/JiQseOHaNBgwY6Cwx7eXkRHx+v7Xk5duxYtiTCy8tLu4/S4ujRo8TGxmJpaUm7du2UDkcQsqlQoQJ+fn4YGhoSHh7Otm3bKKBVnwC5wvOMH+WkycoCFn8lkiah4CUmJrJq1SpCQ0PR19enr083fppViU0LVdSrCXEJ8OE3MHQy3FFmMqlQiPJdAHPs2LEAfP7559mee5XB4RqNhokTJ9KmTRvq168PyGv7GBkZZbtW7ODgQGRkpLbN00lT1vNZzz2vTXx8PCkpKZiamuo8l5aWRlpamvZxfHz8S51TUYqKiuLkyZMAdO3aFWNjY4UjEoScVapUCR8fH9avX8/Vq1cxNTWlS5cuBTJde8UWWLVN/pb/03RwFROahAL26NEjAgICiI+Px9TUlH79+uHs7Aw8VThzLXy/FA6fAc9R8MFIGOkLBmLkRKmQ7x4njUaT6+1VZtSNGzeOS5cusXr16pfeR0GZPXs21tbW2lvlypWVDum5NBoNO3fuRKPRUKtWLWrWrKl0SILwXNWqVaNHD3lk7dmzZwuk9/fkv3JvE8DU0dCx5SvvUhB03Lp1i+XLlxMfH0+5cuUYNmyYNmnKYqAPYwfBrr/AvQmkpMKXC6HvOLgaplDgQoF6pUV+C8r48ePZunUr+/fv1/kldHR0JD09ndjYWJ32Dx8+xNHRUdvm2Vl2WY9f1MbKyipbbxPAtGnTiIuL096Ke8HP8+fPExERgZGRkRjXJJQYdevWxcPDA4BDhw5x/vz5l95XxCN4eyZkqqFXJ3mhVkEoSOfPn9dOaqhcuTLDhg177vqsrpXkwpnf/Fc4898QuXDm3MU5F84USg5FEydJkhg/fjwbN25k3759VK1aVef5Zs2aYWhoyN69e7XbQkJCuHPnDu7u7gC4u7tz8eJFoqKitG12796NlZWVtlq2u7u7zj6y2mTt41nGxsZYWVnp3IqrhIQEDhw4AECHDh2wtLRUOCJByLvmzZtr/x8GBQVx7dq1fO8jNR3emi4vq1K3OsydLAbkCgVHkiSCg4MJCgpCkiTq1auHv79/jl+6n6VSySUL9vwtlzDIVMMvK6Dbm3DiQhEELxSKfM+qK0jvvPMOK1euZNOmTTpr31lbW2t/Kd9++222b9/O0qVLsbKy4t133wXkgdAglyNo3LgxTk5OzJ07l8jISIYNG8abb77J119/Dcjdq/Xr12fcuHG88cYb7Nu3jwkTJrBt2za8vLxeGGdxnlW3ceNGQkNDcXJyYujQoaKsv1DiSJLErl27OH/+PPr6+vj5+eHq6prH18IHc2D9LihnBVt+g8qOhRuvUHZkZGSwbds2bX3BNm3a0KZNm5f+O7vzkFw4M+qx/Hhob/myspVFQUUsQDGsHF6gB8/ll2/JkiWMGDECkAtgfvDBB6xatYq0tDS8vLz49ddftZfhQK4t9fbbbxMcHIy5uTnDhw9nzpw5GBj8f+x7cHAwkyZN4sqVKzg7OzN9+nTtMV6kuCZOoaGhbNy4ET09PUaMGIGdnZ3SIQnCS9FoNGzatInQ0FCMjIwYNGiQzv/x3PwVALN+AX09+GcutG1WBMEKZUJSUhIbNmzgwYMH6Onp0a1bN+3EpVcRlwizF8mTGAAcK8CXE0WB1oJUohKnnGaolQbFMXFKS0tj8eLFJCYm8tprr9GhQwelQxKEV5KZmUlAQAC3b9/GzMyMIUOGaEuT5OTIWRg2GdQa+GwcjPIrwmCFUi06OpqAgADi4uIwMTGhb9++uLi4FOgxjp2Hj76D8Pvy4x4dYNYEsMv9V17Io2JXx2nChAk5bk9KSqJ79+6vHJCQNwcPHiQxMREbGxtat26tdDiC8MoMDAzo27cvDg4OJCcns2bNGhISEnJsezcSxs2Sk6Z+XeGNgq/HK5RRt2/fZvny5cTFxWFjY8PQoUMLPGkCcG8MQX/CO4PlHtNtB8BjBKzdIV+CFoqvfCdO27ZtY8aMGTrbkpKS8Pb2zrY+nFA4Hjx4wNmzZwG5kKehoaHCEQlCwTA2NqZ///6UK1eO+Ph41q5dS2pqqk6b5BR461N4Eg8N3WD2B2IwuFAwLl68yNq1a0lLS6NSpUoMGzaM8uXLF9rxTIzlMU6bF0H9/wpnTp4LQz4UhTOLs3wnTrt27eKPP/7ghx9+AORZXV27dkWlUuVp+RLh1ajVau37XL9+/TwPohWEksLc3JwBAwZgYWGhvWSSkZEByN/Ep3wLV8KgQjn47XP5w0cQXoUkSRw8eJDt27ej0WioXbs2AwcOxMysaMrO168JmxbCtDFgbCRfhu76Bvy+Vp6JJxQv+U6cqlevzs6dO/niiy/48ccf8fT0xMjIiB07dmBubl4YMQpPOXXqFI8ePcLU1JROnTopHY4gFApra2v8/f0xNjbm/v37bNq0CbVazW+rYct+ucjgwpngZK90pEJJl5mZyZYtW7RFWN3d3endu7fO5KKiYKAv1x/b9Re0bgKpafDVf4Uzr9wo0lCEF3jpweHHjh2ja9eutGrViq1bt5bKQeFZisvg8CdPnvDXX3+RmZlJjx49CmSGhyAUZ/fu3WPNmjXyMACzjizc2BJJUvHlRBjWR+nohJIuOTmZDRs2cP/+ffT09PDy8qJhw4ZKh4UkyWOdvlwI8YnyGKixA2HCcDAxUjq64q9YzKpr0qRJjqUDbt++jb29vU7SlDX2pjQpDomTJEmsXbuW8PBwqlSpwoABA0TNJqFMuHHjBn8t20fA0ddJyzRhYHeJOR+qxLgm4ZXExMSwbt06YmNjMTY2pm/fvlSpUkXpsHQ8fAwzf4Ltco1jqlWG2e/Da40VDavYK+zP7Dz1Rfr4+BT4gYX8uXLlCuHh4RgYGODl5SWSJqHMcHSqwaFQZ9IyTXCwuU+v1vdRqcRCdMLLu3v3Lhs2bCA1NRVra2v8/PyoUKGC0mFl41BeviQddBg+/QFu3oUBk2BwT3k8lCicqQxFC2CWFEr3OKWkpPDHH3+QkpJC+/btc10qRhBKG40Gxs6QPzjKWabTu9nvmJsk0a1bt2JxSUUoeS5fvqwdBF6xYkV8fX1LxPjcuET45ndYsUV+7FABvngPvNoqG1dxVOzqOJ06dYoTJ05k237ixAlOnz5dIEEJuvbt20dKSgoVKlSgZUvxTVsoO35aLidNRobw1xwjOneQx/Xt3LmT0NBQhaMTShJJkjh8+DBbt25Fo9Hg5ubGoEGDSkTSBGBtAV+/D2vmQ1VneBgtr9H49kyIilE6urIl34nTuHHjuHv3brbt9+/fZ9y4cQUSlPB/t2/f5tKlSwB4e3ujr6+vcESCUDR2H4Hvl8j3v5wITevKC1k3aNAASZLYvHlzjn+LBOFZmZmZbNu2jSNHjgDQqlUr+vTpUyJr4L3WGHYuhnH/Fc7cfgA8hsOa7aJwZlHJd+J05coVmjZtmm17kyZNuHLlSoEEJcgyMzMJCgoC5Pe3UqVKCkckCEXjxh2YKK/Rzes+MOC/RQlUKhXe3t7UqFEDtVpNQEAAUVFRisUpFH8pKSmsXbuWy5cvo1Kp8PLyomPHjiV6nKiJMUwZLS9q3aCWPPNuyrcw+AO4fV/p6Eq/fCdOxsbGPHz4MNv2iIiIIq97UdodPXqUJ0+eYGFhIdaiE8qMuEQY/SkkJkOrhvI6dE/T09Ojd+/eODs7k56eztq1a4mNjVUkVqF4e/LkCcuXL+fu3bsYGRnRv39/GjdurHRYBaZeDQj8FT4eKydTR8+B5yhYtFoUzixM+U6cPD09mTZtGnFxcdptsbGxfPzxx3Tt2rVAgyvLHj16pB1L1qVLF4yNRXlkofRTq+G9L+XZQ0728OtMMMzh+5ihoSG+vr7Y2dmRlJTEmjVrSExMLPJ4heLr3r17LFu2jJiYGKysrBg6dChVq1ZVOqwCZ6APYwbArj+hTVO5cObs36DP23BZFM4sFPlOnL777jvu3r1LlSpV6NSpE506daJq1apERkYyb968woixzJEkiaCgIDQaDTVq1KBWrVpKhyQIReL7pbD/hLzsxO+fy8uq5MbExAR/f39sbGyIjY1l3bp1pKWlFVmsQvF15coVVq9eTUpKCo6OjgwbNgw7OzulwypUVSrBiu/g2ylymYJL16HXGPjmDzmZEgpOvhOnSpUq8e+//zJ37lzq1q1Ls2bNWLBgARcvXqRy5cqFEWOZc/78ee7fv4+RkZF2HUBBKO22HYCfl8v3v/kQGri9+DUWFhb4+/tjbm5OVFQU69evF4uNl2GSJHHs2DG2bNmCWq2mZs2aDBo0CAuLslHwSKUC/26w92/o0QHUGvh1JXi/CcfPKx1d6SHqOOVBUdZxSkhIYPHixaSnp+Ph4UHz5s0L9XiCUBxcuymvyZWcCqP94dO38/f6hw8fsnLlStLT06lZsyY+Pj7o6eX7e6FQgqnVaoKCgrh48SIALVq0oGPHjmX692DXYfh0gVy6AOTCmR+NkUsblGbFro4TQFhYGO+++y5dunShS5cuTJgwgbCwsIKOrUzau3cv6enpVKxYMcfZi4JQ2sTGw+jpctLUthl89Fb+9+Hg4ICvry/6+vpcv36doKAgxHfCsiM1NZV169Zx8eJFVCoVnp6edO7cuUwnTQCebWHPEhjSS368cit0GQE7DykaVomX79+qoKAg6taty8mTJ2nYsCENGzbkxIkT1KtXj927dxdGjGXGjRs3CAkJ0U65Luv/6YXSL1MN47+AOw+gckX4ebo82PVluLi40Lt3b1QqFf/++y8HDx4s2GCFYik2Npbly5dz+/ZtjIyM8PX1pUmTJkqHVWxY/Vc4c+0P8lp3UY9hzGdyRf6Hj5WOrmTK96W6Jk2a4OXlxZw5c3S2f/TRR+zatUss8vuS0tPTWbx4MQkJCbRq1YqOHTsWynEEoTj5ahH8vgZMTWDjz1Cn+qvv88KFC+zcuROATp06iWr7pdj9+/fZsGEDycnJWFhY0L9/f+zt7ZUOq9hKTYcf/4Hf/itXYGUBn4yV66SVpqG0hf2Zne/EycTEhIsXL1KzZk2d7aGhoTRs2JDU1NQCDbA4KKwfgkaj4d69eyQmJhISEkJoaCjW1taMGjWqRFa0FYT8CNwD730l3/91BvToWHD7Pn78OAcOyEvK9+jRg/r16xfczoVi4dq1a2zbto3MzEztpVpLS0ulwyoRrtyQC2Ze/G/VIvcmMOcDcC0lNZYLO3HKd8VKOzs7zp8/ny1xOn/+vMj08yEkJIS9e/eSkJCgs71u3boiaRJKvYuh8h9ugHFDCjZpAnlJjaSkJE6fPs327dsxNTWlevUC6M4SFCdJEidPniQ4OBiA6tWr07t3b4yMjJQNrASp+1/hzCXr4bu/4Ng58HwD3h8Bb/q//OXysiLfg2hGjx7NW2+9xTfffMOhQ4c4dOgQc+bMYcyYMYwePbowYix1QkJCCAwMzJY0ARw7doyQkBAFohKEohH9RB5jkZYOnVrBByML/hgqlYrOnTtTr149JEkiMDCQe/fuFfyBhCKVNXMuK2lq1qwZ/fr1E0nTSzDQl2ew7v5LnpSRlg6zf4feb8s1oITc5ftSnSRJ/PDDD8ybN48HDx4A4OTkxOTJk5kwYUKprDlUkN1+Go2GRYsW5Zg0ZbG0tGTs2LFicLhQ6mRkwtAP4fgFeYX3TQsLd2q0Wq1mw4YN3Lx5E2NjY4YMGVLqCyGWVmlpaQQGBhIeHq5NjEW5loIhSRAQBF/8CnEJ8uLBbw2AicPlpVxKmmI3xulpWR/+pf26ckH+EO7cucOqVate2G7QoEG4uLi80rEEobiZ8SMs3QgWZvKlgppVCv+Y6enprFmzhgcPHmBhYcHQoUOxtrYu/AMLBSYuLo6AgACio6MxNDSkd+/e1KhRQ+mwSp1HMTDzJ9gaLD92rQRzPgT3xkpGlX/Fso5TFktLy1KfNBW0vK6nJdbdEkqbtTvkpAlg/rSiSZoAjIyM8PPzo0KFCiQmJrJmzRqSkpKK5uDCK4uIiGDZsmVER0djYWHB4MGDRdJUSOxs4ZcZsPhLcKwA4fdh4CSY+p28+LYgy9Pg8CZNmuT5ElxpLEdQkPJa+r+sLBEglA3nrsAn8+X774+QC/MVJVNTU/z9/Vm+fDlPnjwhICCAgQMHisWzi7nQ0FC2bNlCZmYmdnZ2+Pn5FfrqDQJ0bQOtGsnr3C3fDKu3wd5j8MV70K290tEpL0+Jk4+PTyGHUXY4OztjaWn5wjFOzs7ORRiVIBSeh49hzAxIzwCvtvDuMGXisLS0xN/fnxUrVhAZGcnGjRvx8/PDwCDfk4uFQiZJEqdPn2bfvn0AVK1alT59+ohEtwhZWcBXk6CPh9zjdPOuXDTTux18PgEcKigdoXLEWnV5UNDXS7Nm1eXGx8cHN7c8rHAqCMVcWjoMfB/OXpYvzQX+Ko9vUlJERASrV68mPT0dNzc3evfuLSZiFCMajYY9e/Zw7tw5ABo3bkzXrl3Fz0hBqenw0zJYtOq/wpnm8PFYGNijeBbOLHZjnE6dOsWJEyeybT9x4gSnT58ukKBKOzc3N3x8fLKND7O0tBRJk1CqzPxJTpqszOGPL5VPmgAqVqxI37590dPTIyQkhN27d4t17YqJtLQ01q9fr02aOnXqhKenp0iaFGZiBJNHwdbfoFFtiE+Cj+bJX4rC7ysdXdHLd49Ty5YtmTJlCn5+fjrbN2zYwDfffJNjUlXSFUXlcAsLC5ydncUfCKHUWLEZPp4vfyNdOgc6FrOVT65du8amTZsAaN26Ne3atVM4orItPj6e9evXExUVhYGBAb169aJWrVpKhyU8Q62GJRvkwpkpqWBsBJNGyDWhikvhzGJXjsDCwoJ///2XatWq6Wy/desWDRs2fO7YnZKqKNaqE4TS5NRFGPS+XLfpo7fg7UFKR5Szc+fOsWvXLgC6dOlCs2bNFI6obHr48CEBAQEkJiZibm6Or68vFStWVDos4TnuRMDH38Oh/y401asJ33wIDYpBrlvsLtUZGxvz8OHDbNsjIiLEIEtBEIh4BG/PkJOmnh1h7EClI8pdkyZNaNtWnuK3Z88erly5onBEZc+NGzdYsWIFiYmJVKhQgWHDhomkqQRwqQjL5sK8qWBtCZevQ5+3YfZvck9UaZbvxMnT05Np06YRFxen3RYbG8vHH39M165dCzQ4QRBKltR0eGs6PHoCdarBt1OK5+DRp7Vu3ZqmTZsCsG3bNm7duqVwRGXHmTNn2LBhAxkZGbi6uoripCWMSgV+3rB3KfTqBGoNLFoNXqPgSCmuTJTvS3X379+nffv2PH78mCZNmgDyAr8ODg7s3r2bypUrF0qgShKX6gThxSQJPvxGXrrBxgq2LJK/lZYEkiSxefNmrl27hqGhIQMHDsTJyUnpsEotjUbDvn37OHPmDAANGzbE09MTff1iMkhGeCl7jsr12iKj5ccDusMnY+UeqaJU7MY4ASQlJbFixQouXLiAqakpDRs2ZNCgQRgaGhZ4gMWBSJwE4cWWrIeZP8vrXP0zV144tCRRq9UEBAQQHh6OqakpgwcPpkKFMlysppCkp6ezZcsWbty4AUDHjh1p2bJlqVzntCxKSJILZy6T511gZyvXfereoehiKHZjnADMzc156623+OWXX/juu+94/fXXXyppOnjwIL169cLJyQmVSpWtttGIESNQqVQ6N29vb502MTExDBkyBCsrK2xsbBg1alS25Ur+/fdf2rVrh4mJCZUrV2bu3Ln5jlUQhNwdPScvEArwydslL2kC0NfXp2/fvlSsWJGUlBTWrl1LfHy80mGVKgkJCaxcuZIbN26gr69Pnz59aNWqlUiaShFLc/hyIgT8CNUry+vfvT0T3voMHv7XE6VWw7HzsGmv/K9arVy8L+OV5r5bWVlx8+bNl359UlISjRo14pdffsm1jbe3NxEREdrbswvkDhkyhMuXL7N79262bt3KwYMHeeutt7TPx8fH4+npSZUqVThz5gzffvstM2fO5Pfff3/puAVB+L+7kfDOTHl8Q7+u8Iav0hG9vKx17WxtbUlISGDt2rWkpKQoHVapEBUVxbJly3j48CFmZmYMGjSI2rVrKx2WUEhaNIDti+WVAgz0IegQdBkhX8prPUheA2/Cl/K/bQbBjoNKR5x3r1Q53NLSkgsXLmQrTfBSgahUbNy4UWd5lxEjRhAbG5trle2rV69St25dTp06RfPmzQHYuXMn3bt35969ezg5ObFw4UI++eQTIiMjMTIyAuCjjz4iMDCQa9eu5Sk2calOEHKWkgr9xsOVMHkacsCPYFIKVsWIj49n2bJlJCYmUrFiRQYOHKj9+yHk382bN9m0aRPp6enY2trSv39/bGxslA5LKCLXbsLUb+F8Lh+5Wf2NC2cVzFp4xfJSXVEKDg7G3t4eNzc33n77bR4/fqx97tixY9jY2GiTJpBrsejp6WkLcR47doz27dvr/NHz8vIiJCSEJ0+e5HjMtLQ04uPjdW6CIOiSJJjyrZw0lbeB378oHUkTyL3pAwYMwMTEhIiICAIDA1GXtOsJxcS5c+cICAggPT0dFxcXhg0bJpKmMqZ2NVi3QF5BICdZvTezfi4Zl+1eKXEaOnRoofbAeHt7888//7B3716++eYbDhw4QLdu3bR/wCIjI7G3t9d5jYGBAba2tkRGRmrbODg46LTJepzV5lmzZ8/G2tpaeyuNMwUF4VX9tgY275O74RfNAif7F7+mJKlQoQJ+fn4YGhpy69Yttm3bJpZmyQdJkti3bx+7du1CkiTq16+Pv78/JiYmSocmKODMFXmpltxIyDXgTl4sspBe2itVrFy4cGFBxZGjgQP/XzmvQYMGNGzYkOrVqxMcHIyHh0ehHXfatGm8//772sfx8fEieRKEpxw4Kc+cAZj5LrRsqGw8haVSpUr4+Piwfv16rl69ipmZGR4eHmIw8wtkZGSwdetWQkNDAWjXrh3u7u7ifSvDoh6/uE1+2ikpT4nTjz/+mOcdTpgw4aWDeZFq1apRoUIFbty4gYeHB46OjkRFRem0yczMJCYmBkdHRwAcHR2zVTrPepzV5lnGxsYYG5eSaw6CUMDC78P4L0CjkVdHH9pb6YgKV7Vq1ejRowdbtmzhzJkzmJmZ0bp1a6XDKrYSExPZsGEDERER6Ovr0717d+rWrat0WILC7MsXbDsl5Slxmj9/fp52plKpCjVxunfvHo8fP9aW43d3dyc2NpYzZ85o15jat28fGo2GVq1aadt88sknZGRkaEsm7N69Gzc3N8qVK1dosQpCaZSYDKM/hfhEaFJXrs9SFjoR6tatS3JyMnv37uXQoUOYmZnRuHFjpcMqdh49ekRAQADx8fGYmprSr18/nJ2dlQ5LKAZaNoCKdhD56P9jmp6mAhzt5HbFXZ4Sp8JagiAxMVFbBC3rOOfPn8fW1hZbW1tmzZqFr68vjo6OhIWFMWXKFGrUqIGXlxcAderUwdvbm9GjR7No0SIyMjIYP368TtXfwYMHM2vWLEaNGsXUqVO5dOkSCxYsyHMyKAiCTKOB9+dAaLj8rfC3WfLK6GVF8+bNSU5O5tixY+zatQtTU1Pc3NyUDqvYCA8PZ+PGjaSnp1OuXDn69+8vvpwKWvr6MGO8vI6lCt3kKeu714zxcrvi7pXKEbyq4OBgOnXqlG378OHDWbhwIT4+Ppw7d47Y2FicnJzw9PTkiy++0BnsHRMTw/jx49myZQt6enr4+vry448/YmFhoW3z77//Mm7cOE6dOkWFChV49913mTp1ap7jFOUIBAF+XAbz/gIjQ1jzAzQtg1dfJEkiKCiICxcuoK+vT//+/alSpYrSYSnuwoUL7Nq1C41Gg7OzM/369cPU1FTpsIRiaMdBefZcxKP/b6toJydNBVGKAIrJkitPD5R+ke+///6VAiqOROIklHV7jsKoT+T7cyfLa1CVVRqNhk2bNhEaGoqRkRGDBg3KdbxkaSdJEgcPHuT48eOAfEmzW7duGBi80rwjoZRTq+XZc1GP5d7rlg0KtqepsD+z8/Tbfe7cuTztTMyYEITS58YdeO8r+f7rPmU7aQLQ09OjV69erFu3jjt37rBu3TqGDBmCra2t0qEVqYyMDLZt20ZISAgAbdq0oU2bNuJzQHghfX1wb6x0FC9P0Ut1JYXocRLKqvhE6PMO3LwLrRrCinlgKDoTALlQ7qpVq3j48CHW1tYMGTIES8siXgZeIUlJSWzYsIEHDx6gp6dHt27dqF+/vtJhCQIgKocLgqAQjUbuabp5Vx6D8OtMkTQ9zdjYWDsAOi4ujrVr15Kamqp0WIUuOjqaZcuW8eDBA0xMTBgwYIBImoQy5aX+DJ4+fZq1a9dy584d0tPTdZ7bsGFDgQQmCIKyvl8K+47LM+f++AIqiAlS2Zibm+Pv78/y5cuJjo4mICCAAQMGaEuflDa3b99m48aNpKWlYWNjg5+fH+XLl4DCO4JQgPLd47R69Wpat27N1atX2bhxIxkZGVy+fJl9+/ZhbW1dGDEKglDEth+An5bJ9+d8CA3ErPtc2djY4O/vj7GxMffv32fTpk2lcl27ixcvsnbtWtLS0qhUqRLDhg0TSZNQJuU7cfr666+ZP38+W7ZswcjIiAULFnDt2jX8/f1xcXEpjBgFQShC127CB3Pk+2/2h35dlY2nJLC3t8fPzw8DAwPCwsLYuXNnqVnXLmvm3Pbt29FoNNSuXZuBAwdiZmamdGiCoIh8J05hYWH06NEDACMjI5KSklCpVEyaNInff/+9wAMUBKHoxMbD6OmQnAptmsK0MUpHVHI4OzvTp08fVCoVly5dIjg4WOmQXllmZiZbtmzh2LFjgLwSQ+/evUW5AaFMy3fiVK5cORISEgB5AcxLly4BEBsbS3JycsFGJwhCkclUy2vQ3XkAlSvCL5+BQQmo4luc1KhRg27dugFw8uRJTpw4oXBELy85OZk1a9Zw9epV7cy59u3bi3IDQpmX768N7du3Z/fu3TRo0ID+/fvz3nvvsW/fPnbv3o2Hh0dhxCgIQhGY+wccOg2mJvJg8HJiyOJLadCgAcnJyQQHBxMcHIypqSkNGzZUOqx8iYmJISAggCdPnmBsbEzfvn1FhXRB+E++E6eff/5ZO+X2k08+wdDQkKNHj+Lr68unn35a4AEKglD4AvfAb2vk+/OmQp3qysZT0rVq1Yrk5GROnjzJzp07MTU1pWbNmkqHlSd3795lw4YNpKamYm1tjZ+fHxUqVFA6LEEoNkQBzDwQBTCF0uxiKPi+C2npMG4wTBmtdESlgyRJ7Nixg4sXL2JgYIC/vz+VK1dWOqznunz5Mjt27ECtVlOxYkV8fX0xNzdXOixByJdiVwDz9ddfZ8mSJYSFhRV4MIIgFK3HsTDmMzlp6tgKPnhD6YhKD5VKhbe3NzVq1CAzM5P169cTFRWldFg5kiSJI0eOsHXrVtRqNW5ubgwaNEgkTYKQg3wnTkZGRsyePZuaNWtSuXJlhg4dyuLFi7l+/XphxCcIQiHJyIR3ZsH9h1DVGX78tGAX2hTkde169+6Ns7MzaWlprF27ltjYWKXD0pGZmcm2bds4fPgwIF9m7NOnT6kt4ikIr+qlL9Xdv3+fgwcPcuDAAQ4cOEBoaCgVK1bk3r17BR2j4sSlOqE0mvkTLNkAFmYQ+CvUFGN/C01qaiorV67k0aNH2NjYMHTo0GLRm5OSksLGjRu5e/cuKpUKT09PGjdurHRYgvBKit2luizlypWjfPnylCtXDhsbGwwMDLCzsyvI2ARBKCRrd8hJE8D8aSJpKmwmJib4+/tjbW1NbGystgK3kp48ecLy5cu5e/cuRkZG9O/fXyRNgpAH+U6cPv74Y1q3bk358uX56KOPSE1N5aOPPiIyMpJz584VRoyCIBSgc1fgk/ny/UkjwLOtouGUGRYWFgwYMAAzMzOioqJYv349mZmZisRy7949li1bRkxMDFZWVgwdOpSqVasqEosglDT5vlSnp6eHnZ0dkyZNol+/ftSqVauwYis2xKU6obSIioGeY+BhNHi2gd8+B72X7ncWXsbDhw9ZuXIl6enp1KxZEx8fH/SK8Idw9epVtm3bhlqtxtHREV9fXywsLIrs+IJQ2Irdpbpz587xySefcPLkSdq0aUOlSpUYPHgwv//+O6GhoQUeoCAIBSM9A8Z+JidNNarA99NE0qQEBwcHfH190dfX5/r16wQFBRXJunaSJHHs2DE2b96MWq2mZs2aDBo0SCRNgpBPr1zH6cKFC8yfP58VK1ag0WhK5argosdJKA0+/h5WbAErc9i8SJ5JJygnJCSETZs2IUkS7u7utG/fvtCOpVarCQoK4uLFiwC0aNGCjh07FmlPlyAUlcL+zM535XBJkjh37px2OYHDhw8THx9Pw4YN6dChQ4EHKAjCq1uxWU6aVCr4cbpImooDNzc3PD09CQoK4tixY5iamtKiRYsCP05qaiqBgYHcvn0blUpFly5daNq0aYEfRxDKinwnTra2tiQmJtKoUSM6dOjA6NGjadeuHTY2NoUQniAIr+rURZjxk3x/ypvQqZWy8Qj/17hxY1JSUjh48CD79u3DzMyMevXqFdj+Y2NjCQgI4PHjxxgZGdG7d2+qVxfr6QjCq8h34rR8+XLatWsnLlkJQgkQ8QjeniEXu+zRAd4epHREwrNee+01kpOTOX36NNu3b8fExKRAkpsHDx6wfv16kpOTsbCwoH///tjb2xdAxIJQtuX7AnePHj1E0iQIJUBqurycyqMnUKcafDdVvlQnFC8qlYrOnTtTt25dNBoNgYGB3L9//5X2GRISwqpVq0hOTsbBwYHXX39dJE2CUEDEyEBBKIUkCT75Hi5cAxsr+P1LMDNVOiohNyqViu7du1OtWjUyMzMJCAjg0aNH+d6PJEmcOHGCwMBAMjMzqV69OoMHD8bS0rIQohaEskkkToJQCi3dAAFBcrmBXz4Dl4pKRyS8iL6+Pn369MHJyYnU1FTWrl1LXFxcnl+fNXMuODgYgGbNmtGvXz+MjIwKKWJBKJtE4iQIpczRc/DFr/L9T8ZC22bKxiPknZGREX5+fpQvX57ExETWrl1LcnLyC1+XlpZGQEAAFy5cQKVS4eHhQZcuXUS5AUEoBOJ/lSCUIncj4Z2ZoNZA364wyk/piIT8MjU1xd/fHysrK2JiYli3bt1z17WLj49n+fLlhIeHY2hoSL9+/WjevHkRRiwIZYtInAShlEhJhbemw5N4aFAL5nwgBoOXVFZWVvj7+2NqakpkZCQbN24kPT2dO3fucOXKFe7cuYNGoyEiIoJ//vmH6OhoLCwsGDx4MDVq1FA6fEEo1V65cnhZICqHC8WdJMGEL2HzPihvA1t/AycxiarEi4iIYNWqVWRkZGBgYKCzKLCJiQnp6eloNBrs7Ozw8/MTf58EgWK4Vp0gCMXPb2vkpMlAHxbOFElTaVGxYkVtNfGnkyaQK4JrNBrs7e0ZMmSISJoEoYiIxEkQSrgDJ+GbP+T7M8ZDq0bKxiMUHI1Go11fLjcpKSkYGhoWUUSCIIjESRBKsPD7MP4L0GhgQHcY1kfpiISCdO/ePRISEp7bJiEhgXv37hVRRIIgiMRJEEqopBQY/SnEJ0KTuvDFe2IweGmTmJhYoO0EQXh1InEShBJIo4H3Z0NoONiXh99mgbGoc1jqWFhYFGg7QRBenUicBKEE+nkF7DwERobw2+fgUEHpiITC4Ozs/MLlUiwtLXF2di6iiARBUDRxOnjwIL169cLJyQmVSkVgYKDO85Ik8dlnn1GxYkVMTU3p0qUL169f12kTExOjnVFiY2PDqFGjsnVb//vvv7Rr1w4TExMqV67M3LlzC/vUBKHQ7DkK3y+R73/+HjStq2w8QuHR09PDw8PjuW08PDxEhXBBKEKK/m9LSkqiUaNG/PLLLzk+P3fuXH788UcWLVrEiRMnMDc3x8vLi9TUVG2bIUOGcPnyZXbv3s3WrVs5ePAgb731lvb5+Ph4PD09qVKlCmfOnOHbb79l5syZ/P7774V+foJQ0G7cgYlfy3WbhvWBQT2UjkgobG5ubvj4+GTrebK0tMTHxwc3NzeFIhOEMkoqJgBp48aN2scajUZydHSUvv32W+222NhYydjYWFq1apUkSZJ05coVCZBOnTqlbbNjxw5JpVJJ9+/flyRJkn799VepXLlyUlpamrbN1KlTJTc3tzzHFhcXJwFSXFzcy56eILyyuARJ6jRMklw6SpLfBElKS1c6IqEoqdVq6fbt29Lly5el27dvS2q1WumQBKFYKuzP7GLbv3vr1i0iIyPp0qWLdpu1tTWtWrXi2LFjABw7dgwbGxuddZmyFrY8ceKEtk379u11Vgj38vIiJCSEJ0+e5HjstLQ04uPjdW6CoCSNRu5pCrsLFe3kIpdGonRPmaKnp4eLiwt169bFxcVFXJ4TBIUU2/95kZGRADg4OOhsd3Bw0D4XGRmJvb1uiWQDAwNsbW112uS0j6eP8azZs2djbW2tvVWuXPnVT0gQXsH3S2HvMXnm3B9fQIVySkckCIJQNhXbxElJ06ZNIy4uTnu7e/eu0iEJZdj2A/DTMvn+nA+ggRjSIgiCoJhimzg5OjoC8PDhQ53tDx8+1D7n6OhIVFSUzvOZmZnExMTotMlpH08f41nGxsZYWVnp3ARBCSG34IM58v1RftDPU9l4BEEQyrpimzhVrVoVR0dH9u7dq90WHx/PiRMncHd3B8Dd3Z3Y2FjOnDmjbbNv3z40Gg2tWrXStjl48CAZGRnaNrt378bNzY1y5cT1DqH4io2HNz+F5FRo0xQ+Hqt0RIIgCIKiiVNiYiLnz5/n/PnzgDwg/Pz589y5cweVSsXEiRP58ssv2bx5MxcvXuT111/HyckJHx8fAOrUqYO3tzejR4/m5MmTHDlyhPHjxzNw4ECcnJwAGDx4MEZGRowaNYrLly+zZs0aFixYwPvvv6/QWQvCi2Wq4d0v4M4DcHaEXz4DA32loxIEQRAULUewf/9+Cch2Gz58uCRJckmC6dOnSw4ODpKxsbHk4eEhhYSE6Ozj8ePH0qBBgyQLCwvJyspKGjlypJSQkKDT5sKFC1Lbtm0lY2NjqVKlStKcOXPyFacoRyAUta8WymUH3Lwl6fJ1paMRBEEoOQr7M1slSZKkYN5WIsTHx2NtbU1cXJwY7yQUuk17YcKX8v1fPoOenZSNRxAEoSQp7M/sYjvGSRDKokvXYcq38v13BoukSRAEobgRiZMgFBOPY+Gt6ZCaBh1bwYdvKB2RIAiC8CyROAlCMZCRCe/MgvsPoaoz/Pgp6IvB4IIgCMWOSJwEoRj4aiEcPw/mpvD7F2BtoXREgiAIQk5E4iQIClu3E5ZskO/P/xhquSoajiAIgvAcInESBAWdvwqffC/fnzgcvNoqG48gCILwfCJxEgSFRMXAmM8gLQM828B7rysdkSAIgvAiInESBAWkZ8DbMyAyGmpUge+ngZ743ygIglDsiT/VgqCAmT/B6UtgZQ5/fAGW5kpHJAiCIOSFSJwEoYit2AwrtoBKBQs+hWqVlY5IEARByCuROAlCETp1EWb8JN+fPAo6v6ZsPIIgCEL+iMRJEIpI5CN5XFNGJvToIC+pIgiCIJQsInEShCKQmg5vfQaPnkDtavDdVPlSnSAIglCyiMRJEAqZJMm1mi5cAxsreTC4manSUQmCIAgvQyROglDIlm6EgCC53MAvn4GLk9IRCYIgCC9LJE6CUIiOnYcvfpHvfzwG2jZTNBxBEAThFYnESRAKyb1IeGcmqDXQtyu82V/piARBEIRXJRInQSgEKakwejrExEH9mjDnAzEYXBAEoTQQiZMgFDBJginfwpUbUN4Gfv8CTIyVjkoQBEEoCCJxEoQC9vta2LwPDPRh4Uyo5KB0RIIgCEJBEYmTIBSgg6dgzu/y/c/GQatGysYjCIIgFCyROAlCAbl9H8Z/ARoNDOgOr/soHZEgCIJQ0ETiJAgFICkF3vwU4hKgSV344j0xGFwQBKE0EomTILwiSYL3Z0NoONjZwm+zwNjof+3de3xMd/oH8M9kcjG5TCLXGeQiLgmKFSqGdnVXWtRal25rs1lNS7s/xCW1tVRVpBa9KKur2u1l2SpVqkpp3CUu6xJ3KSIICRKpSxJTlcvM8/tjNqdG0IlGJjM+79drXp055ztnnienbT455zvn2LsqIiK6HxiciH6huYuAtdsAN1fgX68DIYH2roiIiO4XBieiX2DTTuCdf1ueT00GOraxazlERHSfMTgR3aOTecCYaZZTdYP7AfF97F0RERHdbwxORPeg1Aj8ZRJw7QegczvLpQeIiMj5MTgR1ZDZDCRPB07lA/ogYF4K4O5m76qIiKguMDgR1dDsBZa5TR5ultupBPnbuyIiIqorDE5ENZC2FXh3oeX5Gy8D7aLsWw8REdUtBiciG2XnWq7XBABD/wAMfMK+9RARUd1jcCKyQXGp5crg128AXTsAE4fZuyIiIrIHBiein2EyAaOmAnkXgCY64L0UwFVt76qIiMge6nVwmjJlClQqldUjOjpaWX/jxg0kJSUhICAA3t7eeOqpp3Dx4kWrbeTl5aFPnz7w9PREcHAwxo0bh8rKyrpuhRzYWx8DW/cCmgbAR1MBf197V0RERPbiau8Cfk6bNm2wceNG5bWr608lv/TSS1izZg2WLVsGX19fjBw5EgMHDsSOHTsAACaTCX369IFOp8N///tfFBQU4Nlnn4WbmxumT59e572Q41m1GfhgieX5zL8BrZvbtx4iIrKveh+cXF1dodPpqi0vKSnBJ598gsWLF+O3v/0tAGD+/Plo1aoVdu3ahS5dumD9+vU4evQoNm7ciJCQEPzqV7/C1KlTMX78eEyZMgXu7rwTK91ZVg4w7i3L8xF/An73G/vWQ0RE9levT9UBQE5ODho1aoTIyEgkJCQgLy8PALBv3z5UVFQgLi5OGRsdHY2wsDDs3LkTALBz5060bdsWISEhypiePXuitLQU3333Xd02Qg7lcjHwl9eAG2VA987Ay0PsXREREdUH9fqIU2xsLBYsWICoqCgUFBQgNTUVjz76KLKyslBYWAh3d3f4+flZvSckJASFhYUAgMLCQqvQVLW+at2dlJWVoaysTHldWlpaSx2RI6ioBJJeB85fBCIaA/+cBKg5GZyIiFDPg1Pv3r2V5+3atUNsbCzCw8OxdOlSaDSa+/a5M2bMQGpq6n3bPtVv0z8Adh4AvDSWyeC+PvauiIiI6ot6f6ruZn5+fmjZsiVOnjwJnU6H8vJyFBcXW425ePGiMidKp9NV+5Zd1evbzZuq8sorr6CkpER55Ofn124jVG99uRb493LL89kTgZZN7VsPERHVLw4VnIxGI06dOgW9Xo+OHTvCzc0NmzZtUtZnZ2cjLy8PBoMBAGAwGHDkyBEUFRUpYzZs2ACtVovWrVvf8XM8PDyg1WqtHuT8Dh4DJs6yPE9OBHo+Yt96iIio/qnXp+pefvll9O3bF+Hh4bhw4QJSUlKgVqsRHx8PX19fDB06FGPHjoW/vz+0Wi1GjRoFg8GALl26AACeeOIJtG7dGoMHD8Zbb72FwsJCTJo0CUlJSfDw8LBzd1SfFF0B/m8yUFYBPN4VGPOsvSsiIqL6qF4Hp3PnziE+Ph6XL19GUFAQHnnkEezatQtBQUEAgNmzZ8PFxQVPPfUUysrK0LNnT8ybN095v1qtxurVqzF8+HAYDAZ4eXkhMTERr7/+ur1aonqovAIYngIUXgKahVlO0bk41LFYIiKqKyoREXsXUd+VlpbC19cXJSUlPG3nhF6dDXy2CtB6ASvfByJD7V0RERHdq/v9O5t/V9MDbfE3ltCkUgFzJjE0ERHR3TE40QMr8wgw+V3L83FDgd92sW89RERU/zE40QOp8HvLvKaKSuDJ7pZbqhAREf0cBid64Nwot3yD7vurQHQkMHO85VQdERHRz2FwogeKCDBpNnDwOOCntVwZ3Ov+XYSeiIicTL2+HAHRL2UyAXuOAEWXgeAA4NgpYNlay+UG3psMhDWyd4VERORIGJzIaaVtBVLnAgXfV1838f+ARzrWfU1EROTYGJzIKaVttUz+vtNFypqE1Gk5RETkJDjHiZyOyWQ50nSn0KQCkPqeZRwREVFN8IgTOYUffgTyC4C8C8DWvbc/PVdFYFm/5whg+FVdVUhERM6AwYkcgtkMXLwMnD0P5BX873HB8s/8AuDS1Zpvs+hy7ddJRETOjcGJ6g3j9f8dNbopFOUVAPkXgPxCy81478ZPC4TpAc8GwK5DP/95wQG1UzcRET04GJyozphMlqNGeReAsxd+CkhVYely8d3f76oGGodYLiEQpv/fo5HlEaoHfL1/+pxu8Zarg99unpMKgC4I6Ny2lhskIiKnx+BEtcp43fpo0c2n087ZcNSoodY6GIU2+ikg6YMs4ennqNVAykjLt+pUsA5PVRcITxlpGUdERFQTDE5UIyYTUHjpltNpF356fqXk7u93VQNNdD+FoVA9EF511EgHaL1rp87evwbeT61+HSddkCU09f517XwOERE9WBicqJprP1QPRlWn084VWm6Mezf+vtbBSDmlprccNaqrIz29fw080c36yuGd2/JIExER3TsGpweQyWQ5CnPrqbSquUdXS+/+fjfXn44ahf4vFIU3+um1j1fd9GELtZqXHCAiotrD4GRHt95HrTaPhpQabx+M8gqA8xd//qhRgJ91MLr5qJEukEdtiIjowcTgZCe3u4+avgbzbyqrjhpd+Okr+zdPyP65o0bubpajRlWn0qrmGVWFJW/PX9YfERGRM2JwsoM73Uet8HvL8vdTLeHJ6qjRLcHo/EVLeLqbwIbV5xhVPQ8J4FEjIiKimmJwqmN3u49a1bJRUwFPDVBy7e7bcnezfBMt9KY5RjdPyvbS1Hb1REREDzYGpzq258jd76MGWOYfVYWmoIY3XcvolvlGIQGAC2/TTEREVGcYnOqYrfdHe+UvwOD+PGpERERUn/B4RR2z9f5o7VsxNBEREdU3DE51rHNby7fnVHdYr4JlPe+jRkREVP8wONWxqvuoAdXDE++jRkREVL8xONlB1X3UdEHWy3VBP12KgIiIiOofTg63E95HjYiIyPEwONkR76NGRETkWHiqjoiIiMhGDE5ERERENmJwIiIiIrIRgxMRERGRjRiciIiIiGzE4ERERERkI16OwAYiAgAoLS21cyVERER0N1W/q6t+d9c2BicbXLt2DQAQGhpq50qIiIjIFpcvX4avr2+tb1cl9yuSORGz2YwLFy7Ax8cHKtWdbs9bv5SWliI0NBT5+fnQarX2LqfWsT/Hxv4cG/tzbM7eX0lJCcLCwnD16lX4+fnV+vZ5xMkGLi4uaNKkib3LuCdardYp/8Oowv4cG/tzbOzPsTl7fy4u92caNyeHExEREdmIwYmIiIjIRgxOTsrDwwMpKSnw8PCwdyn3BftzbOzPsbE/x8b+fhlODiciIiKyEY84EREREdmIwYmIiIjIRgxORERERDZicHJgM2bMwMMPPwwfHx8EBwejf//+yM7Othpz48YNJCUlISAgAN7e3njqqadw8eJFO1VcM++//z7atWunXGvEYDAgLS1NWe/Ivd3OG2+8AZVKheTkZGWZI/c4ZcoUqFQqq0d0dLSy3pF7q3L+/Hn8+c9/RkBAADQaDdq2bYu9e/cq60UEkydPhl6vh0ajQVxcHHJycuxYse0iIiKq7T+VSoWkpCQAjr//TCYTXnvtNTRt2hQajQbNmjXD1KlTrW7T4cj7D7Dc9SI5ORnh4eHQaDTo2rUrMjMzlfWO1N/WrVvRt29fNGrUCCqVCl9//bXVelt6uXLlChISEqDVauHn54ehQ4fCaDTWvBghh9WzZ0+ZP3++ZGVlycGDB+XJJ5+UsLAwMRqNyphhw4ZJaGiobNq0Sfbu3StdunSRrl272rFq261atUrWrFkjJ06ckOzsbJk4caK4ublJVlaWiDh2b7fas2ePRERESLt27WTMmDHKckfuMSUlRdq0aSMFBQXK4/vvv1fWO3JvIiJXrlyR8PBwee6552T37t1y+vRpWbdunZw8eVIZ88Ybb4ivr698/fXXcujQIfn9738vTZs2lR9//NGOldumqKjIat9t2LBBAMiWLVtExPH337Rp0yQgIEBWr14tubm5smzZMvH29pY5c+YoYxx5/4mIPPPMM9K6dWvJyMiQnJwcSUlJEa1WK+fOnRMRx+rv22+/lVdffVW++uorASArVqywWm9LL7169ZL27dvLrl27ZNu2bdK8eXOJj4+vcS0MTk6kqKhIAEhGRoaIiBQXF4ubm5ssW7ZMGXPs2DEBIDt37rRXmb9Iw4YN5eOPP3aq3q5duyYtWrSQDRs2SPfu3ZXg5Og9pqSkSPv27W+7ztF7ExEZP368PPLII3dcbzabRafTydtvv60sKy4uFg8PD/n888/rosRaNWbMGGnWrJmYzWan2H99+vSRIUOGWC0bOHCgJCQkiIjj77/r16+LWq2W1atXWy2PiYmRV1991aH7uzU42dLL0aNHBYBkZmYqY9LS0kSlUsn58+dr9Pk8VedESkpKAAD+/v4AgH379qGiogJxcXHKmOjoaISFhWHnzp12qfFemUwmLFmyBD/88AMMBoNT9ZaUlIQ+ffpY9QI4x/7LyclBo0aNEBkZiYSEBOTl5QFwjt5WrVqFTp064emnn0ZwcDA6dOiAjz76SFmfm5uLwsJCqx59fX0RGxvrMD1WKS8vx2effYYhQ4ZApVI5xf7r2rUrNm3ahBMnTgAADh06hO3bt6N3794AHH//VVZWwmQyoUGDBlbLNRoNtm/f7vD93cyWXnbu3Ak/Pz906tRJGRMXFwcXFxfs3r27Rp/He9U5CbPZjOTkZHTr1g0PPfQQAKCwsBDu7u7VbnIYEhKCwsJCO1RZc0eOHIHBYMCNGzfg7e2NFStWoHXr1jh48KDD9wYAS5Yswf79+63mHVRx9P0XGxuLBQsWICoqCgUFBUhNTcWjjz6KrKwsh+8NAE6fPo33338fY8eOxcSJE5GZmYnRo0fD3d0diYmJSh8hISFW73OkHqt8/fXXKC4uxnPPPQfA8f/dBIAJEyagtLQU0dHRUKvVMJlMmDZtGhISEgDA4fefj48PDAYDpk6dilatWiEkJASff/45du7ciebNmzt8fzezpZfCwkIEBwdbrXd1dYW/v3+N+2VwchJJSUnIysrC9u3b7V1KrYqKisLBgwdRUlKCL7/8EomJicjIyLB3WbUiPz8fY8aMwYYNG6r9VegMqv5yB4B27dohNjYW4eHhWLp0KTQajR0rqx1msxmdOnXC9OnTAQAdOnRAVlYWPvjgAyQmJtq5utr1ySefoHfv3mjUqJG9S6k1S5cuxaJFi7B48WK0adMGBw8eRHJyMho1auQ0+2/hwoUYMmQIGjduDLVajZiYGMTHx2Pfvn32Ls2h8VSdExg5ciRWr16NLVu2oEmTJspynU6H8vJyFBcXW42/ePEidDpdHVd5b9zd3dG8eXN07NgRM2bMQPv27TFnzhyn6G3fvn0oKipCTEwMXF1d4erqioyMDLz77rtwdXVFSEiIw/d4Mz8/P7Rs2RInT550iv2n1+vRunVrq2WtWrVSTkdW9XHrN80cqUcAOHv2LDZu3IgXXnhBWeYM+2/cuHGYMGEC/vjHP6Jt27YYPHgwXnrpJcyYMQOAc+y/Zs2aISMjA0ajEfn5+dizZw8qKioQGRnpFP1VsaUXnU6HoqIiq/WVlZW4cuVKjftlcHJgIoKRI0dixYoV2Lx5M5o2bWq1vmPHjnBzc8OmTZuUZdnZ2cjLy4PBYKjrcmuF2WxGWVmZU/TWo0cPHDlyBAcPHlQenTp1QkJCgvLc0Xu8mdFoxKlTp6DX651i/3Xr1q3a5T9OnDiB8PBwAEDTpk2h0+mseiwtLcXu3bsdpkcAmD9/PoKDg9GnTx9lmTPsv+vXr8PFxfpXoFqthtlsBuA8+w8AvLy8oNfrcfXqVaxbtw79+vVzqv5s6cVgMKC4uNjqaNvmzZthNpsRGxtbsw/8RVPbya6GDx8uvr6+kp6ebvW14evXrytjhg0bJmFhYbJ582bZu3evGAwGMRgMdqzadhMmTJCMjAzJzc2Vw4cPy4QJE0SlUsn69etFxLF7u5Obv1Un4tg9/vWvf5X09HTJzc2VHTt2SFxcnAQGBkpRUZGIOHZvIpZLSLi6usq0adMkJydHFi1aJJ6envLZZ58pY9544w3x8/OTlStXyuHDh6Vfv3719uvet2MymSQsLEzGjx9fbZ2j77/ExERp3LixcjmCr776SgIDA+Vvf/ubMsbR99/atWslLS1NTp8+LevXr5f27dtLbGyslJeXi4hj9Xft2jU5cOCAHDhwQADIrFmz5MCBA3L27FkRsa2XXr16SYcOHWT37t2yfft2adGiBS9H8KABcNvH/PnzlTE//vijjBgxQho2bCienp4yYMAAKSgosF/RNTBkyBAJDw8Xd3d3CQoKkh49eiihScSxe7uTW4OTI/c4aNAg0ev14u7uLo0bN5ZBgwZZXePIkXur8s0338hDDz0kHh4eEh0dLR9++KHVerPZLK+99pqEhISIh4eH9OjRQ7Kzs+1Ubc2tW7dOANy2Zkfff6WlpTJmzBgJCwuTBg0aSGRkpLz66qtSVlamjHH0/ffFF19IZGSkuLu7i06nk6SkJCkuLlbWO1J/W7Zsue3vu8TERBGxrZfLly9LfHy8eHt7i1arleeff16uXbtW41pUIjddJpWIiIiI7ohznIiIiIhsxOBEREREZCMGJyIiIiIbMTgRERER2YjBiYiIiMhGDE5ERERENmJwIiIiIrIRgxMRERGRjRiciOgXe+yxx5CcnGzvMugOFixYAD8/P3uXQeQUGJyISMEAVH8w7BDVTwxORER1qLy83N4lENEvwOBERACA5557DhkZGZgzZw5UKhVUKhXOnDkDAMjIyEDnzp3h4eEBvV6PCRMmoLKy8o7bWrNmDXx9fbFo0SIAQH5+Pp555hn4+fnB398f/fr1U7Zd9dn9+/fHzJkzodfrERAQgKSkJFRUVChj5s2bhxYtWqBBgwYICQnBH/7wh7v289FHHyE0NBSenp4YMGAAZs2aVe0IzsqVKxETE4MGDRogMjISqampVn2pVCp8/PHHGDBgADw9PdGiRQusWrXKahtZWVno3bs3vL29ERISgsGDB+PSpUvK+sceewwjR45EcnIyAgMD0bNnTwDArFmz0LZtW3h5eSE0NBQjRoyA0WgEAKSnp+P5559HSUmJsi+mTJkCACgrK8PLL7+Mxo0bw8vLC7GxsUhPT7eqacGCBQgLC1N6v3z58l1/VkRUA7/8nsVE5AyKi4vFYDDIiy++KAUFBVJQUCCVlZVy7tw58fT0lBEjRsixY8dkxYoVEhgYKCkpKcp7u3fvLmPGjBERkUWLFomPj4988803IiJSXl4urVq1kiFDhsjhw4fl6NGj8qc//UmioqKUO9EnJiaKVquVYcOGybFjx+Sbb74RT09P+fDDD0VEJDMzU9RqtSxevFjOnDkj+/fvlzlz5tyxl+3bt4uLi4u8/fbbkp2dLe+99574+/uLr6+vMmbr1q2i1WplwYIFcurUKVm/fr1ERETIlClTlDEApEmTJrJ48WLJycmR0aNHi7e3t1y+fFlERK5evSpBQUHyyiuvyLFjx2T//v3y+OOPy29+8xurn423t7eMGzdOjh8/LsePHxcRkdmzZ8vmzZslNzdXNm3aJFFRUTJ8+HARESkrK5N//OMfotVqlX1RdRf3F154Qbp27Spbt26VkydPyttvvy0eHh5y4sQJERHZtWuXuLi4yJtvvinZ2dkyZ84c8fPzs+qdiO4dgxMRKW4OQFUmTpwoUVFRYjablWXvvfeeeHt7i8lksnrf3LlzxdfXV9LT05WxCxcurPb+srIy0Wg0sm7dOhGxBKfw8HCprKxUxjz99NMyaNAgERFZvny5aLVaKS0ttamPQYMGSZ8+fayWJSQkWIWHHj16yPTp063GLFy4UPR6vfIagEyaNEl5bTQaBYCkpaWJiMjUqVPliSeesNpGfn6+AJDs7GzlZ9OhQ4efrXnZsmUSEBCgvJ4/f361sHP27FlRq9Vy/vx5q+U9evSQV155RURE4uPj5cknn7RaP2jQIAYnolriat/jXURU3x07dgwGgwEqlUpZ1q1bNxiNRpw7dw5hYWEAgC+//BJFRUXYsWMHHn74YWXsoUOHcPLkSfj4+Fht98aNGzh16pTyuk2bNlCr1cprvV6PI0eOAAAef/xxhIeHIzIyEr169UKvXr2U02e3k52djQEDBlgt69y5M1avXm1V144dOzBt2jRlmclkwo0bN3D9+nVl2+3atVPWe3l5QavVoqioSNnGli1b4O3tXa2GU6dOoWXLlgCAjh07Vlu/ceNGzJgxA8ePH0dpaSkqKyurffatjhw5ApPJpGy3SllZGQICAgBY9tetvRsMBqxdu/a22ySimmFwIqJa0aFDB+zfvx///ve/0alTJyVoGY1GdOzYUZnvdLOgoCDluZubm9U6lUoFs9kMAPDx8cH+/fuRnp6O9evXY/LkyZgyZQoyMzPv+ZtnRqMRqampGDhwYLV1DRo0sKkuo9GIvn374s0336y2Db1erzz38vKyWnfmzBn87ne/w/DhwzFt2jT4+/tj+/btGDp0KMrLy+8YnIxGI9RqNfbt22cVMgHcNrwRUe1jcCIihbu7O0wmk9WyVq1aYfny5RARJQzt2LEDPj4+aNKkiTKuWbNmeOedd/DYY49BrVZj7ty5AICYmBh88cUXCA4OhlarvefaXF1dERcXh7i4OKSkpMDPzw+bN2++bfCJiopCZmam1bJbX8fExCA7OxvNmze/55piYmKwfPlyREREwNXV9v+d7tu3D2azGe+88w5cXCzf0Vm6dKnVmNvtiw4dOsBkMqGoqAiPPvrobbfdqlUr7N6922rZrl27bK6NiO6O36ojIkVERAR2796NM2fO4NKlSzCbzRgxYgTy8/MxatQoHD9+HCtXrkRKSgrGjh2r/NKv0rJlS2zZsgXLly9XrgeVkJCAwMBA9OvXD9u2bUNubi7S09MxevRonDt3zqa6Vq9ejXfffRcHDx7E2bNn8emnn8JsNiMqKuq240eNGoVvv/0Ws2bNQk5ODv71r38hLS3N6nTj5MmT8emnnyI1NRXfffcdjh07hiVLlmDSpEk2/7ySkpJw5coVxMfHIzMzE6dOncK6devw/PPPVws9N2vevDkqKirwz3/+E6dPn8bChQvxwQcfWI2JiIiA0WjEpk2bcOnSJVy/fh0tW7ZEQkICnn32WXz11VfIzc3Fnj17MGPGDKxZswYAMHr0aKxduxYzZ85ETk4O5s6dy9N0RLXJ3pOsiKj+yM7Oli5duohGoxEAkpubKyIi6enp8vDDD4u7u7vodDoZP368VFRUKO+7dVL50aNHJTg4WMaOHSsiIgUFBfLss89KYGCgeHh4SGRkpLz44otSUlIiIpbJ4f369bOqZcyYMdK9e3cREdm2bZt0795dGjZsKBqNRtq1aydffPHFXXv58MMPpXHjxqLRaKR///7y97//XXQ6ndWYtWvXSteuXUWj0YhWq5XOnTsr3+QTsUwOX7FihdV7fH19Zf78+crrEydOyIABA8TPz080Go1ER0dLcnKyMhn+dhPuRURmzZoler1eNBqN9OzZUz799FMBIFevXlXGDBs2TAICAgSA8i3G8vJymTx5skRERIibm5vo9XoZMGCAHD58WHnfJ598Ik2aNBGNRiN9+/aVmTNncnI4US1RiYjYN7oREd1/L774Io4fP45t27bZuxQicmCc40RETmnmzJl4/PHH4eXlhbS0NPznP//BvHnz7F0WETk4HnEiIqf0zDPPID09HdeuXUNkZCRGjRqFYcOG2bssInJwDE5ERERENuK36oiIiIhsxOBEREREZCMGJyIiIiIbMTgRERER2YjBiYiIiMhGDE5ERERENmJwIiIiIrIRgxMRERGRjRiciIiIiGz0/4j99mWBm0ZMAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"recompute ms: ['1668', '3219', '1080', '2762']\n",
"cached ms: ['195', '325', '2754', '1497']\n"
]
}
],
"source": [
"# viz: how the gap widens with generated length (the O(T^2) vs O(T) story, measured)\n",
"lengths = [8, 16, 32] if FAST else [16, 32, 64, 96]\n",
"rec_ms, cac_ms = [], []\n",
"for n in lengths:\n",
" p = make_batch(1)[0][:, :PREFIX + 2]\n",
" rec_ms.append(time_call(greedy_baseline, p.clone(), n, runs=2) * 1e3)\n",
" cac_ms.append(time_call(generate_cached, cached, p.clone(), n, runs=2) * 1e3)\n",
"fig, ax = plt.subplots(figsize=(6, 4))\n",
"ax.plot(lengths, rec_ms, \"o-\", color=\"#888\", label=\"recompute (O(T^2) attention)\")\n",
"ax.plot(lengths, cac_ms, \"o-\", color=\"#1E40FF\", label=\"KV cache (O(T) attention)\")\n",
"ax.set_xlabel(\"tokens generated\"); ax.set_ylabel(\"wall-clock ms\")\n",
"ax.set_title(\"The gap widens with length\"); ax.legend(); plt.tight_layout(); plt.show()\n",
"print(\"recompute ms:\", [f\"{m:.0f}\" for m in rec_ms])\n",
"print(\"cached ms:\", [f\"{m:.0f}\" for m in cac_ms])"
]
},
{
"cell_type": "markdown",
"id": "36cb0c9b",
"metadata": {},
"source": [
"> **What is the interpretation of this plot?** AnswerThe recompute curve bends upward (superlinear: each new token re-runs an ever-longer context), while the cached curve is closer to a straight line (each token does roughly constant new work plus a growing-but-cheap attention read). The vertical gap between them is the amount of redundant computation the cache eliminates, and it grows with length. On CPU with tiny tensors the curvature is gentle; the shape is the lesson.\n",
"\n",
"> **Key takeaways**\n",
"> - The KV cache caches per-position K and V so each decode step computes K,V for one token instead of the whole context.\n",
"> - Prefill needs a causal mask; single-token decode does not (the last-position trick).\n",
"> - Cached generation is token-identical to recompute under greedy decoding (`torch.equal`), and measurably faster, with the gap widening as the sequence grows.\n",
"> - The classic bug is feeding the wrong absolute position to the position embedding during decode. A `torch.equal` check against the recompute baseline is the guard.\n"
]
},
{
"cell_type": "markdown",
"id": "8e6bc5df",
"metadata": {},
"source": [
"## Part 4 — Quantization: fewer bytes per weight\n",
"\n",
"Inference is memory-bound (Part 1), so the most direct speedup is to move fewer bytes. Quantization stores weights in lower precision: FP32 (4 bytes) -> BF16/FP16 (2) -> int8 (1) -> int4 (0.5). Halving the bytes per weight roughly halves the memory traffic, which on a memory-bound workload translates almost directly into speed.\n",
"\n",
"The basic recipe is *linear quantization*. Map a float tensor's range onto a small integer range with a scale `s` (and optionally a zero-point `z`):\n",
"\n",
"$$x_\\text{int} = \\text{round}\\!\\left(\\frac{x}{s}\\right), \\qquad \\hat{x} = s \\cdot x_\\text{int}$$\n",
"\n",
"For symmetric int8, `s = max(|x|) / 127`, and the integers live in `[-127, 127]`. The rounding is the only place information is lost.\n"
]
},
{
"cell_type": "markdown",
"id": "93b4b90d",
"metadata": {},
"source": [
"### Exercise 17.3 — Symmetric int8 quantize / dequantize\n",
"`Difficulty 2/5 · ~12 min`\n",
"\n",
"Implement `quantize_int8(x)` returning `(q, s)` and `dequantize_int8(q, s)` returning the reconstruction. Use symmetric absmax scaling: `s = x.abs().max() / 127`, round, clamp to `[-127, 127]`, store as `int8`. Then the check verifies the theoretical guarantee: the worst-case round-trip error is at most `s/2` (you can be off by half a step) and the mean squared error is near the uniform-quantization-noise value `s^2 / 12`.\n"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "cd910544",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:13.876611Z",
"iopub.status.busy": "2026-06-10T19:27:13.876517Z",
"iopub.status.idle": "2026-06-10T19:27:13.912108Z",
"shell.execute_reply": "2026-06-10T19:27:13.911733Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 17.3 int8 round-trip + error bound: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def quantize_int8(x):\n",
" \"\"\"Symmetric absmax int8 quantization. Returns (q: int8 tensor, s: python float).\"\"\"\n",
" # TODO 1: s = x.abs().max() / 127 (as a python float; guard the all-zero tensor with a tiny floor)\n",
" s = None\n",
" # TODO 2: q = round(x / s), clamp to [-127, 127], cast to torch.int8\n",
" q = None\n",
" attempted(s, q)\n",
" return q, s\n",
"\n",
"def dequantize_int8(q, s):\n",
" # TODO 3: reconstruct the float tensor: q.float() * s\n",
" out = None\n",
" attempted(out)\n",
" return out\n",
"\n",
"def _int8_roundtrip():\n",
" torch.manual_seed(SEED)\n",
" x = torch.randn(4096) * 0.7\n",
" q, s = quantize_int8(x)\n",
" assert q.dtype == torch.int8, f\"q should be int8, got {q.dtype}\"\n",
" assert int(q.abs().max()) <= 127, \"values must clamp into [-127, 127]\"\n",
" xhat = dequantize_int8(q, s)\n",
" max_err = (x - xhat).abs().max().item()\n",
" assert max_err <= s / 2 + 1e-6, \\\n",
" f\"max round-trip error {max_err:.4g} exceeds the s/2 bound {s/2:.4g} — check your rounding\"\n",
" mse = ((x - xhat) ** 2).mean().item()\n",
" assert mse < s ** 2 / 8, \\\n",
" f\"MSE {mse:.4g} too large; uniform quant noise is ~s^2/12={s**2/12:.4g}, well under s^2/8\"\n",
"\n",
"check(\"17.3 int8 round-trip + error bound\", _int8_roundtrip)"
]
},
{
"cell_type": "markdown",
"id": "4c23c3eb",
"metadata": {},
"source": [
"Hint 1 (conceptual)The scale maps the largest-magnitude value onto 127. After dividing by `s`, round to the nearest integer, then clamp so a tie or float fuzz cannot push past 127. The dequant is just multiply by `s` again; there is no zero-point in the symmetric scheme.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"s = (x.abs().max() / 127).item()\n",
"s = max(s, 1e-8) # avoid divide-by-zero on an all-zero tensor\n",
"q = (x / s).round().clamp(-127, 127).to(torch.int8)\n",
"out = q.float() * s\n",
"```\n",
"\n",
"\n",
"Help — \"max error exceeds s/2\"You probably did not round, or you rounded after casting to int8 (which truncates toward zero, giving up to a full step of error). Round in float first, then clamp, then cast: `(x/s).round().clamp(-127,127).to(torch.int8)`.\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "5a96f615",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:13.916403Z",
"iopub.status.busy": "2026-06-10T19:27:13.916315Z",
"iopub.status.idle": "2026-06-10T19:27:13.971573Z",
"shell.execute_reply": "2026-06-10T19:27:13.971195Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 17.3 int8 round-trip + error bound\n",
"scale s = 0.02261\n",
"max round-trip error = 0.01130 (bound s/2 = 0.01130)\n",
"MSE = 0.000043 (uniform-noise theory s^2/12 = 0.000043)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines the quantizers; the check below re-verifies the reference.\n",
"def quantize_int8(x):\n",
" s = max((x.abs().max() / 127).item(), 1e-8)\n",
" q = (x / s).round().clamp(-127, 127).to(torch.int8)\n",
" return q, s\n",
"\n",
"def dequantize_int8(q, s):\n",
" return q.float() * s\n",
"\n",
"check(\"17.3 int8 round-trip + error bound\", _int8_roundtrip, required=True)\n",
"torch.manual_seed(SEED)\n",
"x = torch.randn(4096) * 0.7\n",
"q, s = quantize_int8(x)\n",
"xhat = dequantize_int8(q, s)\n",
"print(f\"scale s = {s:.5f}\")\n",
"print(f\"max round-trip error = {(x-xhat).abs().max():.5f} (bound s/2 = {s/2:.5f})\")\n",
"print(f\"MSE = {((x-xhat)**2).mean():.6f} (uniform-noise theory s^2/12 = {s**2/12:.6f})\")"
]
},
{
"cell_type": "markdown",
"id": "cbea57df",
"metadata": {},
"source": [
"> **Interpretation.** The measured max error sits right under `s/2` and the MSE lands near `s^2/12`, the textbook value for uniform quantization noise. Quantization error is not mysterious: it has a known statistical character, which is why you can reason about how aggressive you can be before accuracy moves.\n"
]
},
{
"cell_type": "markdown",
"id": "c0a7120d",
"metadata": {},
"source": [
"### Granularity: per-tensor vs per-channel\n",
"\n",
"One scale for the whole tensor is *per-tensor* quantization. The problem: if one row of a weight matrix has much larger values than the others, that row's magnitude sets the scale, and every other row is squished into a few integer levels. The fix is *per-channel* (per-row) quantization: one scale per row, so a loud row no longer ruins the quiet ones. This is the granularity tradeoff, and it foreshadows why LLM.int8() needs special handling for outlier features.\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "df7679d3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:13.975631Z",
"iopub.status.busy": "2026-06-10T19:27:13.975521Z",
"iopub.status.idle": "2026-06-10T19:27:13.994446Z",
"shell.execute_reply": "2026-06-10T19:27:13.993747Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"per-row mean abs error\n",
" per-tensor : ['0.0198', '0.0186', '0.0193', '0.0182', '0.0170', '0.0180', '0.0187', '0.0180']\n",
" per-channel: ['0.0007', '0.0008', '0.0007', '0.0182', '0.0005', '0.0007', '0.0005', '0.0006']\n",
"\n",
"The quiet rows (everything but row 3) are far better preserved per-channel.\n"
]
}
],
"source": [
"# viz / measurement: a weight matrix where one row is 30x louder than the rest.\n",
"torch.manual_seed(SEED)\n",
"W = torch.randn(8, 256) * 0.1\n",
"W[3] *= 30.0 # one outlier row, like an emergent outlier feature\n",
"\n",
"# per-tensor: a single scale\n",
"q_t, s_t = quantize_int8(W)\n",
"W_t = dequantize_int8(q_t, s_t)\n",
"\n",
"# per-channel: one scale per row\n",
"def quantize_per_channel(W):\n",
" s = (W.abs().amax(dim=1, keepdim=True) / 127).clamp_min(1e-8) # (rows, 1)\n",
" q = (W / s).round().clamp(-127, 127).to(torch.int8)\n",
" return q, s\n",
"q_c, s_c = quantize_per_channel(W)\n",
"W_c = q_c.float() * s_c\n",
"\n",
"err_t = (W - W_t).abs().mean(dim=1) # per-row mean error, per-tensor scheme\n",
"err_c = (W - W_c).abs().mean(dim=1) # per-row mean error, per-channel scheme\n",
"print(\"per-row mean abs error\")\n",
"print(f\" per-tensor : {[f'{e:.4f}' for e in err_t.tolist()]}\")\n",
"print(f\" per-channel: {[f'{e:.4f}' for e in err_c.tolist()]}\")\n",
"assert err_c[0] < err_t[0] * 0.5, \"per-channel should sharply reduce error on the quiet rows\"\n",
"print(\"\\nThe quiet rows (everything but row 3) are far better preserved per-channel.\")"
]
},
{
"cell_type": "markdown",
"id": "392ffe40",
"metadata": {},
"source": [
"> **Interpretation.** Under per-tensor scaling, the loud row 3 sets the scale and the seven quiet rows lose most of their resolution. Per-channel scaling gives each row its own scale, so the quiet rows recover. The cost is storing one extra float per row, negligible against the weight bytes saved. This is the same insight that drives per-group quantization in GPTQ/AWQ and the outlier handling in LLM.int8().\n"
]
},
{
"cell_type": "markdown",
"id": "84c5fb37",
"metadata": {},
"source": [
"### A real int8 matmul on CPU\n",
"\n",
"The from-scratch quantizer proves the math; PyTorch's `torch.ao.quantization` runs an actual int8 kernel on CPU. We quantize a Linear layer dynamically and confirm two things: the output stays close to the float layer, and the storage is genuinely smaller.\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "4611d058",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:13.995433Z",
"iopub.status.busy": "2026-06-10T19:27:13.995351Z",
"iopub.status.idle": "2026-06-10T19:27:14.024263Z",
"shell.execute_reply": "2026-06-10T19:27:14.022549Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"mean relative error fp32 vs int8 Linear: 0.0000\n",
"weight storage: fp32 262,144 B -> int8 65,536 B (4x smaller)\n"
]
}
],
"source": [
"# deeper: dynamic int8 quantization of a Linear via torch.ao (CPU int8 path)\n",
"import torch.ao.quantization as tq\n",
"torch.manual_seed(SEED)\n",
"lin = nn.Linear(256, 256, bias=False).eval()\n",
"xq = torch.randn(32, 256)\n",
"with torch.no_grad():\n",
" y_fp = lin(xq)\n",
"qlin = tq.quantize_dynamic(lin, {nn.Linear}, dtype=torch.qint8)\n",
"with torch.no_grad():\n",
" y_int8 = qlin(xq)\n",
"rel_err = (y_fp - y_int8).abs().mean() / y_fp.abs().mean()\n",
"print(f\"mean relative error fp32 vs int8 Linear: {rel_err:.4f}\")\n",
"assert rel_err < 0.05, \"dynamic int8 should track the float layer within a few percent on well-behaved inputs\"\n",
"# byte accounting: int8 weights are 1 byte vs 4 for fp32\n",
"fp_bytes = lin.weight.numel() * 4\n",
"int8_bytes = lin.weight.numel() * 1\n",
"print(f\"weight storage: fp32 {fp_bytes:,} B -> int8 {int8_bytes:,} B ({fp_bytes/int8_bytes:.0f}x smaller)\")"
]
},
{
"cell_type": "markdown",
"id": "3d9b4618",
"metadata": {},
"source": [
"> **Caveat:** int8 *weights* are 4x smaller than fp32 and 2x smaller than BF16, which is the memory win. But the int8 *matmul kernel* is not always faster than the float one (on many CPUs and GPUs the float path is more optimized). LLM.int8() is explicitly a memory-savings technique that can be slightly slower per step; it lets you fit a model you otherwise could not. Always separate \"fewer bytes stored\" from \"faster kernel\": they are not the same claim.\n",
"\n",
"> **Key takeaways**\n",
"> - Linear quantization maps a float range onto integers with a scale; the round-trip error is bounded by `s/2` and averages `s^2/12`.\n",
"> - Per-channel granularity protects quiet rows from a single loud row, the seed of the outlier-feature problem at scale.\n",
"> - int8 weights are genuinely smaller (the memory win); a faster kernel is a separate, hardware-dependent question.\n"
]
},
{
"cell_type": "markdown",
"id": "44d32d92",
"metadata": {},
"source": [
"## Part 5 — Accounting that scales: KV bytes, MQA/GQA, MFU\n",
"\n",
"You now have the cache and the quantizer. The last skill is the back-of-the-envelope arithmetic that tells you, before you deploy, how much memory the cache eats and how much of the chip you are actually using.\n"
]
},
{
"cell_type": "markdown",
"id": "179f6bdb",
"metadata": {},
"source": [
"### Exercise 17.4 — KV cache memory and grouped-query attention\n",
"`Difficulty 2/5 · ~12 min`\n",
"\n",
"The KV cache stores, per layer, per key/value head, one K vector and one V vector of size `d_k` for every token. Implement `kv_cache_bytes(n_layers, n_kv_heads, seq_len, d_k, dtype_bytes=2)`.\n",
"\n",
"Then grouped-query attention (GQA) shrinks it: instead of `n_heads` separate K/V heads, share `n_kv_heads < n_heads` of them across the query heads. Multi-query attention (MQA) is the extreme `n_kv_heads = 1`. The cache shrinks by exactly `n_heads / n_kv_heads`. The check uses Llama-2-13B-MHA numbers (40 layers, 40 heads, d_k 128, 2048 tokens) and the Llama-2-70B GQA ratio.\n"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "6f793a2a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:14.025262Z",
"iopub.status.busy": "2026-06-10T19:27:14.025173Z",
"iopub.status.idle": "2026-06-10T19:27:14.033738Z",
"shell.execute_reply": "2026-06-10T19:27:14.033230Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 17.4 KV bytes + GQA/MQA reduction: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def kv_cache_bytes(n_layers, n_kv_heads, seq_len, d_k, dtype_bytes=2):\n",
" \"\"\"Bytes for one sequence's KV cache. Factor of 2 because we store both K and V.\"\"\"\n",
" # TODO 1: n_layers * n_kv_heads * seq_len * d_k * 2 (K and V) * dtype_bytes\n",
" total = None\n",
" attempted(total)\n",
" return total\n",
"\n",
"def _kv_accounting():\n",
" # Llama-2-13B at 2048 ctx, MHA (n_kv_heads = n_heads = 40), BF16\n",
" mha = kv_cache_bytes(n_layers=40, n_kv_heads=40, seq_len=2048, d_k=128)\n",
" gb = mha / 1e9\n",
" # 40 * 40 * 2048 * 128 * 2 (K,V) * 2 (bytes) = 1.678e9 bytes\n",
" assert 1.6 < gb < 1.8, f\"expected ~1.68 GB per sequence, got {gb:.3f} GB\"\n",
" # GQA with 8 KV heads (Llama-2-70B style) is exactly 40/8 = 5x smaller\n",
" gqa = kv_cache_bytes(n_layers=40, n_kv_heads=8, seq_len=2048, d_k=128)\n",
" assert abs(mha / gqa - 5.0) < 1e-6, f\"MHA/GQA should be exactly 5x, got {mha/gqa:.3f}\"\n",
" # MQA (1 KV head) is 40x smaller\n",
" mqa = kv_cache_bytes(n_layers=40, n_kv_heads=1, seq_len=2048, d_k=128)\n",
" assert abs(mha / mqa - 40.0) < 1e-6, f\"MHA/MQA should be exactly 40x, got {mha/mqa:.3f}\"\n",
"\n",
"check(\"17.4 KV bytes + GQA/MQA reduction\", _kv_accounting)"
]
},
{
"cell_type": "markdown",
"id": "082e51a7",
"metadata": {},
"source": [
"Hint 1 (conceptual)Walk the dimensions: every layer has `n_kv_heads` KV heads; each head stores `seq_len` tokens; each token is a `d_k`-vector; you store both K and V (the factor of 2); each number is `dtype_bytes` bytes. Multiply them all.\n",
"\n",
"Hint 2 (the line)`total = n_layers * n_kv_heads * seq_len * d_k * 2 * dtype_bytes`.\n",
"\n",
"Help — \"expected ~1.68 GB but got something far off\"Check the factor of 2 (K and V) is present and that `dtype_bytes` is 2 for BF16. 40 * 40 * 2048 * 128 * 2 * 2 = 1.678e9 bytes. If you got half that, you dropped the K-and-V factor; if you got double, you used 4 bytes.\n"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "fc9c632c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:14.034667Z",
"iopub.status.busy": "2026-06-10T19:27:14.034536Z",
"iopub.status.idle": "2026-06-10T19:27:14.039669Z",
"shell.execute_reply": "2026-06-10T19:27:14.039175Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 17.4 KV bytes + GQA/MQA reduction\n",
"MHA (40 KV heads) : 1.678 GB / sequence -> 107.4 GB at batch 64\n",
"GQA-8 : 0.336 GB / sequence -> 21.5 GB at batch 64\n",
"MQA (1) : 0.042 GB / sequence -> 2.7 GB at batch 64\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines kv_cache_bytes; the check below re-verifies the reference.\n",
"def kv_cache_bytes(n_layers, n_kv_heads, seq_len, d_k, dtype_bytes=2):\n",
" return n_layers * n_kv_heads * seq_len * d_k * 2 * dtype_bytes\n",
"\n",
"check(\"17.4 KV bytes + GQA/MQA reduction\", _kv_accounting, required=True)\n",
"for label, kvh in [(\"MHA (40 KV heads)\", 40), (\"GQA-8\", 8), (\"MQA (1)\", 1)]:\n",
" gb = kv_cache_bytes(40, kvh, 2048, 128) / 1e9\n",
" print(f\"{label:18s}: {gb:.3f} GB / sequence -> {gb*64:6.1f} GB at batch 64\")"
]
},
{
"cell_type": "markdown",
"id": "e7fdffd1",
"metadata": {},
"source": [
"> **Interpretation.** At MHA, one Llama-2-13B sequence's cache is ~1.68 GB, so batch 64 wants ~107 GB just for the cache, more than an 80 GB H100 has even before the weights. GQA-8 cuts that to ~21 GB; MQA to ~2.7 GB. This is the entire reason Llama-2-70B and Llama-3 use GQA: the cache, not the weights, is the binding constraint at serving scale.\n"
]
},
{
"cell_type": "markdown",
"id": "e577bee4",
"metadata": {},
"source": [
"### Model FLOPs Utilization (MFU) for our own model\n",
"\n",
"MFU asks: of the chip's peak FLOP/s, what fraction did your run actually use? It is `achieved FLOP/s / peak FLOP/s`. For a transformer forward pass, the standard estimate is `2 * params * tokens` FLOPs (the factor of 2 is multiply + add; this is the forward-only count, the same approximation Karpathy uses in nanoGPT). We compute it for our own model's generation and read it against the H100's peak. Spoiler: it is tiny, because single-token decode is memory-bound.\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "54d556c3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:14.040546Z",
"iopub.status.busy": "2026-06-10T19:27:14.040470Z",
"iopub.status.idle": "2026-06-10T19:27:15.202502Z",
"shell.execute_reply": "2026-06-10T19:27:15.202187Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"params 159,040 · generated 64 tokens in 150.4 ms (CPU)\n",
"forward FLOPs ~ 2*P*tokens = 2.036e+07\n",
"achieved ~ 1.353e+08 FLOP/s\n",
"MFU vs an H100's 9.90e+14 FLOP/s peak: 0.00001%\n"
]
}
],
"source": [
"# MFU for our tiny model's cached generation, read against H100 peak (an illustrative number).\n",
"P = param_count(cached)\n",
"prompt = make_batch(1)[0][:, :PREFIX + 2]\n",
"n_gen = GEN_TOKENS\n",
"t = time_call(generate_cached, cached, prompt.clone(), n_gen, runs=3)\n",
"# Forward FLOPs ~ 2 * params per token (Karpathy's nanoGPT estimate; forward-only)\n",
"flops_done = 2 * P * n_gen\n",
"achieved = flops_done / t\n",
"peak_h100 = HW[\"H100-80GB\"][0]\n",
"mfu_vs_h100 = achieved / peak_h100\n",
"print(f\"params {P:,} · generated {n_gen} tokens in {t*1e3:.1f} ms (CPU)\")\n",
"print(f\"forward FLOPs ~ 2*P*tokens = {flops_done:.3e}\")\n",
"print(f\"achieved ~ {achieved:.3e} FLOP/s\")\n",
"print(f\"MFU vs an H100's {peak_h100:.2e} FLOP/s peak: {mfu_vs_h100*100:.5f}%\")"
]
},
{
"cell_type": "markdown",
"id": "f2fec9b3",
"metadata": {},
"source": [
"> **Common confusion:** this MFU is microscopic for two reasons stacked together. First, we ran on a CPU, not the H100 we divided by, so the comparison is illustrative, not a real H100 measurement. Second, even on the right hardware, single-stream greedy decode is memory-bound (Part 1), so its MFU is inherently low, often a few percent on real serving even when everything is tuned. The lever that raises decode MFU is batching many concurrent requests, which moves the MLP up the roofline. MFU near 100% is a *training* number, not a single-stream *decode* number.\n",
"\n",
"> **Key takeaways**\n",
"> - KV-cache bytes = `layers * kv_heads * seq_len * d_k * 2 * dtype_bytes`; the cache, not the weights, dominates at scale.\n",
"> - GQA shrinks the cache by exactly `n_heads / n_kv_heads`; MQA is the extreme.\n",
"> - MFU = achieved / peak FLOP/s. Single-stream decode is intrinsically low-MFU because it is memory-bound; batching is the fix.\n"
]
},
{
"cell_type": "markdown",
"id": "27767b14",
"metadata": {},
"source": [
"## Safety lens\n",
"\n",
"Every optimization here changes the attack surface, and the simplest one to demonstrate is **quantization-induced behavior drift**. A model can have near-identical perplexity at int8 and yet behave differently on specific inputs, because quantization collapses the low-magnitude directions in weight space. Egashira et al. (2024) showed an adversary can hide behavior in exactly those directions so it survives FP16 evaluation and only activates after quantization. We do not stage an attack, but we measure the mechanism: quantization changes the argmax on some inputs even when average error is tiny.\n"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "338acadb",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:15.210275Z",
"iopub.status.busy": "2026-06-10T19:27:15.210133Z",
"iopub.status.idle": "2026-06-10T19:27:15.350578Z",
"shell.execute_reply": "2026-06-10T19:27:15.350198Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"mean logit error (tiny): 0.0018\n",
"top-1 prediction flip rate: 0.35% of inputs change their answer\n",
"Average error looks negligible, yet some decisions flip. Perplexity-style averages hide this.\n"
]
}
],
"source": [
"# Measure behavior drift: how often does int8-quantizing a layer flip the top-1 prediction?\n",
"torch.manual_seed(SEED)\n",
"clf = nn.Linear(64, 8, bias=False).eval() # a tiny \"classifier head\"\n",
"W_fp = clf.weight.detach().clone()\n",
"qW, sW = quantize_per_channel(W_fp)\n",
"W_q = qW.float() * sW\n",
"probe = torch.randn(2000, 64)\n",
"with torch.no_grad():\n",
" top1_fp = (probe @ W_fp.T).argmax(-1)\n",
" top1_q = (probe @ W_q.T).argmax(-1)\n",
"flip_rate = (top1_fp != top1_q).float().mean().item()\n",
"mean_logit_err = (probe @ W_fp.T - probe @ W_q.T).abs().mean().item()\n",
"print(f\"mean logit error (tiny): {mean_logit_err:.4f}\")\n",
"print(f\"top-1 prediction flip rate: {flip_rate*100:.2f}% of inputs change their answer\")\n",
"print(\"Average error looks negligible, yet some decisions flip. Perplexity-style averages hide this.\")"
]
},
{
"cell_type": "markdown",
"id": "bb1f7468",
"metadata": {},
"source": [
"> **Interpretation.** The mean logit error is small, but a non-zero fraction of inputs change their top-1 answer. On a language model those flips concentrate on the edge cases adversarial prompts target. The operational rule: **re-run your full safety eval after every quantization step, not just perplexity.** Two other surfaces this chapter creates, covered in depth in Ch 24: prefix-cache *timing* side channels (a cache hit is faster, which leaks whether another tenant prompted the same prefix) and speculative-decoding timing oracles (acceptance rate is visible in response latency). The mitigations are per-tenant cache namespacing and fixed-rate response buffering.\n"
]
},
{
"cell_type": "markdown",
"id": "5600ded4",
"metadata": {},
"source": [
"## Test yourself\n",
"\n",
"Three parts: concept self-checks with folded answers, two auto-checked problems, and a capstone. Solutions are folded; try before you peek. Every answer is in this notebook; if unsure, re-run that section.\n"
]
},
{
"cell_type": "markdown",
"id": "b827c68a",
"metadata": {},
"source": [
"### Part A — Concepts\n",
"\n",
"1. What does arithmetic intensity divide by what, and what is the H100's ridge? AnswerFLOPs divided by bytes moved. The ridge is peak FLOP/s divided by peak bandwidth, ~989.5e12 / 3.35e12 = ~295 FLOP/byte. Below it: memory-bound. Above it: compute-bound. (Part 1.)\n",
"2. Why is prefill compute-bound but single-token decode memory-bound, given they run the same weights? AnswerPrefill pushes `T` tokens through one weight read (intensity scales with `T`), so it is compute-bound. Decode reads the full weight to produce one token (intensity ~1), so it is memory-bound. (Exercise 17.1, the roofline plot.)\n",
"3. During cached decode, why is no causal mask needed? AnswerThe single new query is the most recent position, so every cached key is at a position less than or equal to it; causality holds automatically. The mask only matters during prefill. (Part 3, the last-position trick.)\n",
"4. The deliberate-failure cell printed a baseline and a \"broken\" sequence that diverge after the first generated token. What single line caused it? Answer`pos = torch.tensor([0])` on every decode step instead of `pos = torch.tensor([idx.shape[1] - 1])`. The cache held correct keys; the position fed to `pos_emb` was wrong, so the logits drifted. (Part 3 deliberate failure.)\n",
"5. In the timing plot, why does the recompute curve bend upward while the cached one is nearly straight? AnswerRecompute re-runs an ever-longer context each step (O(T^2) attention work across the run); the cache does roughly constant new work per token (O(T) total). The widening vertical gap is the redundant computation the cache removes. (Part 3 timing viz.)\n",
"6. Quick: write the symmetric int8 scale for a tensor `x` in one expression. Answer`s = x.abs().max() / 127`. Round `x/s`, clamp to `[-127, 127]`, cast to int8. The round-trip error is at most `s/2`. (Exercise 17.3.)\n",
"7. Why does per-channel quantization beat per-tensor when one row is an outlier? AnswerPer-tensor uses one scale, so the loud row sets it and the quiet rows lose resolution. Per-channel gives each row its own scale, so a loud row no longer ruins the quiet ones. (Part 4 granularity demo.)\n",
"8. Llama-2-13B's MHA KV cache is ~1.68 GB per 2048-token sequence. Why do Llama-2-70B and Llama-3 use GQA-8? AnswerGQA-8 shrinks the cache by exactly `n_heads / n_kv_heads`. At serving scale the cache (batch x per-seq bytes), not the weights, is the binding memory constraint, and GQA is near-lossless. (Exercise 17.4.)\n",
"9. The MFU we computed for our model was a tiny fraction of a percent. Name the two stacked reasons. Answer(1) We ran on CPU but divided by an H100's peak, so it is illustrative. (2) Even on the right hardware, single-stream greedy decode is memory-bound, so its MFU is intrinsically low; batching is the lever. (Part 5.)\n",
"10. The safety cell showed a tiny mean logit error but a non-zero top-1 flip rate after quantization. What is the operational takeaway? AnswerAverage-error metrics like perplexity hide decision flips that concentrate on edge cases. Re-run the full safety eval (not just perplexity) after every quantization step. (Safety lens.)\n"
]
},
{
"cell_type": "markdown",
"id": "659fcfbc",
"metadata": {},
"source": [
"### Part B — Auto-checked problems\n"
]
},
{
"cell_type": "markdown",
"id": "a79f4919",
"metadata": {},
"source": [
"**Exercise 17.B1 — The last-position trick for next-token logits** · Difficulty 2/5 · ~8 min\n",
"\n",
"During decode you only ever need the logits of the *last* position to pick the next token. Computing the head over the whole sequence wastes a vocab-sized matmul per position. Implement `next_token_logits(net, idx)` that returns only the last position's logits, shape `(B, vocab)`, by slicing the hidden state to the last position *before* the head. The check confirms it equals the full forward's last slice and that the output is 2D.\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "58f43ba5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:15.363469Z",
"iopub.status.busy": "2026-06-10T19:27:15.363344Z",
"iopub.status.idle": "2026-06-10T19:27:15.431085Z",
"shell.execute_reply": "2026-06-10T19:27:15.430827Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 17.B1 last-position logits: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def next_token_logits(net, idx):\n",
" \"\"\"Return only the final position's logits, shape (B, vocab). No cache; this is about the head slice.\"\"\"\n",
" B, T = idx.shape\n",
" pos = torch.arange(T, device=idx.device)\n",
" x = net.tok(idx) + net.pos(pos)\n",
" for blk in net.blocks:\n",
" x = blk(x) if isinstance(blk, Block) else blk(x, use_cache=False)\n",
" x = net.ln_f(x)\n",
" # TODO: slice to the LAST position BEFORE applying the head, then run the head -> (B, vocab)\n",
" last_logits = None\n",
" attempted(last_logits)\n",
" return last_logits\n",
"\n",
"def _last_pos():\n",
" torch.manual_seed(SEED)\n",
" net = TinyGPT(); net.eval()\n",
" idx = make_batch(2)[0]\n",
" with torch.no_grad():\n",
" full = net(idx) # (B, T, vocab)\n",
" got = next_token_logits(net, idx)\n",
" assert got.dim() == 2, f\"expected (B, vocab), got shape {tuple(got.shape)}\"\n",
" check_close(got, full[:, -1, :], atol=1e-5, msg=\"last-position logits must equal the full forward's last slice\")\n",
"\n",
"check(\"17.B1 last-position logits\", _last_pos)"
]
},
{
"cell_type": "markdown",
"id": "613b437d",
"metadata": {},
"source": [
"Hint 1The head is `net.head`. Take `x[:, -1, :]` (shape `(B, d_model)`) and pass it through `net.head` to get `(B, vocab)`. Slicing before the head means the head runs on one row per sequence instead of `T`.\n",
"Solution\n",
"\n",
"```python\n",
"last_logits = net.head(x[:, -1, :]) # (B, d_model) -> (B, vocab)\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "873ccf85",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:15.432615Z",
"iopub.status.busy": "2026-06-10T19:27:15.432533Z",
"iopub.status.idle": "2026-06-10T19:27:15.491237Z",
"shell.execute_reply": "2026-06-10T19:27:15.482410Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 17.B1 last-position logits\n",
"last-position logits match the full forward's final slice, at a fraction of the head FLOPs.\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines next_token_logits; the check below re-verifies the reference.\n",
"def next_token_logits(net, idx):\n",
" B, T = idx.shape\n",
" pos = torch.arange(T, device=idx.device)\n",
" x = net.tok(idx) + net.pos(pos)\n",
" for blk in net.blocks:\n",
" x = blk(x)\n",
" x = net.ln_f(x)\n",
" return net.head(x[:, -1, :]) # slice to last position, then head -> (B, vocab)\n",
"\n",
"check(\"17.B1 last-position logits\", _last_pos, required=True)\n",
"print(\"last-position logits match the full forward's final slice, at a fraction of the head FLOPs.\")"
]
},
{
"cell_type": "markdown",
"id": "5d25e18d",
"metadata": {},
"source": [
"**Exercise 17.B2 — Speculative-decoding acceptance, greedy version** · Difficulty 3/5 · ~12 min\n",
"\n",
"Speculative decoding has a small draft model propose `K` tokens; the target verifies them in parallel and accepts the longest correct prefix. In the *greedy* case, \"accept token `i`\" means the target's greedy argmax at that position equals the draft's proposed token. Implement `count_accepted(target_preds, draft_tokens)` returning the length of the leading run where they agree (stop at the first mismatch). This is the bookkeeping at the core of every spec-decode loop; the speedup is `accepted + 1` tokens per target pass.\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "f44f1166",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:15.492293Z",
"iopub.status.busy": "2026-06-10T19:27:15.492198Z",
"iopub.status.idle": "2026-06-10T19:27:15.504486Z",
"shell.execute_reply": "2026-06-10T19:27:15.504247Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 17.B2 acceptance run-length: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def count_accepted(target_preds, draft_tokens):\n",
" \"\"\"target_preds, draft_tokens: 1D int tensors of length K (one batch row).\n",
" Return the number of leading positions where they match, stopping at the first mismatch.\"\"\"\n",
" K = len(draft_tokens)\n",
" # TODO: walk i = 0..K-1; while they agree, increment; break on the first disagreement.\n",
" n = None\n",
" attempted(n)\n",
" return n\n",
"\n",
"def _accept():\n",
" # 3 of 4 agree, then a mismatch: accept exactly 3 (do NOT count the matching 4th after a break)\n",
" tgt = torch.tensor([5, 9, 2, 7, 4])\n",
" drf = torch.tensor([5, 9, 2, 0, 4]) # mismatch at index 3, even though index 4 matches\n",
" assert count_accepted(tgt, drf) == 3, f\"expected 3 (stop at first mismatch), got {count_accepted(tgt, drf)}\"\n",
" # all agree -> accept all K\n",
" assert count_accepted(torch.tensor([1,2,3]), torch.tensor([1,2,3])) == 3\n",
" # first token wrong -> accept 0\n",
" assert count_accepted(torch.tensor([1,2,3]), torch.tensor([9,2,3])) == 0\n",
"\n",
"check(\"17.B2 acceptance run-length\", _accept)"
]
},
{
"cell_type": "markdown",
"id": "a5cf35b3",
"metadata": {},
"source": [
"Hint 1This is a \"length of the longest matching prefix\" problem. Loop; the moment `target_preds[i] != draft_tokens[i]`, return `i`. If you never break, return `K`. The subtlety the check guards: a later match after a mismatch does NOT count, because once you reject a token the rest of the draft is discarded.\n",
"Solution\n",
"\n",
"```python\n",
"n = 0\n",
"for i in range(len(draft_tokens)):\n",
" if int(target_preds[i]) == int(draft_tokens[i]):\n",
" n += 1\n",
" else:\n",
" break\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "ee4d190d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:15.511039Z",
"iopub.status.busy": "2026-06-10T19:27:15.510957Z",
"iopub.status.idle": "2026-06-10T19:27:15.528354Z",
"shell.execute_reply": "2026-06-10T19:27:15.524213Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 17.B2 acceptance run-length\n",
"acceptance run-length stops at the first mismatch; speedup per target pass = accepted + 1 tokens.\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines count_accepted; the check below re-verifies the reference.\n",
"def count_accepted(target_preds, draft_tokens):\n",
" n = 0\n",
" for i in range(len(draft_tokens)):\n",
" if int(target_preds[i]) == int(draft_tokens[i]):\n",
" n += 1\n",
" else:\n",
" break\n",
" return n\n",
"\n",
"check(\"17.B2 acceptance run-length\", _accept, required=True)\n",
"print(\"acceptance run-length stops at the first mismatch; speedup per target pass = accepted + 1 tokens.\")"
]
},
{
"cell_type": "markdown",
"id": "0d17ad2e",
"metadata": {},
"source": [
"### Part C — Capstone: a measured cached generator + a roofline read\n",
"\n",
"Assemble the pieces into one deliverable and a short written read.\n",
"\n",
"**Deliverables**\n",
"1. A `CachedGPT` (already built) and a `generate_cached` that you have proven matches `greedy_baseline` token-for-token.\n",
"2. A timing comparison of cached vs recompute generation at two different generated lengths, reported as a speedup ratio.\n",
"3. A one-paragraph roofline read: pick a real model/hardware pair (e.g. Llama-3-8B on an H100) and state, using the formulas from Parts 1 and 5, whether single-token decode is memory- or compute-bound and what the KV cache costs at batch 32, context 4096.\n",
"\n",
"**Self-assessment (pass / partial / fail)**\n",
"- (a) `generate_cached` matches `greedy_baseline` under `torch.equal` (Exercise 17.2 check is `[ ok ]`).\n",
"- (b) Your timing shows cached < recompute and the speedup *grows* with generated length.\n",
"- (c) Your int8 quantizer's max round-trip error is `<= s/2` (Exercise 17.3 check is `[ ok ]`).\n",
"- (d) Your roofline read names a ridge value and classifies decode correctly (memory-bound).\n",
"- (e) The notebook runs top-to-bottom and every solution check prints `[ ok ]`.\n",
"\n",
"My solution (reference roofline read)\n",
"\n",
"```python\n",
"# Llama-3-8B (32 layers, 32 q-heads, 8 kv-heads (GQA-8), d_k=128) on an H100.\n",
"ridge = HW[\"H100-80GB\"][0] / HW[\"H100-80GB\"][1] # ~295 FLOP/byte\n",
"decode_intensity = mlp_intensity(B=1, T=1, D=4096, Fdim=14336) # ~1 FLOP/byte\n",
"print(f\"ridge {ridge:.0f}, decode intensity {decode_intensity:.1f} -> \"\n",
" f\"{'memory-bound' if decode_intensity < ridge else 'compute-bound'}\")\n",
"kv_gb = kv_cache_bytes(n_layers=32, n_kv_heads=8, seq_len=4096, d_k=128) / 1e9\n",
"print(f\"GQA-8 KV cache: {kv_gb:.2f} GB/seq -> {kv_gb*32:.1f} GB at batch 32\")\n",
"```\n",
"\n",
"The read: decode intensity (~1) is far below the ridge (~295), so single-token decode is memory-bound; you cannot fix it with more FLOPs, only more bandwidth or more batching. The GQA-8 cache is ~0.5 GB per sequence, ~17 GB at batch 32, context 4096, which fits alongside the ~16 GB BF16 weights on an 80 GB H100. Had this been MHA (32 KV heads), the cache would be 4x larger (~67 GB) and would not fit. That single 4x is why Llama-3 ships GQA.\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "b4dea079",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:27:15.529433Z",
"iopub.status.busy": "2026-06-10T19:27:15.529337Z",
"iopub.status.idle": "2026-06-10T19:27:27.719457Z",
"shell.execute_reply": "2026-06-10T19:27:27.719177Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"n= 24: recompute 745.8 ms · cached 37.3 ms · speedup 19.98x · identical=True\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"n= 64: recompute 1933.0 ms · cached 498.4 ms · speedup 3.88x · identical=True\n"
]
}
],
"source": [
"# Capstone scaffold: run the measured comparison. Fill in your own roofline read in the markdown above.\n",
"for n in ([8, 16] if FAST else [24, 64]):\n",
" p = make_batch(1)[0][:, :PREFIX + 2]\n",
" tr = time_call(greedy_baseline, p.clone(), n, runs=2)\n",
" tc = time_call(generate_cached, cached, p.clone(), n, runs=2)\n",
" # correctness alongside the timing: identical tokens at this length too\n",
" same = torch.equal(greedy_baseline(p.clone(), n), generate_cached(cached, p.clone(), n))\n",
" print(f\"n={n:3d}: recompute {tr*1e3:6.1f} ms · cached {tc*1e3:6.1f} ms · speedup {tr/tc:.2f}x · identical={same}\")"
]
},
{
"cell_type": "markdown",
"id": "4415fc41",
"metadata": {},
"source": [
"## Reflection\n",
"\n",
"Write ~150 words on the dumbest bug you hit and how you found it. The off-by-cache-length position bug in Part 3 is the canonical one: the cache stores correct keys, so nothing crashes, the output just quietly drifts after the first generated token, and the only thing that catches it is a `torch.equal` check against a recompute baseline. Did your cached generation match the baseline on the first try? If not, what did you print to localize it (the position each step? the cache length? the first index where the two sequences disagree)? Nobody grades this. Writing it is how the debugging move becomes yours: when a \"fast\" path silently disagrees with a \"slow\" reference, the slow reference is the oracle, and bisecting on the first divergent token is faster than re-reading the code.\n"
]
},
{
"cell_type": "markdown",
"id": "0073a649",
"metadata": {},
"source": [
"## Going further\n",
"- Stanford CS336, Lecture 10 (inference): the arithmetic-intensity math, KV cache, GQA/MLA, speculative decoding, paged attention. This chapter's spine.\n",
"- Tim Dettmers, *LLM.int8() and Emergent Features*: why int8 quantization gets hard at scale, and the outlier-feature finding that the per-channel demo here foreshadows.\n",
"- Lilian Weng, *Large Transformer Model Inference Optimization* (2023): the encyclopedic overview; pair with this notebook for breadth.\n",
"- vLLM docs, *PagedAttention* and *continuous batching*: the production-canonical serving stack; the realistic home of the cache you built.\n",
"- Karpathy, `nanoGPT` model.py and `llama2.c`: the clean reference for what an inference inner loop actually looks like.\n",
"- Tri Dao et al., *FlashAttention*: the IO-aware attention rewrite and the online-softmax trick (a streaming softmax in one pass).\n",
"\n",
"## What this enables\n",
"- **Ch 18 — Generative Models**: diffusion inference reuses these patterns (batch-packing the denoising steps, quantizing the UNet), though its attention has no autoregressive cache.\n",
"- **Ch 19 — RL + RLHF**: PPO/DPO generate samples every training step, so a fast cached generator makes RLHF affordable; this notebook's `generate_cached` is the seed.\n",
"- **Ch 24 — Safety + Red-Team**: the timing side channels and quantization drift sketched in the Safety lens get red-team-grade depth there.\n",
"- **The gap this leaves**: we proved the cache correct and faster on a tiny CPU model. We did not implement paged attention, continuous batching, or speculative decoding end-to-end (only the acceptance bookkeeping in 17.B2). On a real serving stack those are where the throughput multipliers actually come from.\n"
]
},
{
"cell_type": "markdown",
"id": "78b9dde5",
"metadata": {},
"source": [
"---\n",
"*Built top-to-bottom. If every check above printed `[ ok ]`, you've reproduced the chapter. Total running time and last-verified stamp written by CI.*\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "obvix-nb",
"language": "python",
"name": "obvix-nb"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"obvix": {
"title": "Ch 17 — Efficient Inference"
}
},
"nbformat": 4,
"nbformat_minor": 5
}