{
"cells": [
{
"cell_type": "markdown",
"id": "d013efb8",
"metadata": {},
"source": [
"# Ch 14 — NLP with RNNs and Attention (notebook)\n",
"\n",
"`[← 13 sequences-and-time-series]` · **this notebook** · `[15 transformers-from-scratch →]`\n",
"\n",
"Runs top-to-bottom in ~5 min on free Colab CPU. Last verified 2026-06-11.\n",
"\n",
"**What you'll build**\n",
"- A character tokenizer for Shakespeare, then a tiny byte-pair encoder you train yourself and watch merge ` th` into one token.\n",
"- A character-level GRU language model trained on ~80KB of Shakespeare, that you then sample from at different temperatures.\n",
"- The attention operation from scratch, three ways: Bahdanau additive, Luong multiplicative, and the unified scaled dot-product form, each cross-checked against a torch reference.\n",
"- A deliberate failure: dot-product attention at large `d_k` collapsing to a one-hot, then the one-line `/ sqrt(d_k)` fix that rescues it.\n",
"\n",
"**How this notebook works.** Code cells with a `# TODO` are yours to fill in. Run the cell to grade yourself: `[ ok ]` passed, `[FAIL]` shows what went wrong, `[ -- ]` means not attempted yet. Every exercise has a hint ladder (open only as many as you need) and a folded solution below it. The notebook runs top-to-bottom even if you fill in nothing — the folded solutions redefine the functions so later cells work. See Ch 00 for the full protocol.\n"
]
},
{
"cell_type": "markdown",
"id": "818bf878",
"metadata": {},
"source": [
"## Before you start\n",
"\n",
"1. A model has a 256-symbol vocabulary (one per byte) versus a 50,000-token subword vocabulary. For the same sentence, which produces the *longer* integer sequence, and which has the *larger* embedding table? AnswerByte-level gives the longer sequence (one token per byte, so ~5 tokens for \"hello\") but the smaller table (256 rows). Subword gives a shorter sequence (often one token per common word) but a far larger table (50k rows). Tokenization trades sequence length against vocabulary size; this notebook makes that trade concrete.\n",
"2. You sample from a language model and divide the logits by a temperature `T` before softmax. As `T → 0`, what does sampling become? As `T → ∞`? Answer`T → 0` sharpens the distribution toward the argmax: sampling becomes greedy (always the single most likely token). `T → ∞` flattens it toward uniform: sampling becomes a coin flip over the whole vocabulary. Temperature is the knob between \"boring and safe\" and \"chaotic\".\n",
"3. Predict before you run: attention takes a softmax over *which* axis of the score matrix, the queries or the keys? AnswerOver the keys. For each query you want a probability distribution over the source positions it could attend to, so each query's scores must sum to 1. Softmax over the wrong axis is the single most common attention bug; we will check it explicitly.\n"
]
},
{
"cell_type": "markdown",
"id": "45ed497d",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Three setup cells: imports plus a soft version report, then the seed/config block with the house check harness, then the embedded dataset. No installs are needed on Colab; torch and numpy are preinstalled.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "e9137ed5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:55.790001Z",
"iopub.status.busy": "2026-06-10T19:52:55.789903Z",
"iopub.status.idle": "2026-06-10T19:52:56.622694Z",
"shell.execute_reply": "2026-06-10T19:52:56.622007Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"numpy 2.2.6 · torch 2.12.0+cpu\n",
"device cpu (this notebook is CPU-canonical; any GPU section prints-and-skips)\n"
]
}
],
"source": [
"import numpy as np\n",
"import torch\n",
"import matplotlib.pyplot as plt\n",
"print(f\"numpy {np.__version__} · torch {torch.__version__}\")\n",
"if np.__version__ < \"2.0\":\n",
" print(\"WARN: written for NumPy 2.x; older versions may differ in the last digit\")\n",
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
"print(f\"device {device} (this notebook is CPU-canonical; any GPU section prints-and-skips)\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "231b1c7d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:56.623610Z",
"iopub.status.busy": "2026-06-10T19:52:56.623493Z",
"iopub.status.idle": "2026-06-10T19:52:56.632752Z",
"shell.execute_reply": "2026-06-10T19:52:56.632226Z"
}
},
"outputs": [],
"source": [
"import os, random, math\n",
"SEED = 0\n",
"FAST = bool(os.environ.get('NB_FAST')) # CI smoke mode: ~10x fewer steps, identical code paths\n",
"rng = np.random.default_rng(SEED) # the one numpy RNG we pass around\n",
"torch.manual_seed(SEED); random.seed(SEED)\n",
"torch.set_num_threads(min(4, os.cpu_count() or 1)) # cap threads: steadier timing, avoids CPU thrash\n",
"SAMPLE_SEED = SEED + 10 # offset seed for generation, so sampling never perturbs training\n",
"\n",
"# ── house self-check harness (identical across all chapter notebooks) ──\n",
"import numpy as _np\n",
"\n",
"def check(label, test_fn, required=False):\n",
" \"\"\"Run one self-check. test_fn raises AssertionError (with a teaching\n",
" message) on failure, NotImplementedError if the stub is unfilled.\n",
" required=True is used only in solution cells; it is what CI grades.\"\"\"\n",
" try:\n",
" test_fn()\n",
" except NotImplementedError:\n",
" if required:\n",
" raise AssertionError(f\"{label}: reference solution incomplete\")\n",
" print(f\"[ -- ] {label}: not attempted yet — fill in the TODO above, then re-run.\")\n",
" return False\n",
" except AssertionError as e:\n",
" if required:\n",
" raise\n",
" print(f\"[FAIL] {label}: {e}\")\n",
" return False\n",
" print(f\"[ ok ] {label}\")\n",
" return True\n",
"\n",
"def attempted(*vals):\n",
" \"\"\"Treat None placeholders as 'not attempted'.\"\"\"\n",
" if any(v is None for v in vals):\n",
" raise NotImplementedError\n",
"\n",
"def check_shape(x, want):\n",
" assert tuple(x.shape) == tuple(want), \\\n",
" f\"shape {tuple(x.shape)}, expected {tuple(want)} — check your reshape/transpose order\"\n",
"\n",
"def check_close(got, want, atol=1e-5, rtol=1e-4, msg=\"\"):\n",
" g, w = _np.asarray(got, dtype=float), _np.asarray(want, dtype=float)\n",
" assert g.shape == w.shape, f\"shape {g.shape} vs expected {w.shape}. {msg}\"\n",
" bad = ~_np.isclose(g, w, atol=atol, rtol=rtol)\n",
" assert not bad.any(), \\\n",
" f\"{bad.mean():.2%} of values wrong (max diff {abs(g - w).max():.3g}). {msg}\"\n",
"\n",
"# ── small house helpers (defined here, never imported) ──\n",
"def param_count(module):\n",
" \"\"\"Total trainable parameters in a torch module.\"\"\"\n",
" return sum(p.numel() for p in module.parameters() if p.requires_grad)\n",
"\n",
"def entropy(p, axis=-1, eps=1e-12):\n",
" \"\"\"Shannon entropy in nats of a probability vector/row (numpy or torch->numpy).\"\"\"\n",
" p = np.asarray(p, dtype=float)\n",
" return float(-(p * np.log(p + eps)).sum(axis=axis).mean())\n",
"\n",
"def check_close_torch(got, want, atol=1e-5, msg=''):\n",
" \"\"\"torch-aware allclose with a teaching message.\"\"\"\n",
" got = got.detach() if hasattr(got, 'detach') else got\n",
" want = want.detach() if hasattr(want, 'detach') else want\n",
" assert torch.allclose(torch.as_tensor(got, dtype=torch.float32),\n",
" torch.as_tensor(want, dtype=torch.float32), atol=atol), \\\n",
" f'tensors differ by up to {(torch.as_tensor(got)-torch.as_tensor(want)).abs().max():.3g}. {msg}'"
]
},
{
"cell_type": "markdown",
"id": "95683458",
"metadata": {},
"source": [
"> **Note:** seeds make this notebook's printed numbers reproduce on CPU. Library versions and BLAS thread counts can shift the last digit or two, and a training loss of 2.41 versus 2.43 means nothing broke. Quoted numbers hold for the pinned environment; treat them as a range, not a target. Generation cells use a separate offset seed (`SAMPLE_SEED`) so re-sampling never disturbs training reproducibility (Karpathy's `+10` convention).\n"
]
},
{
"cell_type": "markdown",
"id": "03a81fb5",
"metadata": {},
"source": [
"### The anchor dataset: tiny_shakespeare (embedded)\n",
"\n",
"The whole notebook runs on one corpus: an ~80KB excerpt of *tiny_shakespeare* (the opening of *Coriolanus*). It is embedded directly in the cell below as a compressed, base64-encoded string and checksummed on load, so the canonical path touches no network. One optional cell further down fetches the full ~1MB file from a commit-pinned URL and falls back to this excerpt on any failure (the Tier-3 dataset pattern: immutable source, checksum, offline-proof embed).\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "777c13cd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:56.633753Z",
"iopub.status.busy": "2026-06-10T19:52:56.633643Z",
"iopub.status.idle": "2026-06-10T19:52:56.648480Z",
"shell.execute_reply": "2026-06-10T19:52:56.648166Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"loaded 81,924 characters, 61 distinct symbols\n",
"First Citizen:\n",
"Before we proceed any further, hear me speak.\n",
"\n",
"All:\n",
"Speak, speak.\n",
"\n",
"First Citizen:\n",
"You are all resolved rather to die than to famish?\n",
"\n",
"All:\n",
"Resolved. resolved.\n",
"\n",
"First\n"
]
}
],
"source": [
"import zlib, base64, hashlib\n",
"\n",
"# tiny_shakespeare excerpt, zlib-compressed then base64-encoded.\n",
"# decoded text sha256 is asserted on load below.\n",
"_SHAKES_B64 = \"\".join([\n",
" \"eNqMvdt240ayLfqOr4Ce+ELpA6gHDVXZ1aV1bJe3q7pr+BEkkiQsEGDjIpr99TvmnJEJgKre64yxVlslkbhkRsZ1xoxPVdcP+cdq\",\n",
" \"qP4Tmk32IezbLuSXkJ+7dhdCmRfNNd+P3XAM3To/hqLLTyHvz6F4fciy57reZF/xj3X63aflFf9sx7ywSxZ1nXehb+s3u2hX4Hr5\",\n",
" \"0OZlFfLhWDT4eV+cqv745Jf9wz/8kL72/uL85zq/2j1em/aSfyyqsc9/Lbod/lv1+e5YhX0emnC64g521/wc2nMd4sN/D/zmyq5y\",\n",
" \"iT++v88vYcjtgq+VvcSxOq1tVUr7/Ar/LN5Cvmu7Ji+GvB27vL00tnjVzm7x0q+GvMjfQldWuyG+2G9tfsIiD0X9WjWHvG1Ww2Ne\",\n",
" \"2y2qId+GvGybsMmLS3Fd83/vsuxr2LV2x/Q8Xxrbo7Yr1/mhbct8p9/37x/cXo+Lv9u1YzPYyp/btkufX2tBisEetyqanld7yL4f\",\n",
" \"7VWKcTi2XTVc837s9qEaentQu+lY2/aFugr22mO/yas9LnLN9JdrFex/bam248CL9+M5dPt6tAvZCh+rOuAtL6EL2eXY1razp8Cl\",\n",
" \"P1WH45AfxtD3vGC8Ca92HE9FE+rrY+bXtd08Vs0rvoj3G1qTJJPNDe9Zh6JpdJ1iyIr9vrbV7+06et92+1fY2V7tuV0mcqGzZzNh\",\n",
" \"Kez/mqxq3kIztB0F5lx0Q7Ub66Kzj+HbVZcX27Epi2YXHnGBrB/3+9Dh37xGfiiqxmXtlLvgdPYizQEXsI9cquGY4dbn6jXYMwUd\",\n",
" \"uK3tsR2trrBfbnI7hnzYQ1v2ku2XjCdM17BbHEe7YscPbrtQmCw07YA/2AcgA/iD3/fhvQR913bZwYkHPdjlTQrq+poXeAeI0fw4\",\n",
" \"RfF99j/aOcj3uNPG1MKql5xfTXoP8aTZ+5zapqiH6w8e4GPb9FVpL4BnuEDibB/e7NjYbgc7VT3PAd8CL0z57a5P7yT8X7jpJdT1\",\n",
" \"Iw/lju+1xc1N3psBz3KoTFbxuBDvrAvntuPy2JmXONnNeQ6uPT7Wh3rPTbLL4HzaAo3lD97gNxxQXED7gtU/FSZqVTv29fX9WXzJ\",\n",
" \"++Ka22u0eOe1XprvarfSyxYnfheK1jRjmVWD1tI+GJoS0t2Oh2Pet/vh3h6m31XB5K40ldzkO1Oi25DN3hu3w2GztbxZRr9+zutn\",\n",
" \"pg6LPvDvp5aKGStpF9gGHgCTCK7BIw7w7mhfzirIrclWPkAlYrNtm6thLAPOFa70VnXD+EPJ89e258WSHUN9pjjbd5piGLsgfe4q\",\n",
" \"Ky8yCIU+cXrIYU5OY09Bb1p7tytf88jDt2vfwmAr+IO13+cv+p7dc20/NxB53N/ecVt0nb2KPbjddOyLobKlfczi1uyLsR7sfSkS\",\n",
" \"pgzPNTWJ7UvV8cFMooK9vH3LdWdv22QKh5rpGPrwlH+D3uHSQujzdqUDYlrRljKHcmk2trr2MkNxlfm1xzDhs6/Y1/1AfSzO1dDW\",\n",
" \"d34SP1J3Qmu8f+GvJiJ3dsWWf+91nR+pAbPs1/xXs5ANzvrzoavO58I0WxP8YGAJivqCw1Hb+pbZwoje3PYzNAG+e7T/6SG1ENiN\",\n",
" \"243CZaXDn2ADTJLtZX79+beff3v551eXDTNsrytbYTPZLq4m3mtKgMnlk70UvnngKcq+85wWWG2c/nrc9lrtUzEMoXvS4cSO24pe\",\n",
" \"8ZX3D/3FdPF27CvaDIihicXYQOlGRW66yWTTFD5ND0x+dizs+DSvNTaJKhkKpaER48G+QDTsJPIgle0609mBJpfjYEJyyVdmI+zF\",\n",
" \"SpPG/gHPLXGGlTbDYpvd9fIw+qFr7U7Q9MOx3+hB+iNW9FXXzOafK7pTD6v4sFxd0yynord10fLSedh3pkRK/KKa9q0J9iZbM1L9\",\n",
" \"2pbYboIjaWaPi96ZhnwL/dOPvA2davMGq24drTO+10BDwExdF4/0kg9BV1/PHqS1i+6OZnOHYlvjmuYtfMbb3bgrdmKxofmnlmak\",\n",
" \"M3XQ2EmFz2kHjXYZ2yOb2NNDGI5SLydbZdOLMBzZV7vga4D/huubjwvFxtPOi9qRtLfFp+tqz8+ckhHEF/5ozTnBpwZ4Mse2h/HB\",\n",
" \"Itk17N3aJoM8XqSMh4I2f9cVOzp/A3SoqYue4jt22z77spd/6FtZw88peltEeMxwlaHpTfl22fP5DIfc3o8PWp3OoazssAxaEjyc\",\n",
" \"vzMfAb6EvITlOkIoXuGX0a/V6r2aguyjG6MvQabWUqJQ2g/5c20vsU4e/mBOUA/bapp1a2e3qM2fNwX37VhR8+nc8tVwNm2rte94\",\n",
" \"c2xJXxd4Rz6q3cAE2NXktLQUBRozfKHGru0ZSkBMj4HrgFWUr3bCnsH3r8IPTMLHeKmxv8u/dWOAisFBvNM5bMLKHho3LP1j2TWY\",\n",
" \"syO5gleXQha9g7zD3s5suMeOhj6zbT6d7PsUpkNnMvOotQ4lXVJdd4T/aXbYfGW6JviNvdIjDUtR52VRwSmzQCy5zGaphswOqp0P\",\n",
" \"uz/CtJlA2rb6I5nZfoO9waJn5yqYI2cShfUcB9teXpimzE6bSdF45regnvGs0vSmih7yF3r5JsSmj0wD4e2pJc9raSJIOh0wfKej\",\n",
" \"O+i6HjZDn9lCVmmcp/P/s0Tj6lYdXuEeWnjSMmYOmtK8j35yrtbZlw52m9ba3t3UwL41v9Ue01ViVCrZsy1BGCyIsWDLgqpq4MG3\",\n",
" \"7+KOVJgIaeEIPWYfRuotOvLyRu3upiXPZvLtUMOG8EAjPLANyr61WEnTTwzy6moYaq30j8KwurbvM17kiZWG3OQmUenl/WTiuJsw\",\n",
" \"7NutvZhilLLqTXh2QXJU+MtsRx5Y3F4eXMbjVFrcZMphsczfePTgCNqXTUPgLDZpi7ZtebUdO4XT1sTOAu+tPe5qKVP4FUTlaBsR\",\n",
" \"170acGWEvY3JJ0+jhT+jec+2fnAvu3CCGL3oGJ+q0i7mZxr3tANX2pJB5MbGBNoee23KGGu8G8/b1jZG1jWYN2mfMjUEtUdBsj9k\",\n",
" \"v+COdbGl5sfKRN9i7bpmSF4X3qMboRr77Cd7sj7ovtgNLNlb1fP841M7fN9ic7NIIdTr7Bl3Po3DyNhIEWF1pkrCS5rZrGBSs382\",\n",
" \"7iwUZ/qDuoOFnxZtmm/o4RDUO48SjjKXgXZfC2zf6M0vWpX39/9FhmRXGaDzoybPZZh26Gm+61/x2dsj8ZB/lxCZ+eHR6U8WlEN9\",\n",
" \"0j1xpWcisu/ak06wBZq9Ih25/CYD9nxmYdZ2uttXafEXniwqt/Q0unb2LDsLbUxv7P6ehtBcO9vFGpH+2UL9EgeKJqvqPYhhZEOh\",\n",
" \"VORum1A1UAXYhF6yF5o3+y7Dhy7sQnUeHvWYfStHYl9ZAINnkNmvq0PDQ0WXDv6VQu1iyKj2YRVwEPvRlqPof+wx0kTyFaHpuBFP\",\n",
" \"OZzXO1qvV77X/a4zDzJQyMq1i/GhMitnT30NssnwcE3R1YjPYKr1OTO1esa2Lk1rr+mimpCF0hMcB31uMAdhDPwoxNvEDgefGyy5\",\n",
" \"P40NHQK5x2eqQhjvPntxpwjf3Rdbsxlr5XLkBl0hgTeOuf26ecpWn2DBEXnI7cXDX7SxZkinD0pK9Z130clRYfrV8wSdqU0si6Rm\",\n",
" \"G5IJWpWQzBZeOT0BKMe5Brl5Sh6R/8d9seK23SeEuAcsyzrmr3iU7YSea7vrWu+rXMJwcziflv4rTUKyNy/0SKHnt/YGti6FnQE7\",\n",
" \"fvf3dtL4YMnwyGDYC/xukR7C+Lxgfmwdr0BLke6eBO0H0hhWHQwt/H5ThsgjLrT/b+2ADeN+redO/6MEmcfEDMxbfFEYCpgRO3lQ\",\n",
" \"c7hA3hX9UToeuy4TgHMpz2fsk+raZCt4U9gymDkzoGZSWzOhuFZy8lf5v0cTUXvJtX0eC/Oi8+tevu2O3by23bKnLQZlmqKWwhpa\",\n",
" \"KAI7Zy5I2yj1YwcdhsfeMfsQdoU9n12zOLkLmbyy6KshADu/18ZuV6t9vIsZMeqgNRI4iOegvI4dEzHytswmxVjEtIKpRBPan6kp\",\n",
" \"Yx5sjAc7nvEYVRaTQaR3KGMzv7y5kM2rjq+5A8yP2a0s4pD+UJCgkI0OCz5IkbNlR/xjiuUtmGHLPkGfI8GYVtluzoyLLTMk35RH\",\n",
" \"s7vCj+6CTuaVS7zxFfNHgumBMofZ3wW6/++DydX9PbVDj7TBzH84Fd0rdMcP7NvzldbtkbZizf9dSLFJCZ9gdnOPNrOvIehwvWDD\",\n",
" \"3AHKcRJsoUNh3rCdEfwVkRNNlD1vMZZV9GD1TmmJ7CJbC20WArmvmdzf44P0CZBmfuO5irlu28LmYSUV2CvZYA+wGt7roRdl5exp\",\n",
" \"dGo2OZIB5jfU/i1bu6cbB24yWPYQfzAG6DyjzKXXCvPJdImZvXQr+khbF/4uEOrjmhavuAHqZyEMYp4ezg3lCh7pATmqwxGG9Ftr\",\n",
" \"hjG6ZReEJzGZRfdG0bVcjr15GKh2nEeLUnb2gE3AGb2kM+zrC88bp8oz0faCnWetog9yypU05dt52o9/m+KEB627MhRyo102qU6Q\",\n",
" \"N7GLMD3JNSv63hal/kFG+WX5jTvm5Ra/WmzNJ3cf1p4uRqrDl8QMI73RrTnn+C9iKfwgrcN0KxJhKC10dLorLCBOmm3pSjn8gM9A\",\n",
" \"ALBeRW9RdRTYjqkypUKpdbBI3WiK4RdzOPh1vpj99lLBHzKReTMrW6AggCXnSdA+FOU1JjsqCzqWCbVNJnGjv9yheKfcpidMzm3V\",\n",
" \"sJxi3zKL9khhxSow1al8AW1LgYzhZ4s5kU9AYscrC0gBPv/x0QUd6k6buZLqUCZPgmW+oR2DAVFg3rWoFq3pBq7txbfbKJYsclWD\",\n",
" \"SVnUyu3ZvHRb3OxXf+cYXdqCbn+cyeIzM97gFXjGUHN7mD3tZ1ejdAJYZkgfi8kTzwLta76FWaYmMKO6PbYdohiX20uqx+C+65TI\",\n",
" \"0OvJ8Db2VuZm7fSTxeJPinCQW9vveUCVUsmmdDP2uI91nrF8yOMjm7/YD72fq7W0fi6riCfB2fUkS4sMEf7d+/OZDD/6F/bt31AV\",\n",
" \"hxB6sxNXzwTZCWUKA/vMF2iiA7droTD2kEyZbv62gh2xmP5osmGWuglK2aF62DDl33k5wS6DKIGCi4LORelrJd3MPNKN6sdtOQaW\",\n",
" \"cqgvYj7IXvove2dUE1T7eKBrWQYP9Xm8kQPOfoq/4s4fmfn1LFU3BXQ8BNmzibmZCjPI8NA88lOyyq5sr+mnnBGG3BfttXlFXaq7\",\n",
" \"WABaT3tThjOsaPZPLNBVHvobZDbvL9VJFUTsCE2BmaGS73kMFxTNLhbemCuuD9kmH6EbPxd2NK6BiS57mD9NhTFKCKzZmTkYhxAd\",\n",
" \"np3t1wGhvP26lEHZFSp8+7mVzBc9s9lpjdbZZ/vEWzX/gI5O0SG196NDLenWbvcwcW/0+swLd0cnlUmQZdx110VSQk+j3DyXfG1B\",\n",
" \"uNKkMdv5GsKZL2b3KC78FLbADF7wGugeUZWtM49Rw1PzNHtSpPRCQET3TuvHqr8+hYWHi9s/KvvQ7j03Zq6Ax3qoplceDNMjXSoT\",\n",
" \"7NEqnO5SHYDR5JXFAkADZIVwdGJ2L/Qj/PqWNYDMH5rJ9mo1rxjhkcygSI3ASiDdQek3V7OKbqNkdlejEPIo7b0vJOoZ/k4FZ14V\",\n",
" \"snQoodNzNI+uq8yk9Mxssl6KDAlUK11Tyo+t8ZaVkhhV90PRlFPBGutnz4akzoeAcNIdkXZrG1xCJ4V5ecRzjBkzql5juvPtdLmo\",\n",
" \"aix2zSx/VcbSfTciFy7nbYDrhnjAfMFeYAom6KhcCgtP7MWuOiUxQR8l0r6EPw8Id5C2foMOLKBxDkdkGl48dDxTM+AxGsBB5iFZ\",\n",
" \"wWQariRwjOuIlo4/0kzmro1FGSxGk4Mh31fQA1Rk5Z7DS0W+pAsDzRvcXC87IlXV91j0XWu2okR+lCnOF8SmIcgH81A3+emyGrZz\",\n",
" \"7flpYZY9NQIDTDDOBtU4yWvmG1OVngyGZSiaeyAUuisk6XBcMYlua4mUdOi20bA5imHbta+M0gBsMWXTu5NTtuZ60oUw3exfOQVX\",\n",
" \"Lkx9waM9mVAe41cSagKOAkTMxNXOaQRUIDnOejmSlpuUN4TyOdqB7PUyb8o+RTFUWgB+cKyBy9eLMa9kqshjHRjRdLwA0EI4C1Sq\",\n",
" \"9n4W+pgJQyHvdQoJIVgMedseSBlejoK4bSFHLTJ9zLWdC+QMHAHThXRSinMPydP6U/i4PfSd3cqacDWppnJqIS7IwwzuNdlVwmms\",\n",
" \"C9Wxb3M/iDJm7zSXjU8MkbrKZFMlozLsQwom3sba9D+c3LI99e76Rn25O7ZmjjdYFtNO/zOyBv3BTimS/F+rXcVf/CvU/A2V1Ivq\",\n",
" \"nbav9/erryWcKSXdumJLa3D0t38L7v+OjUnzflUmW2LROUvPUKBv5nKsvEBzUoWA/hrUUnUyG8GyiK2g9kBBUmcPIGmmzyCDY/EK\",\n",
" \"j6pZp7Hr5CLAJegOI9y8ZTRXsdDsUjE3Av9o4U15moj1ffy074oD03hwlQNcYDs0G/lgdo9bnM7MP+3ChqHxwuw+La6CxWvgOyA7\",\n",
" \"xPwyPvmvtu5hgwthHFD8mz8nMyuHuqDpXA0sSDco+Cr0OxYrnNOG8oDDxAwmDvJ1Dgl7yC10X+fKpuKQ12a9p1rdV0W8m8xfbW13\",\n",
" \"giOJ9NKwTKbZLkAptkSfKQT5X97gWyzlo3hjPhS9kbGugYIY91VZjVGjUCDO4+ABfQ6E4AtykTmtz9uVQA0CB2R3dH6pBl9Yt2MM\",\n",
" \"zTyBpymKE9JJOqh2NCB7qqIcIQ0fv/z6Ikn5M77gHvofsZwJx1HVnfgi3+m5F/Uez6b/HgkPnDKsAXU7xW8P8ipPqmRcafRQqXlr\",\n",
" \"a4aK0DrZFzzK6ap6H8+GOX8bR9kUjAmyb/FNFFnw5mMzvN88W2iLat1VjzuZPbMErLP1sT3ppCtk6lVoXCzEC1WQfGElcO2uwO7N\",\n",
" \"V4IFD6YAmUjT0wEnZd7GYP5DZVok/wXuifa2HU0DFvXAchB3YBdS7d0hARKJFTyhIHjPmqE3o3RGy0/yZhBwmza1s/Xt5ZvSruvl\",\n",
" \"yXzM6GAAnqj3hrXbmbaDT8QEJtEjtEgrGmLXVANrlseKyqfqE1hloVS+rHkw7s2ylHfvNoFBFAwZxbG9dQ5Dx/omVKuKDVJsyCEo\",\n",
" \"o+cle9Vv/QVn2/PbTTx/c/dpi+j4wPmiZULZtr1sFseV4n6Cnwj/+pGICZcKZh4GwQlgRA9NEQ2g2ZgGqiP/3nb9sTrvx1rpL2ID\",\n",
" \"FEIWTOGdAc2iC04VviEwaO2PArzay0d/qe+AD7wxiG5kLiDnRa9gtppr3A9//PObJwSOjIvyYB5ivbyca0jqhR1C1mYym9E9EbCK\",\n",
" \"+dF4TbvL68pzAxV8eFXpw5UQnOn6trQP1DH4FMtr2Kt4lQ90WE5AcBFeSKXGQtdZCFpz77syeU+LJ/8Q7k/t7tUdhxJiAf9hdnVs\",\n",
" \"IMIRuFxUGmV4Y1w4qY0DqmG2a+2kL7a0wrYvVUHFMd3xK+OECAjMvpkjXSccBVIsFkiYsDBL2ZeICD2lX5QMhARTtOU1xU7gMnLd\",\n",
" \"9sjM7itNDFyBidFnomnNpw2eUIYv2r764yGpCVRK6YFl1FSzV/9UwEq74xkBknlRwdl5aRBUnSL0SUBZyh7r+bYTuCF81A8Bhhm3\",\n",
" \"PAJB3dCrFVKGZTAhf7ZXIBoQHCPdqzgpgnAhQbQuphV3CMrMH/lKY7xdlFTsCYhoXMegglLXQavGCuw4MBxBspuVhlzRX2lvsAtw\",\n",
" \"b4JAYbT5CMqRardPR+D96gvqJ0eAtswE2Sn2mrCrrbvVUrgQoMVKHHPNh1ZFgOyL0nYywz3wFiYI3M94K7obcCoF8DLDAKR6125/\",\n",
" \"tFUATCLOrpnFT59YAfXG9IqfBL/4Q+Ylh3izACUjB1g4Uhb4mRrxRWV6avFs3FJdPq24cEaQjoJWHccKD46LLo7BLwGO2xGiuU54\",\n",
" \"heyzb7yJ/rmA8agUC+kjVSMx2BemB5nyZLMBxIcVGVth4tkHwYwPFlrLJaBKi4dYcf9s7exRAOpp6creqPev7XqRZqUnmZwoeQqM\",\n",
" \"TOaVC0RYwFZVKsfHOgQNN138I2GNsSqAho1/fnr5Sd4AaovUXkzlPilGoa5Wqjk4rA6r2zYJASgkmR4oIcW3XfStUAAk1Go3EJWP\",\n",
" \"Z6UM76puN57emHlunvLVN0eJ7onOQex8MFtu/gd0yItjigbu2+MszHe/rOw3+YtKFOYEHiOwsZYKgC1+zK84EUyx0p95QPkr+qvY\",\n",
" \"o37FeBNBiaAZ/BgfisBVWBuG8qz4FDjNph5CL0/dYYFSy2bo5TDL9qSqkXKeum43WozNQDcemnUSc+28rSXbbFK9Pu41ahREUgdX\",\n",
" \"YsrJy0e+cckK5UrdIghZydwZE7hdYG9Hmc+E1YwXI1bofL4vI4St7dRG10LKy7aUv3bg4Kz5oAVW6WH1TqLhAlnAgKq7p6L36GxB\",\n",
" \"XUAYJAC3mH8o23E7cAf0tkWvcgoUgUN0qIMm4f2t7ZiATnUqrDEBbPhOTFi6C0b8GuSoRxkXoSrxUcBvZQ5TRMWMKRIiiqGcVNvw\",\n",
" \"VEXmL3CEqrDDv4YvbQ7vyXaTCesCWSwTfjz/Q/7hOoFwkBx+wDsnTWa36OxxVunYFuiGktWzV6cBIZYUdVZ6mhczzWFKcfFIOfzD\",\n",
" \"gcueL9q3Un5CqKedkGc5UyaxgEPbXPV9hfr/sQoxOlMVHtm61LcFtSUXZ8TB/Nh2VVtX7ApQhjYMSpxv1f4G9HnKFwX4TNkHIolV\",\n",
" \"ADC5eBRKwE9xgmyw4smc7ZtQRJLQhCBdyIE57BIe4f5wCl+8l4S1zV3oYPuRS33I6TfDI1ibtj2xL2OHqm0b0ymenxCmlZB9iqGO\",\n",
" \"mUJLHhCkAgGUVPVayXuZpwesx0V1vmUjHSoCOy7gKYQBgA3o0gvSatswXKBsx34K/nlAFEGZ50YItDwr87qaNkIk2cIQYckox1Y9\",\n",
" \"1fndfI1oDNKJiGa6L/bhfZDxyV7eUQO3QjT7E287+/e/vvzyz19/e3lGSiO2CNjG0DBDt8JePuYsm1PvxuLhnjmFjE6aSSIWnWD1\",\n",
" \"3n5gVxzSyuiYI+jZVn/st8QwvmSxxGb6rK5QUA1/td7josrutqczyrAMbQ8hM2fV31461M90OG3hRwrh5Y03W/jJ+GZGa4M7UTFI\",\n",
" \"G5osow4DqHSg4wGFZbKcIboL3T0sILs+S68t1nqNli90aU/btZTPFTlXOeSo1dfqnjjX4+41yCk6FP9RcYudjPwSTkEBg4nrId1v\",\n",
" \"vpc9OxQdKgZF7D/yFYJjDHAcS0/wXxj+42wc6SsfAY8zjZa/rBnmQ6WjhnCk98DVuriVZ2+dqg/MrZtvo8VWg1TWwN0fhgixP1co\",\n",
" \"b4T7WDBhMtWTKchPw1vNusAWEZoBd0fMQQU+s+gdkkvFinAXb4A6UlYWTHhfYr31UkxlGRZd9wXUMKwHcgOhzuzE5i/KZttVHrUA\",\n",
" \"FzoXORWUPWnDqA67b1FWn23NmfJoqS1ecdyJUEM5ei7bUDfIQHKhKckmV3+11wS4oiejNiRHD5tWvzczUgPozdDlgu/ow/Z+/lk0\",\n",
" \"xzDRX6aGOjsqBfzJf7388Y+XX3DeAEHAB2lt5AjOowSAhcrC3tetGuB802FFUilP2Bfv6ZvnfukC6gw+2pkb/CxdZtlhLpJZkDGg\",\n",
" \"9KrOZntqANAz4sC7gBIBnvEFYtv+h7BSlMMBKqKRu/JIZYXQz6xpNfBSerWbeG/zsWr0V8dJTQnNF17e+6HxY6iJXkVfNMqb12iI\",\n",
" \"qi6LzXvyouyKb209noeRPYOxTTfGYsmB/4cJTh0ucKWQS7AlVYL3FwSk/ypqOzKFmudUSnyrUGwU7HW2WbOSkQqBp+A2BPXEwF64\",\n",
" \"0xVbvVSqDHXmqCDGOL8GGs7enWW3Ta7lpSlRyTS3c010F44PdUvyBNxgX923sWOH+gdFE818PCQFoeLLfHZ/HJtGCdvTZv4cvd/G\",\n",
" \"YoTTmZBGxVM7weRHcydWCCDzCHJSWa2/UweretJYz9zjpjF2TH9EHIy/wgParHRYgdq58syq0EhMkCoSR1fB6EQ6U8nJrnvIthb6\",\n",
" \"fYCaOBYAKwz3p0IGBLn+on+VV3eya3/pqJBb1JAccHCsaISnDf68fJ67/Ev+P+O5GoTi0J/u5jv7TLXO6kTb1nfsrYAKka51XSHw\",\n",
" \"x6GqPSvVtefjdeNoOYQlNFufw27cFt7C03tvqinrV7On9rchIs1XKhnjxMFw6uL6wIrefQDUWs0NaOE4V4OvGh8/ex7yf3Rocm5i\",\n",
" \"xZdA81NDHM43qEg/EOvYOb9XC+62EvbJnAbvt5ytHBvGsHo49dQIXSkBBDg6Sexi9T4HgYRN0yaJXvHpldjBaqEHix4Q01hKGbPM\",\n",
" \"EHYgWvjX8y8//4Fr/YpCcAkDum3R30aVCCMrb3hxHr+atzZIsy5eYqrnoHE4uGdo/74imzq/G7IPjrLD7Z4S6se2u9ojN0iY7T3c\",\n",
" \"NhZzRCfQyUHvw2Vqa32GxQObBHIpdjSoGveFnQRTyLxNxOB4d40p36flU0MXvi4fNiJIuQzpTWcrP1e5eFVmDhmXT0kW6HtoH94g\",\n",
" \"Y+k1rX+/O5rE36uTcr40X1ZykjqH+6s3zUSzR+aRyX8LCE05MBotMoJvvEFp214fcl3ATgm28YUyH9LGn/J2ZWJZmnHE5rLSE52i\",\n",
" \"WBfaMP0sPwet4GaY0X8m9oeGMASouuLC63WjubB7OD0FzijSnOYKDqjOXTcOCYGZhffOFAkA3cpU4Hwd2oxonEdvdMGF7Nez3yGC\",\n",
" \"nH44ojv9FHyV/WM7hKVsJoq/YR5TyQtbbZMHhBRNVxzkTazxAfbODuaSxFZ5ZgfpZ/XmyoZBFZVBzVaP+Rd2bxUduwz4FjhjkNkT\",\n",
" \"MtsBrfaL8wmGDVQ2+2kTT63y4mm7o22rCy9MFo5IohVaHK9ndXyuJ3mMF1ET9wRUiShMrArAEy8zBCVhfPZRNuaxd8rMkh2ofYgF\",\n",
" \"bMe2YisaZeinZ0A9ajoTj7GdDRrVfYaybbvFG/5287fFEn09esC3ps7mj0ud4uvT2I23ji89e7PDo44Db/6mAnqGDE8Pn145jqhJ\",\n",
" \"5eBO/UheG0wP+amKeEnIexOmCI0hz9jA0LSNRWcm1BDAdZZa7g7R31HdwxYHWkSRQQ2NWjU3apJrxpIt1cfZ3hBd++ZzHAbvutQV\",\n",
" \"aS5A9YFSqwWWxF+r+uDUB4c2FrMWCopt2lMwulB4KfNIeoe32P62ZsFgUEUWTdAK9GaLhEJyjIUizC3/PTRmbM6BfYgztBrB8FiR\",\n",
" \"a9HJIPfnkc7LP+tr3weL2mKMiqOHl9rbsmQv9gS7It+PNUGeCOb6B6TZA6UtQlzZYIaOo0wcAGY8KxybiBe0TTyEbj2V+MUNQ28z\",\n",
" \"A7UOYWay7RapAUbwMUEl5GSigoATwZTLfzsCa2RRShTD7fkqF9bZqaDrsDzwKLFim3Hk4z1Ojg6lQKdOoPD3zn4kBAnYighEdud2\",\n",
" \"8VRflg/FUMXpPBRO2O7MH+Nf5qHUlBEmVIb8L/IpeMP6Y7rAyXEdODkZ9H2NPDA5Cn54Tnn/p+X7suQB7HpFt9IC0seU4C5iS4Jn\",\n",
" \"rYCo/YZWIOaMNwu3W0ALojCuWtjHCOXMWCKbMACz7p9KiXXn3sBSnAWWIn+OGu+Z+Nq4CwB9Udzmlel8mHHI5jk+x00BJej5QGd3\",\n",
" \"wNmAfDE3lzlESGCvMgIkyNTUgVqKyiiPgB6szzqDRDmXgj2UhKNv45le/zfh/IdHVSY5I7pRf6Cr2224RjQrobuZnhUbTqW/UCS/\",\n",
" \"BGkhpj/X1Gwbgi9VkEWtnz/z0sgBlVVPRLLwOMyGnKofnQH3T5S97mOe6SH/FN08eWAIXx7cwqlqS+8TD/KQ/95Vykf8q+rM+TCf\",\n",
" \"O6eeH47XDOXYExNKtD4rWp8HlSFbVaH+6/mGs+k+mC/edLRjm/UDF7Q/uiEAIvD2Pb+nN7DIa0oXJiDDnywfe58KjtmDubOX4iBb\",\n",
" \"5pWaEw/uL89/fON3zFE/shriaWqyMcwvSvUO/O78W8+HLiyRwl8hRXD0WCbws3ISqEJVmHcIL7RyMb3zVoWL7BCfUAgAgC+hfZeP\",\n",
" \"i7Jesot6cMSnVbMA3VDtbcerPNS9RxzpItgSN/Xwtv2HSMm0Ma3eRGzDycWcMDq5t4AUlcifXwMP2tcRTUbe+3ppFvBpgPKKjk3z\",\n",
" \"dRVS2Q3sD8vlQKgdK4JoUJbTXzBzMYeHNW5MLiH1YprKWNUFAwOPz1H8hj/yG07Tr0XX65y7bFNZmAb69wgoMPNP3asXIC/utfWn\",\n",
" \"lgZNIYi3UXcmk57pZJfdt1ZMSbKPwMmlZrs7t35bAQIslICeN1UM0OQMy1b1wn3w5Z0upa6n5pdZyUX+BCN4WeE9EWTsB3FitYat\",\n",
" \"m9+UbsAvY+o0UiGg1ad7vROBga1Ynz0jO4B0It5WcTn+yEzyQ/5dETFxsenh1tkfThWoPFoivTkzeWfvN543OgaAK6UG0aASF7I9\",\n",
" \"ZPZbe1LQxP5cNU1EfKo94jHB7M0Xaryk4kU0vYUa51267pxOoZoW99HeWbQDzl+k2OI12EubvB5U2TBtAbdMRnAua1/c9/KmKoYi\",\n",
" \"6QCpTa9p0SW2lW6OPAVI8CGPV7IT99je3cIb96LcEFiBVWJkO33tJwtI6Y0IR2TfIWB9tJTyW1Aw9siNODVPWPHXSvycAbPVRvkV\",\n",
" \"TCWWbwXld8v2YpplfzZ0agq7gyekEjbrhrKNqhwsyqcdVR8TcvgXwr9x8wtiePl+TMrBs2A3ej91Zikz2btzFvuGVIeQb7L25pnY\",\n",
" \"tBhCLfMdyiU099npMpAzKg5VGyXFbDtKIXXlYAMWt/+UK3oKqRCuPKFJDhp57+8/tJU3XVpEh0a27PeaCQV+rF0tHWCRlqhpbFVm\",\n",
" \"n0TI6YuNTDeDbDZOoTspevgLnqJLRYw6NN4dedTsuVVBZPuWa6Vt7Dq3pz97szFsIAPmGIR2Y6NmWTU/eNnqTLYWxRjFcJf/Xo/I\",\n",
" \"T3J16/qOC3i0J3cs4yObbFEuL12u2PQDhLuEa68lVbJghMKzZ7sDWZmcPFBEHYJg0tmXbj3vkFE5jJk5tz+erCYDQJgA9hHlqo3b\",\n",
" \"CGWPrl3Fivb7x1lXv/pW9gUP+5TCiy2qOlIXdNSsExxfuFZxNJgyPxLtBzGFUf9KXSs8D1UYlQBjMvyaRRT3nVjT7N1JiOVpv5Jo\",\n",
" \"LIYR7RNVGQSHO6lzP30UOVNzy2MPNtewFOIL5YsJ1yPWCTQ5tfX9kTQspuAf6Sy8zMqr8XNANLy8/zqx3ZMzBF3M08eYWuVfp2QV\",\n",
" \"J1/Mz4Dob6YAY/+BF/EmnFlkpPxKzgZB1+m118LGvnsbLFXsEdVqRNweM3FmsWvynUk5wvYKpUQohW1UaljsR9O5zTr7WBfns/Ls\",\n",
" \"EYY6gL1TDnYqfdH/ngFCIvGOdO/0pl88fyQtdkckjUfHxNiVaL1UCjIALAVTrES2eANi4ntrKnAtWYWBRHzCHC1i6P2QkDtMSxXd\",\n",
" \"dmx2Nd9UnWW20tUB/xn8W2tBzOkqtmqgKSzmNI/Yr3xBXFlEtpLEffDR3AngHcmQxYAaDFROwBC6DkG/oOZoTx7QAtQ7sMLbcq75\",\n",
" \"oUPcameSupJ4geFILjTVZs92xZEgEGljFPxGMBCoY9oiAG9hD5EJDIrtVe9ZzaDyesc9oqp2lGJGbsOO7Glb/+Bs/GLPRJlbbKAy\",\n",
" \"uwkHLfzePgCkJ/d4z/QlNY/YkfJicfb+UJ3um4hiTU6QLepERvuHKg9++vyTwrnDlwUw+wgMb/yLGdWOHXJgGL7Dp6jbnKzQnr1+\",\n",
" \"W3YSsNIWImslwcJdHxux4NT+J8bNSDr3qKcUSmzaGSjtv8eTOaMjcYf9Wl0VDcIL/DN76Vp1rKKUihwszmodhj4STTbocZEFyT6M\",\n",
" \"oHX15rvYvnvxwP0Ue+fQWB8b8AiVv3oQJCi9kykjz7N7pafIDEDs+TrdyfSbPnTXTV7WPANBv+MO5Xjbv5nnp2aaiIuAJQVQk32o\",\n",
" \"E1bp98i4lrIVyoqYsooQN3pEa3omQKXZyamYMxpJ2JDyDbQAUWWg+RqN0oA93C5RQhP256qreJLYjmOeRYohZsjYScmqPcPbf1Db\",\n",
" \"qS3oXPWEBl7z8DfepHdGVMErW+ROW6a4TJzQCC3BdPJDdnkr1bRs0DibTPbMdJhOeERQLI8ZV8b+QVOYKTitSnZk7ec5BSGGxHbw\",\n",
" \"YiLXnolOVIxwPl77alfUqgsKaNHK+uJC9p9UakM9NyZzhYGbubbL8NVB1gW0e2tqH8iET7K16+wTVHkJgJIdM1LdReHypCCeS0A+\",\n",
" \"eCsWBP1a9YdRHamm089gluv6lQd/d/kHpL8PLNkT+/h71/bnQB7urb5zLg5h6ehfPRJEnRY2Qe8voUByhoBxVpcPdmpME8DpmOU0\",\n",
" \"ZqEGs8QmCRUyikmNoSmNapUP4ExTEa8BdyIM97zJY/ZRtk3l/GjnRFzTpeZDhO6RcuCWyFSc3CA7du7zqcnkA8lPgzAIp/jWdqS8\",\n",
" \"mwMBw2PmpKPuK+xVJ9fhA4Fc5ajYFsSJEcUoU7mmIxO7ZcXsC5u4QU2WhOfwmuyI9OsFKFJ+qFe0gK4yD5twuMjIOoIm88OVhLAd\",\n",
" \"GvjFVG2nna1jBzIxxM8zt6p+cV63j0zl/roP+Z+RcxTIuTuyfcROdXVFiK2U+YX+GEnpo3/fetjNvGmiNT4xx5ntTUcPIM1UxbAj\",\n",
" \"XvRXiz6YsBFitWhe0V3TF7uu2pPnHiKIFNj7vsLINI938DSr3pKBqPfN4tp0GFK+VpnWCD8Ur4gItCUgbF2bvK7kV5doZm90VWIn\",\n",
" \"XpQGhygt2sm+RTYjhEFIXqN7S7n9CROy/CvVEg8MMk1MQVbg9ydSaPnmz9tW2e4jazNeyFo8QCrkKCB7VDq5vsZkU9xRZk6U3CJg\",\n",
" \"vffOMzgOSl6h6sWzGe/Ht2eehceVJQDz29AiuXzMr+dqIlnwRL0FzjX0Lm5g8teH9VRWotuwE6btAmfZlhGY7LYTEP5EuScF2ZoM\",\n",
" \"C4JOyTP3tguvV4tmMyLvT1dHcC2W6DspC67Ms7rwEhAgfc1+e8cO7+viuiqf8i88EbEJy6ljT+dZyOCVGlE4a9jFPQkxGUTDRYNl\",\n",
" \"mJushdBQn54Zw0NZeSlO/miEHA2syM26Lrxj2TMFIl30x1k5g5+CaSfjCEWjnrP5YzDJQQ8ubuT0XGDOcr4yar2JVsEtJRSIs+SK\",\n",
" \"9wfmpUy8wTeJ1C/ryI6wQ6/ZNbBRBYTSqB1I1HrFGijqtcCRE6VqhxJB7ynAX40fwQ2a8TxUJLJ1bjnGQzykBTAi5vN3zcpbwkqo\",\n",
" \"34Xe/8QIF2+BCLFq8R44EGSYk/5a1HvmHdDPfSRNhcIEthCKmix5ZdiZ9PbsBigFAhJJElvYi1jNjb8Kf+OcZn+YHYH/dEhYBFQg\",\n",
" \"CdBC/dr2pokhJtf80R6VKNWk/3xrmkLB7B9MXzBmcgrVfXHhNcxvuB4FUmziYIv+yEAyQUrZDagW5ZvDkxxVtE7BRfZMGnukTyEG\",\n",
" \"5krWLMDzUZ0+TVc5hqeJBCbVrNMq/+KPw0j3EQeQDDsNmspk8aFa1XyH0MPrFnByQhOJMJnZ31fgNyg8KXb3rkHydAf+BPYhgrtD\",\n",
" \"fKXE9MUGf2Si+onTcTuWBz9fos/qvSMlZZQX60ZQKhMvkRpAJfip2zoOPHC10QkBibrv01SRpQV5mO9CLMwkJFTNbL97REzpPyHR\",\n",
" \"xFQtGMd2LMVxXyIBO+uY7eLsp37qyFgFLVn1RelMX7Fx3OPZ7A8hNZ0TTP2wpDe+LaUQ/SAbJPv7JDXG7GTjLR9ArMwa6ekDRst1\",\n",
" \"8lQ6CIeW6gpoSw56WU/t4N/8TshYOHfNW3Fhldkbp54b0yHM8ScKCXIG8AaPzJgqazNvPCMDhXI7zrJxbM/L8hXZ/uaEKOaoJWZk\",\n",
" \"vnmfkPjRR9Oi8qOTlk0O35EmUoCslJR8Ay47bpJIe1sM9Ri7iRRzSviWFRgk6quF5NTEkfUoBTCMLVCB8FV5dJ/Hvw8ZLEMtrpzY\",\n",
" \"1McSgMUsJk5RRRWVUyd78alQvl751ZIpfzy00pDMHGg1TXB+5E9F4hvWVv+MyFdEheNukONQ+MHH2h7lqhTgegf5YcWPrOXgIxRE\",\n",
" \"wovDOsB2/1NoHERU9CiYbZTPV2GFRCLyZUyYYyobcwRQkqoid1cERc+ip3Zqzrs6wQxgQ/jVRStF+nN0M/UqUjBHYmrouY+zjEgG\",\n",
" \"wTkP43YAPUsxROJztKmxi8e86+8uRgJCMgTuT+y/eoQB56gtNHj+onIav8teBa+sNXnFKVpwlZiVjtwSr7nqKzRbyBReML8BnnOJ\",\n",
" \"oQVhEo7ZEJiVIOqxjODZyselMqeFA5Uy379laAa//bvyDD4JJfbGsAm36s9gsWHvKcmfmHedmmrNv4BE4+p3EwdhIRFUveHJW7MY\",\n",
" \"wMLP2oaIU4NOiB1nqjaLPrKXD+pu7JOAM/qAV77MLa+DfAy4j7PKiAL0eLaQyWD9ii4mVRDfDAV+78ny/Mx6iiNI0grxBmqAMAMg\",\n",
" \"yms7xJn8wrreJH7y1JcdCX7T8BGnaRQ8i8kIwdrJZiucRiAKZFU+5L/Xk4loVTN+9BVnFQ/3YOHZwoqyY+g0aKoKm9pTaY0VyaDj\",\n",
" \"Yn45WnmWwcqvLEfflth+nSWm0O/Zg7KbzZizmQ5sz/6p4hSCSpTzE3BjgTRwUogcws0NZ7MeiKamBrByRJ/QWiZEyjZSSSCzx4NK\",\n",
" \"hQ6i3nVqkPZTviNxfxVSPcb0wmY2kAbLS4NOAEKhZsfcm4zY03UJQtAnk72eZp7oMWfIhF+qMBJ0O5gLi62WBItf9V3q+PPU5M1C\",\n",
" \"yTUVtVl2WOHg4DfIHXWa+vMYiTyUDdgh1HFdmy+DmBc1JyZ6ER4Op60NYoXlsiGByd8hHxh9pCJSrdyzOE/Pb+rQ+67eXqWxhRh9\",\n",
" \"Nnu2y9n5CYhYd0aK8EXVy1lchOB0XzixKGh0HvJP1d/6tXpBF/5lojqQS9f56EA/LABZw8lNsb/aCssWEAooM6CjFq2FFJU9cHTJ\",\n",
" \"CfkMTVWws0WeOLguF1Q7ETkyNSQfheMRTYzdna2mL9HtiiFW9P0d1KDqYxlmgIGX2DG2EZaY48Su3rgQDYcyEoEEHPAi0OTx6K7x\",\n",
" \"NU57y77Tf8c0D+YLGTy5pHgK8GYHI50NPqF2CkeoeBRt2vY8EVx3xeGwYn+XWfKrF1rUeuVpEnOzz7RueGyTW6T/OLgjj6x/nv5e\",\n",
" \"TwxOLJYjMEG06f29DMtCKn0uCYH2HG0gzr/Y3Ba9QHrcV8Sbqz5BblDBjU8YE3r8FDqonRhkleyrmXZ33xPvsjQqYj/YkZC4kTjl\",\n",
" \"oI8JTW/bnqYU6WvOn6Pz3Y1ow155WGC/LU+svakr0Js5Un5R1K4hVvnQpWEa/d+jncUyzuRsQRqQpl5kJSC4MXaKFLlpRsaePFwm\",\n",
" \"b9ug+T9+7ifQIujXeHvQLc5MZRWRIPnqe/Dmi3jYqJ/YBM9UvjdbelHwYUVuQmTjYgqLzuAJeqZOrMx79AQhV/RGjNAINnpUn0sH\",\n",
" \"oyyrpV6kWZPnLM0IGAj2uwQPB8+Y3dhiNlmRsmfbAJYVDg+IR/t3wK0JifQW5Q2O38lpmFCJPbIBEknZQQcq/G2xtFpRRfo+9S8h\",\n",
" \"TzXVOQ4dhA2ZkYdot5ACiUMp0i8eY/uWs4SZYSF6cTSPfPFhFGDSh8mAmGalTdCXozp780DCWp+RIX5igOw+01V8g0OzCo5YNf/4\",\n",
" \"Hf1Y6trDdEH2a9PVjOpA9Llskube0/chHek3xnD1qJGAgXkmKDe1++N4W4hbo3d5aW1sxVGk9mJG+m1XYB3wcaHhUrNE2YqY0bPG\",\n",
" \"fSUqH3Jyy8WZCPfPVef155YaTXtkUQ1YzTlsQg3TwI5h7cR2tBGkGgJ4Q58JZdVzoshsvAPNu4P0uwCHVRiGxV7f3zu8KlIKxOG6\",\n",
" \"y+BUulE8GqPK+QSsz/CH/QmJNxb35E86dC1NLlgtvTlnX4hg5zWhro6xH4QaIw5p5cfeVhccGxju6NaV8yWn+1CvMPR4yL/sU/BM\",\n",
" \"xGhUi+0+xceSNwqiAL0AtwxcW9Hrq8qPro2xC4m7ZS+XHBN93zxAFZnjd/hvTRwlSgfMHlCbvo23I+RtHecbT3z18BGpJ91vHcQV\",\n",
" \"xr53BZXL/Uj9ZuuofTg4KvqAEVKkdAN60n0WJwu8z0jLb4MGy14T8exGzpcJGLx8jKLCKgtqNDpZnz8t4e9Tzdc5VKUSSBxF7o7W\",\n",
" \"CSfjY/+qPACiqYKp/TQMaT1j40frc4HE7G8icGDmEZrwTplcliIkdqoARvall1kOy1MFTnfOAhMce47A6OOmsVUxoBlZc6FRKi5q\",\n",
" \"czD3zB/1rSD4d65QB0DyQB/Wc/Kpp/ewFoXFmaDUsjP/uk6B69YTLAUHgnbD6IPeYvPQXRzBzA744nqXf5q6ZxIq2pwf0wgoTzep\",\n",
" \"ug424rXqHcwN8mCWYYs6zKUDzOPBjIrrHGwTIi67WBCXD5rIEEd7bcQ1Acje/nQXin4dZZVJ992uLk6aR5ofr+fQbc2J3EHmnole\",\n",
" \"edFITlaYvEPyGJt8zOoOIpmKSq4vYEgkPrVm4s0yOG0bud08Q/koX50kBCk3PG+3p2aGbjDBraGARk83T8XCbkTT1W3z1xreW1Qv\",\n",
" \"qVuLkTqkExxjMuGriXjTG57TXFKxi4oFqD2zeb47rZ14ptjVcnsCarjo/iJnSH1NXTJkk44qfcuGDvImCbbX5p7YiLqMECHPzy24\",\n",
" \"UUz7FJ3TAZps2Q472/mGiqt9TYBinbS1BIpMIXJUFnNGEcNBKiM+3tFokWNs6CjeKD7S4hKyoL44/LVCJeFT5IHzxkg3Mx6BrFMa\",\n",
" \"f3ZpH1pWF3E8DsTOx51Ik8Mp+/hsgpK7Usk/fvnj5csvz7/98+td/gFJKyjtoizFfyzqAuiRNKx2wSijpykadkVPl0oNdmiRsfMn\",\n",
" \"TUjtg8xDoRnbwGLMm72AAeMckch5ZSbagrEjDmrTcraKObpO1+PqGwyuECF6DLasU+dliHBUH3SDVSWxEat8O6YKpqMQ3zhCGfFs\",\n",
" \"alwhMIjB19L6tusEBzUN7IMU0hRw2gHk4ymBF0/6XLpK+J0/HAGpAacs9nNAwvqGSSv7ddTkjxh4Amq70fwiIWJIxKTBc32MDdiI\",\n",
" \"hVnscPp98PngIjWRNifLrcaHySF/ie2g84L3tLWJ62cbDkqXklSSTrEfLbiNf9AMlsrCnjuogxr4gz3h5yb+AqLAth84K9XbRN0c\",\n",
" \"L5WaWd3V8KiImorGx22gcv90K3nQ5qzq1JwoLd/DF1BYNw4W0agHNrk/4pDpaTn1zeIy+t+7TqnsXJ2Psd/6zEgENcEPyr00U0If\",\n",
" \"sbb3ZtgboUvHWYQ7+ukW114wFHVFZY+S44YTs/49ujySD5UdZBofzENLjqGyPS0W5YuGK2H9LI6/c/7gSNHIvuMEa2vW00wQc554\",\n",
" \"vcSAKkj5Q/6TD1xiQZRyuGh6ilQnmvB9Cjfr/uEaiSbuYPkOLdmVCwhigQboaygoUebLth3nJ6CCVT5oIvCF00YujFw14np60X+0\",\n",
" \"jAKnwzZHj41ptsZb1aNQViIz6KJC6u+XYWZNnX5iaDdx4PaUQPnmOT8+G7zNu3do0dW3mLz16VSrUiOmWj9PSHD4CPHZlT/G394l\",\n",
" \"tugXp0tT0i/m7nzssrO8ptYGYcacpxmdEulyPAXLO6tUQsd7uIrKzD27swiPBcOm7e1216dcxO8VO/pTDu1lwVed0HAbFA/MeQPK\",\n",
" \"opjCcF7MBd0TSMjetjNqNd+DS5hQSbDRupZgTnaZRFwXatFh0cdgdZjpCzT/heQfEl+jQkC51tByxL907Sh7R/SvqgskcuIrYI5c\",\n",
" \"bZFfqmpW5EI4Fp6IE/XWS2KfxHBdtBk58h1FHFIDC0tjPgrIzFX6gCmaDaI4tySG9kK/yAWv4HJxldABervXRNtDiLD5G8Hju4nx\",\n",
" \"ESNm5sIFqGHoEimpw7t74ucfcnNTRJhs3z+30FvmDGuyJDnB0nTtngWR7dXJqVrqAc8e7evEDVsNcPI2agOskSrnT0WzG0Y751F0\",\n",
" \"GyW1MFNHTWgISfiDs1WvnWuYDfUK4QFW7NUBX3nDTQKnrbOfT9uiMzXpYzA93ODMZbkPnOo9nr1a3LVI1OGqmGRzcH/IrKmZRndZ\",\n",
" \"2cfJjFnCqG089/YiDi9qwovYO2E5xOQfY7ltF1PWLHGs41hV+AGLjh1T4ec0A71pyaJOHQBq6iNdIiHqjyJ+JVesndWH/B9tLAtN\",\n",
" \"oOFf0MvtzBmEdNTlo5N8RH5W1UFxKDGM+YM8Pwx34b46/vxGwr7Hdvkrx5Y9LXLsOEvKfIYy9lrsrioWHhApbmZMBxYHiVmQ/VTx\",\n",
" \"wQErq/v7eyHaRChaJhbwyIUrunsyI/kYEG8tKVQWtn3QhNr+zPAei/dXC+K06/tXSo6MajTLMXzFeBhZNDUbyo602dCDN++4H9r7\",\n",
" \"2GYf+XP/QSAWIA7l081wTsR5rc869qemeEf32znBYwx7zVhaxpKnmuqCBpwB7wCKMsbZhc/5iii0aoKSzh/k90R+yDlAbXAT31po\",\n",
" \"hhsuuMaxEBYjbBdXeNYs8Yn6+zGG6poEM2WcXY6RqV5MgnuYM6mvOGrAbpIauBWDF6GP9Rko8MUj+LdYWV18q+aUI/8an5wtcMOl\",\n",
" \"zdhwRUxUs1Ep4RTjTMEkaD89i9979GDaczjOhjIvnuLFKZiD6aSTD32KkY9m0zUzrAeeIWUdYCE05Ieg0sVq+Ffh8ODxnMUbaUoN\",\n",
" \"k3KkW72knf+5P4Pyqhai+YxQZ7bMz8qFnuk2QDcSJ8iRgq0Jzv8yuQQO+ybNXLy0E3Gz50Ez5wsvkz+dhrDkHoCZkI8RB5Wx0nF/\",\n",
" \"FBQfmCNd+ymtN7hROCtT4VK8/OJIxemvfKiifnUrUfJx1/f3l4inUlLbdKAJ59NySy+zadu3k0j5EF6k83KP5qo8emMpMTKekAGW\",\n",
" \"fk/Git2uYJ8Sbw+GckKp0A6ja5RBs/FilmSTxTxKNwdV9F6xrhqvp8U3QPJSaZZaqVPkxIcEHBwiRnRwAguUM7L4WU/wcOODkJA6\",\n",
" \"IdsaKcMovWj00N9J4j+T0O+MXZ0G06Ej787FBCFzOtj5Sgk8GEmaO59T7cxiV/pmTJzNkDxRkSA5SEJ/9HuAu/FQB6cEKTiSpQqR\",\n",
" \"3L0FLLFoBrWP4X2YLlUDMh/BNcNccDb5l3ReMx8LDa4K3gGDGOD/O2jbXBjvm+VfQdZmzmiRBjqNA4E8RNfDAxq7NzGjT0kF5dXv\",\n",
" \"summwlXdzZY7DrnmKj+9E08Gl9dUmE18zlCOndvEsSHrPDH+3M51bOix7weUCE/FAelxwu5Q4ScEvK057SxDJtP5DBcK51cLxMUQ\",\n",
" \"HjGLSnEx6NSwNuzDjWSYo5AyYVvW3sdT22nYu5dS16nLmcMUYZRYkRvPAvYNigMd4TCQwk1lHtNsBTC62TfMtKbT/ugDy3wKxSlI\",\n",
" \"21cnzEJAC3XVZJq6mDpG6STEYWB2AQQyVyXEq9SRmKlJ1ORsgAuJFrJ2IEd1Sl2y8wO1ESahs1SWBfcaMiJu9huvyBTNrHLrHIeZ\",\n",
" \"f8Yu0BCc4hVDgcFe8lEjU1ltxixJc3eyk22gWIbpg6pe86sFRXzBS6uKLUb69oQE9Znv3/19oqxi8hC//uW6G7uDaVnzzLyN0gLo\",\n",
" \"5pXfifygQ0tqFfQs1cSIlHxvcRdpzl6260Ryx+QeyQ8eFM+u0hDljojdY3UWjVDmIbSQpq5649RoodL3lYNACoKqQDt3nihx9a2/\",\n",
" \"2i6yCXnq72pSAhJHUt9lQ4Q0sqK5Vd+/V3PoX9y23tnTxhUjFANFKS+WsoPeexPFxGLKvqxFNJZFzqipQKhZ2miHJ6Qp4gWZ5bXX\",\n",
" \"0QucPUtzqmwNd21/ikNseo0oj+OZNA/g5vQ9SWIsNDpRFwvageF+WK0GPsMwSm8eOLDIwzg+AmrgpoAhYSxCbEP2wyP+tBy94faU\",\n",
" \"FGww3skYzL620ArQw5r25m1bTvIVh/eSpCSO16Laprkw767SiGAShMFrem3QI7rinDyeCgvtelqIDGkmvl3hxXGMgme4jcMGFr3Y\",\n",
" \"3UAfKZNvEXnPsTJyhUjDx7yxfbHvw3APthsk+xKTK4i+RXEkaeIoxP6qdUVm7RycZ73IvJnSubIxrDyQvfF7VOxFF2ZP5acgPpYa\",\n",
" \"pMguoSlfsT8kMbnbnpnTR+rFdCh2qC6tnXHiNTghA7XaaTyd2AgPwN14nmDPV9T6DgkjgnrgxPFHwnP00bV6TrlH9hAoS9+j7R+c\",\n",
" \"U2Dxf7cm7D+Nqpf8IkAXwrCXWawn+etvUhFD05HT88fZor5nmEoCBcZZbK2/c5xWKgmR7YfjWgC2yxBsNrvxdphMFOa4HxRk1gnM\",\n",
" \"I2+jdTEvSzYFvaA0QLEaqdjeEQ7mKVhYXHRX87AaZmHdXfbMx3La1+jA9pj+cC3FBiBSWnZxpE7sr8xS/kya3hwbZPhxTvpxi4Gu\",\n",
" \"vVPw4V0kaVRoYncjPNS+7hGqugbWs7kjl8gqk12KAzl+EmQMebZ+NkNZv4ijl8Xz3npKjUtSuOIsehVoxj385G3L7PSqz3ZqMqdT\",\n",
" \"qEU26WhPW3U1YTn7HhFTsXu9z/qiLJHS+tPnKkaF3hfq24nOrWqJY/kIRjp12NntTM/am1cnR7ZWvb/o2QwQd8xsk+jC85/CaPKV\",\n",
" \"hribErSNh9FrGJL3XrzBEnE17edVOClRhPiorIYC6Hp1wiOLU96XbI3MFmZwowMRl9f9CT6gt887DQwNPckx5LRr3qgt/WkqSzJB\",\n",
" \"UF+zFJxvItvc1sePOnyBlZYwYX/UCEnI1In9XuyR5vQzbKFQbBZuxVkCGjkquLxmEg5HuAJNq290CM28c9hDStmzbHKzTUpAvrKg\",\n",
" \"gv88ic3kep7IqJu2tjjbzjIhwgCDFpRZFHuV/xkbwY+RkFu6pZ+Lu2kIhnd9gcpnwYN9nYYb+u09rUZySFWx2MSN2IQPsn0/ZPWb\",\n",
" \"L7AZqPVEvF04uYPD7MBT37Z32f/2SBgWkiZlLikByVrhWIQijhwS0RjY2LxnchCGL3MaIdVu3MCpq02qPSXpfSIzdGRWKN8ZA7kb\",\n",
" \"P5+SFWEysZkEs/0CUQZ0ejezR7PN0tslkr8/EYw4EH+68/wbosFk3W01LFNU80+Bk6kSWoGJPKTm9fLQ+kjKZqSdW8HQ1GCsrGLj\",\n",
" \"E8uE85cpNJdvoGSJiMDO0sYFH8MRYNcCoGLIfu666szjat7cP4o6sHZEsN4JvA27cYAh5hpT4cbJXN2biQ5Qzu0+n0ZWzPEYhXBX\",\n",
" \"9yVBwOY+ItObST/DMpVPcZiDxcsqTxCymSBm4R05adPG/1/I0pe189z41x7noqpKr63yzQZ8RWpRjaw0T9UQezsQMiEI36h32XZ0\",\n",
" \"lb1VACMzzOQSwK4NT56XJBzOLZ5qH3P65pXPwtjMlIJQX57YHMiVwo3Ey09hWFsAtBGH1i/VQZ/7bKTqzDaJqX5LuFR9XQ6pWLCQ\",\n",
" \"sojfOyPSzTxXpvGydDEMEmj3+6Xomi4g5bWe2QstWkjWi2b0SfT+NxlJ+tjjiRGiofQqzTqR8WLdp2kZfZvp/j6vKNEUHelhTMXw\",\n",
" \"BD60B9UkkCENyjpR5KhHgJtGPwcICXBNDx8WS/QP51yGqZh0cM4zfvX/eZxdLgOqVWe4nyihIxB4NjInHmuQiTaOvzHHPrXdOhhs\",\n",
" \"oxUajhFnmCmzRLolROzOloOJswSKi7Xm+kNG1KiLYiYqzjMUwSbn85yWmUT/4F1eXOcMWEzDR6x8xJGR2hzVQvO90Jqv3Ox0UhO+\",\n",
" \"goO6nZPMPr8wDSg33gGWAqN1WZog/kl0nsHHO01VIVqSWT9wUie2haW4qKcEVnRU7ua+laxVhsPmq+7EgfLO5cbRAaPv+f5G87Ei\",\n",
" \"K+eogxg7x7y3jYLiCoBap0F2TyaryRW3q3bMJqkHKA5MSxWWSHcvf9nJ3lrvxENzKwbFmLol3Ci1ZZv6HWsy3pgZ7/49oouLJb0j\",\n",
" \"QY/+XFuLlZYuPOj69DckCf1QXdr4ywHNGmsOe4dUZU3VpGo+ouIfTDVYT+zGmBSLpG/4+xxU3zf5vJgner3fAys1hxPfiiVIb8T3\",\n",
" \"rC/wbTbOT3BA6ZHDgBq1NK96OeuR8NP7vYkQfbiZ2TMbhAjqTVVwUw3VH5495iHzIaYiZYqp7qOXVzWGQOSDsNGb7Ce17IutChR3\",\n",
" \"kQGpIo2/KYi3K+WibIV/fIxATfnGsfV1DWIAjrNKaq5hj5zpKHudz1A05Sb7/+j4snnf6Q9RHmL3T5R5oO8mrsxJfUaWuDQDivrn\",\n",
" \"Au9YvIeaKCtVRR61+RBpIXmhdRyCEJM+E6YOM/vqOM9Hj6hZVaFcIu+IzPv/+dk5Zsfxs1GjP8KMs2JoRkvZRhWif9BWsYAiJRI1\",\n",
" \"nx+qdoubm31ZpzHoaUa1ndiBQr0qk5liD8onNUOcE23SO8c4zmCKnSmoy5OFyvuSE17JnXquPkUBUCXYgHuB0OV6N0ryX+or96tc\",\n",
" \"EwAsqEw1PN3fT4sY+0J9olAI9lciwcj/eqWDffPueKyuYPNUbGxAt05V36lTISJpjpoFYD4FRweV7MQGgweol7BIIsl03uFwXgkd\",\n",
" \"6E3IphHtuB6f8mctT8kBRpzBrOIGoV4lM4uTFHOSiVrZvGqoSitgTZiSdatS1pNfuOuY6bcluHlfXK2O6OGr+XrZF8HkI2945Oy/\",\n",
" \"mdynLCGMpVqgWtZnNrmwbsnD1Q38NwnBr1En17BSBoefLm5qfs+JiRrr3RfTZfqHqe3dVtYJ+fQL7shaadiJzBS0pFcQHvssoed8\",\n",
" \"R8Y04SJtCalnuxZOYMJ0+OZ5rmU+qYfLGPNWSiTqswSLxvpPi7Cm8UkPkSi1IJ5HDK+JnEBpCvvOzoLq+wEcKYwYU2AnLj6HfQAl\",\n",
" \"cgAOSYBHHuMugFxLCZe02JG7ZaOWWWhLOzes/LEmFv8RaR41T5lgHZScnDukrpd8HT+jptU51mJxctzrp9UixKabaW9y7V/UFtG5\",\n",
" \"cTmgrH4jjwsh0yGD5QfYEktfNUcoGSHwuWtgPgh9akLdjhW7hyMeeG8m5rohhEpcgoqZHWqAyR3MSCnIfJnNB0V4kPrruPY7WPfI\",\n",
" \"SBpu3l7WSRMEvIVtMWluG4kcGVI2g1eFMNJL7HE9FicRJaZM6rJn4EuTOqs8RzmrUJIumexGvWcQ5QBPwcO2DpyN2mP7RA/OugPQ\",\n",
" \"+0m6mZZQWbsDGwdBYjguZgHZQdgVZ2bXyCTJdytMvHfdlSyJyjWZuzz0U4bjldN10ClQvwITUTUgjepIbRo4JXP32tmRXZEn6MgI\",\n",
" \"O+zIpQUmt48189cxocaeXvYDXv1xzXFEm69tWv1KhrUGunOtN9R+rIBAFUVmz+EpK0edd1UJvBWDeAfcvRVdJRAYao/hb7Fr8gRh\",\n",
" \"NACc6mlSB8Hn87Xr0ekIT7dBpt5Oep/91GqGtKml1l/i3J4xFRzqo2XwjdLBuN9D0i9Mhb6NtcXEzA5VUCRtFycSl2jqRD785IN0\",\n",
" \"LoXYmo7VEJxJBL3KSHY4m3kFUPf9obgAF2aLHsSeoC/bvpI1tOJg99+PbdiCKgskUVjz1woB5SZN83Sma6dzhWrCbrdsYjqooOPr\",\n",
" \"DGAi0c59jTHfO3PcIfmY74lQY0zjRHSAD4WaMGiC0eOCWBaytih3f1lSAi9ojNkBNtazqsG3yIklTkRA0tbZTyoyK8XBid6miwiS\",\n",
" \"XNzpc6IaGALK1OZFooRoN+uZ/+FLqH1XhEfJwUw4ZhoYttuXQg6xFfQX1TA5G3DyRmdP/eJjAGKA6wNvF0/3U9RVMzonFiGEilVn\",\n",
" \"AUOqdaQeuMbB85AJ04ykPvUyNXUc8OBhSIzMjmPxeJHRhObhXVLbss+8/uYNMGomqcSJelKq7iXnyFUaG8LnAe597j2ZxQBUXC6r\",\n",
" \"OX4vDr0hGFeuTwLOt/kUKmrH1z4m3MeNhuxZpHUe2s05M1nF9AFv51Gr1xRn9r5irOMoP9ukE0CW6yNJKnGeaSmKREUL0rjKYfYK\",\n",
" \"7hK9tke3tv1R65uZ4cwJwQCWAEJiMLsb+OKL2AB0cbQsKvun92MBz86+jIsMiFiLK41Gd3+jH6sh5l7gbYvWmL1I0WCWoa+6lKBh\",\n",
" \"XWD5dBrcMiU/dbdjPKoaH4bd9wpVbAtGGxlAoA24bHejJ+Dj6/Gd07z4KDnLGw9TR76eWqFhHMMSG6uh53NCqtBMF8c0TLf6SpQU\",\n",
" \"QwGOtmtHNe+CHlrMaMVoZ7GrNPibVCQNDywcJw16Hw+H4Bhhba4y0+wFNYethKroBzH/ckqDwJanx4iNXbmmSXuobgrxUvGkjOig\",\n",
" \"izFH5PQ6k9K5Exf3T1Wv1ruhSh/wfhNOo6hj+da2F9Eq1atn1zRY9Vyog/fLPkaHZFIW7ruSKUuNmyTpdtE6YfS5Z18u4O69sD//\",\n",
" \"LXo1aELFI5JcHFfYeqHcbAjY8SOzX8ci9kVhY+9nQmyOTGUuIa8VcU9CYyCQ9G0IHCwqmkBkZi9xNHHvVW9zL1ouo6PiBYuZNo9l\",\n",
" \"r1hUmNrtYQjZxcAiHQWY7h6wS4XzOa1YsjWtePW+RY2yP3CegXk+4exwx61nMu1wQdrYqxQSxygm2I3bbR3ilcmuVhf/iY+MhIoT\",\n",
" \"ZGI53Zueg9FW/QyNs+QZjaFJ71TRNz7jQ87T5w0TUt0xhTJx7FKtTh3xQYj5vBxPW2Zn5LjMPJ8USGzrqkntHtLfcEcxCGEgJfm+\",\n",
" \"HsG2SDzZOvtF7BsC7FVlJMHrTaHtnXfN/uc1dDsxav4zTudUceCMyrPTCkl92X3FmaUd+h+7DVqWzY8apzZ+N5g9jyAZqHg89VdJ\",\n",
" \"ZPTGMbQlwljYM9xjWWRv0Gk2m+cQN0g89MtlV2+l1LRTMuXsZaV/cJ0oF72vlMyI4v6If0BacOkEqCcr9thGYP0X8T8vMRPTxJua\",\n",
" \"ipdcMRz7Kq4xWNRssqjMJz8l9vt0SVLQTrMLxeOi4qd34JgtUeYSxclZNsajKJmp98/qo40Kp1YT7ZVPzsIGkmuHaBrHTuJ5hUlU\",\n",
" \"k1CiGNApf3j/7J8Qf8eBf6LiZFTSXCOsODQTrXsWm+rLm5x16zSY6gWXnnex6oA00BWH1OWhOBYf9nWL2ZPsEnui2a5FBo2EIXE+\",\n",
" \"u9nHQVO5zvyS5ulTO01VSWA44Ics2DNQz53tQQR2oQxJAI138E535Cgw1H2qU5ow3MccszBkNSb1xIpCwvzMINM+jFYINnqMGtNq\",\n",
" \"d4SbRTuD8DGj6SO8Ey1OIax+IBdSyMjtChA5e2ovtumMxqdv2nVGM6WKQVmhcQqpxivKedXfg/gaZHZ9PTJ6E02rrvaNCx2e6DXq\",\n",
" \"I644TxpFxb6Dvgsa10FGsom0GNXgzRjEvSFnnsUhh7Y9U0e9RfyRccfBwu7k2GqSqd7CuYdcmbxWHCmIwEkv48qfrjsOAlY/QssX\",\n",
" \"bSxsIRQzYdHHeA3WD6xFVQ/FpdZ7l3atyz51IPtg09uj9NnjFocUlU5jX1/jlkd6nMicmRU9+OBiQm3r7I4wpXaRA7NemoHSs3S4\",\n",
" \"Ro8GXQiWSsfz2QfHkdoi+JyB2cHMtq355AOJDLyCR4JHH92ERDL9yCIGJ8WQCRgeac0n2JEyA6zyJ2EQyVKbmbg2kd3Ho6B0CKjH\",\n",
" \"14ks1EHzWfpznOqlGW9pjWPaRjBB+oDDVI8E4Tgnz6hev44NqfApUAMTXwzj5ar5a0QP79BGjAKD9QsLOvqaw5LXrm0sVtNwV1JC\",\n",
" \"yJSF2Pd2rsfda2ZX4qQxrcl2BFBxos8WB04xeLj2I80+q17wXCjwK2LCH8NTFClywj3jVJ8ocNsO4zxVpW1zdyIeYMmmLh9EdIRc\",\n",
" \"32WXPmMmToFJ46JyzoQ5m0aY4LaMCcBfd38SWpsESCTl2V9n+oxETM58pG5VEHTmgiLGMmY6B5NSVhOI5ipkv8IWJ/hykSDMGDnY\",\n",
" \"9d7g1dONnJhlU9RLFYMY3VPrcPyJXDPreS+a+jSjYD2JBXA5CsjbfRr1wpIIqPGAnmwxDwQFmP/GI0FDeGKIpLGnwTGerYNSPA6M\",\n",
" \"pEg+v84PjKI+MaA+vBuI+BX+oqdT4/AUOFXetHXUiBedoHo2CDpOfyRIKk4y9HkT5K0pA3Rn9SY4B7r5qyHOL7l4YoG0MpAUu8ND\",\n",
" \"9itnxCXe6KhrvjtzRKID6Hgc8Y/AmZRpQvtaBEemRtn14hZj6l5xnwXVaorZlaw+jC3h2notYOH0fY8TNzC+BuEQXeJCfUg0bWwp\",\n",
" \"dz0UZ1yYuX0h2WjhtKxeS+OTqhbr457CxHmBJzhtWQOYgo/KoylPP8xHc8D/HpTNoSfDar0LwLOWqJtIymZWCjKTjAreePABSGVU\",\n",
" \"1jd9cXQWOd9pSD8/ZrOp1zH1nvw86dU5jWsi8UrDkBmpzGD0PJ1OS1xfxSOBvgIiB0cBQCOsgpF5A3o1nwcTorpMPRrVcNM9mcuB\",\n",
" \"Vb6dKyGqikohlZOaperBNpTyix/iAKH45Gs9+oNKrhODq1AU708YCvXTYX6M4czRS+BYlRsSNVHM+LDcee2D8o1BiS1711ku3CzL\",\n",
" \"HyLhvkYQmW5QC8TuCSVsA/pPABF5ISDLMwIpj4Oq9Qv5xH2GUUnyY4K3xRvPUcO3lXPWu1mHA4eBkwYyCzHjBj8xVUpM0L4OTvLD\",\n",
" \"WzyIKwoUWaEUW3zS5JquaL+jk5DNdhF56ejAnxI/Plmi3zcBazA0S/oksl2ypNwuI/zHfteBZ9fr/gmK04+NXo+tOJyb63TZnIkj\",\n",
" \"R7VCH4zdLQn/Kc3ghiA30HerJQ4vKcFFf7Q2/jTWQ3WuOVinPxcXdbzvBIVzn5IjBDxl4cXcQT21VPFk57vEPgsEO+brOsImvnrE\",\n",
" \"dEdmpro6bRUYS41JishBjRKuSKL8BavhKf+9a3fsTp4N45qRm0ZqAtTS38BCt/HsKBBoaZYOjkqk8/NeVjZ+rTAqM0BN5i9D7r3+\",\n",
" \"SqyI0yHytwDHgIAqf6u6mI2Qjimrg8VaPgOAG91tEl5THAwnzjZJpT0vTcSsHNN1H8iMfqidnTJ0YJII5UOOhJmFPlCDhNj6MYjw\",\n",
" \"KadvoyxF5gHmmwWbtOtyUGxMpU8DTjY+PJZwnd0A5ULyzlPqjY60nC/uZMHpjPw8TrjNR+HnybB/Kv7TNhVCqqOQhSV6VJRa6qqe\",\n",
" \"o1RqNKdN8KUNbU8gsVT23JC+554lt5WP55sj1uQ8IaVQhUv2tQ4XbweK48E2cWGQN4JffKSXs3aeQJ4l5o+Vd3xtwLdSNRELdeUM\",\n",
" \"wmKIq4xvC4zgcduFT+Q7h7AogBRFVH6lGj/4gRnPn49Lc6fS/NISnBSE3N5vl612rZmB/DNKK+O5gpIF819zz8maHNrh0NxL8TdX\",\n",
" \"kmjqPhR6vTjIphvlEROLRsGJwxnYe0HbBVykA4J8nkEsNjiY14n+HBuX6CDFnjGhW338DZK8WWp9lKCrxuAIxh6d8z53RPNElVTb\",\n",
" \"yivv4BWFvwsUbVNq3eeLZd/YCBZgpRTz9YruyGgT0vRl81Le4C17WxGaVtGX3mq8BongRfkuRicQ7TP1PAQTQv4kRhnScyrteDpH\",\n",
" \"qmKo+KrkEWIsgvmQj7I1bMdEFG6GBmtLwJ/DFGxRRXgLl7sPcW4RXUlKARKGLgAlFTH4sNBWSXhcnKq6Kn2QS2c+LzFxcbvEDxDT\",\n",
" \"ApqfEIvhmPXC2hic26qx4LIwC4Jf7ITs3TudkKb+qCzLdnkS7oAFxo9MFoFMLnIIpMOgubsmQitWuzhJpyFihgBEBYTgJPZ5RPbg\",\n",
" \"QBCAhiWDkJurW141n/VRigj9YzxunI7AmZelgxOzP8I9GfoDR5IT4IMyfXCI5d5CvAMp2MTTMBtKotc9JtZn5aSyP0a8bGAhI45E\",\n",
" \"cfKLOMmZg0/BSKWw/QyC9gGMRCy5b7yhWXPvgdNB3IVIyglYnXc1V7mV4FW6aogyYcPpzOJVUVvsnQLqLGTLDfg5htt37/zBqdSt\",\n",
" \"IjU6aTybtK8iOY4q3fL/L2I46mMrwAykgnjrzGnbhH5wWGnhytNpvRxGQ5fD1LTwdpOX5MGuh0ancerRppnbqLHgDQCUaf4iGPOZ\",\n",
" \"WqbGdp4C24hH5wTuKS6y6DoqKQ0o5cpgnS3HTCGwjTsNHtLwmPcO/MpLt0oJbOacqNpL6Oz3/jfHNs4BpNk8s0bErMauvaPDESx/\",\n",
" \"vYjEUyukU9rj+RkLk6M9YSKWfqXFr+3FHVTWLuFV17Gp1nMbS1DhyxD7aplDkYfD8Tmtq+tFSu5Ht1xM/vlFeh/npkbfn7r1yf20\",\n",
" \"XhCtZb+PHPUuQOOl8WHLYq9ygAWDXi9/OipV4cbKlDgGEA9tQiZwdON+35lt3OTnWTwotDcyswi+Zboive8UgH/V1Nb4niJ/nJVD\",\n",
" \"6UEyqFK2/eIzrK7QJYFA9L/YcYK2qmAyvgSq/z56N4HCv9UwJws6tDyRc7YpkWUx9cVNb72fIcaVs8ZJoXpjyMb4P1aP2KN1ut0z\",\n",
" \"UdCdyXg9J+YR4WelAjOBEci/aOKFSSI80YF9PalnJMlEiq3hSfqIy1tySBNgzD1I/a+cQDxiCokp3VjEs7gV3vjFWy+AcW8AFUYk\",\n",
" \"suO0AIf1ORwHFObrxOTL4CJ2FsyT3vY5U0DT0KtOQ36aGtjmaZd+0uSxiTYaxQtyY0NvhaZMG5Fmqt3ET1+maWC+l6dNTJsR5ayq\",\n",
" \"CY+vGeOeWheL/1cr66gdZHZeIxO4cLNyT4TW3nxjhhr806kOsIzIy0L/lc7IG5b7NmMcEZDzmphg2Y0EzXmHfg0R7iC5FsdH+6J7\",\n",
" \"HYda9tQk1lxPo5ne+pr4H725KR7ad33g4iqeJt6dsGGSdgZ6NC0w1JuoN4QBYj5knUVYu8bWtXFaxDiNsf+oSarAmyFb7k/fpvci\",\n",
" \"kyXPObkVVPr0fE2JyVmyi143SVf7LgiaMPiacOPIF80Sn39QM95Yl9bsOKdhYHotr5yLSg3hReZzR4Q4jymbaRpJ2YrqRy/CThwL\",\n",
" \"Zifwkk+JRz+9z5WCpeS7iciHgAhE2rFa4ZUT1mumS8gOxOmBj+SWNPeHLPHTxSXa0y0IsynqXoP9CBycTkCx24XzwNKvThB63iaS\",\n",
" \"fIyaYt6iSyNP4klmcoKfURZtKpV4Y3Sc913EzEc8o+mbm9hBfol+j9pXlNTsE1+m6PIWm7QN03PFz7+XL/c0Y+56qudO9fQ1213p\",\n",
" \"hYKkKZsmJ2lPMdlBaD8xrfrATUU/HTllM0+2A3hi7pQLKmHhY4K0Xe+RAQC6Or79f5dLplTxfRLI09ktEB0kwk4njChFKUW4DqJV\",\n",
" \"2G7+jFyL/1yMwJrGPxQY8yTEol/GTFTvwxhNgTinDdgzWnCROX0MqM8TI0+lsQjpy3GzOQbZK9MZM2mvI3hs5HhyI8Ghyako9qWe\",\n",
" \"czEwyUAkietsguvEYQPIhTTBJwgS4z6x2BZD1kaGkUiZgSxIqicAYWyOxg8UxTe+B7FP7VPuTndxjZ36f2EmE32canBE2Z7dsDfb\",\n",
" \"xXTw1dchNRGQeLRV1yV5ydRlLqplitcjIR2QXaaTLij5U7SITd4CsU1hUSOtElUKcADpM0kfyNnGJ1sTnudLCFf8R3rx6IgJe8f3\",\n",
" \"b4EJrC1nNVCG+Qz79hAH+ESyBVK6sNJxCjVqtEVC1TvNaBku6gjmzDbibrFVXUD7SkK/7EinMrmNOHS0RGIUgrpA79APXiNCrxyN\",\n",
" \"6nVon/JqoZA4cGQC/If3R+zZJ5YWNKN9W7+pdD3R08nwPOUfxiF2AwPQV4iFKo3YC864FDv7yPWUEUsnM6C39xl0N/X02M9Z9Jne\",\n",
" \"poij6n1Y8mcPg9mUs45JneiizzG0G2UNgQLYBpT1LSTKvYrlmgjZd52ReYc2mDjow6iRegaw1gB5/L5l5yAQr5fW+8YIfwWiAO2j\",\n",
" \"AmJRvTJJJGfDwuVrdo6k6yDaia3MCbvkTIkaQySOu0z+E3scvXLOPKCGOvu2SOD8d9Fazuq/cUBjHCrzEnUHSy2i33Av25z9bfQk\",\n",
" \"xOqvAHUdI9Vlg2vq7Esr24ne4ejtwvyd+Km+KZrGfvYCPqUBGe+o2r9HilrN73jKVi85IhK6Mav7+985PNid4LsUtckmIqzUInCf\",\n",
" \"vYnAvLDN/f1q1pCY4jVwb8f6j/fJTBM0PS5V63D2lT13PiNyGpgWsPtiQVopPOyKRhB9wQSrPlU4sW+ctPKwWq6lY+bYQHfHg80V\",\n",
" \"mFKTXjrfTBw3DqamO0osOFR4JD5/x86f/gheDHDQgJxmIh6fmyTC8vlEzJ1xvi0rCDEdI87KN3TRSlOa2KzC6Zbda0X6DDIN+JBC\",\n",
" \"EczY420SLfB6iuVXgeQF6S9iWo18XQLB37zXh8qjKcyTiChl8FrR0YkzGwF6CbZN+Q5kZw+c0nDU7NpT6CNp4sPESKZhHSMbkyEt\",\n",
" \"TpgPNcBTqBr5e1+lVP3vMXmfEbV9hDdC/0qh9OpdQxtUIqUDEKfhx8q+m33itrXyKhXGMUWza1U/etLPjvC7zi5ZdbfU/V7MXIuW\",\n",
" \"QlqZ7aMUO/LdgQaRKU/6rORiozrC8AGlMW7unATbmWUUk6SJMonujW46iq/QyQenIH8v1t99NGAT5WYdkxCcFzwvwABh+vTOHRbK\",\n",
" \"pdLkEfhN6GMafNjDbdcdf3mXe2lYt/NiwrFYDZs4XyjVnbPeyXJjujshF64RyAUm8mIQK6kDA6nXXY4uTj3oNLk/cgAiO/WKMZoj\",\n",
" \"a8TNuhAQGGzgufCnPHYn+tBW8BL4N2VVMuwgSnkvSZvjHJ+Q5i+rML7fWc28cP7gRLjZlj9yv56bOCgu+hmsza/v7xO8NzkYN68R\",\n",
" \"U1MqY8sTJPH3UMyrUsOY5gN3mRvKYUq3Jcj5Or5gmq2o/BaIug6qj3+S67ZYcxmvCEYUVmHi66L1SFNyJ/bn5Rd+hGsITXU4FU//\",\n",
" \"j5syEDIHAbjEg4MnOjI1kGMl4icy/1wnmjrOGxYr+OMcYzE4jbbwwtmPUM03zxjDDZCao2Z3ChOFH41EO/bzYVuZKteI3nRhmVji\",\n",
" \"PJ1iyS1yRISC04XsJE6kv166iS1bIcH+yunW2QxG6eG6T/koZrMvhEbXI2diAlDuS8xu6oDvS7PYbUzE+cxxcyIdExDRnCfioYVu\",\n",
" \"iEwI68gWdQZzptl7LmXV2OXHggysjU/V2bKIFRNwbErKvHq+D9VQXx/jKA5fFr/w7DPOMHdBm+vJY0JayNjlGQvQDgLKULEdKiA2\",\n",
" \"3eeWCse4nMWYrFmu/OaYMI+wnwvk96SeRcdKtusxStkMoU4fFE+SxY/4ceTCVTwH/1XaU8aUGHfXq4lC2k/aww9nTNFzAuxHcLkE\",\n",
" \"JNdMVzXYxexONqMtG9Pk8SnjpnaiZOukfiLW19ni49P3c6qeaNf+amMSLr71LRMEScTIg6Cb3mUflJdRwg0ptYqbNPh4UlN43Vvw\",\n",
" \"fu5d5+UAZpPd2IjRWFOnXPU8MPKNMxcvrQV7SPQiCooH+8VVqeZVfOMgJizI53bLdfiJszuEPWi9bLSOjb+mStTZiB4ohIwfVS3Y\",\n",
" \"FT5NQfWF71PlxZvqkiFySqU0FQZ9mgKklHAbCJYeqn+PsRtLRAwVkvG2fOchUlaYZJhFgTJSEX4rdnCQ7tQ8tOdBg6+6UTBSgTiC\",\n",
" \"7WfuUE6ecLATYA/6dp3mVOMiscM3Nvh4eePQYskS6P7i7AwsHzz48JmC0Ge6g498M3yckbPGrqzUuOFj4imaKGZ5COwUoFC3rUpm\",\n",
" \"f06iupkOh5+yxcCex+w7fACVCecfU2JvdiF2UyHJHX2ZC3K6/wEqobTTHcEZ9hpYhL76e9G65TQYHQDTj+8eiWbhJ7z0Kfl7vafj\",\n",
" \"ID3+I99x9j30KdNWxZBlrp2+Vn/PFchndYuw+4HGdu09iQwUMX5uhtk/IqYfMg3+evOhoF8JRDkuPEVXZvWs6Kq+tYn6JE7244mf\",\n",
" \"aOHZZk6cNtXjMvchpp6Z/nhWIT+ShIp+C5mg9aJIc6NCvs/9t7tbomVtDjO2XNO6OlXq9U+6OlWOsp+bcpTiSg6VHjUu0GZRlF3H\",\n",
" \"TAzPBEnRi+4VzuUbCy10S+xcthwKwEFNE+XbrfZOZdDm/UwSVxmmjWZYaHfGqh681eazulMvoC9UVHmqZpxrTOyOcEVWqlzjcTR+\",\n",
" \"ZxoatuTxXCYnMIcsTl7QK9yT4/L2abtF2fw23CuQPQZf8yH2uh+KjkOMFtf5M6bv3jv0KpAiqk4QFFRd2IEPc8eywZUZTHnX2R+B\",\n",
" \"pMdxWvLs2Zd1d1wzYnqVM25AaBPnagDnc1jOhmA+jS50rIX8oI73ySeP+sgNP6OcKXFxAjAhqwDbgLUQsJ7toxdQe9n7phlIs9sL\",\n",
" \"CqTOfk3mVbIpRN4HHBmirh6yNBxkTkytZ12s+5z79iQ86t2U19o5LXzFia/vo0p/tbkLof2bLRmzHElvHDWWMVImXx3EH36Uq5d6\",\n",
" \"IMwYqUYfCDg2HsCZhuNkrM9iil6VykMIGaipgLMnex/IfVSCq77iCvvatGTgJQAXjrwBN+/7WytCWsIT1SeErE4A3eesm44DIccf\",\n",
" \"vdJvJEMKYiwB8Q1WLMadEGEfhFNcQeWimYzAz6PexKa2zYwHQ6rOPDy+OnE40keESHWKkmnVknu5J5h88idn3Q8YtwH26iAsAE14\",\n",
" \"L5aQSWP/Rv7UR0WtDRGnyofdJl6C5lJ4K6EeY4aFE68Unj2bcgOPE+KNAl0MDk+4FN5NBf3LtVgjV3pjHdcrrptdfpOvCtRVHPPC\",\n",
" \"kD2Srafi8qxmc0bbE0K8x7mPMbn1Dyvx1V+CSClD6WPZP3sqP1/NZrLe+gGb6U8e/Z5unWCh9ZKOB/djNn/aLA10To13qTvZnq2I\",\n",
" \"vbpkfwBp+vXpZm/ziJkJrpuqQ9OKYLNVi+g6+2Jue08iHmZYINsIEnZHkD+Z9yybTk7BfuqpmVdO5nCC2C+SyFmHlmCRE1AL0o6d\",\n",
" \"/CBYqIm0EvISK+zaOoE2zxYRXBPRU9TtA2GFjuwUwTbIHMWUBeT8a4ik/t4xxCpabI0XetxUaeJ90/hm8wlfJr7LBJOzMOvRGwih\",\n",
" \"ybuOxYrsWf0gnLPTDqHZCRVC8inPzfmDvuxn51fUFmgcPDQFO2nlbJgBAatGO5WOyGlerRfeJVFBAASRda2PrGWqkz/lsxSG9IQd\",\n",
" \"R5nSif8EWtTZYCtOS2ZvzAz8F6EkqfdOKqLqJzq/hnPaRON3k5p/5xATSwWeH45RkVVhq60at3of3TS0NAnr7GvMQ1+niLtOQ3nn\",\n",
" \"TojaVtOLrhcyhiN8X5SAVZZrtTYjbFupm9QhrMQLcN6tppOwpUtuY+QVV+OmeVM/6ywt1sHM8kn9oGkS/Uk4/OfeqX5wZ0cv4s8g\",\n",
" \"zvKUnA4GEqlffDpULD/7IBV9i487duRoxKqvvUyB/jSU182b7bxvnyW4OmTfrrGahqQvYyNqvHEYZn8AYm+d3YoMh8Cq6QWNbJig\",\n",
" \"mNqQUa3ouGhiiuCVzKeuMR+inHkAP1V62zTd+rPsZI8hHY64A1i4Cz6a5mQBa8T306AiFJr8BDlQcVYbHSexJEWcqX1/xj6yJea+\",\n",
" \"6n0qIXM+35OmAaQ3Ink43hSQgKtkREWvFiwaaLWVwyWmMeSo1dFfepkjNRqDsG0++xIVSLEIY+kAL4Dv+/SObAKBw8+sLl6Ac3Iq\",\n",
" \"GqTpQ/cUVY53dn3ZJ3Ls5HLYB+U6QP8zx2VnShqvH0N5L74SPubTD6z1qk890RVaVB/j/OyIuUIZ5IcJbrEFpc+5V61YnDGi8026\",\n",
" \"CoiNseQZf+9gmRm9QCEsvjqBQzy1S9hUZT6NOx9pOKVPxjgSZ4A9oFpdTyAs4ar8Mky3OI+EPN3CXYkZN2QEWp4IvPT8TzIijwkQ\",\n",
" \"zGgt0v/wZaVCyWIjZt2Og0A08Zczg0npYx4bbcjM37Cg5OztnhhgNxcWT82cYgupY+rZqNoXexPlKGOOc3hr7akpB8ni85BSswV1\",\n",
" \"E4iyrROO05vsaaxJh9Ho3DxCqOwjokJD4RQI4u8auK5xYDp8MSjRxMwUlKwZazgFPim02KVjzpz0oThJZgddtVzSO5y9JSjCp9fs\",\n",
" \"8dDeCGAcGMoG2/neM+Rp2IC9MXpSAHSLCpMO2KHasqnL4uGGzeH40ZXOvuBglux579wNjPEHktVpWTRypOvjHM1IUVPY+RfpJzbF\",\n",
" \"x8d1KeXwmKCENdBkHdbFtAvmtZUVt81nH0V2uWIa9pJ9LGJXdNzCPNFez2DUEQUjmYbH9Svbg/guyiefzHsveTtNW8AMX9tbtqan\",\n",
" \"YimZScSL4VyNTtfkLMcVz9DvXWh3u/FcxV6ZS3xmryeoht/OPIppoN9aDUWMtmda89Cxg7XVQZpREG441H6IvKpa4vkcUAQxZ5w0\",\n",
" \"Yv6Gh/wroEqB/Sw5Fo0WUhKtyS/2A7FYChqLRg5x9xbmPAMSXuQDKOdVMwbHS6twyYF3kOdz5zSpyuUqjaXRIO4Jsue/wHLundwQ\",\n",
" \"ygptMDJkz81uYgZY57+NpwIt9rTdnLbck2H7aJGYdlQcO59buJM+H8KjEhyeR4dcm3Y4xSf5fdzWHCdjz/9/RnO4RkH5nJzXRxvB\",\n",
" \"8eYwLi/og47UlP9YkYOU41OoLfLsG3W24GI5W6fkGuEaOHaY63pDNhkU2Zk62Tmt1LcJQUA9o9aLODLD/HM0hXdOrqVxh8xPHoiX\",\n",
" \"FwCZiXRCoQkVZ5AiJl/gRDs+yybqmZQwBifdV0w8iuSVrjUi21qKSM/stnNSxpXHG/sK/X4edYjMY9K53re1TLOl1ut4UMV7BG9C\",\n",
" \"OKH7e3DQnwWxpniZJVLNeHsVHNv9N1PicsL0yPXV46gpXdihd1gjPEdAZW+zY9NULmdRl9nr201ktMIMAftOaIYZIYvrnyU5lywT\",\n",
" \"CYgfNWP3NKK5TW64l3TIYAhY1H84tV40vWwSx/KKK1htKg62A1fROkYtcnzFEBnEPIjBbOjG/D4Ri5kZHXsM0iB3RrvVeWZWFNWT\",\n",
" \"jmfzxqVFhnIppd8WKyTCr2mFxB7A99pHcweai+K0ZLV4dFLIKtLAgXGI74NSCOmTKoEnK6rd9dQK5kMUiR1uG4d4LlKjY13bKUoD\",\n",
" \"V9hGBDmiViWhaOCg7l+e//gWuRE4YOGkMGqi4GPEUcTmDgYspZrOLhXVDIdPiojq5im+trrxnDFGtbWt42AH1eggeyQ/p4zOeAIR\",\n",
" \"PJ2H2JHisMLOXvyBJBzgL4N1WDTEfYvEaxdisfE2CeuAIpaTxCfYSFdqdDTT3uAm7kN057ZEXDHPFGZ3mr1hoexMXOj5in5p6Hjd\",\n",
" \"H0byqnpHI17CaQipkEj1PA8MfKGmcegacGMmr6rNE8n+RHLFowBz5ZuN8/t0Yag6AVefzRCNt61FXzmkhTUKe4KbrWdm75Qi6Pn3\",\n",
" \"zL5pmObiK2b05K96YuYUvMLQa3x9qx8e3XDs57XNSOqtEWkecJlQ/Sk+Byp0KJh1JDqK9EhkO9C8moFlGlAqtGeSnAG5Cwy9JoEo\",\n",
" \"XzllQlJwDTaU5t8jWMzf4YMGXzjvJz0uFin99QfF9v7o/U3FNOsFXGaRX8zryGp3jylM7BUzt4kpn43XD/a84BZde1EkjhP5b01N\",\n",
" \"pJDxMHMCubOPE/DvDSunMKfnSg1Gp0eVfb0T59z5RCtiPyM1q/k1z9PEy8i/xOIsbOVCG/6Ozr0FGmAhPsWdN/72sfFszrnrCrOk\",\n",
" \"knWSMdqJzX+/pOCxpgn6NK7Vvv1027mZqDon1fBZbDcMaT09kXgsb5gql7NdnfOFE8KWb/cWfO+ZDe1CKqH2T++6UL/5Hq5VJb0U\",\n",
" \"18c0jSea3Tl18pJXO1b0Cg6ax2R4hMC+S+98/aE9I3HL7VO86ibRvLbqtjv1uYs1OJ872plt+FUoRbJQ4zVjglxdo5ipW0XeXeQg\",\n",
" \"IlmDF+KqXmnDBeMasigKeJXFFoTAVoqT2OMIx6qT7LIucM27sZ7A1DMs61NKkDjUxd0M7t0kCjr+p7W3CP+XjsvYGFhG9KWYwcBa\",\n",
" \"Y57VucYoVzvCpq63Dpaq6xmLs2D32VfBF5xMjoM+hLYhCruP5Xe8EWmuvWtBZwC/nSfI0LO88nnzfIKFFOw6oQC9wjW1VdtD1Uwp\",\n",
" \"M5uFMJrmGxLnPX7s5HKoXBfO4HV7hJ/blEXtJ4L8fxaMzMhQ/29d17LctpFF9/wKcDPcQPoAaqGyM55ENbGdilzlyrJJQhJiEmDh\",\n",
" \"YQ7/fvqcc2+jASpVSazIfALdt+/jPBx/ZrFUoOt4Ut+Jz0sldBPwlCq6yinuLQwGbjYxpQY42ezNGl0JU3YJvrRDAorJseRm86lN\",\n",
" \"KP00I1ejBIn7I9vB8bgCTD6jL65vcX3YV4J7q7+Ce7Yb+9o+fP6ZxgYPLFefglhCpehnTGPZa7/9sqaV4BCjNJx6LFhZA5txbEfU\",\n",
" \"zcbS9kloPMqwQDTFyHTXTlZJSKDKz4f75excHmDmPehdNp4AlyQsZksqnp8SwRHuAelX0EBHPBAnOlD/pJxg+HUjiqZoTxAV+Z5a\",\n",
" \"3aFz+lfpkhVCH3buX4UgJWg1jeo9GPep51Zyq1yRvVjBZdTh9I2nzS4h4HynZ1lhHkDDDnms7J5j3lp8Uxl3HOh7EYOGgPHYfBLn\",\n",
" \"kVhprsi3+rejWZXDA6DWCwUVl2I37mLuGdgyOoI/53Mjyq+SWmb1BSeyy/wdJa5k4bCD1yqaKH12tcGyk1lUlYFBEIMFj4NYJc6c\",\n",
" \"iQY3qOIPt+IIX1zWiEpKFAwCUE+PnePhL6xPnhjaHJl5X3y+msPt1EB9mhB5dWciZf3WMw+Uf5DFK0EU+XFHpVLDOApQzujyZ/WK\",\n",
" \"zPlU6T0PIoBbkCld/5okol1lwvjVSXOuragzDvx4aiQpNilQXBC3xw6528bOUNblcmGUHMb+hxQbu2oX74fskV3rHBNR852b6rKJ\",\n",
" \"jiou8bGVY5UNyC6bgyBR/T5Ix6pEXw2ym8ckkWta1WUGqouv4DX6d5h9BGOUuqqVzEs7N9fw/uAkHWi9bDsDiGiMl3YuNgEmQe5i\",\n",
" \"drNI1MGm7lApXm6Cy96WDGt/pTUa2C/Sh7CemUPy+jcZC0lsh8vwxTorMWRQsZFdaKbzSpJQooyxhog5GTYRNZAGU3uA0nu193su\",\n",
" \"57C+oq4KtP4zIZW6P2APWjAehLpmR7biAEIy7ZU5HoWrqLuDrmU+V/gruYcv9CrFv0+DxkCjFnK6m7jiuBfJR2gmuHXNCUtMIxZm\",\n",
" \"FXwBajx8r3xveCTDsbkwoP1OdATrtWwgtyC2/MJfruXw8aSWw0B3EvfbqA90KpZLC1cpdN0TxRQh9sSVe1gvTCSYTeEvVPrpvml+\",\n",
" \"TTebmv0612eKeQhvvP0+UxBelq/Z66xjyRo6H/T1Rcyrh4R+x3vHyxJzzZMJXhAAE3Zx445DVWz4iTazuiBe5ZDZre6Bhlu8/4Yf\",\n",
" \"YLNefRWOUSM1WsZD7hcXsKMncr+WbjjWhqRl8dCuQjCBqrBJR5QTmAoNTzp2NcVv10MXCvd4i/dODh+WKHfevEh9R8iZnGkhrK9V\",\n",
" \"biyN3pnTyhvyP00J66nV7LKD2MhM8sSy47wbWjzEsmFcB5quQc3HrrNOX4g7CEQaAY3DDzv5UYY1FeUCHwvhGSZFBRkD/Qy1wdFt\",\n",
" \"5rSvHrBR4uUG0ewSvD061YMx2GJP4AU9n4CtVsPYWdmI00teOqk9+NYzUmY5dSA10Rs5wumdUJWs5ZJDPOEY0o+p0g3beoF4tRcW\",\n",
" \"VNf1s1pXdXH48FHt64kdTE9CIOmEdY/JBmwy3XesYNvK7rgVQeG1RkHl+ljG5SRMDCe1tPHP46AOqK8ALndnQthqTzExaE0i2Wrk\",\n",
" \"IyQGLhb/hdyUX+EFVt0XtuVdNXiNve1Fd6KUFbuY+uvKIDGBuQld6OnBwaxDHb1Lm7u+8AKOMax8MfBRLDfQiS8pi0KiPAbOIyZ2\",\n",
" \"BIpST83l5CV4x7E5rzpvi41geXHMjadNUeTzU37EUQ/4vVp7nrGbwZaSGI4xx1hxHyddI3Sh3ixT6JoJzxMPPU1TvM4CDIwMQrZO\",\n",
" \"waFlhzNd63KesOlTXvLj2PP2GwAszirrNmfHAhtynOx50DPQ1JOMXbiILfuBv1kMvbuYzCiteamkW9eNdQqpBCnNNIkxTntLGgt6\",\n",
" \"W3Jcvjryn+dir2lOJtO0OIkICuDlPAHlFGCsWWosmdjvg8n7JqUn2ywTUTVe/5WD3TTWQVMaHZJSfUEMaypWUUAu2tGk6QoxJ6Ys\",\n",
" \"Tj/3rYVQ17NMvm3xAn5CkL54T7vBbHh2jZiiG4jIUASTEzlY0sF8xbH1repIeFF9CAJjMrKcVpfWUtyVQuFt/CNZo5rjFMeuYc50\",\n",
" \"HPrJLVzmIcSDKvZIJbU0425m1SYgIyR/2McgGaR3uMgh1fWVgGnWtTaZNOL41QMdmx3770J0pgpPx8/ApSJGeEzuXtiEjCWCJmGF\",\n",
" \"bQFz9Gge2W/29QbpvbpnkzjJzRxqOBVlgm0Cs3bxKl8fC8R/wdmq//GuuqOPl/tJ8JkZ5bbY2LjQQfY1iCNTLzaJPbTH4yTEgKE4\",\n",
" \"8lfddgaOURDoQ4Wpen+/KQhIo5YfAqd5obkFmuoPODmmIl+y33AukoQdz9mASTXeiSoKTGpr9fYQV1YfOyakZz0byiGpM2yW9ZzA\",\n",
" \"mHWTAhgbXchPK8oWozX/elyo0UkIqmoQc7JY8Im/sOoerhp3pmT4jjL0wMYF/Fe8iXudvNihPcmALmK9ODQw1SpXz6CxXcSprAwK\",\n",
" \"HI5r7R9JTronu2t/JkoRfZs95eeHNF57OiNiddgDM5AoMUw0EYpc8ERmbvEp9UBLGjIly3hLGgNH7Y9YhZkV3LUKlnZhmZtbwJT1\",\n",
" \"xJDvJmntiTqZ8W8bqsbxhCwn+qK5uOJOH+ltNQK+xH48LXfRntomXbO4mXahY2VZD26dzRSbta/xSlSK8Bn/yICMh9JfJhaXWpTM\",\n",
" \"YLHyXgSFatjnjYeYT79dgzsm8Q1sMxtIn/I2IMAhQiq287lSMoSroqFtQQWBpIX1mONWfaGOvVpC1Eq0mWog5MKAFvEUw3X5ezyd\",\n",
" \"oWQAXK61mqY08vx27eu9y2K7bQtVY9N9B7YZwyp6Hh2pgmlZtAca8enUzH5IjQoG0GO9/6H4Q/i29mXtR5/KGzN6TW0i6AS/kkWG\",\n",
" \"2DHhs8iS7cz93Y8WwTWooDpUr5jRJEq9VWRoWQGu/UXA6jpZnyZJMqNbtIcEIy1TS4YOe3g9qmECzNK1x5lF5G/xbCN4P8WAHKvY\",\n",
" \"6zRpXG4WkwDIU5uPF+1cNWMHok1/2ZPVd5PTFBeaQpQaWw1SrIw78XgyT+3vqixdtyPjOjldq5ewVRqbPWIcS5fOAxALB/b4bbTu\",\n",
" \"uQ/8oo/JVdKDPBPlreqeqRFkItquBTSQRWWWo0m9ZUdbaIgxX0oVQBfL1K7CYDX2utY6fsOhzGrFJ2bkZu2MM1Gnl9W7GX6gUxZU\",\n",
" \"d65dtzHnuH6YqT/KW4pmFT2NoOfef6nf+qgxVz55CvEK0S/xrV3nkqsOrTtUc9z1r22ZXODt5mxXXGCIUfCRN2LW6sMwBOUX1vO1\",\n",
" \"lYF9hjr+pyTOP8xQ70Ap7Qm631IjuhSTrHs1kqAUo6X6M0gXprLFt+xUKedF7/C1DZAUeU71+b9+QS5OelkMF8P1Rnn3wyvNncFp\",\n",
" \"xvGOIv39lzcxLIbgdUGdVUN8vOmQj6cgKJirr6O54lwTGW7hkHs8o0sFhWISb2Ysx6+Nzk+CMVUAxEwBjtOLF6HwxduMocuT2WGT\",\n",
" \"PAElI9JeZk6Iz1V8W7lcf/jENbHOWUD/Rg3pTfo1CVPT/95e23AmevCy+GG92vikdL0pNn9MnZX4f/5m+Fm9rrgmN6vNc3wIhrT4\",\n",
" \"/cduHPTT1K2fP3e1+aMiLuyc/YGHPBNwpAG8/XLZXWOHS4J/yBwfxYgy9zWplz5AOEr1K8JDPDAfirnQOrsOk1Tp3Mp2PVMbVmsu\",\n",
" \"fpKZT49/38WthfgBvw8RRvYV8lukCQl9GjL46VaPLOwd+uyPm4lSkOqc7AUGU4q7zig0FJulgVHGVQgOmiam13GMci8g72OS5Qin\",\n",
" \"KncbnqWh/4EKwIv9xwYjZvdgrVqZf5YudRZT+BjBbzvaTKVoYJ9E2V0N9mjiaJg1LMhwjmSwJ5jT88SgzEQQ4qFeLudO/sRsT328\",\n",
" \"po6KaVkENQB0WlTM9lipZy+26bP2UD+j4sk/xVqls0v3QSoO80HnYko2fT+7mo6r5dfFBXmQWrCnF/Qqs8hMzGTQUIRGAWN3tW+j\",\n",
" \"iWBFZECsVfdkNSE96+WeW7nl5xlRhVOXkR9/cUpZVdwrd8su49eOCdnoEDMT9c0QLdMjTNyQoniHVv3Wc4ecjFhwd+22zHXW2i/S\",\n",
" \"Maa+5ne/TSK4WDuvTA5eiVCF/M0BpP7ZF3RpketpynM8uH3b6qPbutpF7gAp/ha6M3qVpr9qnWsYRTw1sjBza+YYdhI+JKGhP9l5\",\n",
" \"3ns8n8WIv+SAkaDGBHqs50LsDDSssoTz+piVDimsleZ4o9KIgoZIFPT221Ueg2cvPzch0hxnj2sS12RvpQb9GbmMz10svmBIH09H\",\n",
" \"U2fn5EZJd8wkrCSUsUFP9fEabiStDKp+xxXnCW58AvPr3r1z4d8patnCOtSViaK5sJSUVp0W5NpFpHSdKlmxxC3kZTXGYol9bzPE\",\n",
" \"y8wPiroSp/m4ezpv1XLD9V0XE8wIfwWgbxFYPc7pCIvvPIsDSDSm+48LW07V4M4QUw8S6mQ2QLC88CdcKv+QE5Q3OcH0ZZA0vlZJ\",\n",
" \"al1IJDRSQTCJh56NCK5rNpm8JG2Ifyc3bqIhpRjvxJ9Xt9H6nPBR3Ddxazwkid3QS4/DyURwnpas1OyiP7/F454+QYQsy4BveAft\",\n",
" \"5dT2uK139QFKg+euzlUs9EYlcX96nat9YzPfEx5uP3bGtWfnYn4atp2LPdHum3cTxyruliUcEK5MPPLtdDGXA1y/NlqRTGypdOAj\",\n",
" \"6VvkYyacSMBz6OI/yNTu7txdIog9wEZ1DJ1/UuOk1vT77s7wFctnaGxjz4qFBK1lLQy3HXAsmxyBPe+ff9T3e6CufxL5M1YrAOFi\",\n",
" \"kop9o1JeXWvK6RDXBRHupn1nChlT7Bdg5OX8C5SD1d4ibQ3XpJg9t7vSw6zmoWvAeHbVRf8y9CFKCl7XKrBAXWF24vFUInNoe/Oe\",\n",
" \"twf66BA9FLC0MFnYP9hcrnljod9PxhgY+jrIujYp/X4CehIiiGP9JcTjfZ8JXvBkSXZChKOGV1PxfbTjkFeWvQM4a65+R2RGq6Lr\",\n",
" \"xjMORnJGdMSjnN8lx7t00ylqYD7mt65tPFhs7drcAEHTvYlPV1ZxF21MQP/Vu3X0v8ZqZjXN40gK31tjx1thfZY8kJ61P5Il8MKx\",\n",
" \"tOSvZ/eVYs/S/L9QWPhDkYoVS1cw4n9jYEFTLkcvz41NcgYD4VoWIaw5Ix8W0tnTOMHl4r5UZ4r9uVkVTLmgWYt8hxM8aTukNpD5\",\n",
" \"rcuziuxVWQEZYFg3JTO4IanPcZ98A6nL8pJBI4gLrrSpSUzpsMbYdTU6IJ+I/j9JBpU7cTbyM/JkyMpSFDdHGaOuUzzPrupt1Am7\",\n",
" \"ap6c3D4kLodvqE3Wha+4yd58bZIEBgsWnGMDe4q40RdqCZXdnzemD2eOtSeRq0pD16Ga1ROIR+YgFRqfpj4u1YksPKXECSEtr+sm\",\n",
" \"4Uw2fBqdo3gfzwSZnmjddvWrOig84beJ0hyzzxpyRGpIfZgJ1MVfeVOCAyei7ExSIn4qd23VFOFnxf6jQ9HVj7F82JU8gNmBYxAG\",\n",
" \"bYTU3L+nEqO35GwO47yMG5cA72E23Nz0Dg4Wx7hKfuGYsOSJh796ae3eZoGUYW6JU25+rf+4yUbNeQSI2xg82n2Z4SG9+QrPakgr\",\n",
" \"8vqfWjrTXuKeD81wa2VDx0bgCzcTkVNo1pj4nxPPZZB+dr8f5+B2T765VLdopxIcl41DzW9PE9SGMIXnlpjtJ207xiEQJZcr7RcJ\",\n",
" \"ayXmLvqBS2C9UaFn0k6pR5xrcy3UYfJ/53pLL6XPTfyub6SUnDREUVKx7zH5fdrwiAGlTJpogh2q1ECrLR6f0pjL9Wcd1iKAclrn\",\n",
" \"MKM33pnhZWOhKMZFfZpDa9koQRVcwb8ApZdoFWbfkQFzKFzY81zRkcDAwTmCWp0UMar+hiI53fdYxRGZtRslCqQxRundCSpT6aEw\",\n",
" \"zsFI0CS5GUpz6XNpJx4qzGgOIvoQln3XmI5RjhO9TOOBLG+dqJ/xXI9BpzoYIlbHfrIFWX0zrQ65vxs01lkR+BhV07W0W4v1Pg4m\",\n",
" \"mAgD9U1Nck0NYu7a8BwMsaQKp8n881IgxYpJEy2mL/P2NWUKQqrthqwBH5PnwU/nrFNaiqcZ6GvqeUFvVaq9zgNGVUMQ4mMv5xeY\",\n",
" \"HqeUHPOi+MDrvR2ZBOWk+RozXTMLSJIqAbSF/8rfuHBRmroytCeBkClax7UHwicDKtb3gYVtS0v62vq7Bo0QSwrPEqAU+QpQY2hn\",\n",
" \"3N3J0pNOjvWwNEo3bSXr+0BgqExNvXpwQbLEdv5u0DSQhGXkQRnMjaZWYgjhwsRUCLTahJ6omPZNucxtYwfvAnHy4kfIWxWf4xvG\",\n",
" \"HRYu8HS/ZPIfx3bJxH4aJrysCtqllZtQHnZs0fpReAqO/F6RusT7ceDlNbJJ4y30uH0EmmcB7JvrEvqZqBklLhCeE472jxi6xsrb\",\n",
" \"CIRYt5JVR6/r6Namat6gPdsPJjRj9HpPrGhBO8huy1Vens9ElEyIyXwYoM4MP0aZGjX3uthDHQPKHa5AdZDIy1QVMEK+CMBZMSbS\",\n",
" \"maHpUU5uDuJ98lAxo7W2Zf929a0mNg+85rMLgLPXHIvze3cEJp0GP/b9g74tXSEq4aREZYTP50/ch52gDKNbwQL4LPY4dxfrUpWP\",\n",
" \"2V14msxR+hbH2/8B+yCygw==\",\n",
"])\n",
"\n",
"def load_shakespeare():\n",
" \"\"\"Decode + checksum the embedded excerpt. Idempotent, no network.\"\"\"\n",
" text = zlib.decompress(base64.b64decode(_SHAKES_B64)).decode(\"utf-8\")\n",
" want = \"a4537202f380020cbd7c71aa471f18b2e40d4bfba83061ea25c1d4e0c4418fd6\"\n",
" got = hashlib.sha256(text.encode(\"utf-8\")).hexdigest()\n",
" assert got == want, f\"corpus checksum {got[:8]} != {want[:8]}; the embedded payload was corrupted\"\n",
" return text\n",
"\n",
"text = load_shakespeare()\n",
"print(f\"loaded {len(text):,} characters, {len(set(text))} distinct symbols\")\n",
"print(text[:180])"
]
},
{
"cell_type": "markdown",
"id": "991661d8",
"metadata": {},
"source": [
"> **Interpretation.** ~82,000 characters, 61 distinct symbols (uppercase, lowercase, punctuation, newline). The whole vocabulary fits in a list you can print. That is the appeal of character-level modeling for a first language model: the tokenizer is trivial, so all the difficulty lives in the model, which is exactly where we want it for learning.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "8e529b95",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:56.649427Z",
"iopub.status.busy": "2026-06-10T19:52:56.649349Z",
"iopub.status.idle": "2026-06-10T19:52:56.651848Z",
"shell.execute_reply": "2026-06-10T19:52:56.651613Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"USE_FULL_CORPUS is False; using the embedded excerpt (the default canonical path)\n"
]
}
],
"source": [
"# deeper: optional full-corpus fetch from a commit-pinned URL. Off the canonical\n",
"# path: if it fails (offline, rate-limited), we keep the embedded excerpt.\n",
"USE_FULL_CORPUS = False # flip to True to train on the full ~1MB tiny_shakespeare\n",
"if USE_FULL_CORPUS:\n",
" import urllib.request\n",
" PINNED_URL = (\"https://raw.githubusercontent.com/karpathy/char-rnn/\"\n",
" \"master/data/tinyshakespeare/input.txt\")\n",
" FULL_SHA256 = \"86c4e6aa9db7c042ec79f339dcb96d42b0075e16b8fc2e86bf0ca57e2dc565ed\"\n",
" try:\n",
" raw = urllib.request.urlopen(PINNED_URL, timeout=30).read()\n",
" assert hashlib.sha256(raw).hexdigest() == FULL_SHA256, \"checksum mismatch\"\n",
" text = raw.decode(\"utf-8\")\n",
" print(f\"fetched full corpus: {len(text):,} chars\")\n",
" except Exception as e:\n",
" print(f\"fetch failed ({e}); staying on the embedded excerpt\")\n",
"else:\n",
" print(\"USE_FULL_CORPUS is False; using the embedded excerpt (the default canonical path)\")"
]
},
{
"cell_type": "markdown",
"id": "6433dda6",
"metadata": {},
"source": [
"## The map\n",
"\n",
"> **Part 1 — Tokenization: bytes to integers.** Build a character codec, then a byte-pair encoder you train yourself, and watch a 256-symbol vocabulary grow merges like ` th`.\n",
"> **Part 2 — Embeddings: integers to vectors.** The `nn.Embedding` lookup table, what its rows learn, and why similar tokens end up near each other.\n",
"> **Part 3 — A char-RNN language model.** A GRU that predicts the next character, trained on the excerpt with teacher forcing and the four-comment loop, with an experiment log.\n",
"> **Part 4 — Attention from scratch.** The seq2seq bottleneck, the softmax-over-keys operation, Bahdanau additive and Luong multiplicative scoring, the unified Q/K/V form, and a deliberate large-`d_k` failure you fix.\n",
"> **Part 5 — Temperature sampling.** Turn the trained model into a generator and feel temperature move it from greedy to gibberish.\n"
]
},
{
"cell_type": "markdown",
"id": "940fc261",
"metadata": {},
"source": [
"## Part 1 — Tokenization: bytes to integers\n",
"\n",
"> **Objectives.** Build a reversible character codec for the corpus. Understand the three tokenization families and their length-versus-vocabulary trade-off. Train a tiny byte-pair encoder and verify its merges on a hand-checkable string.\n",
"\n",
"Before any model sees text, the text becomes a sequence of integers. *Tokenization* is the choice of how to split text into pieces (tokens) and assign each an ID. The three families you should be able to name:\n",
"\n",
"- **Word-level**: split on whitespace and punctuation. Small sequences, but the vocabulary explodes and any unseen word is out-of-vocabulary.\n",
"- **Character-level**: one token per symbol. Tiny vocabulary, never out-of-vocabulary, but sequences are long and most compute goes on redundancy.\n",
"- **Subword (byte-pair encoding, WordPiece, SentencePiece)**: the dominant choice since GPT-2. Start from characters or bytes, repeatedly merge the most frequent adjacent pair. Common words become single tokens; rare words become subword sequences.\n",
"\n",
"We use character-level for the model (trivial codec, all difficulty in the model) and build BPE as a side exercise to feel why subword won.\n"
]
},
{
"cell_type": "markdown",
"id": "1d92b2b0",
"metadata": {},
"source": [
"### A character codec\n",
"\n",
"A character tokenizer is two lookup tables: `stoi` maps each symbol to an integer, `itos` maps back. Sorting the distinct symbols makes the mapping deterministic across runs.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "050834fb",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:56.652678Z",
"iopub.status.busy": "2026-06-10T19:52:56.652606Z",
"iopub.status.idle": "2026-06-10T19:52:56.655968Z",
"shell.execute_reply": "2026-06-10T19:52:56.655517Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"vocab_size = 61\n",
"first 10 symbols: ['\\n', ' ', '!', '&', \"'\", ',', '-', '.', ':', ';']\n",
"encode('First'): [16, 43, 52, 53, 54]\n"
]
}
],
"source": [
"chars = sorted(set(text)) # deterministic: sorted, so IDs are stable across runs\n",
"vocab_size = len(chars)\n",
"stoi = {c: i for i, c in enumerate(chars)}\n",
"itos = {i: c for i, c in enumerate(chars)}\n",
"\n",
"def encode(s): return [stoi[c] for c in s] # str -> list[int]\n",
"def decode(ids): return \"\".join(itos[i] for i in ids) # list[int] -> str\n",
"\n",
"print(f\"vocab_size = {vocab_size}\")\n",
"print(\"first 10 symbols:\", chars[:10])\n",
"print(\"encode('First'):\", encode(\"First\"))"
]
},
{
"cell_type": "markdown",
"id": "952f50b3",
"metadata": {},
"source": [
"> **Predict:** what is `decode(encode(\"To be, or not to be\"))`? Run the next cell. If a codec is not exactly reversible, every downstream number is suspect, so we assert the round-trip rather than eyeball it.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "af43c69b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:56.656848Z",
"iopub.status.busy": "2026-06-10T19:52:56.656772Z",
"iopub.status.idle": "2026-06-10T19:52:56.658813Z",
"shell.execute_reply": "2026-06-10T19:52:56.658395Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"'To be, or not to be'\n",
"[ ok ] codec round-trips exactly\n"
]
}
],
"source": [
"roundtrip = decode(encode(\"To be, or not to be\"))\n",
"print(repr(roundtrip))\n",
"assert roundtrip == \"To be, or not to be\", \"codec is not reversible — stoi/itos disagree\"\n",
"print(\"[ ok ] codec round-trips exactly\")"
]
},
{
"cell_type": "markdown",
"id": "fd26cc59",
"metadata": {},
"source": [
"> **Interpretation.** The codec is a bijection on the symbols that appear in the corpus. It has no notion of meaning; \"Q\" and \"q\" are simply two of the 61 integers. All the structure the model learns will come from the *sequence* of these integers, not from the integers themselves.\n"
]
},
{
"cell_type": "markdown",
"id": "80e20e07",
"metadata": {},
"source": [
"### Exercise 14.1 — Encode the corpus to a tensor\n",
"`Difficulty 1/5 · ~5 min`\n",
"\n",
"Encode the entire `text` into a 1-D `torch.long` tensor named `data`. This is the array every later batch will be sliced from. The check verifies the length, dtype, and that decoding the first 20 entries returns the first 20 characters.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "7cba41e6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:56.659569Z",
"iopub.status.busy": "2026-06-10T19:52:56.659505Z",
"iopub.status.idle": "2026-06-10T19:52:56.663613Z",
"shell.execute_reply": "2026-06-10T19:52:56.663035Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 14.1 corpus tensor: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# data: a 1-D LongTensor of token IDs for the whole corpus, shape (len(text),)\n",
"data = None # TODO: torch.tensor(encode(text), dtype=torch.long)\n",
"\n",
"def _check_data():\n",
" attempted(data) # 'not attempted' until you fill in `data`\n",
" assert data.dtype == torch.long, f\"dtype {data.dtype}, expected torch.long\"\n",
" check_shape(data, (len(text),))\n",
" assert decode(data[:20].tolist()) == text[:20], \"decoding the head does not match the text head\"\n",
"\n",
"check(\"14.1 corpus tensor\", _check_data)"
]
},
{
"cell_type": "markdown",
"id": "038477b6",
"metadata": {},
"source": [
"Hint 1 (conceptual)You already have `encode`, which turns a string into a list of ints. Wrap that list in a `torch.tensor` and set the dtype so indexing later gives integer IDs, not floats.\n",
"\n",
"Hint 2 (the line)`data = torch.tensor(encode(text), dtype=torch.long)`\n",
"\n",
"Help — \"expected dtype long but got float\"You omitted `dtype=torch.long`. Embedding layers index with integer tensors; a float index raises later, not here, so set the dtype now.\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "bd57b507",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:56.664329Z",
"iopub.status.busy": "2026-06-10T19:52:56.664256Z",
"iopub.status.idle": "2026-06-10T19:52:56.671286Z",
"shell.execute_reply": "2026-06-10T19:52:56.670987Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 14.1 corpus tensor\n",
"data: (81924,) torch.int64 · first 20: [16, 43, 52, 53, 54, 1, 13, 43, 54, 43, 60, 39, 48, 8, 0, 12, 39, 40, 49, 52]\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines data; the check below re-verifies it.\n",
"data = torch.tensor(encode(text), dtype=torch.long)\n",
"\n",
"check(\"14.1 corpus tensor\", _check_data, required=True)\n",
"print(f\"data: {tuple(data.shape)} {data.dtype} · first 20: {data[:20].tolist()}\")"
]
},
{
"cell_type": "markdown",
"id": "513d5155",
"metadata": {},
"source": [
"### Byte-pair encoding, from scratch\n",
"\n",
"Character-level keeps the vocabulary tiny but the sequences long. Byte-pair encoding (BPE) buys back sequence length: it repeatedly finds the most frequent adjacent pair of tokens and merges it into a new token. After enough merges, ` the` is one token, while a rare word stays several. GPT-2 uses byte-level BPE; this is the warm-up, and Ch 15 has the full version with regex pre-tokenization and special tokens.\n",
"\n",
"The three operations are small enough to read in full: count adjacent pairs, merge one pair everywhere, and loop until the target vocabulary size.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "505ed078",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:56.672152Z",
"iopub.status.busy": "2026-06-10T19:52:56.672074Z",
"iopub.status.idle": "2026-06-10T19:52:56.674801Z",
"shell.execute_reply": "2026-06-10T19:52:56.674544Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"pair counts: {(97, 98): 2, (98, 97): 1}\n",
"merge (a,b)->256: [256, 256]\n"
]
}
],
"source": [
"from collections import Counter\n",
"\n",
"def get_pair_counts(seq):\n",
" \"\"\"Count every adjacent pair in a token sequence. seq: list[int] -> Counter.\"\"\"\n",
" return Counter(zip(seq, seq[1:]))\n",
"\n",
"def merge(seq, pair, new_id):\n",
" \"\"\"Replace every occurrence of `pair` in `seq` with the single token `new_id`.\"\"\"\n",
" out, i = [], 0\n",
" while i < len(seq):\n",
" if i < len(seq) - 1 and (seq[i], seq[i + 1]) == pair:\n",
" out.append(new_id); i += 2\n",
" else:\n",
" out.append(seq[i]); i += 1\n",
" return out\n",
"\n",
"# micro-demo on a hand-checkable string: \"abab\" -> pair (a,b) is most frequent\n",
"demo = [ord(\"a\"), ord(\"b\"), ord(\"a\"), ord(\"b\")]\n",
"print(\"pair counts:\", dict(get_pair_counts(demo)))\n",
"print(\"merge (a,b)->256:\", merge(demo, (ord(\"a\"), ord(\"b\")), 256))"
]
},
{
"cell_type": "markdown",
"id": "9656c10e",
"metadata": {},
"source": [
"> **Interpretation.** On `abab` the pair `(97, 98)` appears twice and merging it gives `[256, 256]`: a sequence half as long, with one new symbol. That is the whole idea of BPE in miniature. Real BPE just runs this thousands of times on a real corpus.\n"
]
},
{
"cell_type": "markdown",
"id": "18c3870f",
"metadata": {},
"source": [
"### Exercise 14.2 — Train a byte-pair encoder\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"Implement `train_bpe(byte_seq, num_merges)`. Starting from a list of byte values, perform `num_merges` greedy merges: each round, find the single most frequent adjacent pair, assign it the next free ID (starting at 256), and merge it everywhere. Return `(merges, final_seq)` where `merges` is an ordered list of the `(pair, new_id)` you applied.\n",
"\n",
"The check runs it on a string where the answer is hand-derivable: in `\"abababab\"`, the first merge must be `(a, b)`.\n",
"\n",
"Harder: after the first merge to `ab`, the second most frequent pair becomes `(ab, ab)`; predict the third merged sequence before you look.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "34831ffd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:56.675591Z",
"iopub.status.busy": "2026-06-10T19:52:56.675519Z",
"iopub.status.idle": "2026-06-10T19:52:56.679184Z",
"shell.execute_reply": "2026-06-10T19:52:56.678742Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 14.2 bpe first merge: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def train_bpe(byte_seq, num_merges):\n",
" \"\"\"byte_seq: list[int] in 0..255. Returns (merges, seq).\n",
" merges: list of ((tok_a, tok_b), new_id) in the order applied.\n",
" seq: the fully merged sequence.\"\"\"\n",
" seq = list(byte_seq)\n",
" merges = []\n",
" next_id = 256\n",
" for _ in range(num_merges):\n",
" # TODO 1: count adjacent pairs with get_pair_counts; stop early if there are none\n",
" counts = None\n",
" attempted(counts)\n",
" if not counts:\n",
" break\n",
" # TODO 2: pick the single most frequent pair (Counter.most_common(1))\n",
" best = None\n",
" # TODO 3: merge `best` into `next_id` across seq, record (best, next_id), bump next_id\n",
" attempted(best)\n",
" seq = merge(seq, best, next_id)\n",
" merges.append((best, next_id))\n",
" next_id += 1\n",
" return merges, seq\n",
"\n",
"def _check_bpe():\n",
" merges, seq = train_bpe(list(\"abababab\".encode()), num_merges=1)\n",
" assert merges[0][0] == (ord(\"a\"), ord(\"b\")), \\\n",
" f\"first merge {merges[0][0]}, expected (97, 98): (a,b) is the most frequent pair in 'abababab'\"\n",
" assert len(seq) == 4, f\"after merging (a,b) once, 'abababab' (8 tokens) should be 4 tokens, got {len(seq)}\"\n",
"\n",
"check(\"14.2 bpe first merge\", _check_bpe)"
]
},
{
"cell_type": "markdown",
"id": "3b876f4f",
"metadata": {},
"source": [
"Hint 1 (conceptual)Each round is exactly the two helpers you already have: `get_pair_counts` to find the winner, then `merge` to apply it. The only bookkeeping is the next free ID, which starts at 256 (bytes 0-255 are taken) and increments once per merge.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```\n",
"counts = get_pair_counts(seq)\n",
"if not counts: break\n",
"best = counts.most_common(1)[0][0] # the pair, not the count\n",
"seq = merge(seq, best, next_id)\n",
"merges.append((best, next_id)); next_id += 1\n",
"```\n",
"\n",
"\n",
"Help — \"most_common(1) gives a count, not a pair\"`Counter.most_common(1)` returns `[(pair, count)]`: a list of one (pair, count) tuple. You want `[0][0]` to reach the pair itself.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "2dedbd38",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:56.680086Z",
"iopub.status.busy": "2026-06-10T19:52:56.680007Z",
"iopub.status.idle": "2026-06-10T19:52:57.264922Z",
"shell.execute_reply": "2026-06-10T19:52:57.264480Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 14.2 bpe first merge\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"50 merges; corpus went from 81924 bytes to 56029 tokens\n",
"first merges: ['e ', 'th', 't ', 's ', 'ou', ', ', 'd ', 'er']\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines train_bpe; the check below re-verifies it.\n",
"def train_bpe(byte_seq, num_merges):\n",
" seq = list(byte_seq)\n",
" merges, next_id = [], 256\n",
" for _ in range(num_merges):\n",
" counts = get_pair_counts(seq)\n",
" if not counts:\n",
" break\n",
" best = counts.most_common(1)[0][0]\n",
" seq = merge(seq, best, next_id)\n",
" merges.append((best, next_id))\n",
" next_id += 1\n",
" return merges, seq\n",
"\n",
"check(\"14.2 bpe first merge\", _check_bpe, required=True)\n",
"\n",
"# run it on real text: a few merges on the corpus bytes\n",
"N_MERGES = 20 if FAST else 50\n",
"merges, merged = train_bpe(list(text.encode(\"utf-8\")), num_merges=N_MERGES)\n",
"print(f\"{len(merges)} merges; corpus went from {len(text.encode())} bytes to {len(merged)} tokens\")\n",
"# show the first few merges as readable strings\n",
"vocab = {i: bytes([i]) for i in range(256)}\n",
"for (a, b), nid in merges:\n",
" vocab[nid] = vocab[a] + vocab[b]\n",
"print(\"first merges:\", [vocab[nid].decode(\"utf-8\", \"replace\") for _, nid in merges[:8]])"
]
},
{
"cell_type": "markdown",
"id": "3497be34",
"metadata": {},
"source": [
"> **Interpretation.** The first merges are the corpus's most frequent digraphs. On Shakespeare you typically see common pairs and leading-space combinations near the top. Each merge shortens the sequence and grows the vocabulary by one, which is the length-versus-vocabulary trade made explicit, merge by merge.\n",
"\n",
"> **Common confusion:** the leading space matters. Most real tokenizers attach the space to the *front* of a word, so ` the` (with a space) and `the` are different tokens. Prompts that look identical in a chat box can tokenize differently depending on where the space landed, which is why you always inspect the actual token IDs when debugging surprising model behaviour.\n"
]
},
{
"cell_type": "markdown",
"id": "9ee3d034",
"metadata": {},
"source": [
"> **Key takeaways**\n",
"> - Tokenization maps text to integers; the choice of granularity trades sequence length against vocabulary size.\n",
"> - Character-level is trivial to implement and never out-of-vocabulary; we use it for the model so the difficulty lives in the model.\n",
"> - BPE is three small functions: count pairs, merge the top pair, repeat. Each merge shortens sequences and grows the vocabulary by one.\n",
"> - A leading space is part of the token; ` the` and `the` are different IDs.\n"
]
},
{
"cell_type": "markdown",
"id": "719c14da",
"metadata": {},
"source": [
"## Part 2 — Embeddings: integers to vectors\n",
"\n",
"> **Objectives.** Understand `nn.Embedding` as a learned lookup table. See that it is exactly a one-hot matrix multiply, and that the rows, not the lookup, are what carry meaning.\n",
"\n",
"A token ID is just an index. To do arithmetic the model needs a vector. `nn.Embedding(vocab_size, d_embed)` is a learned table of shape `(vocab_size, d_embed)`; row `i` is token `i`'s vector. The lookup is the whole operation. What makes embeddings interesting is what the rows *become* after training: similar tokens drift to similar positions, and famously `king - man + woman ≈ queen` falls out of word2vec without anyone designing it in. That geometry is a consequence of the training objective, not of the lookup.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "8c984d3a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:57.266123Z",
"iopub.status.busy": "2026-06-10T19:52:57.266026Z",
"iopub.status.idle": "2026-06-10T19:52:57.269340Z",
"shell.execute_reply": "2026-06-10T19:52:57.268961Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"input ids : (2, 3)\n",
"output vecs: (2, 3, 8)\n",
"row for 'T' equals table row stoi['T']: True\n"
]
}
],
"source": [
"emb = torch.nn.Embedding(num_embeddings=vocab_size, embedding_dim=8)\n",
"ids = torch.tensor([[stoi['T'], stoi['o'], stoi[' ']],\n",
" [stoi['b'], stoi['e'], stoi['.']]]) # (2, 3) batch of token IDs\n",
"vecs = emb(ids) # (2, 3, 8) one vector per token\n",
"print(\"input ids :\", tuple(ids.shape))\n",
"print(\"output vecs:\", tuple(vecs.shape))\n",
"print(\"row for 'T' equals table row stoi['T']:\",\n",
" torch.allclose(vecs[0, 0], emb.weight[stoi['T']]))"
]
},
{
"cell_type": "markdown",
"id": "e6433945",
"metadata": {},
"source": [
"> **Interpretation.** Indexing `(2, 3)` IDs returns `(2, 3, 8)`: the embedding adds a trailing dimension of size `d_embed`, leaving the batch and time axes untouched. The lookup is exactly `table[ids]`. Nothing is learned yet; the rows are random until training shapes them.\n"
]
},
{
"cell_type": "markdown",
"id": "10a541d1",
"metadata": {},
"source": [
"### Exercise 14.3 — Embedding is a one-hot matrix multiply\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"`nn.Embedding` looks like a lookup, but it is identical to one-hot encoding the IDs and multiplying by the table. Implement `embed_via_onehot(ids, table)` that returns the same thing `table[ids]` does, but via an explicit one-hot matmul. The check compares your output to the lookup elementwise.\n",
"\n",
"Understanding this equivalence is what lets you reason about gradients flowing back into an embedding table: only the rows that were looked up get a gradient.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "6c523995",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:57.270363Z",
"iopub.status.busy": "2026-06-10T19:52:57.270296Z",
"iopub.status.idle": "2026-06-10T19:52:57.273235Z",
"shell.execute_reply": "2026-06-10T19:52:57.272843Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 14.3 embedding=onehot@table: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import torch.nn.functional as F\n",
"\n",
"def embed_via_onehot(ids, table):\n",
" \"\"\"ids: (B, T) long. table: (vocab, d). Return (B, T, d) via one-hot @ table.\"\"\"\n",
" vocab = table.shape[0]\n",
" # TODO 1: one-hot encode ids -> (B, T, vocab) float, with F.one_hot(...).float()\n",
" onehot = None\n",
" # TODO 2: matrix-multiply onehot @ table -> (B, T, d)\n",
" out = None\n",
" attempted(onehot, out)\n",
" return out\n",
"\n",
"def _check_onehot():\n",
" table = emb.weight.detach()\n",
" got = embed_via_onehot(ids, table)\n",
" check_shape(got, (2, 3, 8))\n",
" check_close_torch(got, table[ids], msg=\"one-hot @ table must equal the lookup table[ids]\")\n",
"\n",
"check(\"14.3 embedding=onehot@table\", _check_onehot)"
]
},
{
"cell_type": "markdown",
"id": "08a0b05e",
"metadata": {},
"source": [
"Hint 1 (conceptual)One-hot encoding turns each ID into a row of zeros with a single 1 at the ID's position. Multiplying that row by the table selects exactly one table row. Do it for all `(B, T)` IDs at once with a batched matmul.\n",
"\n",
"Hint 2 (the lines)`onehot = F.one_hot(ids, num_classes=vocab).float()` then `out = onehot @ table`. The `@` broadcasts over the leading `(B, T)` dimensions.\n",
"\n",
"Help — \"expected scalar type Long but found Float\" or a matmul dtype error`F.one_hot` returns a `long` tensor; matmul against a float table needs `.float()`. Cast the one-hot, not the table.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "4073f31e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:57.274020Z",
"iopub.status.busy": "2026-06-10T19:52:57.273955Z",
"iopub.status.idle": "2026-06-10T19:52:57.276505Z",
"shell.execute_reply": "2026-06-10T19:52:57.276164Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 14.3 embedding=onehot@table\n",
"[ ok ] the lookup is a one-hot matmul; gradients reach only the rows that were indexed\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines embed_via_onehot; the check re-verifies it.\n",
"def embed_via_onehot(ids, table):\n",
" vocab = table.shape[0]\n",
" onehot = F.one_hot(ids, num_classes=vocab).float() # (B, T, vocab)\n",
" return onehot @ table # (B, T, d)\n",
"\n",
"check(\"14.3 embedding=onehot@table\", _check_onehot, required=True)\n",
"print(\"[ ok ] the lookup is a one-hot matmul; gradients reach only the rows that were indexed\")"
]
},
{
"cell_type": "markdown",
"id": "55f3d9a5",
"metadata": {},
"source": [
"> **Key takeaways**\n",
"> - `nn.Embedding(V, d)` is a `(V, d)` learned table; indexing it adds a trailing size-`d` axis.\n",
"> - It is mathematically a one-hot encode followed by a matmul, so only looked-up rows receive gradient.\n",
"> - Meaning lives in the rows after training, not in the lookup itself.\n"
]
},
{
"cell_type": "markdown",
"id": "e45090b5",
"metadata": {},
"source": [
"## Part 3 — A character-level GRU language model\n",
"\n",
"> **Objectives.** Build a GRU that predicts the next character. Make training batches by sliding a window over the corpus. Train with teacher forcing and the four-comment loop, log the loss, and read the experiment log as your expected-value reference.\n",
"\n",
"A language model assigns a probability to the next token given the previous ones. The character-level recipe: embed each character, run a GRU over the sequence to get a hidden state per position, and project each hidden state to a distribution over the vocabulary. The GRU (Ch 13) is the recurrence; here it is a black-box `nn.GRU`, because the chapter's new idea is attention, not the recurrent cell.\n"
]
},
{
"cell_type": "markdown",
"id": "76920e6d",
"metadata": {},
"source": [
"### Making batches by sliding a window\n",
"\n",
"Training data for a language model is `(context, next-char)` pairs. We pick `block_size` characters as the context and the same window shifted right by one as the targets, so position `t` predicts the character at `t+1`. Picking random start offsets gives independent training examples from one long stream.\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "5d102929",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:57.277243Z",
"iopub.status.busy": "2026-06-10T19:52:57.277169Z",
"iopub.status.idle": "2026-06-10T19:52:57.281173Z",
"shell.execute_reply": "2026-06-10T19:52:57.280797Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"train 73,731 chars · val 8,193 chars\n",
"x (32, 64) y (32, 64)\n",
"x[0,:1] predicts y[0,:1]: ' and end' -> 'and end,'\n"
]
}
],
"source": [
"BLOCK_SIZE = 64 # context length: how many characters the model conditions on\n",
"BATCH_SIZE = 32 # sequences per batch\n",
"n = int(0.9 * len(data))\n",
"train_data, val_data = data[:n], data[n:] # a held-out tail, never trained on\n",
"print(f\"train {len(train_data):,} chars · val {len(val_data):,} chars\")\n",
"\n",
"def get_batch(split, gen):\n",
" \"\"\"Return (x, y) each (BATCH_SIZE, BLOCK_SIZE); y is x shifted right by one.\n",
" gen is an explicit torch.Generator so batches are reproducible.\"\"\"\n",
" src = train_data if split == \"train\" else val_data\n",
" ix = torch.randint(0, len(src) - BLOCK_SIZE - 1, (BATCH_SIZE,), generator=gen)\n",
" x = torch.stack([src[i:i + BLOCK_SIZE] for i in ix])\n",
" y = torch.stack([src[i + 1:i + BLOCK_SIZE + 1] for i in ix])\n",
" return x, y\n",
"\n",
"g = torch.Generator().manual_seed(SEED)\n",
"xb, yb = get_batch(\"train\", g)\n",
"print(\"x\", tuple(xb.shape), \"y\", tuple(yb.shape))\n",
"print(\"x[0,:1] predicts y[0,:1]:\", repr(decode(xb[0, :8].tolist())), \"->\", repr(decode(yb[0, :8].tolist())))"
]
},
{
"cell_type": "markdown",
"id": "0e042daf",
"metadata": {},
"source": [
"> **Interpretation.** `x` and `y` are both `(32, 64)`. `y` is `x` shifted by one, so every one of the `32 * 64` positions is a supervised next-character prediction. One batch yields 2,048 training signals, which is why character LMs learn from a tiny corpus.\n"
]
},
{
"cell_type": "markdown",
"id": "ef29c826",
"metadata": {},
"source": [
"### Exercise 14.4 — Build the char-RNN forward pass\n",
"`Difficulty 3/5 · ~20 min`\n",
"\n",
"Fill in `CharRNN.forward`. The pieces are wired in `__init__`: an embedding, a GRU, and a linear head. Your job is the three-line forward: embed the IDs, run the GRU, project to logits. Return logits of shape `(B, T, vocab)` and the final hidden state.\n",
"\n",
"The checks verify the output shape on a random batch and that an untrained model's loss is near `ln(vocab_size)` (a uniform model's loss), which is the sanity value every language model starts from.\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "5f88df2a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:57.281916Z",
"iopub.status.busy": "2026-06-10T19:52:57.281848Z",
"iopub.status.idle": "2026-06-10T19:52:57.287454Z",
"shell.execute_reply": "2026-06-10T19:52:57.287002Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 14.4 rnn output shape: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 14.4 rnn init loss ~ ln(vocab): not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class CharRNN(torch.nn.Module):\n",
" def __init__(self, vocab, d_embed=64, d_hidden=128):\n",
" super().__init__()\n",
" self.embed = torch.nn.Embedding(vocab, d_embed)\n",
" self.gru = torch.nn.GRU(d_embed, d_hidden, batch_first=True)\n",
" self.head = torch.nn.Linear(d_hidden, vocab)\n",
"\n",
" def forward(self, ids, h=None):\n",
" # ids: (B, T) long. Returns logits (B, T, vocab) and hidden (1, B, d_hidden).\n",
" # TODO 1: e = self.embed(ids) # (B, T, d_embed)\n",
" e = None\n",
" # TODO 2: out, h = self.gru(e, h) # out: (B, T, d_hidden)\n",
" out = None\n",
" # TODO 3: logits = self.head(out) # (B, T, vocab)\n",
" logits = None\n",
" attempted(e, out, logits)\n",
" return logits, h\n",
"\n",
"torch.manual_seed(SEED)\n",
"model = CharRNN(vocab_size)\n",
"\n",
"def _check_rnn_shape():\n",
" logits, h = model(xb)\n",
" check_shape(logits, (BATCH_SIZE, BLOCK_SIZE, vocab_size))\n",
"\n",
"def _check_rnn_init_loss():\n",
" logits, _ = model(xb)\n",
" loss = F.cross_entropy(logits.reshape(-1, vocab_size), yb.reshape(-1))\n",
" expected = math.log(vocab_size)\n",
" assert abs(float(loss) - expected) < 0.5, \\\n",
" f\"init loss {float(loss):.2f} should be near ln({vocab_size})={expected:.2f} (a uniform model); a big gap means the head or embedding is mis-wired\"\n",
"\n",
"check(\"14.4 rnn output shape\", _check_rnn_shape)\n",
"check(\"14.4 rnn init loss ~ ln(vocab)\", _check_rnn_init_loss)"
]
},
{
"cell_type": "markdown",
"id": "5771fb3c",
"metadata": {},
"source": [
"Hint 1 (conceptual)Three layers, three lines, in order: embedding, then GRU, then linear head. `nn.GRU` with `batch_first=True` takes `(B, T, d_embed)` and returns `(output, h_n)` where `output` is `(B, T, d_hidden)`.\n",
"\n",
"Hint 2 (the lines)\n",
"\n",
"```python\n",
"e = self.embed(ids)\n",
"out, h = self.gru(e, h)\n",
"logits = self.head(out)\n",
"```\n",
"\n",
"\n",
"Help — \"init loss is 8.0, not ~4.1\"A loss far above `ln(vocab)` usually means the head output dimension is wrong, or you returned activations instead of logits. Print `logits.shape`; the last axis must be `vocab_size`. Cross-entropy expects raw logits, not a softmax.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "ef258236",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:57.288464Z",
"iopub.status.busy": "2026-06-10T19:52:57.288390Z",
"iopub.status.idle": "2026-06-10T19:52:57.306116Z",
"shell.execute_reply": "2026-06-10T19:52:57.305560Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 14.4 rnn output shape\n",
"[ ok ] 14.4 rnn init loss ~ ln(vocab)\n",
"CharRNN parameters: 86,269\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines CharRNN.forward via a clean re-definition of the class.\n",
"class CharRNN(torch.nn.Module):\n",
" def __init__(self, vocab, d_embed=64, d_hidden=128):\n",
" super().__init__()\n",
" self.embed = torch.nn.Embedding(vocab, d_embed)\n",
" self.gru = torch.nn.GRU(d_embed, d_hidden, batch_first=True)\n",
" self.head = torch.nn.Linear(d_hidden, vocab)\n",
"\n",
" def forward(self, ids, h=None):\n",
" e = self.embed(ids) # (B, T, d_embed)\n",
" out, h = self.gru(e, h) # (B, T, d_hidden)\n",
" logits = self.head(out) # (B, T, vocab)\n",
" return logits, h\n",
"\n",
"torch.manual_seed(SEED)\n",
"model = CharRNN(vocab_size)\n",
"check(\"14.4 rnn output shape\", _check_rnn_shape, required=True)\n",
"check(\"14.4 rnn init loss ~ ln(vocab)\", _check_rnn_init_loss, required=True)\n",
"print(f\"CharRNN parameters: {param_count(model):,}\")"
]
},
{
"cell_type": "markdown",
"id": "0a8987a9",
"metadata": {},
"source": [
"> **Note:** the parameter-count print after a module is a habit worth keeping. It is the first thing to check when a model will not fit or will not learn, and it catches dimension typos that shape checks miss.\n"
]
},
{
"cell_type": "markdown",
"id": "1f22dfb8",
"metadata": {},
"source": [
"### The training loop\n",
"\n",
"Teacher forcing makes this simple: at every position the model conditions on the *true* previous characters (they are right there in `x`), so the loss is a clean per-position cross-entropy. The loop is the four-comment skeleton used in every chapter: forward, backward, update, track. We re-seed at the top so re-running this cell alone reproduces the loss log.\n",
"\n",
"> **Runtime:** this cell takes about 1-3 minutes on CPU at full fidelity, a few seconds under `NB_FAST`.\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "40fadc86",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:52:57.306996Z",
"iopub.status.busy": "2026-06-10T19:52:57.306917Z",
"iopub.status.idle": "2026-06-10T19:53:18.389972Z",
"shell.execute_reply": "2026-06-10T19:53:18.389467Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"step 0 · train 3.981 · val 3.989\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"step 250 · train 1.658 · val 1.744\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"step 500 · train 1.467 · val 1.643\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"step 750 · train 1.379 · val 1.665\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"step 1000 · train 1.305 · val 1.686\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"step 1250 · train 1.217 · val 1.661\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"step 1499 · train 1.183 · val 1.746\n",
"\n",
"final val loss 1.746 (random baseline ln(vocab) = 4.111)\n"
]
}
],
"source": [
"STEPS = 60 if FAST else 1500 # training iterations: ~25x fewer in CI smoke mode\n",
"EVAL_EVERY = 20 if FAST else 250\n",
"EVAL_BATCHES = 3 if FAST else 10 # eval is cheap in smoke mode, thorough at full fidelity\n",
"LR = 3e-3 # Adam step size; 3e-3 is a stable default for this size\n",
"\n",
"torch.manual_seed(SEED)\n",
"model = CharRNN(vocab_size)\n",
"opt = torch.optim.Adam(model.parameters(), lr=LR)\n",
"gen = torch.Generator().manual_seed(SEED)\n",
"\n",
"@torch.no_grad()\n",
"def eval_loss(split, batches=EVAL_BATCHES):\n",
" model.eval()\n",
" losses = []\n",
" for _ in range(batches):\n",
" x, y = get_batch(split, gen)\n",
" logits, _ = model(x)\n",
" losses.append(F.cross_entropy(logits.reshape(-1, vocab_size), y.reshape(-1)).item())\n",
" model.train()\n",
" return sum(losses) / len(losses)\n",
"\n",
"loss_log = []\n",
"for step in range(STEPS):\n",
" x, y = get_batch(\"train\", gen)\n",
" logits, _ = model(x) # forward\n",
" loss = F.cross_entropy(logits.reshape(-1, vocab_size), y.reshape(-1))\n",
" opt.zero_grad(set_to_none=True) # zero grads (the gradient-accumulation footgun, avoided)\n",
" loss.backward() # backward\n",
" torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0) # clip exploding RNN grads\n",
" opt.step() # update\n",
" if step % EVAL_EVERY == 0 or step == STEPS - 1: # track stats\n",
" tr, va = eval_loss(\"train\"), eval_loss(\"val\")\n",
" loss_log.append((step, tr, va))\n",
" print(f\"step {step:4d} · train {tr:.3f} · val {va:.3f}\")\n",
"\n",
"print(f\"\\nfinal val loss {loss_log[-1][2]:.3f} (random baseline ln(vocab) = {math.log(vocab_size):.3f})\")"
]
},
{
"cell_type": "markdown",
"id": "48b95537",
"metadata": {},
"source": [
"> **Interpretation.** Loss starts near `ln(61) ≈ 4.11` (uniform) and falls. At full fidelity (1500 steps) val loss lands around 1.7-2.1; under `NB_FAST` (60 steps) it only reaches ~3.0-3.5, which is correct, not broken, because the smoke run does a few percent of the training. The experiment log below records both so you know what to expect for your setting.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "306fb1bf",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.390913Z",
"iopub.status.busy": "2026-06-10T19:53:18.390739Z",
"iopub.status.idle": "2026-06-10T19:53:18.481818Z",
"shell.execute_reply": "2026-06-10T19:53:18.481319Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"this run: STEPS=1500, final val loss 1.746, params 86,269\n"
]
},
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
setting
\n",
"
expected val loss
\n",
"
wall time
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
NB_FAST (60 steps)
\n",
"
~3.0-3.5
\n",
"
~3-10 s
\n",
"
\n",
"
\n",
"
1
\n",
"
full (1500 steps)
\n",
"
~1.7-2.1
\n",
"
~30 s-3 min
\n",
"
\n",
"
\n",
"
2
\n",
"
random baseline
\n",
"
4.11 (=ln vocab)
\n",
"
0 s
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" setting expected val loss wall time\n",
"0 NB_FAST (60 steps) ~3.0-3.5 ~3-10 s\n",
"1 full (1500 steps) ~1.7-2.1 ~30 s-3 min\n",
"2 random baseline 4.11 (=ln vocab) 0 s"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# the experiment log: the learner's expected-value reference (Karpathy's loss-log convention)\n",
"import pandas as pd\n",
"log_table = pd.DataFrame([\n",
" {\"setting\": \"NB_FAST (60 steps)\", \"expected val loss\": \"~3.0-3.5\", \"wall time\": \"~3-10 s\"},\n",
" {\"setting\": \"full (1500 steps)\", \"expected val loss\": \"~1.7-2.1\", \"wall time\": \"~30 s-3 min\"},\n",
" {\"setting\": \"random baseline\", \"expected val loss\": f\"{math.log(vocab_size):.2f} (=ln vocab)\", \"wall time\": \"0 s\"},\n",
"])\n",
"print(f\"this run: STEPS={STEPS}, final val loss {loss_log[-1][2]:.3f}, params {param_count(model):,}\")\n",
"log_table"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "5e0fb91b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.482790Z",
"iopub.status.busy": "2026-06-10T19:53:18.482652Z",
"iopub.status.idle": "2026-06-10T19:53:18.556052Z",
"shell.execute_reply": "2026-06-10T19:53:18.555613Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAGGCAYAAACNCg6xAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAbHFJREFUeJzt3Xd4VFX+x/H3pPcGSegdqaEXAyq9KhCsi0hRbAgCIojsrgq6Ctjrz8YqrsLigjRRQEQQQTqEXqS3hBaSEBJS7++PIUOGFGZgkkn5vJ7nPszcOXPne5IQPpx77rkmwzAMREREROSGXJxdgIiIiEhJoeAkIiIiYiMFJxEREREbKTiJiIiI2EjBSURERMRGCk4iIiIiNlJwEhEREbGRgpOIiIiIjRScRERERGyk4CRSxs2YMQOTycTmzZudXUqZNmnSJEwm0029N/t7ePToUccWJSK5KDiJiNPVqFEDk8lk2Xx9fWnTpg3/+c9/crVdtWqVpd2WLVtyvT506FD8/Pys9nXs2BGTyUSfPn1ytT969Cgmk4m33367wBqTk5OZNGkSq1atsq9zIlKqKDiJSLHQrFkzvv32W7799lsmTZpEQkICQ4YM4csvv8z3PZMmTbLrMxYvXpxn2LJFcnIykydPLrTg9M9//pOUlJSbeu+gQYNISUmhevXqDq5KRK6n4CQiRSI5ObnA1ytXrswjjzzCI488wvjx41mzZg1+fn689957ebZv1qwZixcvZuvWrTZ9frVq1QgODmby5Ml2134zLl++bFd7Nzc3vLy8buqzXF1d8fLyuulTfSJiOwUnkVLu1KlTDBs2jEqVKuHp6UnNmjUZPnw4aWlpVu1SU1MZO3YsoaGh+Pr60r9/f86dO2fVZuHChdx9992WY9WuXZvXXnuNzMxMq3YdO3akcePGbNmyhbvuugsfHx/+/ve/21V3aGgo9evX59ChQ3m+/uyzzxIcHGzzqJO/vz/PPfccP/74o81hK9vRo0cJDQ0FYPLkyZZThdmfnX168NChQ/Tu3Rt/f38GDhwIwB9//MEDDzxAtWrV8PT0pGrVqjz33HO5RpfymuNkMpkYOXIkCxYsoHHjxnh6etKoUSOWLl1q1S6vOU41atTgnnvuYc2aNbRp0wYvLy9q1aqV5+nPHTt20KFDB7y9valSpQr/+te/+PrrrzVvSiQPbs4uQEQKz+nTp2nTpg3x8fE8+eST1K9fn1OnTjF37lySk5Px8PCwtM0OIq+88gpHjx7l/fffZ+TIkXz//feWNjNmzMDPz4+xY8fi5+fHb7/9xssvv0xiYiJvvfWW1WdfuHCBXr168be//Y1HHnmE8PBwu2rPyMjg5MmTBAcH5/l6QEAAzz33HC+//DJbt26lRYsWNzzm6NGjee+995g0aRKLFi2yuZbQ0FA+/fRThg8fTv/+/bn33nsBaNKkiVW9PXr04I477uDtt9/Gx8cHgDlz5pCcnMzw4cMpV64cGzdu5KOPPuLkyZPMmTPnhp+9Zs0a5s2bxzPPPIO/vz8ffvgh9913H8ePH6dcuXIFvvfgwYPcf//9DBs2jCFDhvDVV18xdOhQWrZsSaNGjQBzsO7UqRMmk4mJEyfi6+vL9OnT8fT0tPnrI1KmGCJSag0ePNhwcXExNm3alOu1rKwswzAM4+uvvzYAo2vXrpZ9hmEYzz33nOHq6mrEx8db9iUnJ+c6zlNPPWX4+PgYV65csezr0KGDARifffaZTXVWr17d6N69u3Hu3Dnj3Llzxs6dO41BgwYZgDFixAirtitXrjQAY86cOUZ8fLwRHBxs9O3b1/L6kCFDDF9fX6v3dOjQwWjUqJFhGIYxefJkAzC2bNliGIZhHDlyxACMt956q8Aaz507ZwDGK6+8kuu1IUOGGIDx4osv5notr6/ZlClTDJPJZBw7dsyy75VXXjGu/5UMGB4eHsbBgwct+7Zv324AxkcffWTZl/09PHLkiGVf9erVDcBYvXq1Zd/Zs2cNT09P4/nnn7fse/bZZw2TyWRs27bNsu/ChQtGSEhIrmOKiGHoVJ1IKZWVlcWCBQvo06cPrVq1yvX69aeFnnzySat9d955J5mZmRw7dsyyz9vb2/L40qVLnD9/njvvvJPk5GT27dtndTxPT08effRRm+v95ZdfCA0NJTQ0lIiICL799lseffTRXCNZOQUGBjJmzBgWLVrEtm3bbPqc0aNHF9pcp+HDh+fal/NrdvnyZc6fP0+7du0wDMOmmrt27Urt2rUtz5s0aUJAQACHDx++4XsbNmzInXfeaXkeGhpKvXr1rN67dOlSIiMjadasmWVfSEiI5VSjiFhTcBIppc6dO0diYiKNGze2qX21atWsnmefIrt48aJl3+7du+nfvz+BgYEEBAQQGhrKI488AkBCQoLV+ytXrmx1KjAhIYHY2FjLFhcXZ9W+bdu2LF++nKVLl/L2228TFBTExYsXrY6Rl9GjRxMUFGTzXKebCVu2cHNzo0qVKrn2Hz9+nKFDhxISEoKfnx+hoaF06NAByP01y8v13xcwf29yfl9u5b3Hjh2jTp06udrltU9EFJxE5CpXV9c89xuGAUB8fDwdOnRg+/btvPrqq/z4448sX76cadOmAeYRrpxyjrSAOeBUrFjRsmXPE8pWvnx5unbtSo8ePXj++ef57rvvWLBgAR988EGBdd/sqFNQUJBDR508PT1xcbH+lZqZmUm3bt346aefmDBhAgsWLGD58uXMmDEDyP01y8uNvi+F9V4RyZsmh4uUUqGhoQQEBLBr1y6HHG/VqlVcuHCBefPmcdddd1n2HzlyxKb3v/DCC5bRKSDfSd/Z7r77bjp06MAbb7zBU089ha+vb75tx4wZw/vvv8/kyZMJCgq6YS3ZYWvSpEkMGTLEpvpv5lL/nTt3cuDAAb755hsGDx5s2b98+XK7j1VYqlevzsGDB3Ptz2ufiGjESaTUcnFxISoqih9//DHP26nYO+qQPXqR831paWn83//9n03vb9iwIV27drVsLVu2vOF7JkyYwIULFwpcBBOuBaGFCxcSHR1tUz1jxowhKCiIV1991ab22VfJxcfH29Qe8v6aGYZxw1G0otSjRw/WrVtn9XWLi4tj5syZzitKpBjTiJNIKfbGG2/wyy+/0KFDB5588kkaNGhATEwMc+bMYc2aNTaNzmRr164dwcHBDBkyhFGjRmEymfj2228L9bRPr169aNy4Me+++y4jRozA3d0937bZSw1s3769wNGpbIGBgYwePdrm03Xe3t40bNiQ77//nttuu42QkBAaN25c4Byy+vXrU7t2bcaNG8epU6cICAjghx9+sGl+UlF54YUX+O677+jWrRvPPvusZTmCatWqERcXp0U1Ra6jESeRUqxy5cps2LCB+++/n5kzZzJq1Cj+85//0LFjR8sIiq3KlSvH4sWLqVixIv/85z95++236datG2+++WYhVW82btw4Tpw4ccMRkKCgIMaMGWPXsceMGUNgYKDN7adPn07lypV57rnnGDBgAHPnzi2wvbu7Oz/++CPNmjVjypQpTJ48mbp16+a5CKWzVK1alZUrV9KgQQPeeOMN3n//fYYMGcJjjz0GcNOrmYuUViZDswRFROQ6Y8aM4fPPPycpKSnfSeYiZZFGnEREyrjrb/9y4cIFvv32W+644w6FJpHraI6TiEgZFxkZSceOHWnQoAFnzpzh3//+N4mJibz00kvOLk2k2FFwEhEp43r37s3cuXP54osvMJlMtGjRgn//+99Wy06IiJnmOImIiIjYSHOcRERERGyk4CQiIiJiozI3xykrK4vTp0/j7++vhd1EREQEwzC4dOkSlSpVynXPyeuVueB0+vRpqlat6uwyREREpJg5ceIEVapUKbBNmQtO/v7+ALz44ouMHj3asnryhg0b+PPPP2ncuDHdunWztP/4449JT0/nscces6wwvGXLFlavXk39+vXp1auXpe1nn31GSkoKgwYNonz58gDs2LGDFStWULt2bfr27WtpO336dC5dusSAAQOoUKECAHv37mXp0qVUrVqV+++/39L2m2++IS4ujvvvv98S+g4ePMiPP/5IpUqVeOihhyxtZ82axZkzZ+jXrx+1atUC4OjRo8yfP5/Q0FCrm6z+73//49SpU9x9993cdtttAJw6dYr//e9/BAUF8eijj1razp8/n6NHj9K9e3caNWoEwNmzZ5k5cya+vr48+eSTlraLFy/mr7/+olOnTjRr1gww3/vqm2++wdPTk2eeecbSdtmyZezZs4c777yTVq1aAXDp0iWmT5+Oi4sLo0ePtrRdsWIFO3bs4PbbbycyMhKAK1eu8OmnnwIwatQoy5ozq1evZsuWLbRs2dJyZVBmZiYffvghAMOHD7esiLxu3TrWr19PkyZN6NKli+XzPvjgA7Kysnj88cctPzebN2/mjz/+oGHDhvTo0cPS9v/+7/9ITU1lyJAhhISEABAdHc3KlSupW7cu99xzj6XtF198weXLlxk4cCBhYWEA7N69m19++YUaNWrQv39/S9uvv/6a+Ph4HnzwQSpXrgzAgQMH+Omnn6hcuTIPPvigpe13333HuXPn6N+/PzVq1ADg8OHDLFy4kPDwcB5++GFL2++//57Tp0/Tp08f6tSpA5h/YcydO5eQkBCrG9/OnTuXEydO0LNnTxo0aABAbGws//3vf/H39+fxxx+3tF20aBGHDh2iS5cuNGnSBIDz58/z7bff4u3tzdNPP21pu2TJEvbt28ddd91luW9dQkICX331Fe7u7owcOdLSdvny5ezatYt27drRtm1bAJKTk/n8888BeO655yxtV65cSXR0NG3atKF9+/aA+Z56n3zyCQAjRozAw8MDgLVr17Jx40aaNWtGp06dLMd47733AHjqqaf0O0K/I/Q7ogz8jrhw4QK1atWyfB8LUuaCU/bpOS8vLwICAiy/FH19ffHy8sLHx4eAgABLey8vL1xdXQkICLDs9/Pzy7etYRj4+/vfsK23tzfp6ek2t/Xy8sqzrbe39w3b+vv759nWx8cHLy8v/Pz8LPsTExNtbpuSkmJz24yMDLy8vPD09Myzra+vr2W/yWTCy8sLFxcXq7bZ36OcbT08PCy/3AICAiy/FPNqm5mZadU2+3FebbO/n1lZWQQEBFj+MhX0c2IymWz6OfH29iYzM9Pm7/2VK1du+uckv+99dtuc3yN7fk4uX75sc9vU1FS8vLwsf+cKamsYBl5eXri7u9/w58TNzc3q+1nQz0laWppV2+zgVND3Prutfkfod4R+R5SN3xGATVN4ytxyBImJiQQGBnLu3DnKlStn+SJlZmaSmZmJi4sLbm7X8mT2F9Pd3d2hbdPT0zEMAzc3N8v51KysLDIyMjCZTFY3My0ObTMyMsjKysLV1dXyi8eetoZhkJ6eDmD5R8vRbfP6utvT1pnf+9L8c1IU3/tb/Tkpqu+9fkc4/vup3xHO/96Xht8RCQkJBAUFkZCQYBWy8lJmg5MtXxwREREp/ezJBlqOQERERMRGZW6Ok4iI3LzMzEzL6Q+RksLd3d1hN6xWcBIRkRsyDIPY2Fji4+OdXYrITQkKCqJChQq3vIajgpOIiNxQdmgKCwvDx8dHCwhLiWEYBsnJyZw9exaAihUr3tLxik1wmjp1KhMnTmT06NG8//77+babM2cOL730EkePHqVu3bpMmzaN3r17F12hIiJlTGZmpiU0lStXztnliNjN29sbMK8tFhYWdkun7YrF5PBNmzbx+eefWxbDys+ff/7JgAEDGDZsGNu2bSMqKoqoqCh27dpVRJWKiJQ92XOaste0EimJsn9+b3WOntODU1JSEgMHDuTLL78kODi4wLYffPABPXv2ZPz48TRo0IDXXnuNFi1a8PHHHxdRtSIiZZdOz0lJ5qifX6cHpxEjRnD33XfTtWvXG7Zdt25drnY9evRg3bp1+b4nNTWVxMREq63QpSYX/meIiIhIkXNqcJo9ezZbt25lypQpNrWPjY0lPDzcal94eDixsbH5vmfKlCkEBgZatkK9we/hLWS81otL7z9VeJ8hIiJOUaNGjQLn4ErZ4LTgdOLECUaPHs3MmTMt95IpDBMnTiQhIcGynThxotA+68+D5XGL3YPPsTWQcLbQPkdERGzTsWNHxowZ45Bjbdq0yepmxVI2OS04bdmyhbNnz9KiRQvc3Nxwc3Pj999/58MPP8TNzY3MzMxc76lQoQJnzpyx2nfmzBnLncPzkn3DyJxbYWnQtjpbU5rjasri3K+LC+1zRETEMQzDICMjw6a2oaGhmiAvzgtOXbp0YefOnURHR1u2Vq1aMXDgQKKjo/O8VDAyMpIVK1ZY7Vu+fDmRkZFFVXaBggNhd1AUAJkbFji1FhGRsm7o0KH8/vvvfPDBB5hMJkwmEzNmzMBkMrFkyRJatmyJp6cna9as4dChQ/Tr14/w8HD8/Pxo3bo1v/76q9Xxrj9VZzKZmD59Ov3798fHx4e6deuyaNGiIu6lFDWnBSd/f38aN25stfn6+lKuXDkaN24MwODBg5k4caLlPaNHj2bp0qW888477Nu3j0mTJrF582ZGjhzprG7kEtLpHjIMVypc3o5x5oizyxERcTjDgOQU52z23Jb+gw8+IDIykieeeIKYmBhiYmIs81xffPFFpk6dyt69e2nSpAlJSUn07t2bFStWsG3bNnr27EmfPn04fvx4gZ8xefJkHnzwQXbs2EHv3r0ZOHAgcXFxt/LllWKu2CyAmZfjx4/j4nIt27Vr145Zs2bxz3/+k7///e/UrVuXBQsWWIJWcXBX5/Ks/eFOOviu4twvCwgb9JyzSxIRcaiUK9DASesO7/0ZfLxtaxsYGIiHhwc+Pj6WKR379u0D4NVXX6Vbt26WtiEhITRt2tTy/LXXXmP+/PksWrSowP+cDx06lAEDBgDwxhtv8OGHH7Jx40Z69uxpb9ekhChWwWnVqlUFPgd44IEHeOCBB4qmoJvg7wsHw6PokLQKt60L4JExoLVPRESKlVatWlk9T0pKYtKkSfz000/ExMSQkZFBSkrKDUecci7c7OvrS0BAgOXWHlI6FavgVFpU7tad5B+8CUk7inF0O6aazZxdkoiIw3h7mUd+nPXZjuDr62v1fNy4cSxfvpy3336bOnXq4O3tzf33309aWlqBx3F3d7d6bjKZyMrKckyRUiwpOBWCDnf48ut/utPHf6H5dN1TzZxdkoiIw5hMtp8uczYPD488r9K+3tq1axk6dCj9+/cHzCNQR48eLeTqpCRy+srhpZG3F5yq2g8An90/QqZtl7qKiIhj1ahRgw0bNnD06FHOnz+f72hQ3bp1mTdvHtHR0Wzfvp2HH35YI0eSJwWnQlKn511cyAjBL/M8mXvXOrscEZEyady4cbi6utKwYUNCQ0PznbP07rvvEhwcTLt27ejTpw89evSgRYsWRVytlAQmw7Dn4s6SLzExkcDAQBISEgp1MczUNJj/+D/5m/+3nKtzL6HPvVdonyUiUpiuXLnCkSNHqFmzZqHe6UGkMBX0c2xPNtCIUyHx9IBzdaIACDi0DNJSnFuQiIiI3DIFp0LUtGdLjqdVxdO4TGb0cmeXIyIiIrdIwakQtWtpYtkV8yTx+BULnFuMiIiI3DIFp0Lk5gqXG0cBEHTyd0jSMvwiIiIlmYJTIbu9R112pTTClQzSN/3k7HJERETkFig4FbLWEfBrRhQASb8vdG4xIiIicksUnAqZqysYLfqSZZgIPrcJLpx0dkkiIiJykxScikDH7hVYlxwJQNo6jTqJiIiUVApORaBZA/jdiALgytoFULbWHBURESk1FJyKgMkE3pG9uJLlSUDiATi119kliYiIDWrUqMH777/v7DKkGFFwKiI9ugbwW1JnANL+XODcYkREROSmKDgVkQa1Yb17FAAZGxaB7rotIiJS4ig4FRGTCcrf0YmEzAB8rsTAwQ3OLklEpFT74osvqFSpElnX/Ue1X79+PPbYYxw6dIh+/foRHh6On58frVu35tdff3VStVJSKDgVoV5dPPk5sTcAqWsXOLcYEZGbZRiQmuyczY6Lax544AEuXLjAypUrLfvi4uJYunQpAwcOJCkpid69e7NixQq2bdtGz5496dOnD8ePHy+Mr5qUEm7OLqAsqVsdvvSNYgCzMUX/DOmvgruns8sSEbFPWgqMbeCcz353L3j62NQ0ODiYXr16MWvWLLp06QLA3LlzKV++PJ06dcLFxYWmTZta2r/22mvMnz+fRYsWMXLkyEIpX0o+jTgVsRod23I6vSIeGYmwe+WN3yAiIjdt4MCB/PDDD6SmpgIwc+ZM/va3v+Hi4kJSUhLjxo2jQYMGBAUF4efnx969ezXiJAXSiFMRu6ezC4sW9eXp8p9zZe0CvJr1dHZJIiL28fA2j/w467Pt0KdPHwzD4KeffqJ169b88ccfvPfeewCMGzeO5cuX8/bbb1OnTh28vb25//77SUtLK4zKpZRQcCpi1SrCvnJRwOe47/sNUhLBO8DZZYmI2M5ksvl0mbN5eXlx7733MnPmTA4ePEi9evVo0aIFAGvXrmXo0KH0798fgKSkJI4ePerEaqUk0Kk6J4jo1ID9V27DNSsVti1xdjkiIqXawIED+emnn/jqq68YOHCgZX/dunWZN28e0dHRbN++nYcffjjXFXgi11NwcoK7O5pYmBgFXL0Fi4iIFJrOnTsTEhLC/v37efjhhy373333XYKDg2nXrh19+vShR48eltEokfzoVJ0TVAiFoxX7QeabeB5dB/GxEFTB2WWJiJRKLi4unD59Otf+GjVq8Ntvv1ntGzFihNVznbqT62nEyUkiu1RhY3JrTBiweZGzyxEREREbKDg5Sa+7YNHV03WpunediIhIiaDg5CTlg+FCzd6kG254ntkNsX85uyQRERG5AQUnJ+rUJYTfkzqYn2xa4NRaRERE5MYUnJyox53wY1IUAGnrFtp1DyYREREpegpOThToB2n1u5GU6YtHwgk4ssXZJYmIiEgBFJycrEdnb5Zd6gGAsXGBc4sRERGRAik4OVm39vBzchQAmZt+gsx05xYkIiIi+VJwcjJfb/Bu0p5zGeVxuxIHe1c7uyQRERHJh4JTMXB3Fzd+TOgD6HSdiEhJNXToUKKiopz2+YMGDeKNN96wPK9Rowbvv/++0+pxlBv1Iy0tjRo1arB58+YiqUfBqRjo1BZ+SY0CIGv7crhy2bkFiYhIibJ9+3Z+/vlnRo0a5exSipyHhwfjxo1jwoQJRfJ5Ck7FgJcnVGzVlCOpNXDNSIEdy5xdkohIqZOWlubsEgrNRx99xAMPPICfn5+zS3GKgQMHsmbNGnbv3l3on+XU4PTpp5/SpEkTAgICCAgIIDIykiVLluTbfsaMGZhMJqvNy8urCCsuPH26mFhw9RYsOl0nInLrOnbsyMiRIxkzZgzly5enRw/zFczvvvsuERER+Pr6UrVqVZ555hmSkpIs75sxYwZBQUEsW7aMBg0a4OfnR8+ePYmJibG0yczMZOzYsQQFBVGuXDleeOEFjOvW4ktNTWXUqFGEhYXh5eXFHXfcwaZNmyyvr1q1CpPJxLJly2jevDne3t507tyZs2fPsmTJEho0aEBAQAAPP/wwycnJ+fYzMzOTuXPn0qdPnwK/HiaTienTp9O/f398fHyoW7cuixblf6/Uv//977Rt2zbX/qZNm/Lqq68CkJWVxauvvkqVKlXw9PSkWbNmLF261Kr9yZMnGTBgACEhIfj6+tKqVSs2bNgAwKFDh+jXrx/h4eH4+fnRunVrfv3111yfeenSJQYMGICvry+VK1fmk08+sXo9ODiY9u3bM3v27AK/Bo7g1OBUpUoVpk6dypYtW9i8eTOdO3emX79+BSbGgIAAYmJiLNuxY8eKsOLCc0dLWJERZX6ybw1cOu/UekREbiQtLY20tDSrwJCZmUlaWhoZGRkOb3szvvnmGzw8PFi7di2fffYZAC4uLnz44Yfs3r2bb775ht9++40XXnjB6n3Jycm8/fbbfPvtt6xevZrjx48zbtw4y+vvvPMOM2bM4KuvvmLNmjXExcUxf/58q2O88MIL/PDDD3zzzTds3bqVOnXq0KNHD+Li4qzaTZo0iY8//pg///yTEydO8OCDD/L+++8za9YsfvrpJ3755Rc++uijfPu4Y8cOEhISaNWq1Q2/HpMnT+bBBx9kx44d9O7dm4EDB+aqJ9vAgQPZuHEjhw4dsuzbvXs3O3bs4OGHHwbggw8+4J133uHtt99mx44d9OjRg759+/LXX+bbiCUlJdGhQwdOnTrFokWL2L59Oy+88AJZWVmW13v37s2KFSvYtm0bPXv2pE+fPhw/ftyqlrfeeoumTZuybds2XnzxRUaPHs3y5cut2rRp04Y//vjjhl+DW2YUM8HBwcb06dPzfO3rr782AgMDb+n4CQkJBmAkJCTc0nEKwwtvGca2R/sYxjPVDGPl184uR0TEMAzDSElJMfbs2WOkpKRY7Z86daoxdepU4/Lly5Z9a9euNaZOnWr8/PPPVm3feecdY+rUqUZ8fLxl38aNG42pU6caixYtsmr7wQcfGFOnTjXOnj1r2bdt2za76+7QoYPRvHnzG7abM2eOUa5cOcvzr7/+2gCMgwcPWvZ98sknRnh4uOV5xYoVjTfffNPyPD093ahSpYrRr18/wzAMIykpyXB3dzdmzpxpaZOWlmZUqlTJ8r6VK1cagPHrr79a2kyZMsUAjEOHDln2PfXUU0aPHj3yrX/+/PmGq6urkZWVZbW/evXqxnvvvWd5Dhj//Oc/Lc+TkpIMwFiyZEm+x27atKnx6quvWp5PnDjRaNu2reV5pUqVjNdff93qPa1btzaeeeYZwzAM4/PPPzf8/f2NCxcu5PsZ12vUqJHx0UcfWfWjZ8+eVm0eeugho1evXlb7PvjgA6NGjRr5Hje/n2PDsC8bFJs5TpmZmcyePZvLly8TGRmZb7ukpCSqV69O1apVbzg6Beah0sTERKutuOrTCRYkRAGQpdN1IiK3rGXLlrn2/frrr3Tp0oXKlSvj7+/PoEGDuHDhgtXpMB8fH2rXrm15XrFiRc6ePQtAQkICMTExVqex3NzcrEZ8Dh06RHp6Ou3bt7fsc3d3p02bNuzdu9eqniZNmlgeh4eH4+PjQ61ataz2ZX92XlJSUvD09MRkMhX4tbj+s3x9fQkICCjw2AMHDmTWrFkAGIbBf//7XwYOHAhAYmIip0+ftuojQPv27S19jI6Opnnz5oSEhOR5/KSkJMaNG0eDBg0ICgrCz8+PvXv35hpxuj4XREZG5vo6ent7F3hK01HcCv0TbmDnzp1ERkZy5coV/Pz8mD9/Pg0bNsyzbb169fjqq69o0qQJCQkJvP3227Rr147du3dTpUqVPN8zZcoUJk+eXJhdcJjIZvCa6R4yjddwPbYNzh2D0OrOLktEJE/PPfccYA4E2dq2bUurVq1wcbH+f/nIkSNztW3RogVNmzbN1fbpp5/O1TYiIuKmavT19bV6fvToUe655x6GDx/O66+/TkhICGvWrGHYsGGkpaXh4+OT67PBPD/IKKT7ieb8LJPJlOdnZ5/aykv58uVJTk4mLS0NDw8Pmz/LlmMPGDCACRMmsHXrVlJSUjhx4gQPPfRQgZ+Rk7e3d4Gvjxs3juXLl/P2229Tp04dvL29uf/++29qIn9cXByhoaF2v89eTh9xqlevHtHR0WzYsIHhw4czZMgQ9uzZk2fbyMhIBg8eTLNmzejQoQPz5s0jNDSUzz//PN/jT5w4kYSEBMt24sSJwurKLXN1hbZ3hbHm8h3mHZsXOLUeEZGCeHh44OHhYTXS4erqioeHB25ubg5v6whbtmwhKyuLd955h9tvv53bbruN06dP23WMwMBAKlasaJngDJCRkcGWLdfuN1q7dm3L3Kps6enpbNq0Kd/BgZvVrFkzgHz/7bwVVapUoUOHDsycOZOZM2fSrVs3wsLCAPOc40qVKln1EWDt2rWWPjZp0oTo6Oh851GtXbuWoUOH0r9/fyIiIqhQoQJHjx7N1W79+vW5njdo0MBq365du2jevPnNdtVmTg9OHh4e1KlTh5YtWzJlyhSaNm3KBx98YNN73d3dad68OQcPHsy3jaenp+WqveytOLM+XbcQCul/OCIiZVGdOnVIT0/no48+4vDhw3z77beWSeP2GD16NFOnTmXBggXs27ePZ555hvj4eMvrvr6+DB8+nPHjx7N06VL27NnDE088QXJyMsOGDXNgjyA0NJQWLVqwZs0ahx4328CBA5k9ezZz5syxnKbLNn78eKZNm8b333/P/v37efHFF4mOjmb06NGAecSqQoUKREVFsXbtWg4fPswPP/zAunXrAKhbty7z5s0jOjqa7du38/DDD+c5ArZ27VrefPNNDhw4wCeffMKcOXMsn5Htjz/+oHv37oXyNcjJ6cHpellZWaSmptrUNjMzk507d1KxYsVCrqrotGwEO7x6kJLlhcvZQ3Bil7NLEhEpNZo2bcq7777LtGnTaNy4MTNnzmTKlCl2H+f5559n0KBBDBkyhMjISPz9/enfv79Vm6lTp3LfffcxaNAgWrRowcGDB1m2bBnBwcGO6o7F448/zsyZMx1+XID777/fMgfs+pXRR40axdixY3n++eeJiIhg6dKlLFq0iLp16wLmwZFffvmFsLAwevfuTUREBFOnTrWMIL777rsEBwfTrl07+vTpQ48ePWjRokWuGp5//nk2b95M8+bN+de//sW7775rWV4CYN26dSQkJHD//fcXytcgJ5NxCydtU1NT8fT0vOkPnzhxIr169aJatWpcunSJWbNmMW3aNJYtW0a3bt0YPHgwlStXtvxQv/rqq9x+++3UqVOH+Ph43nrrLRYsWMCWLVtsHvpMTEwkMDCQhISEYjv69K9PocnakfQN/BE6D4P7XnZ2SSJShl25coUjR45Qs2bNUrN2XmmTkpJCvXr1+P777wu8wKq0euihh2jatCl///vf821T0M+xPdnArhGnJUuWMGTIEGrVqoW7uzs+Pj4EBATQoUMHXn/9dbvPE589e5bBgwdTr149unTpwqZNmyyhCeD48eNWC45dvHiRJ554ggYNGtC7d28SExP5888/HX6+2Nn6ds5xum7zj5B1c+uXiIhI2eDt7c1//vMfzp8ve2sApqWlERERYblYobDZNOI0f/58JkyYwKVLl+jduzdt2rShUqVKeHt7ExcXx65du/jjjz9Yt24dQ4cO5bXXXiuSme03oySMOBkGdHkkjR/8WhPsFg/PzoT6dzi7LBEpozTiJKWBo0acbFqO4M033+S9996jV69euS4bBXjwwQcBOHXqFB999BHfffddkSW/0shkgl6dPVj86z0MCvkONi1QcBIRESkGbApO2bPfb6Ry5cpMnTr1lgoSs76d4e8/RDEo5DuMbUswPfQv8ND/9ERERJzJrjlO6enp1K5dO9dqneJ49WrCpfCWnEirgik1CXatcHZJIiIiZZ5dwcnd3Z0rV64UVi1ynXs6u7AwsZ/5yaYFTq1FRKSgFaZFijtH/fzafcuVESNGMG3aNKZPn55rtVdxrD6d4ImZUYws/wnG7pWYLseDb5CzyxKRMsbDwwMXFxdOnz5NaGhorlW9RYozwzBIS0vj3LlzuLi43PC2NDdid/LZtGkTK1as4JdffiEiIiLXfYDmzZt3SwXJNTWrgGe129h9pSGNvPbAtp/hjoedXZaIlDEuLi7UrFmTmJgYu5edESkufHx8qFatWp4XudnD7uAUFBTEfffdd0sfKrbr0xkWzu1nDk6bFig4iYhTeHh4UK1aNTIyMsjM1NpyUrK4urri5ubmkJHSW1o5vCQqCes45XQyFu4fdJo/67bDxWTAa39CSGVnlyUiIlJqFNrK4VL0qlSASvUqsSG5rXnH5kXOLUhERKQMu6nZ3XPnzuV///sfx48fJy0tzeq1rVu3OqQwuaZPZ1jwbRSRvuvNp+u6D3d2SSIiImWS3SNOH374IY8++ijh4eFs27aNNm3aUK5cOQ4fPkyvXr0Ko8Yy7+6OsDSpN6lZHnB6H5za5+ySREREyiS7g9P//d//8cUXX/DRRx/h4eHBCy+8wPLlyxk1ahQJCQmFUWOZFxYCDSMCWZnUybxDazqJiIg4hd3B6fjx47Rr1w4w34350qVLAAwaNIj//ve/jq1OLPp2hoUJVxfD3LwQtBCdiIhIkbM7OFWoUIG4uDgAqlWrxvr16wE4cuQIZewCvSLV805YldKFxEx/uHgaDm1ydkkiIiJljt3BqXPnzixaZL6y69FHH+W5556jW7duPPTQQ/Tv39/hBYpZcCC0aenF0sSe5h06XSciIlLk7F7HKSsri6ysLMvtVmbPns2ff/5J3bp1eeqpp255KfPCVtLWccrph2Xww0drmFV9IIZPIKYpm8GteH+9RUREijt7soEWwCxBEpOgzb2Z/F7jdsLdz8KTX0LT7s4uS0REpESzJxvc1DpO8fHxbNy4kbNnz+a62/DgwYNv5pBigwA/uOt2VxYd6MsT5aabT9cpOImIiBQZu4PTjz/+yMCBA0lKSiIgIMDqvi8mk0nBqZD17QSfboriiXLTMXb9iinlEnj7O7ssERGRMsHuyeHPP/88jz32GElJScTHx3Px4kXLln21nRSezrfDYRpzMLU2pvRUiF7q7JJERETKDLuD06lTpxg1ahQ+Pj6FUY/cgI83dG1nYkFClHnH5gXOLEdERKRMsTs49ejRg82bNxdGLWKjPp2wBCdj/5+QcMa5BYmIiJQRNs1xyl63CeDuu+9m/Pjx7Nmzh4iICNzd3a3a9u3b17EVSi4d2kCCRzW2JLegpc9W2LIYOg9zdlkiIiKlnk3LEbi42DYwZTKZyMzMvOWiClNJXo4gp3HTwHvDN7xW8WWo1gQm/OjskkREREoke7KBTYkoe9HLG23FPTSVJn06weLEe8gwXOH4Djhz2NkliYiIlHp2z3GS4qFdC8CvHKuT7jLv0C1YRERECp3dwWnUqFF8+OGHufZ//PHHjBkzxhE1iQ3c3aB3h2uTxNm0AMrWIvAiIiJFzu7g9MMPP9C+fftc+9u1a8fcuXMdUpTYpk8n+OVSd5KzfOD8MTga7eySRERESjW7g9OFCxcIDAzMtT8gIIDz5887pCixTesI8A/xYVni1duu6HSdiIhIobI7ONWpU4elS3OvVr1kyRJq1arlkKLENq6ucE9HWJAYZd6xdTFkZjizJBERkVLN7nvVjR07lpEjR3Lu3Dk6d+4MwIoVK3jnnXd4//33HV2f3ECfznD/D3dyIbMc5S6dh31roFFHZ5clIiJSKtkdnB577DFSU1N5/fXXee211wCoUaMGn376qW7w6wTNG0DFcDd+TLiHoSHfmE/XKTiJiIgUCpsWwARITk7OdX+6c+fO4e3tjZ+fX6EUVxhKywKYOU39AjYs2ML8mveChw9M3QKeupegiIiILRy+ACZA+fLlueeee/jiiy+IjY0FIDQ0tESFptKqT2fYmtKC4+nVIC0Zdi53dkkiIiKlks3Bad++ffTo0YP//e9/1KhRg7Zt2/L666+zc+fOwqxPbNCwNtSuamJBfD/zDl1dJyIiUihsDk7VqlXj2Wef5ddff+XMmTOMGTOGnTt3cuedd1KrVi3GjBnDb7/9ptuuOIHJBPd0yrEY5p7VkBTn1JpERERKo5u65UpgYCADBgxg9uzZnDt3js8++4zMzEweffRRQkNDmTlzpqPrlBu4pxMcSqvDriuNISvDvDSBiIiIONQt36vO3d2d7t2789FHH3Hs2DFWrFjBbbfd5ojaxA631YAGtWB+fJR5h07XiYiIOJzdwWnp0qWsWbPG8vyTTz6hWbNmPPzww1y8eJHmzZvTunVrm4716aef0qRJEwICAggICCAyMpIlS5YU+J45c+ZQv359vLy8iIiI4Oeff7a3C6VWn86wKLEvWZjg8BY4f9zZJYmIiJQqdgen8ePHk5iYCMDOnTt5/vnn6d27N0eOHGHs2LF2HatKlSpMnTqVLVu2sHnzZjp37ky/fv3YvXt3nu3//PNPBgwYwLBhw9i2bRtRUVFERUWxa9cue7tRKvXpBGczwvnzcjvzjs0LnVuQiIhIKWPzOk7Z/Pz82LVrFzVq1GDSpEns2rWLuXPnsnXrVnr37m1ZquBmhYSE8NZbbzFs2LBcrz300ENcvnyZxYuvzd+5/fbbadasGZ999plNxy+N6zjl1G841I35H29XHg8V6sA/fzXPHhcREZE8Fco6Ttk8PDxITk4G4Ndff6V7d/MNZkNCQiwjUTcjMzOT2bNnc/nyZSIjI/Nss27dOrp27Wq1r0ePHqxbt+6mP7e06dMZll7qSRqeEHsQTuY9eiciIiL2szs43XHHHYwdO5bXXnuNjRs3cvfddwNw4MABqlSpYncBO3fuxM/PD09PT55++mnmz59Pw4YN82wbGxtLeHi41b7w8PACR7lSU1NJTEy02kqzezpCkhHA8sQu5h2aJC4iIuIwdgenjz/+GDc3N+bOncunn35K5cqVAViyZAk9e/a0u4B69eoRHR3Nhg0bGD58OEOGDGHPnj12Hyc/U6ZMITAw0LJVrVrVYccujiqEQuuIHGs6bV4EWVpbS0RExBHsnuNU2Lp27Urt2rX5/PPPc71WrVo1xo4dy5gxYyz7XnnlFRYsWMD27dvzPF5qaiqpqamW54mJiVStWrXUznEC+M8CeO3DVLbVb4WfKRFGzYJ67Z1dloiISLFUqHOcXF1dOXv2bK79Fy5cwNXV1d7D5ZKVlWUVdHKKjIxkxYoVVvuWL1+e75woAE9PT8tyB9lbade7A2SYPFkUbz6NqtN1IiIijmF3cMpvgCo1NRUPDw+7jjVx4kRWr17N0aNH2blzJxMnTmTVqlUMHDgQgMGDBzNx4kRL+9GjR7N06VLeeecd9u3bx6RJk9i8eTMjR460txulWvlgaN8ix+m6bUsg/YpTaxIRESkN3Gxt+OGHHwJgMpmYPn06fn5+ltcyMzNZvXo19evXt+vDz549y+DBg4mJiSEwMJAmTZqwbNkyunXrBsDx48dxcbmW7dq1a8esWbP45z//yd///nfq1q3LggULaNy4sV2fWxb06QQTNrfhrFGJsCunYddv0Ly3s8sSEREp0Wye41SzZk0Ajh07RpUqVaxOy3l4eFCjRg1effVV2rZtWziVOkhpX8cpW8IlaHkvPB8yheHlP4OmPeDJL5xdloiISLFjTzawecTpyJEjAHTq1Il58+YRHBx8a1VKoQr0hw5tYMGWKHNw2r0SkhPAJ9DZpYmIiJRYds9xWrlypUJTCdGnE+xLbcDhzHqQkQbbdF8/ERGRW2HziFNOJ0+eZNGiRRw/fpy0tDSr1959912HFCa3rms78PSA/52P4sXwaeZ717Uf4OyyRERESiy7g9OKFSvo27cvtWrVYt++fTRu3JijR49iGAYtWrQojBrlJvn5QJdIWPRnX3Nw+ms9xMdCUAVnlyYiIlIi2X2qbuLEiYwbN46dO3fi5eXFDz/8wIkTJ+jQoQMPPPBAYdQot6BPJziVXoXo9DZgGOZRJxEREbkpdgenvXv3MnjwYADc3NxISUnBz8+PV199lWnTpjm8QLk1nW8HX2/4/lyUeYcWwxQREblpdgcnX19fy7ymihUrcujQIctr58+fd1xl4hBentD9Dvgp8W4ycYeTeyDmgLPLEhERKZHsDk633347a9asAaB37948//zzvP766zz22GPcfvvtDi9Qbl2fTpCQFcSaKx3NOzTqJCIiclPsDk7vvvuuZZHLyZMn06VLF77//ntq1KjBv//9b4cXKLfuzlbmdZ2+Px9l3rF5oXm+k4iIiNjF7qvqatWqZXns6+vLZ5995tCCxPE83KHnnbBwSRdSTb54XjgJhzdD7dbOLk1ERKREsXvEKVtaWhonT57k+PHjVpsUT307wxXDm2VJPc07dLpORETEbnYHpwMHDnDnnXfi7e1N9erVqVmzJjVr1qRGjRqW+9lJ8XN7MygfbF4ME4CtP0FmujNLEhERKXHsPlX36KOP4ubmxuLFi6lYsSImk6kw6hIHc3OF3h1g5oJ2JLqEEnD5HOxZDRFdnF2aiIhIiWF3cIqOjmbLli3Ur1+/MOqRQtS3M/xngRsL4/swKOAr8+k6BScRERGb2X2qrmHDhlqvqYRq2QgqhuY4XbfjF7iS5NSaREREShK7g9O0adN44YUXWLVqFRcuXCAxMdFqk+LLxQXu6QQ7rjThrGtNSL8C239xdlkiIiIlht3BqWvXrqxfv54uXboQFhZGcHAwwcHBBAUFERwcXBg1igP17Qxg0i1YREREboLdc5xWrlxZGHVIEYm4DapVgjnnong25D3Y9wcknoOAUGeXJiIiUuzZHZw6dOhQGHVIETGZzLdg+WRmDQ67NqNWZjRsWQydHnV2aSIiIsWeTafq7F3Y8tSpUzdVjBQN8+k6mBkTZX6g03UiIiI2sSk4tW7dmqeeeopNmzbl2yYhIYEvv/ySxo0b88MPPzisQHG8ejWhbnVYcPEeskyucCwazh5xdlkiIiLFnk2n6vbs2cPrr79Ot27d8PLyomXLllSqVAkvLy8uXrzInj172L17Ny1atODNN9+kd+/ehV233AKTCfp0hne/DmWXyx00yfwdNi2Eu8c4uzQREZFizWQYhmFr45SUFH766SfWrFnDsWPHSElJoXz58jRv3pwePXrQuHHjwqzVIRITEwkMDCQhIYGAgABnl+M0h09Ap8FwX9A83q30HITVhJdXmlOViIhIGWJPNrArOJUGCk7X3P0kHDmUxI6GLXHLugIvLILqTZ1dloiISJGyJxvYvY6TlB59OsHlLD820N28Q5PERURECqTgVIbd3cn857+PRZkfbP4RMjOcVo+IiEhxp+BUhlWtAC0awe+X7uKKezBcOgcH/nR2WSIiIsWWglMZ17cTZODOyvS7zTt0uk5ERCRfdgeny5cvF0Yd4iS9O5ovpJt+JMq8I3oppKU4syQREZFiy+7gFB4ezmOPPcaaNWsKox4pYuHl4PZmsDmlFYmeVSD1Muz81dlliYiIFEt2B6fvvvuOuLg4OnfuzG233cbUqVM5ffp0YdQmRaRvJwATPyf3M+/Q6ToREZE82R2coqKiWLBgAadOneLpp59m1qxZVK9enXvuuYd58+aRkaGrskqanneBm2uO03W7V0HSRWeWJCIiUizd9OTw0NBQxo4dy44dO3j33Xf59ddfuf/++6lUqRIvv/wyycnJjqxTClFIILRvCX+l3cZZn4aQlQHbfnJ2WSIiIsXOTQenM2fO8Oabb9KwYUNefPFF7r//flasWME777zDvHnziIqKcmCZUtj6Xl3Tad7FKPODTQudVouIiEhxZfctV+bNm8fXX3/NsmXLaNiwIY8//jiPPPIIQUFBljaHDh2iQYMGpKWlObreW6ZbruQtMQla3gvBRiwbbrsdEwa8uhbKVXF2aSIiIoWqUG+58uijj1KpUiXWrl1LdHQ0I0eOtApNAJUqVeIf//iHvYcWJwrwg05t4UxGBY773W7euWWRc4sSEREpZuwOTjExMXz++ee0bt063zbe3t688sort1SYFL0+V0/XzT6rq+tERETy4mbvG3x8fMjMzGT+/Pns3bsXgAYNGhAVFYWbm92Hk2KkSyR4e8F3x3szvuHLuJzeD6f2QuUGzi5NRESkWLB7xGn37t3UrVuXIUOGMH/+fObPn8/QoUOpW7cuu3btsutYU6ZMoXXr1vj7+xMWFkZUVBT79+8v8D0zZszAZDJZbV5eXvZ2Q/Lg4w3d2kFiViAHfK8OP2nUSURExMLu4PT444/TuHFjTp48ydatW9m6dSsnTpygSZMmPPnkk3Yd6/fff2fEiBGsX7+e5cuXk56eTvfu3W94W5eAgABiYmIs27Fjx+zthuQj+3TdjJNR5gebF0FWltPqERERKU7sPrcWHR3N5s2bCQ4OtuwLDg7m9ddfL3DeU16WLl1q9XzGjBmEhYWxZcsW7rrrrnzfZzKZqFChgn2Fi006tAF/X5h3ujP/Cg3A7eJpOLQR6t7u7NJERESczu4Rp9tuu40zZ87k2n/27Fnq1KlzS8UkJCQAEBISUmC7pKQkqlevTtWqVenXrx+7d+++pc+Vazw9oMcdkGp4sd2rp3mnTteJiIgANxGcpkyZwqhRo5g7dy4nT57k5MmTzJ07lzFjxjBt2jQSExMtmz2ysrIYM2YM7du3p3Hjxvm2q1evHl999RULFy7ku+++Iysri3bt2nHy5Mk826emplrVZG9dZVGfzuY/vzgcZX6w9SdIT3VaPSIiIsWF3Qtgurhcy1omkwmA7EPkfG4ymcjMzLT5uMOHD2fJkiWsWbOGKlVsX3QxPT2dBg0aMGDAAF577bVcr0+aNInJkyfn2q8FMPOXngGt74OExEz2tYrEM/kMPPkFNO3h7NJEREQczp4FMO2e47Ry5cqbLiw/I0eOZPHixaxevdqu0ATg7u5O8+bNOXjwYJ6vT5w4kbFjx1qeJyYmUrVq1Vuqt7Rzd4PeHWDmj66sd+tLB740n65TcBIRkTLO7uDUoUMHh324YRg8++yzzJ8/n1WrVlGzZk27j5GZmcnOnTvp3bt3nq97enri6el5q6WWOX06w8wf4eP9UXSo+CXsXAEpieCtUToRESm7bmrFyvj4eP79739bFsBs1KgRjz32GIGBgXYdZ8SIEcyaNYuFCxfi7+9PbGwsAIGBgXh7ewMwePBgKleuzJQpUwB49dVXuf3226lTpw7x8fG89dZbHDt2jMcff/xmuiL5aBMBYeVg44VGXK5fB9+EgxC9FCIfdHZpIiIiTmP35PDNmzdTu3Zt3nvvPeLi4oiLi+Pdd9+ldu3abN261a5jffrppyQkJNCxY0cqVqxo2b7//ntLm+PHjxMTE2N5fvHiRZ544gkaNGhA7969SUxM5M8//6Rhw4b2dkUK4OoKd3cAMLEyK8q8U1fXiYhIGWf35PA777yTOnXq8OWXX1pusZKRkcHjjz/O4cOHWb16daEU6ij2TAAr67bshntHwm3+x1le9U4wmeBfGyAo3NmliYiIOIw92eCmRpwmTJhgdV86Nzc3XnjhBTZv3mx/tVJstWgIVcLhwKVqxJVrCYYBWxY5uywRERGnsTs4BQQEcPz48Vz7T5w4gb+/v0OKkuLBZIJ7rt6CZVlqlPmBTteJiEgZZndweuihhxg2bBjff/89J06c4MSJE8yePZvHH3+cAQMGFEaN4kTZi2G+v+MeDBc3OLELYvNe+kFERKS0s/uqurfffhuTycTgwYPJyMgAzGspDR8+nKlTpzq8QHGuRnWgVlU4fCKEM6F3UeHMb+ZRpz7jnF2aiIhIkbNrxCkzM5P169czadIkLl68SHR0NNHR0cTFxfHee+9pvaRSyGSCPldP1y1MjDI/2LzQPN9JRESkjLErOLm6utK9e3fi4+Px8fEhIiKCiIgIfHx8Cqs+KQay5zl9vKMbhocPnD8OR+xbekJERKQ0sHuOU+PGjTl8+HBh1CLF1G01oH4tSEz34Vj5q7dd0SRxEREpg+wOTv/6178YN24cixcvJiYmhsTERKtNSqfs03Xfn48yP9i6GDLTnVaPiIiIM9i9AKaLy7WsZTKZLI8Nw8BkMpGZmem46gqBFsC8OcdOwV2PgLtLBgdatsXl8nkY/jU07uzs0kRERG6JPdnA7qvqVq5cedOFSclVvTI0rQ/b97mxP+QeGlyeYT5dp+AkIiJliN3BqWbNmlStWtVqtAnMI04nTpxwWGFS/PTpBNv3wX9ORzGFGbDjF7hyGbx8nV2aiIhIkbB7jlPNmjU5d+5crv1xcXHUrFnTIUVJ8ZR9dd2snc3ICK4OaSnm8CQiIlJG2B2csucyXS8pKQkvLy+HFCXFU8VQaNMEwMT2gCjzTl1dJyIiZYjNp+rGjh0LmCeEv/TSS1ZrN2VmZrJhwwaaNWvm8AKleLmnI2zcAdOP9KOlywew7w+4dB78yzu7NBERkUJnc3Datm0bYB5x2rlzJx4eHpbXPDw8aNq0KePG6TYcpV3vDjDpY/h5X21SO0fgGbsTtv4EHYY4uzQREZFCZ3Nwyr6a7tFHH+WDDz7QpfxlVGgItGsOa7bARs8o7mSn+XSdgpOIiJQBds9x+vrrrxWayrjsxTA/3d8HTC7m26+cP+7cokRERIqA3cHp8uXLvPTSS7Rr1446depQq1Ytq01Kv553grsbrD0czuVq7cw7NUlcRETKALvXcXr88cf5/fffGTRoEBUrVszzCjsp3YIC4K7WsGId/GGKoidrzMGp57OgnwcRESnF7A5OS5Ys4aeffqJ9+/aFUY+UEH06mYPTRzt70iPkH5jOHIITu6BahLNLExERKTR2n6oLDg4mJCSkMGqREqRbe/D0gF0n/Ems0dW8U6frRESklLM7OL322mu8/PLLJCcnF0Y9UkL4+UDn282Pl6dFmR9sWQRZxfsmzyIiIrfC7lN177zzDocOHSI8PJwaNWrg7u5u9frWrVsdVpwUb306w5LV8OHWjtxXNRBTwlk4sA7q3+Hs0kRERAqF3cEpKiqqEMqQkqhzW/D1hmNnPDh/592E7pllPl2n4CQiIqWU3cHplVdeKYw6pATy9jLPdVrwKyy+HMWjzILopfC3f4G77lsoIiKlj91znADi4+OZPn06EydOJC4uDjCfojt16pRDi5Pir29n85+fbmyNEVwJrlyCnSucW5SIiEghsTs47dixg9tuu41p06bx9ttvEx8fD8C8efOYOHGio+uTYu7OVhDgB2cuuHC6Wj/zzs0LnVuUiIhIIbE7OI0dO5ahQ4fy119/4eV17XRM7969Wb16tUOLk+LPwx163WV+PPdClPnB7pWQnOC0mkRERAqL3cFp06ZNPPXUU7n2V65cmdjYWIcUJSVL9r3rvl5fH6NifchIg20/O7coERGRQmB3cPL09CQxMTHX/gMHDhAaGuqQoqRkiWwO5YLgYiIcrhhl3qnFMEVEpBSyOzj17duXV199lfT0dABMJhPHjx9nwoQJ3HfffQ4vUIo/N1fo3cH8eFZMX/ODv9bDxdPOK0pERKQQ2B2c3nnnHZKSkggLCyMlJYUOHTpQp04d/P39ef311wujRikBsq+u+359ZbJqtzU/2bzIeQWJiIgUArvXcQoMDGT58uWsXbuW7du3k5SURIsWLejatWth1CclRKvGUKE8xJ6HvSFRNDq0wXy6rtvTzi5NRETEYewOTtnat29P9erVqVixIq6uro6sSUogFxe4pxNMnwP/Odabaa4vw6m9cHo/VKrn7PJEREQc4qYWwMzWsGFDjh075qhapITLPl23cH0QGfWvXmqnSeIiIlKK3FJwMgzDUXVIKdCkHlSrBClXINovyrxz00LIynJqXSIiIo5yS8FJJCeTCfp0ND/++kAX8PKDi6fg8Gan1iUiIuIotxSc/v73vxMSEuKoWqQU6HP1dN0vG71Ia9TL/ESn60REpJS4peA0ceJE/P39iY6O5uLFi46qSUqw+rWgTnVIS4f17lHmndt+Mq8mLiIiUsLZHZzGjBnDv//9bwAyMzPp0KEDLVq0oGrVqqxatcquY02ZMoXWrVvj7+9PWFgYUVFR7N+//4bvmzNnDvXr18fLy4uIiAh+/lm39yguTCboe3Ve+Fe7IiEgFC7Hw57fnVqXiIiII9gdnObOnUvTpk0B+PHHHzl8+DD79u3jueee4x//+Iddx/r9998ZMWIE69evZ/ny5aSnp9O9e3cuX76c73v+/PNPBgwYwLBhw9i2bRtRUVFERUWxa9cue7siheSeq8Hpjy2uXIm4upK4TteJiEgpYDLsvDTOy8uLgwcPUqVKFZ588kl8fHx4//33OXLkCE2bNs3zPna2OnfuHGFhYfz+++/cddddebZ56KGHuHz5MosXL7bsu/3222nWrBmfffbZDT8jMTGRwMBAEhISCAgIuOlapWC9n4Tdf8Hnw3bQc10fcPeEKVvA29/ZpYmIiFixJxvYPeIUHh7Onj17yMzMZOnSpXTr1g2A5OTkW14IMyEhAaDACefr1q3LtUp5jx49WLduXZ7tU1NTSUxMtNqk8GWfrpuxJQLCakF6Kmxf5tyiREREbpHdwenRRx/lwQcfpHHjxphMJkuI2bBhA/Xr17/pQrKyshgzZgzt27encePG+baLjY0lPDzcal94eDixsbF5tp8yZQqBgYGWrWrVqjddo9ju7qvBaf12E0mNosxPdLpORERKOLuD06RJk5g+fTpPPvkka9euxdPTEwBXV1defPHFmy5kxIgR7Nq1i9mzZ9/0MfIyceJEEhISLNuJEyccenzJW9UK0LwhGAYsTeln3rl/LSScdW5hIiIit+Cm7lV3//33Wz2Pj49nyJAhN13EyJEjWbx4MatXr6ZKlSoFtq1QoQJnzpyx2nfmzBkqVKiQZ3tPT09LuJOi1bcTbNsDMzfU4P4azeHoNti6GDo95uzSREREbordI07Tpk3j+++/tzx/8MEHKVeuHFWqVGHHjh12HcswDEaOHMn8+fP57bffqFmz5g3fExkZyYoVK6z2LV++nMjISLs+Wwrf3R3NyxNs3Q0X610dddLpOhERKcHsDk6fffaZZZ7Q8uXLWb58OUuWLKFnz56MGzfOrmONGDGC7777jlmzZuHv709sbCyxsbGkpKRY2gwePJiJEydano8ePZqlS5fyzjvvsG/fPiZNmsTmzZsZOXKkvV2RQhZeHtqaV65gYXwfcHGFY9vh7BHnFiYiInKT7A5OsbGxluC0ePFiHnzwQbp3784LL7zApk2b7DrWp59+SkJCAh07dqRixYqWLeeI1vHjx4mJibE8b9euHbNmzeKLL76gadOmzJ07lwULFhQ4oVycJ/vqujlrykP9O8xPNOokIiIllN3BKTg42DLBeunSpZar6gzDIDMz065jGYaR5zZ06FBLm1WrVjFjxgyr9z3wwAPs37+f1NRUdu3aRe/eve3thhSRXh3A1QV2/QVna0WZd25aYJ41LiIiUsLYHZzuvfdeHn74Ybp168aFCxfo1ct8I9dt27ZRp04dhxcoJVtIINzRyvx4bmwP8PCGc0fNp+xERERKGLuD03vvvcfIkSNp2LAhy5cvx8/PD4CYmBieeeYZhxcoJV+fq6frfvjdFyPCvGCqTteJiEhJZPctV0o63XKl6CUkQat7IS0d1vxjBVXnPwb+5eH1DeB6UytiiIiIOEyh3nIF4NChQzz77LN07dqVrl27MmrUKA4fPnxTxUrpF+gHHduYH//v6F3gFwKXzpsXxBQRESlB7A5Oy5Yto2HDhmzcuJEmTZrQpEkTNmzYYDl1J5KXPp3Nfy5c6Y7R/G7zE52uExGREsbu8yQvvvgizz33HFOnTs21f8KECZab/ork1DUSvL3g2Gk4VCGKOnxrvulvWop5wriIiEgJYPeI0969exk2bFiu/Y899hh79uxxSFFS+vh4m8MTwOw9LaFcVUi9DDs0SikiIiWH3cEpNDSU6OjoXPujo6MJCwtzRE1SSmWfrlu8yoTRSrdgERGRksfuU3VPPPEETz75JIcPH6Zdu3YArF27lmnTpjF27FiHFyilR4c24O8LMedgV1AUEXwMe36HpDjzhHEREZFizu7g9NJLL+Hv788777xjuYdcpUqVmDRpEqNGjXJ4gVJ6eHlA9/bwwy/wv211iajaCE7shq0/wV2DnF2eiIjIDdl1qi4jI4Nvv/2Whx9+mJMnT5KQkEBCQgInT55k9OjRmEymwqpTSons03U//w6ZLaPMT3S6TkRESgi7gpObmxtPP/00V65cAcDf3x9/f/9CKUxKpztaQlAAnL8IWzz7gskEhzfDhRPOLk1EROSG7J4c3qZNG7Zt21YYtUgZ4O4Gve8yP/5hYwWoe/VSu00LnVeUiIiIjeye4/TMM8/w/PPPc/LkSVq2bImvr6/V602aNHFYcVI69ekMsxbDktXw+vgo3A78aT5d12OEeQRKRESkmLL7XnUuLrkHqUwmE4ZhYDKZyMzMdFhxhUH3qnO+zExo+yCci4NvJiXScV4ryEiFF3+Gqo2cXZ6IiJQx9mQDu0ecjhw5ctOFiQC4usI9HeHrebBgbQAdG3eG6CXmUScFJxERKcbsDk7Vq1cvjDqkjLmnkzk4/bIG0ib3wyN6CWxZBFEvgours8sTERHJk93BacqUKYSHh/PYY49Z7f/qq684d+4cEyZMcFhxUnq1aAiVw+HUGfjtUid6egdAfCz89C7UaG6+JUtIFfDyvfHBRESkdDAMSEmE+DMQH2P+dyEhFi5e/fOOgRDR1akl2h2cPv/8c2bNmpVrf6NGjfjb3/6m4CQ2cXExn677/HtYtNqLns17w5+zYenH1g39Qq6FqPJVrR+HVAZ3L6fULyIidsrKhMRz5jAUH3s1GJ25Fo6y96el5H+MGs1LXnCKjY2lYsWKufaHhoYSExPjkKKkbOjT2RycVqyHy8PH4esXAmcPw4WT5nWdkhPMt2NJioNj2/M+SGDYdWGqivl5+WoQXBFc3Yu2UyIiZVFq8nUBKMeIUfwZ8/6Es2Bk2XY83yAIrABB4RBUEYIqmLeaLQq1G7awOzhVrVqVtWvXUrNmTav9a9eupVKlSg4rTEq/xnWhZhU4chKW7wwlqt91o5UpiddCVK4/T0DqZfNfxISzcHhL7g8wuZj/wuUMVOWyg1VVCAzXfCoRkYIYhvk/r9efMrs+HKUk2nY8F1cICIPgCleD0XVb9j6P4ns24aZu8jtmzBjS09Pp3Nl8/4wVK1bwwgsv8Pzzzzu8QCm9TCbo0wk+/BZ+XAlR14++egdAlYbm7XqGAZfjr4Uoy3Y1WMWdhPRUuHjKvOXF1R2CK10LUzmDVbkq5r/cWldKREqrjDTzfzyzT5nlPF2WvSWcMbezhYePeaQ/MPzqn9kjRhWujhqFg3/5Ev8fVrvXcTIMgxdffJEPP/yQtDTzF9PLy4sJEybw8ssvF0qRjqR1nIqXA0eh26PmFcW3zINAR93BxzDM59KzQ9T5q39eOGF+fPE0ZKYXfAx3z6sjVTmDVY6A5RusYCUixY9hwJVL+Z8yyw5Kl87bfkz/8teNCuU8hXY1HHn5l9jfifZkA7uDU7akpCT27t2Lt7c3devWxdPT86aKLWoKTsVPj2Gw7zC8OR4e6l1EH5qVaf6f1IUcYSoux+nAi6dvfC7e0zfv0ars+Vbe+vmSQmYY5tGAjDTzfwSyH+fcl54GmTn3X22X377064+XmuP1HPsyM8HNw/wfDHdPcPMs+HGufV62vc/V7hMjpVtWpjnwWI0M5bwC7QxcjIG0ZNuO5+ZhHiHKOSqUHY6yT6cFhpnblWKFugBmNj8/P1q3bn2zbxexuKeTOTj9uLIIg5OLq/k0XXAlqNMm9+uZ6eZfPtefAsx+nHDGPMfq9H7zlhefQOswlT23KuRqyPL0Kdw+imMYBmRlXA0g6VeDRNp1YeK6fdeHl/z2WYWaHO1s3neDUdPSwMX1xuHqRo8tAc/L/ve7uBXdKEralRxziWLMv2euP3WWeNYcnmzhHZD/KbPsuUR+ISV2lMhZbnrEqaTSiFPxc/QUdHjEvETBprlQPtjZFdkg/QrEncp70vqFk5B04cbH8C+fx9WAVx+HVDb/0i6LLEEl1TpspF/J8Tg1j8epuV+z7Mt+nNf7ssNI6nUhJ8coTEn5NeniZg4Jbu7mP109wN3j6uOr+3Ju+e2zvCfna+7W+1xdr33d0q9ujnic/T0qLkwuNoye2fr4anDLSM1xOX6OU2iX422vKTAs/1NmQVfnGek/ZzYrklN1JZWCU/HU52nYsR/q14I7W0GbCGgdAcGBzq7sJl25fDVYnYALx60nrZ8/YdsVKIHheU9aL1fVPFLmyFMYhpFjFOT6cJFf2Lg+qBTQLs+gk/1azuNdfa04/1oyuRQQPtyvjXBYgoaN+2wJNG5Xg1B+gSaPe4mWSFlZOX4+rjgulJWE4ObhnfsKs+s3//I6helgCk4FUHAqnuYshXHTcu+vWx3aNDGHqDZNzKuNlwrJCbnDVM7J6zean+Diav5fZXaQ8g2+Op+lgMByo2BTXGWPorh7XgsLlseeuV9z87w2YuLmAW5e1wKHm2fuQOLuZR1eLKEmn/Cif7BKv8IObq5uuU+ZZT/3DtCpMydQcCqAglPxdeoMbNgBm3bAxp1w8FjuNpXDr4aoCGjdxBysSt3vGMOAyxetJ62fPwFx2acET5p/+RYmy+hGfoHE0zqEuF8XSK6fW5LrtRzHsAo5nte9373EX7osIsWfglMBFJxKjrgE2LQTNu4w/7nrAGRed6FbcIA5SLVuYg5TjeqalzYo1bKy4NI567lVyQm5A4klqHjkEXLyCT3unqXrlI+IiA0UnAqg4FRyXU6BbXvMQWrjTvPjK9cNvPh4QfOG107vtWgI3sV3AVoRESkGFJwKoOBUeqSlw66/ro1IbdoJCZes27i5QsRt10akWkdAkL7tIiKSg4JTARScSq+sLPjr2NURqathKuZc7na31bCecF4prMhLFRGRYkTBqQAKTmWHYcDJM1cnm189vXfoeO52VcJzjEg1gTrVSuGEcxERyZeCUwEUnMq2C/HWE853/5V7wnlI4NUJ51dHpBrVNZ/yExGR0knBqQAKTpLT5RTYutt6wnnqdWvf+XhBi0bXRqSaN9CEcxGR0kTBqQAKTlKQtHTYeeDaWlKbdkJiknUbdzdofJs5SLVpAq0aa8K5iEhJpuBUAAUnsUdWFhw4eu3U3sYdEHs+d7t6Na+d2mvTBCqGFnmpIiJyk0pMcFq9ejVvvfUWW7ZsISYmhvnz5xMVFZVv+1WrVtGpU6dc+2NiYqhQoYJNn6ngJLfCMOBEbI4RqR1w6ETudlUqXBuRat0EalfVhHMRkeLKnmzg1DWWL1++TNOmTXnssce49957bX7f/v37rToWFqbryaVomExQraJ5u6+Hed/5i9fWkdq4A3YfhJOx5m3ecnObckHQKuJamGpYRxPORURKIqcGp169etGrVy+73xcWFkZQUJDjCxK5CeWDoddd5g0gKfnqhPOrI1Lb9pqv5lv2h3kD8PU2TzjPPr3XvAF4eTqtCyIiYqMSeVevZs2akZqaSuPGjZk0aRLt27d3dkkiFn4+cFdr8wbmq/R2Hrg2IrV5l3nC+R+bzRuYJ5xH1LOecB7o77w+iIhI3kpUcKpYsSKfffYZrVq1IjU1lenTp9OxY0c2bNhAixYt8nxPamoqqanXbmiWmJhYVOWKAODpYQ5CrRrD8AHmCef7j+SYcL4Tzpw3j1Jt3Q2fzTafErSacB4BFTThXETE6YrNVXUmk+mGk8Pz0qFDB6pVq8a3336b5+uTJk1i8uTJufZrcrgUF4YBJ2KurSW1aScczmPCedWK19aSahMBtTThXETEIUrMVXU53WxwGj9+PGvWrGHdunV5vp7XiFPVqlUVnKRYOxdnPeF8zyHzSFVO5YPNo1htmkDT+lA5HEJDNOlcRMReJeaqOkeIjo6mYsWK+b7u6emJp6dm3UrJEhoCvTuYN4BLl60nnEfvNV/Nt/QP85bNxQXCQqBimHktKavt6r6wcgpXIiI3y6nBKSkpiYMHD1qeHzlyhOjoaEJCQqhWrRoTJ07k1KlT/Oc//wHg/fffp2bNmjRq1IgrV64wffp0fvvtN3755RdndUGkSPj7Qoc25g3ME8537L82InXgKMSeM993L/a8eduWz7Es4SpHmKoQCpWu/lkxFMLLK1yJiOTFqcFp8+bNVgtajh07FoAhQ4YwY8YMYmJiOH782u3s09LSeP755zl16hQ+Pj40adKEX3/9Nc9FMUVKM0+PazcifuZh877MTPMoVMy5q9tZiDl/9c9z5mAVex4yMnOEq715H9/FxTzqlTNMVQyDiuWtR67cS/yYtYiIfYrNHKeiopXDpSzLzITz8eYQdfrs1T+vhqrssJUdrm4kO1xVLH91xCos75ErhSsRKe7K1BwnEbGdqyuElzNvTevn3SYrC85dtA5TllGsq9uZ85CeYf7zzHlgX97HMpnyGbnKMfdK4UpEShL9uhIRKy4utoWrnCNX2acCT18dsTp99lq4OnvBvBUUrsoH5z1ilb0vvBx4uBdWj0VEbKfgJCJ2y55gHhYCTerl3SYry3yrGavRqutGr2LPmcPVuTjztv0G4cpqrlXO0aswhSsRKRoKTiJSKLLnQIXaEK6yTwuePmc9/yrmvPnPtPRr4WrH/vw/MzQ496nACteNXHl6FEp3RaSMUHASEafJGa4iCghXcQn5j1hlh620dPPcrHMXbxyurg9T5YOv1REaDOWCNe9KRPKmXw0iUqy5uJiDTflgiLgt7zaGYQ5XlisFr861yhm2Ys9Bao5wtfNAwZ8bEnhdoAoxPw/L8Tg0xNzOxcXx/RaR4knBSURKPJMJygWZtxuFq+tHrs7Fwdk48xpY5+LgwkXzQqJxCebtwNGCP9vVBUKCroWrsBDrwJX9OCwEAvx0f0GRkk7BSUTKhJzhqnHd/NtlZcHFxGtzqs7FXR2luvr4fI7HcQnmkJX9/EY83PMOVaHBUD7k2mhWaAj4ejuq5yLiSApOIiI5uLhcC1j1axXcNj0D4uLNI1bXh6qcj89dhMQk8zys02fN2414e1mHqtDg3KNZoSHm17w04V2kyCg4iYjcJHc38wKe4eVv3PZKGpy/PlDlMZp1Ng5Srpi346fN240E+F43gpVHwNKkdxHH0F8hEZEi4OUBVSqYtxu5nGIOWWfj8hjBumh+LTtwpaVD4mXzdujEjY99/aT3nHOwcj4PDjCvNC8i1hScRESKGV9v8K0M1SsX3M4wzIEp5whWzlCVczTrVie9h14/ghUCFa4uROrl6aieixR/Ck4iIiWUyQSBfuatTrWC2+aa9J7ffKybmPQeHJD3fQgt62WVN8/ZEikNFJxERMqAm530nt+E97Nx5uUcUq6YA9nFRNhzKP9jBgXkcbPn8tfCVcVQhSspGRScRETEiq2T3g3DfLVgTM7b5Fy/nYXkKxCfaN4KCleB/nmEq1DrzUfLNIiTKTiJiMhNMZnMYSfQP/9RrOx5WDlXcD+d/fj81cdXw1XCJfO293D+nxnon+M0YI6QlX0LnYqhWgNLCpeCk4iIFJqc87AKCleXLuceqcp5P8KYs+arDbPD1b4CwlWAn3W4yjl6lX1qUOFKbpaCk4iIOJXJZA47AX5Qr2b+7S5dzn2j55yjWLHnzW0Sk8zb/iP5HyvA9+oIVR6nA7PDlZ+P4/sqJZ+Ck4iIlAj+vuBfE267Ubg6V/DNnrPXvUq8XPCyDP6+5isCc54GrHjdCJa/r8O7KcWcgpOIiJQa/r7m7bYa+bdJSr4uXF0duTp97trjxCRzCLt0Gf46lv+x/HzyWH4he97V1X3+vrq5c2mi4CQiImWKnw/UrW7e8pOUbB6tsprMniNcnT5rDldJyeZgVVC48vXOMceqvHnxUG8v8Lm6eXuZrxa0PL76PGcbreJefCg4iYiIXMfPx7yoaEELi15OyWMJhutCVsIlc7uDx8zbzfL0yDtQ5Re0vL2v7c8roOV87qZQZhcFJxERkZvg6w21q5m3/CSnXFt2IfYcxJyHiwnm/ZdTzAuIJl/dsh+npFzbZxjm46SmmbeLiY7vh4d7PiNg14es/IJbHqNl2ftK402lS2GXREREigcfb6hV1bzZyzDMYSn5ijloZYcrS+BKyR26knOEsev3XU6x3p+VZf6ctHTzlnDJsX0Hc3AqMHTls9/XO8eoWY5RtNAQ89IWzqTgJCIiUgyZTOYbKHt5QkigY49tGJCabj26lXJdQCsodF0/Mnb9aFlGpvlz0jMg/eryEI4wbhg8+4hjjnWzFJxERETKGJMJvDzMW7CDQxmYR7CsAleOkJVf6Eq+7tSlVYi7+rg4LP+g4CQiIiIO5eFu3gL9nV2J47k4uwARERGRkkLBSURERMRGCk4iIiIiNlJwEhEREbGRgpOIiIiIjRScRERERGyk4CQiIiJiIwUnERERERspOImIiIjYSMFJRERExEYKTiIiIiI2KnP3qjMMA4DExEQnVyIiIiLFQXYmyM4IBSlzwenSpUsAVK1a1cmViIiISHFy6dIlAgMDC2xjMmyJV6VIVlYWp0+fxt/fH5PJ5PDjJyYmUrVqVU6cOEFAQIDDj1+cqe/qu/peNpTVfoP6Xlr7bhgGly5dolKlSri4FDyLqcyNOLm4uFClSpVC/5yAgIBS94NlK/VdfS9rymrfy2q/QX0vjX2/0UhTNk0OFxEREbGRgpOIiIiIjRScHMzT05NXXnkFT09PZ5dS5NR39b2sKat9L6v9BvW9rPY9pzI3OVxERETkZmnESURERMRGCk4iIiIiNlJwEhEREbGRgpMDffLJJ9SoUQMvLy/atm3Lxo0bnV3SLZsyZQqtW7fG39+fsLAwoqKi2L9/v1WbK1euMGLECMqVK4efnx/33XcfZ86csWpz/Phx7r77bnx8fAgLC2P8+PFkZGQUZVduydSpUzGZTIwZM8ayrzT3+9SpUzzyyCOUK1cOb29vIiIi2Lx5s+V1wzB4+eWXqVixIt7e3nTt2pW//vrL6hhxcXEMHDiQgIAAgoKCGDZsGElJSUXdFbtkZmby0ksvUbNmTby9valduzavvfaa1W0YSkvfV69eTZ8+fahUqRImk4kFCxZYve6ofu7YsYM777wTLy8vqlatyptvvlnYXbuhgvqenp7OhAkTiIiIwNfXl0qVKjF48GBOnz5tdYzS2PfrPf3005hMJt5//32r/SW17w5jiEPMnj3b8PDwML766itj9+7dxhNPPGEEBQUZZ86ccXZpt6RHjx7G119/bezatcuIjo42evfubVSrVs1ISkqytHn66aeNqlWrGitWrDA2b95s3H777Ua7du0sr2dkZBiNGzc2unbtamzbts34+eefjfLlyxsTJ050RpfstnHjRqNGjRpGkyZNjNGjR1v2l9Z+x8XFGdWrVzeGDh1qbNiwwTh8+LCxbNky4+DBg5Y2U6dONQIDA40FCxYY27dvN/r27WvUrFnTSElJsbTp2bOn0bRpU2P9+vXGH3/8YdSpU8cYMGCAM7pks9dff90oV66csXjxYuPIkSPGnDlzDD8/P+ODDz6wtCktff/555+Nf/zjH8a8efMMwJg/f77V647oZ0JCghEeHm4MHDjQ2LVrl/Hf//7X8Pb2Nj7//POi6maeCup7fHy80bVrV+P777839u3bZ6xbt85o06aN0bJlS6tjlMa+5zRv3jyjadOmRqVKlYz33nvP6rWS2ndHUXBykDZt2hgjRoywPM/MzDQqVapkTJkyxYlVOd7Zs2cNwPj9998NwzD/knF3dzfmzJljabN3714DMNatW2cYhvkvqouLixEbG2tp8+mnnxoBAQFGampq0XbATpcuXTLq1q1rLF++3OjQoYMlOJXmfk+YMMG444478n09KyvLqFChgvHWW29Z9sXHxxuenp7Gf//7X8MwDGPPnj0GYGzatMnSZsmSJYbJZDJOnTpVeMXforvvvtt47LHHrPbde++9xsCBAw3DKL19v/4fUEf18//+7/+M4OBgq5/3CRMmGPXq1SvkHtmuoPCQbePGjQZgHDt2zDCM0t/3kydPGpUrVzZ27dplVK9e3So4lZa+3wqdqnOAtLQ0tmzZQteuXS37XFxc6Nq1K+vWrXNiZY6XkJAAQEhICABbtmwhPT3dqu/169enWrVqlr6vW7eOiIgIwsPDLW169OhBYmIiu3fvLsLq7TdixAjuvvtuq/5B6e73okWLaNWqFQ888ABhYWE0b96cL7/80vL6kSNHiI2Ntep7YGAgbdu2tep7UFAQrVq1srTp2rUrLi4ubNiwoeg6Y6d27dqxYsUKDhw4AMD27dtZs2YNvXr1Akp333NyVD/XrVvHXXfdhYeHh6VNjx492L9/PxcvXiyi3ty6hIQETCYTQUFBQOnue1ZWFoMGDWL8+PE0atQo1+ulue+2UnBygPPnz5OZmWn1DyRAeHg4sbGxTqrK8bKyshgzZgzt27encePGAMTGxuLh4WH5hZItZ99jY2Pz/Npkv1ZczZ49m61btzJlypRcr5Xmfh8+fJhPP/2UunXrsmzZMoYPH86oUaP45ptvgGu1F/TzHhsbS1hYmNXrbm5uhISEFOu+v/jii/ztb3+jfv36uLu707x5c8aMGcPAgQOB0t33nBzVz5L6dyCnK1euMGHCBAYMGGC5P1tp7vu0adNwc3Nj1KhReb5emvtuqzJ3k1+5eSNGjGDXrl2sWbPG2aUUuhMnTjB69GiWL1+Ol5eXs8spUllZWbRq1Yo33ngDgObNm7Nr1y4+++wzhgwZ4uTqCtf//vc/Zs6cyaxZs2jUqBHR0dGMGTOGSpUqlfq+S27p6ek8+OCDGIbBp59+6uxyCt2WLVv44IMP2Lp1KyaTydnlFFsacXKA8uXL4+rqmuuKqjNnzlChQgUnVeVYI0eOZPHixaxcuZIqVapY9leoUIG0tDTi4+Ot2ufse4UKFfL82mS/Vhxt2bKFs2fP0qJFC9zc3HBzc+P333/nww8/xM3NjfDw8FLZb4CKFSvSsGFDq30NGjTg+PHjwLXaC/p5r1ChAmfPnrV6PSMjg7i4uGLd9/Hjx1tGnSIiIhg0aBDPPfecZdSxNPc9J0f1s6T+HYBroenYsWMsX77cMtoEpbfvf/zxB2fPnqVatWqW33vHjh3j+eefp0aNGkDp7bs9FJwcwMPDg5YtW7JixQrLvqysLFasWEFkZKQTK7t1hmEwcuRI5s+fz2+//UbNmjWtXm/ZsiXu7u5Wfd+/fz/Hjx+39D0yMpKdO3da/WXL/kV0/T/QxUWXLl3YuXMn0dHRlq1Vq1YMHDjQ8rg09hugffv2uZacOHDgANWrVwegZs2aVKhQwarviYmJbNiwwarv8fHxbNmyxdLmt99+Iysri7Zt2xZBL25OcnIyLi7WvxZdXV3JysoCSnffc3JUPyMjI1m9ejXp6emWNsuXL6devXoEBwcXUW/slx2a/vrrL3799VfKlStn9Xpp7fugQYPYsWOH1e+9SpUqMX78eJYtWwaU3r7bxdmz00uL2bNnG56ensaMGTOMPXv2GE8++aQRFBRkdUVVSTR8+HAjMDDQWLVqlRETE2PZkpOTLW2efvppo1q1asZvv/1mbN682YiMjDQiIyMtr2dflt+9e3cjOjraWLp0qREaGlrsL8u/Xs6r6gyj9PZ748aNhpubm/H6668bf/31lzFz5kzDx8fH+O677yxtpk6dagQFBRkLFy40duzYYfTr1y/PS9WbN29ubNiwwVizZo1Rt27dYndJ/vWGDBliVK5c2bIcwbx584zy5csbL7zwgqVNaen7pUuXjG3bthnbtm0zAOPdd981tm3bZrlyzBH9jI+PN8LDw41BgwYZu3btMmbPnm34+Pg4/bL0gvqelpZm9O3b16hSpYoRHR1t9Xsv51VipbHvebn+qjrDKLl9dxQFJwf66KOPjGrVqhkeHh5GmzZtjPXr1zu7pFsG5Ll9/fXXljYpKSnGM888YwQHBxs+Pj5G//79jZiYGKvjHD161OjVq5fh7e1tlC9f3nj++eeN9PT0Iu7Nrbk+OJXmfv/4449G48aNDU9PT6N+/frGF198YfV6VlaW8dJLLxnh4eGGp6en0aVLF2P//v1WbS5cuGAMGDDA8PPzMwICAoxHH33UuHTpUlF2w26JiYnG6NGjjWrVqhleXl5GrVq1jH/84x9W/2CWlr6vXLkyz7/bQ4YMMQzDcf3cvn27cccddxienp5G5cqVjalTpxZVF/NVUN+PHDmS7++9lStXWo5RGvuel7yCU0ntu6OYDCPHkrgiIiIiki/NcRIRERGxkYKTiIiIiI0UnERERERspOAkIiIiYiMFJxEREREbKTiJiIiI2EjBSURERMRGCk4iIiIiNlJwEhEREbGRgpOIlGpDhw4lKirK2WWISCmh4CQiIiJiIwUnESkV5s6dS0REBN7e3pQrV46uXbsyfvx4vvnmGxYuXIjJZMJkMrFq1SoATpw4wYMPPkhQUBAhISH069ePo0ePWo6XPVI1efJkQkNDCQgI4OmnnyYtLc05HRSRYsHN2QWIiNyqmJgYBgwYwJtvvkn//v25dOkSf/zxB4MHD+b48eMkJiby9ddfAxASEkJ6ejo9evQgMjKSP/74Azc3N/71r3/Rs2dPduzYgYeHBwArVqzAy8uLVatWcfToUR599FHKlSvH66+/7szuiogTKTiJSIkXExNDRkYG9957L9WrVwcgIiICAG9vb1JTU6lQoYKl/XfffUdWVhbTp0/HZDIB8PXXXxMUFMSqVavo3r07AB4eHnz11Vf4+PjQqFEjXn31VcaPH89rr72Gi4sG7EXKIv3NF5ESr2nTpnTp0oWIiAgeeOABvvzySy5evJhv++3bt3Pw4EH8/f3x8/PDz8+PkJAQrly5wqFDh6yO6+PjY3keGRlJUlISJ06cKNT+iEjxpREnESnxXF1dWb58OX/++Se//PILH330Ef/4xz/YsGFDnu2TkpJo2bIlM2fOzPVaaGhoYZcrIiWYgpOIlAomk4n27dvTvn17Xn75ZapXr878+fPx8PAgMzPTqm2LFi34/vvvCQsLIyAgIN9jbt++nZSUFLy9vQFYv349fn5+VK1atVD7IiLFl07ViUiJt2HDBt544w02b97M8ePHmTdvHufOnaNBgwbUqFGDHTt2sH//fs6fP096ejoDBw6kfPny9OvXjz/++IMjR46watUqRo0axcmTJy3HTUtLY9iwYezZs4eff/6ZV155hZEjR2p+k0gZphEnESnxAgICWL16Ne+//z6JiYlUr16dd955h169etGqVStWrVpFq1atSEpKYuXKlXTs2JHVq1czYcIE7r33Xi5dukTlypXp0qWL1QhUly5dqFu3LnfddRepqakMGDCASZMmOa+jIuJ0JsMwDGcXISJS3AwdOpT4+HgWLFjg7FJEpBjReLOIiIiIjRScRERERGykU3UiIiIiNtKIk4iIiIiNFJxEREREbKTgJCIiImIjBScRERERGyk4iYiIiNhIwUlERETERgpOIiIiIjZScBIRERGxkYKTiIiIiI3+H4LN07S8hFJbAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: the loss curve\n",
"steps_, tr_, va_ = zip(*loss_log)\n",
"plt.figure(figsize=(6, 4))\n",
"plt.plot(steps_, tr_, label=\"train\", color=\"#1E40FF\")\n",
"plt.plot(steps_, va_, label=\"val\", color=\"#FF6A1E\")\n",
"plt.axhline(math.log(vocab_size), ls=\":\", c=\"#888\", label=\"random (ln vocab)\")\n",
"plt.xlabel(\"step\"); plt.ylabel(\"cross-entropy (nats/char)\"); plt.legend(); plt.title(\"char-RNN training\")\n",
"plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "54de6ac8",
"metadata": {},
"source": [
"> **Interpretation.** What does this plot show? AnswerBoth curves fall below the random baseline, and train sits slightly under val: the model is learning the corpus's character statistics without badly overfitting at this size. If val turned back up while train kept falling, that would be overfitting and a cue to stop earlier or shrink the model. The gap is small here because the model is tiny relative to the corpus.\n"
]
},
{
"cell_type": "markdown",
"id": "16368ffe",
"metadata": {},
"source": [
"> **Key takeaways**\n",
"> - A char-RNN is embed -> GRU -> linear head, trained with per-position cross-entropy under teacher forcing.\n",
"> - Batches are windows slid over one long stream; `y` is `x` shifted by one.\n",
"> - The four-comment loop is forward / zero-grad+backward / clip+update / track; zeroing grads each step is not optional.\n",
"> - Initial loss near `ln(vocab)` is the universal sanity check; the experiment log is your expected-value reference for FAST versus full runs.\n"
]
},
{
"cell_type": "markdown",
"id": "1f896333",
"metadata": {},
"source": [
"## Part 4 — Attention from scratch\n",
"\n",
"> **Objectives.** Understand the seq2seq bottleneck attention was invented to fix. Implement the softmax-over-keys weighted sum, Bahdanau additive scoring, and Luong multiplicative scoring, and check each against a torch reference. See the unified Q/K/V form, and break dot-product attention at large `d_k` before fixing it with `/ sqrt(d_k)`.\n",
"\n",
"For five years the way you built a translation system was: read the source with an RNN, compress it into one context vector, hand that to a decoder RNN. It worked until the source got past about twenty words, at which point one fixed-size vector could no longer hold the whole sentence and translations degraded. Bahdanau, Cho, and Bengio (2014) asked: what if the decoder could *look back* at all the encoder's hidden states and choose which to attend to at each step? That look-back is **attention**, and it is the operation the rest of this chapter, and most of Ch 15, is built from.\n"
]
},
{
"cell_type": "markdown",
"id": "d09b9bd0",
"metadata": {},
"source": [
"### The core operation: softmax over keys, then a weighted sum\n",
"\n",
"Strip away the names. Attention is: score each key against the query, softmax the scores into weights (one distribution per query, over the keys), then take the weighted average of the values. The only thing you must get right is the softmax *axis*: it normalises over keys, so each query's weights sum to 1.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "a5794410",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.556992Z",
"iopub.status.busy": "2026-06-10T19:53:18.556913Z",
"iopub.status.idle": "2026-06-10T19:53:18.560080Z",
"shell.execute_reply": "2026-06-10T19:53:18.559777Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"weights: [[0. 0. 0.9999]]\n",
"context (≈ value of the dominant key, 9): 8.999319076538086\n"
]
}
],
"source": [
"# micro-demo on a hand-checkable input: one query, three keys, scores [0, 0, 10].\n",
"# the third key dominates, so the weighted sum should be ~ the third value.\n",
"scores_demo = torch.tensor([[0.0, 0.0, 10.0]]) # (1 query, 3 keys)\n",
"weights_demo = torch.softmax(scores_demo, dim=-1) # softmax over keys\n",
"values_demo = torch.tensor([[1.0], [2.0], [9.0]]) # (3 keys, 1-dim values)\n",
"context_demo = weights_demo @ values_demo # (1, 1)\n",
"print(\"weights:\", weights_demo.numpy().round(4))\n",
"print(\"context (≈ value of the dominant key, 9):\", float(context_demo))\n",
"assert abs(weights_demo.sum().item() - 1.0) < 1e-6, \"attention weights must sum to 1 (softmax over keys)\""
]
},
{
"cell_type": "markdown",
"id": "85f552a3",
"metadata": {},
"source": [
"> **Interpretation.** The score 10 against 0 and 0 makes the third weight ~0.9999, so the context is ~9, the third value. Attention is a soft, differentiable `argmax`-and-select: a hard lookup when one score dominates, a blend when they are close.\n"
]
},
{
"cell_type": "markdown",
"id": "9967649e",
"metadata": {},
"source": [
"### Exercise 14.5 — Softmax over the right axis\n",
"`Difficulty 1/5 · ~5 min`\n",
"\n",
"Given a score tensor `scores` of shape `(B, T_q, T_k)`, produce attention `weights` of the same shape that sum to 1 *over the keys*. This is one line, but it is the most common attention bug, so we isolate and check it before building anything on top.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "4852e011",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.561086Z",
"iopub.status.busy": "2026-06-10T19:53:18.561014Z",
"iopub.status.idle": "2026-06-10T19:53:18.564398Z",
"shell.execute_reply": "2026-06-10T19:53:18.564116Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 14.5 softmax over keys: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def attn_weights(scores):\n",
" \"\"\"scores: (B, T_q, T_k). Return weights (B, T_q, T_k) summing to 1 over keys.\"\"\"\n",
" # TODO: softmax over the KEY axis (the last one), not the query axis\n",
" w = None\n",
" attempted(w)\n",
" return w\n",
"\n",
"def _check_axis():\n",
" s = torch.randn(2, 4, 8)\n",
" w = attn_weights(s)\n",
" check_shape(w, (2, 4, 8))\n",
" row_sums = w.sum(dim=-1)\n",
" check_close_torch(row_sums, torch.ones(2, 4),\n",
" msg=\"each query's weights must sum to 1 across keys; did you softmax over dim=1 (queries) by mistake?\")\n",
"\n",
"check(\"14.5 softmax over keys\", _check_axis)"
]
},
{
"cell_type": "markdown",
"id": "900cb4e3",
"metadata": {},
"source": [
"Hint 1 (conceptual)You want a probability distribution over keys *for each query*. The keys are the last axis. Softmax over that axis.\n",
"\n",
"Hint 2 (the line)`w = torch.softmax(scores, dim=-1)`\n",
"\n",
"Help — \"row sums are not 1\" / sums along the wrong axis are 1If `w.sum(dim=1)` is all ones instead of `w.sum(dim=-1)`, you normalised over queries. Switch `dim` to `-1` (the key axis).\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "e35c7918",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.565241Z",
"iopub.status.busy": "2026-06-10T19:53:18.565171Z",
"iopub.status.idle": "2026-06-10T19:53:18.567489Z",
"shell.execute_reply": "2026-06-10T19:53:18.567143Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 14.5 softmax over keys\n",
"[ ok ] softmax over keys; each query attends to a distribution over source positions\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines attn_weights; the check re-verifies it.\n",
"def attn_weights(scores):\n",
" return torch.softmax(scores, dim=-1) # normalise over the key axis\n",
"\n",
"check(\"14.5 softmax over keys\", _check_axis, required=True)\n",
"print(\"[ ok ] softmax over keys; each query attends to a distribution over source positions\")"
]
},
{
"cell_type": "markdown",
"id": "faa998d9",
"metadata": {},
"source": [
"### Bahdanau additive attention\n",
"\n",
"The first attention mechanism (Bahdanau 2014) scores the decoder state `s` against each encoder state `h_i` with an *additive* function: project both, sum inside a tanh, project to a scalar.\n",
"\n",
"$$e_i = v_a^\\top \\tanh(W_a\\, s + U_a\\, h_i), \\qquad \\alpha_i = \\frac{\\exp(e_i)}{\\sum_j \\exp(e_j)}, \\qquad c = \\sum_i \\alpha_i\\, h_i$$\n",
"\n",
"Here `W_a, U_a` are learned matrices and `v_a` a learned vector. The name \"additive\" is the `+` inside the tanh. We implement it in NumPy from scratch, then build the same thing as a torch module and assert they agree, which is the implementation ladder: math -> from-scratch -> library -> agreement check.\n"
]
},
{
"cell_type": "markdown",
"id": "488de177",
"metadata": {},
"source": [
"### Exercise 14.6 — Bahdanau additive scoring from scratch\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"Implement `bahdanau(s, h, W, U, v)` in NumPy returning `(context, weights)`. Shapes: `s` is `(d,)`, `h` is `(T_src, d)`, `W` and `U` are `(d, d)`, `v` is `(d,)`. The check feeds it the same weights as a torch reference module and asserts the contexts match to `1e-5`, and that the weights sum to 1.\n",
"\n",
"Harder: the additive form has three weight matrices and a nonlinearity. Count the matmuls and compare to the dot-product score you build next.\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "7a7ed62e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.568285Z",
"iopub.status.busy": "2026-06-10T19:53:18.568209Z",
"iopub.status.idle": "2026-06-10T19:53:18.573617Z",
"shell.execute_reply": "2026-06-10T19:53:18.573263Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 14.6 bahdanau vs torch: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def np_softmax(x, axis=-1):\n",
" x = x - x.max(axis=axis, keepdims=True) # subtract max for numerical stability\n",
" e = np.exp(x)\n",
" return e / e.sum(axis=axis, keepdims=True)\n",
"\n",
"def bahdanau(s, h, W, U, v):\n",
" \"\"\"s: (d,). h: (T_src, d). W, U: (d, d). v: (d,). Returns (context (d,), weights (T_src,)).\"\"\"\n",
" # TODO 1: score each source position: e_i = v . tanh(W@s + U@h_i)\n",
" # vectorise over i: (s @ W.T) is (d,), (h @ U.T) is (T_src, d); broadcast-add, tanh, dot with v\n",
" scores = None\n",
" # TODO 2: softmax over the T_src axis with np_softmax\n",
" weights = None\n",
" # TODO 3: context = weights @ h -> (d,)\n",
" context = None\n",
" attempted(scores, weights, context)\n",
" return context, weights\n",
"\n",
"def _check_bahdanau():\n",
" d, T = 16, 7\n",
" rs = np.random.default_rng(0)\n",
" s = rs.standard_normal(d); h = rs.standard_normal((T, d))\n",
" W = rs.standard_normal((d, d)) * 0.1; U = rs.standard_normal((d, d)) * 0.1; v = rs.standard_normal(d)\n",
" ctx, w = bahdanau(s, h, W, U, v)\n",
" # independent torch reference computing the identical formula\n",
" st, ht = torch.tensor(s).float(), torch.tensor(h).float()\n",
" Wt, Ut, vt = torch.tensor(W).float(), torch.tensor(U).float(), torch.tensor(v).float()\n",
" e_ref = (torch.tanh(st @ Wt.T + ht @ Ut.T) @ vt)\n",
" w_ref = torch.softmax(e_ref, dim=-1)\n",
" ctx_ref = w_ref @ ht\n",
" assert abs(float(w.sum()) - 1.0) < 1e-6, f\"weights sum to {float(w.sum()):.4f}, expected 1\"\n",
" check_close(w, w_ref.numpy(), atol=1e-5, msg=\"weights disagree with the torch reference\")\n",
" check_close(ctx, ctx_ref.numpy(), atol=1e-5, msg=\"context disagrees with the torch reference\")\n",
"\n",
"check(\"14.6 bahdanau vs torch\", _check_bahdanau)"
]
},
{
"cell_type": "markdown",
"id": "ad45eccf",
"metadata": {},
"source": [
"Hint 1 (conceptual)`W@s` is one vector of size `d`. `U@h_i` is one per source position. Their sum, then `tanh`, then dot with `v`, gives one scalar per source position. Vectorise: `s @ W.T` is `(d,)`, `h @ U.T` is `(T_src, d)`; broadcasting adds the first to every row of the second.\n",
"\n",
"Hint 2 (the lines)\n",
"\n",
"```python\n",
"scores = np.tanh(s @ W.T + h @ U.T) @ v # (T_src,)\n",
"weights = np_softmax(scores) # (T_src,)\n",
"context = weights @ h # (d,)\n",
"```\n",
"\n",
"\n",
"Help — \"shapes (T,d) and (d,d) not aligned\"Use the transposes: `s @ W.T` (not `W @ s`) keeps the row-vector convention so the broadcast with `h @ U.T` lines up. If contexts are close but weights are off, you softmaxed the wrong axis.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "56e83349",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.574569Z",
"iopub.status.busy": "2026-06-10T19:53:18.574441Z",
"iopub.status.idle": "2026-06-10T19:53:18.577153Z",
"shell.execute_reply": "2026-06-10T19:53:18.576818Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 14.6 bahdanau vs torch\n",
"[ ok ] from-scratch NumPy Bahdanau matches the torch reference to 1e-5\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines bahdanau; the check re-verifies it against torch.\n",
"def bahdanau(s, h, W, U, v):\n",
" scores = np.tanh(s @ W.T + h @ U.T) @ v # (T_src,)\n",
" weights = np_softmax(scores) # (T_src,)\n",
" context = weights @ h # (d,)\n",
" return context, weights\n",
"\n",
"check(\"14.6 bahdanau vs torch\", _check_bahdanau, required=True)\n",
"print(\"[ ok ] from-scratch NumPy Bahdanau matches the torch reference to 1e-5\")"
]
},
{
"cell_type": "markdown",
"id": "e1e5417f",
"metadata": {},
"source": [
"### Luong multiplicative attention, and the unified Q/K/V form\n",
"\n",
"Luong, Pham, Manning (2015) replaced the additive score with a *multiplicative* one: a single dot product (optionally with one matrix), `e_i = s^\\top h_i`. Cheaper than the additive form (one matmul, no MLP, no nonlinearity) and trains comparably. This is the operation that became scaled dot-product attention in Vaswani 2017.\n",
"\n",
"Bahdanau and Luong used different vocabularies for the same operation. The unifying names that won are **Query** (what the step asks), **Key** (what each position offers for matching), and **Value** (what each position contributes if matched):\n",
"\n",
"$$\\text{Attention}(Q, K, V) = \\text{softmax}\\!\\left(\\frac{Q K^\\top}{\\sqrt{d_k}}\\right) V$$\n",
"\n",
"In Bahdanau/Luong cross-attention, `Q` is the decoder state and `K = V` are the encoder states. In self-attention, `Q, K, V` are three projections of the same sequence. Same equation, different sources.\n"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "87069b9b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.577994Z",
"iopub.status.busy": "2026-06-10T19:53:18.577924Z",
"iopub.status.idle": "2026-06-10T19:53:18.580815Z",
"shell.execute_reply": "2026-06-10T19:53:18.580471Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"out (2, 5, 32) weights (2, 5, 7)\n",
"[ ok ] scaled dot-product attention shapes check out\n"
]
}
],
"source": [
"# the unified scaled dot-product attention, the operation Ch 15 builds everything from\n",
"def scaled_dot_product_attention(Q, K, V, scale=True):\n",
" \"\"\"Q: (B, T_q, d_k). K, V: (B, T_k, d_k). Returns (out (B,T_q,d_k), weights (B,T_q,T_k)).\"\"\"\n",
" d_k = Q.size(-1)\n",
" scores = Q @ K.transpose(-2, -1) # (B, T_q, T_k)\n",
" if scale:\n",
" scores = scores / (d_k ** 0.5)\n",
" weights = torch.softmax(scores, dim=-1)\n",
" return weights @ V, weights\n",
"\n",
"# shape smoke test before trusting it (lucidrains habit)\n",
"Q = torch.randn(2, 5, 32); K = torch.randn(2, 7, 32); V = torch.randn(2, 7, 32)\n",
"out, w = scaled_dot_product_attention(Q, K, V)\n",
"print(\"out\", tuple(out.shape), \"weights\", tuple(w.shape))\n",
"assert out.shape == (2, 5, 32) and w.shape == (2, 5, 7)\n",
"print(\"[ ok ] scaled dot-product attention shapes check out\")"
]
},
{
"cell_type": "markdown",
"id": "fd3241d0",
"metadata": {},
"source": [
"> **Interpretation.** Q has 5 queries, K/V have 7 keys, and the output is one `d_k`-vector per query (`(2, 5, 32)`) with a `(2, 5, 7)` weight matrix: for each query, a distribution over the 7 keys. The `falsification` is worth noting too: this same function with `mask` would zero out forbidden positions, which is exactly the causal mask Ch 15 adds.\n"
]
},
{
"cell_type": "markdown",
"id": "288812ea",
"metadata": {},
"source": [
"### A deliberate failure: dot-product attention at large `d_k`\n",
"\n",
"The `/ sqrt(d_k)` looks like a cosmetic constant. It is not. The dot product of two random `d_k`-vectors has variance proportional to `d_k`, so at large `d_k` the scores spread out, the softmax saturates toward a one-hot, and the gradient through it vanishes. Watch it break: we compute attention *without* the scaling at growing `d_k` and measure the entropy of the attention weights (high entropy = spread out and healthy, near-zero = collapsed to one key).\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "dc26fe40",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.581502Z",
"iopub.status.busy": "2026-06-10T19:53:18.581436Z",
"iopub.status.idle": "2026-06-10T19:53:18.584898Z",
"shell.execute_reply": "2026-06-10T19:53:18.584646Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"UNSCALED (broken):\n",
" d_k= 4 attention entropy = 2.529 nats (max possible 4.159)\n",
" d_k= 64 attention entropy = 0.582 nats (max possible 4.159)\n",
" d_k= 1024 attention entropy = 0.156 nats (max possible 4.159)\n"
]
}
],
"source": [
"# the BROKEN run: unscaled dot product, entropy collapses as d_k grows\n",
"torch.manual_seed(SEED)\n",
"print(\"UNSCALED (broken):\")\n",
"for d_k in [4, 64, 1024]:\n",
" q = torch.randn(1, 1, d_k); k = torch.randn(1, 64, d_k); v = torch.randn(1, 64, d_k)\n",
" _, w = scaled_dot_product_attention(q, k, v, scale=False)\n",
" print(f\" d_k={d_k:5d} attention entropy = {entropy(w.squeeze().numpy()):.3f} nats\"\n",
" f\" (max possible {math.log(64):.3f})\")"
]
},
{
"cell_type": "markdown",
"id": "2b8822b2",
"metadata": {},
"source": [
"> **Interpretation.** At `d_k=4` the entropy is near the uniform maximum (`ln 64 ≈ 4.16`): attention is spread across keys. At `d_k=1024` the entropy collapses toward 0: the softmax put almost all its mass on a single key, before the model learned anything. A near-one-hot attention has near-zero gradient, so this model would barely train. Now the fix.\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "91f114a8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.585615Z",
"iopub.status.busy": "2026-06-10T19:53:18.585548Z",
"iopub.status.idle": "2026-06-10T19:53:18.588809Z",
"shell.execute_reply": "2026-06-10T19:53:18.588524Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"SCALED by 1/sqrt(d_k) (fixed):\n",
" d_k= 4 attention entropy = 3.512 nats\n",
" d_k= 64 attention entropy = 3.729 nats\n",
" d_k= 1024 attention entropy = 3.811 nats\n",
"\n",
"[ ok ] scaling keeps attention entropy high at every d_k — this is why transformers divide by sqrt(d_k)\n"
]
}
],
"source": [
"# the FIXED run: divide by sqrt(d_k), entropy stays healthy at every scale\n",
"torch.manual_seed(SEED)\n",
"print(\"SCALED by 1/sqrt(d_k) (fixed):\")\n",
"for d_k in [4, 64, 1024]:\n",
" q = torch.randn(1, 1, d_k); k = torch.randn(1, 64, d_k); v = torch.randn(1, 64, d_k)\n",
" _, w = scaled_dot_product_attention(q, k, v, scale=True)\n",
" ent = entropy(w.squeeze().numpy())\n",
" print(f\" d_k={d_k:5d} attention entropy = {ent:.3f} nats\")\n",
" assert ent > 2.0, f\"even scaled, entropy collapsed at d_k={d_k}; the /sqrt(d_k) should keep scores O(1)\"\n",
"print(\"\\n[ ok ] scaling keeps attention entropy high at every d_k — this is why transformers divide by sqrt(d_k)\")"
]
},
{
"cell_type": "markdown",
"id": "7a452df3",
"metadata": {},
"source": [
"> **Common confusion:** this is *not* about the model being untrained. Even with perfectly fine random projections, the unscaled dot product saturates purely from dimension. The single normalisation `/ sqrt(d_k)` is what made dot-product attention robust at the scales transformers wanted; it is the difference between Luong's small-`d` attention and Vaswani's `d_k=64`-per-head attention.\n"
]
},
{
"cell_type": "markdown",
"id": "a627ae10",
"metadata": {},
"source": [
"### Exercise 14.7 — Bahdanau attention as a torch module\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"Implement `BahdanauAttention.forward` (a torch `nn.Module`) for batched inputs: `s` is `(B, d)` and `h` is `(B, T_src, d)`. Return `(context (B, d), weights (B, T_src))`. The check verifies the shapes and that the weights sum to 1, and that for a batch of size 1 it matches your earlier NumPy `bahdanau` when fed the module's own weights.\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "d106e6fe",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.589655Z",
"iopub.status.busy": "2026-06-10T19:53:18.589588Z",
"iopub.status.idle": "2026-06-10T19:53:18.594067Z",
"shell.execute_reply": "2026-06-10T19:53:18.593725Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 14.7 bahdanau module: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class BahdanauAttention(torch.nn.Module):\n",
" def __init__(self, d):\n",
" super().__init__()\n",
" self.W = torch.nn.Linear(d, d, bias=False)\n",
" self.U = torch.nn.Linear(d, d, bias=False)\n",
" self.v = torch.nn.Linear(d, 1, bias=False)\n",
"\n",
" def forward(self, s, h):\n",
" # s: (B, d). h: (B, T_src, d). Return context (B, d), weights (B, T_src).\n",
" # TODO 1: expand s to (B, 1, d) so it broadcasts over the T_src axis of h\n",
" s_exp = None\n",
" # TODO 2: scores = v(tanh(W(s_exp) + U(h))).squeeze(-1) -> (B, T_src)\n",
" scores = None\n",
" # TODO 3: weights = softmax over T_src; context = bmm(weights[:,None,:], h).squeeze(1)\n",
" weights = None\n",
" context = None\n",
" attempted(s_exp, scores, weights, context)\n",
" return context, weights\n",
"\n",
"def _check_ba_module():\n",
" torch.manual_seed(SEED)\n",
" ba = BahdanauAttention(d=16)\n",
" s = torch.randn(3, 16); h = torch.randn(3, 10, 16)\n",
" ctx, w = ba(s, h)\n",
" check_shape(ctx, (3, 16)); check_shape(w, (3, 10))\n",
" check_close_torch(w.sum(-1), torch.ones(3), msg=\"weights must sum to 1 over source positions\")\n",
"\n",
"check(\"14.7 bahdanau module\", _check_ba_module)"
]
},
{
"cell_type": "markdown",
"id": "c041edf4",
"metadata": {},
"source": [
"Hint 1 (conceptual)This is the batched version of Exercise 14.6. The only new mechanics are `unsqueeze(1)` to broadcast the query over source positions, and `torch.bmm` for the batched weighted sum.\n",
"\n",
"Hint 2 (the lines)\n",
"\n",
"```python\n",
"s_exp = s.unsqueeze(1) # (B, 1, d)\n",
"scores = self.v(torch.tanh(self.W(s_exp) + self.U(h))).squeeze(-1) # (B, T_src)\n",
"weights = torch.softmax(scores, dim=-1)\n",
"context = torch.bmm(weights.unsqueeze(1), h).squeeze(1) # (B, d)\n",
"```\n",
"\n",
"\n",
"Help — \"Linear expects input ... got ...\" or weights do not sum to 1`self.W` and `self.U` are `Linear(d, d)`; they act on the last axis, so feed them `(B, *, d)` shapes. If weights do not sum to 1, you softmaxed before squeezing the size-1 score axis, or over the wrong dimension.\n"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "83b4b223",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.594895Z",
"iopub.status.busy": "2026-06-10T19:53:18.594819Z",
"iopub.status.idle": "2026-06-10T19:53:18.598582Z",
"shell.execute_reply": "2026-06-10T19:53:18.598305Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 14.7 bahdanau module\n",
"[ ok ] batched Bahdanau attention: a learnable scorer over source positions\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines BahdanauAttention; the check re-verifies it.\n",
"class BahdanauAttention(torch.nn.Module):\n",
" def __init__(self, d):\n",
" super().__init__()\n",
" self.W = torch.nn.Linear(d, d, bias=False)\n",
" self.U = torch.nn.Linear(d, d, bias=False)\n",
" self.v = torch.nn.Linear(d, 1, bias=False)\n",
"\n",
" def forward(self, s, h):\n",
" s_exp = s.unsqueeze(1) # (B, 1, d)\n",
" scores = self.v(torch.tanh(self.W(s_exp) + self.U(h))).squeeze(-1) # (B, T_src)\n",
" weights = torch.softmax(scores, dim=-1)\n",
" context = torch.bmm(weights.unsqueeze(1), h).squeeze(1) # (B, d)\n",
" return context, weights\n",
"\n",
"check(\"14.7 bahdanau module\", _check_ba_module, required=True)\n",
"print(\"[ ok ] batched Bahdanau attention: a learnable scorer over source positions\")"
]
},
{
"cell_type": "markdown",
"id": "a10c5e53",
"metadata": {},
"source": [
"> **Key takeaways**\n",
"> - Attention scores a query against keys, softmaxes over keys, and weighted-sums the values. The softmax axis is the keys.\n",
"> - Bahdanau additive uses `v . tanh(W s + U h)`; Luong multiplicative uses a dot product; the unified form is `softmax(QKᵀ/√d_k) V`.\n",
"> - The `/ sqrt(d_k)` is not cosmetic: without it the softmax saturates at large `d_k` and gradients vanish.\n",
"> - In cross-attention `Q` is the decoder, `K=V` the encoder; in self-attention all three are projections of one sequence.\n"
]
},
{
"cell_type": "markdown",
"id": "bb76f4e8",
"metadata": {},
"source": [
"## Part 5 — Temperature sampling\n",
"\n",
"> **Objectives.** Turn the trained char-RNN into a generator. Implement temperature-scaled sampling from scratch, and feel temperature move the output from repetitive-and-greedy to high-variance-and-incoherent. This is the qualitative payoff: the model's training shows up as text.\n",
"\n",
"A trained language model is a next-character distribution. To generate, we autoregressively sample: feed the current context, get logits for the next character, sample one, append it, repeat. *Temperature* `T` divides the logits before softmax. `T < 1` sharpens (toward greedy), `T > 1` flattens (toward uniform), `T = 1` samples from the model's own distribution.\n"
]
},
{
"cell_type": "markdown",
"id": "8ca9614f",
"metadata": {},
"source": [
"### Exercise 14.8 — Temperature-scaled sampling\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"Implement `sample(model, start, n_new, temperature)`. Maintain a growing context (a `(1, t)` tensor of IDs), and `n_new` times: forward the context, take the *last* position's logits, divide by `temperature`, softmax, and sample one token with `torch.multinomial`. Append and continue. Return the decoded string.\n",
"\n",
"The checks verify the output length, and a behavioural property: at `temperature → 0` the sampler must become *deterministic* (greedy), so two low-temperature runs from the same seed and start produce the identical string.\n"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "f35d9c57",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.599408Z",
"iopub.status.busy": "2026-06-10T19:53:18.599337Z",
"iopub.status.idle": "2026-06-10T19:53:18.604006Z",
"shell.execute_reply": "2026-06-10T19:53:18.603742Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 14.8 temperature sampler: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"@torch.no_grad()\n",
"def sample(model, start, n_new, temperature=1.0):\n",
" \"\"\"Autoregressively generate n_new characters after `start` (a string).\"\"\"\n",
" model.eval()\n",
" ctx = torch.tensor([encode(start)], dtype=torch.long) # (1, t)\n",
" out_ids = []\n",
" for _ in range(n_new):\n",
" logits, _ = model(ctx) # (1, t, vocab)\n",
" # TODO 1: take the last position's logits -> (1, vocab)\n",
" last = None\n",
" # TODO 2: divide by temperature (guard against 0 with max(temperature, 1e-6)), then softmax over vocab\n",
" probs = None\n",
" attempted(last, probs)\n",
" # TODO 3: sample one id with torch.multinomial(probs, 1) -> (1, 1)\n",
" nxt = torch.multinomial(probs, num_samples=1)\n",
" out_ids.append(int(nxt))\n",
" ctx = torch.cat([ctx, nxt], dim=1) # append and continue\n",
" return start + decode(out_ids)\n",
"\n",
"def _check_sample():\n",
" torch.manual_seed(SAMPLE_SEED)\n",
" s = sample(model, \"The \", 30, temperature=0.8)\n",
" assert len(s) == len(\"The \") + 30, f\"expected {len('The ')+30} chars, got {len(s)}\"\n",
" # behavioural: near-zero temperature is greedy => deterministic across runs\n",
" torch.manual_seed(SAMPLE_SEED); a = sample(model, \"The \", 20, temperature=1e-6)\n",
" torch.manual_seed(SAMPLE_SEED + 1); b = sample(model, \"The \", 20, temperature=1e-6)\n",
" assert a == b, \"at temperature->0 sampling must be greedy and deterministic; it differed across seeds\"\n",
"\n",
"check(\"14.8 temperature sampler\", _check_sample)"
]
},
{
"cell_type": "markdown",
"id": "f7a753ac",
"metadata": {},
"source": [
"Hint 1 (conceptual)Only the last position predicts the next character; the earlier ones are context the GRU already consumed. Index `logits[:, -1, :]`. Temperature divides those logits *before* softmax, so a small temperature makes the largest logit dominate (greedy).\n",
"\n",
"Hint 2 (the lines)\n",
"\n",
"```python\n",
"last = logits[:, -1, :]\n",
"probs = torch.softmax(last / max(temperature, 1e-6), dim=-1)\n",
"nxt = torch.multinomial(probs, num_samples=1)\n",
"```\n",
"\n",
"\n",
"Help — \"multinomial: invalid distribution\" or all-zero probsThat happens if you sampled from logits instead of probabilities, or divided by exactly 0. Softmax the temperature-scaled logits first, and floor the temperature at `1e-6`.\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "9964b207",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.604969Z",
"iopub.status.busy": "2026-06-10T19:53:18.604899Z",
"iopub.status.idle": "2026-06-10T19:53:18.629126Z",
"shell.execute_reply": "2026-06-10T19:53:18.628769Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 14.8 temperature sampler\n",
"[ ok ] temperature sampler: greedy at T->0, the model's own distribution at T=1\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines sample; the check re-verifies it.\n",
"@torch.no_grad()\n",
"def sample(model, start, n_new, temperature=1.0):\n",
" model.eval()\n",
" ctx = torch.tensor([encode(start)], dtype=torch.long)\n",
" out_ids = []\n",
" for _ in range(n_new):\n",
" logits, _ = model(ctx)\n",
" last = logits[:, -1, :] # (1, vocab)\n",
" probs = torch.softmax(last / max(temperature, 1e-6), dim=-1)\n",
" nxt = torch.multinomial(probs, num_samples=1)\n",
" out_ids.append(int(nxt))\n",
" ctx = torch.cat([ctx, nxt], dim=1)\n",
" return start + decode(out_ids)\n",
"\n",
"check(\"14.8 temperature sampler\", _check_sample, required=True)\n",
"print(\"[ ok ] temperature sampler: greedy at T->0, the model's own distribution at T=1\")"
]
},
{
"cell_type": "markdown",
"id": "d2200a3e",
"metadata": {},
"source": [
"### The qualitative payoff: temperature sweep\n",
"\n",
"Now the emotional reward of training. Generate from the trained model at three temperatures from the same start. The text will not be coherent Shakespeare (the model is tiny and trained on 80KB for a minute), but its *character* should change with temperature.\n"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "56086653",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:18.630009Z",
"iopub.status.busy": "2026-06-10T19:53:18.629938Z",
"iopub.status.idle": "2026-06-10T19:53:19.154863Z",
"shell.execute_reply": "2026-06-10T19:53:19.154421Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"--- temperature 0.4 ---\n",
"The corn o' the common patience, / If he had report thou have been the gods and the people, / If is the common motion, I cannot army surfeit of the nobles / To the man'd \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"--- temperature 0.8 ---\n",
"The carion friends and good denery purpose your blood--the people, / We procely to, I knom to sell / in was good nottight there, he would speak. / / BRUTUS: / Say, your good\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"--- temperature 1.3 ---\n",
"The devilous madge and good denery. / / VIRGILIUS: / 'Twornandmen on. His good rememb, he tam; / Ith?-- / PBonsemy, when turth bys? Lart a use spurch to more did best / man'd \n"
]
}
],
"source": [
"N_GEN = 160 if not FAST else 80\n",
"for temp in [0.4, 0.8, 1.3]:\n",
" torch.manual_seed(SAMPLE_SEED)\n",
" out = sample(model, \"The \", N_GEN, temperature=temp)\n",
" print(f\"\\n--- temperature {temp} ---\")\n",
" print(out.replace(\"\\n\", \" / \"))"
]
},
{
"cell_type": "markdown",
"id": "7395608e",
"metadata": {},
"source": [
"> **Interpretation.** Low temperature (0.4) repeats the corpus's most common patterns: lots of spaces, common words, often loops. High temperature (1.3) is more varied but spells less, with odd character runs. Around 0.8 is the usual sweet spot. None of it is coherent here because the model is tiny; the point is that one knob, dividing the logits, slides generation from conservative to chaotic. The full-corpus, longer-trained version (flip `USE_FULL_CORPUS` and raise `STEPS`) produces recognisable pseudo-Shakespeare.\n",
"\n",
"> **Note:** `@torch.no_grad()` and `model.eval()` wrap the sampler. Generation needs no gradients, and `eval()` matters for models with dropout or batchnorm (this GRU has neither, but the habit is free and prevents a whole class of inference bugs).\n"
]
},
{
"cell_type": "markdown",
"id": "21053f35",
"metadata": {},
"source": [
"> **Key takeaways**\n",
"> - Generation is autoregressive: forward, take the last logits, sample, append, repeat.\n",
"> - Temperature divides the logits before softmax; `T→0` is greedy/deterministic, `T→∞` is uniform.\n",
"> - Wrap sampling in `@torch.no_grad()` and `model.eval()`; use a separate seed so it never perturbs training.\n",
"> - A tiny model on 80KB produces texture, not coherence; coherence needs more data, more steps, and (Ch 15) a better architecture.\n"
]
},
{
"cell_type": "markdown",
"id": "588f4efe",
"metadata": {},
"source": [
"## Safety lens\n",
"\n",
"Attention is a learned soft-routing operation, and that has two safety consequences worth a runnable look.\n",
"\n",
"**Adversarial token insertion (the ancestor of prompt injection).** Because attention picks which positions to read, an input token with an extreme key can pull most of the attention mass onto itself, starving the rest of the input. This is structurally the same mechanism as modern prompt injection. We can demonstrate it directly: give one key a large-norm vector and watch it capture the attention distribution.\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "8eafccc2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:19.155968Z",
"iopub.status.busy": "2026-06-10T19:53:19.155861Z",
"iopub.status.idle": "2026-06-10T19:53:19.159815Z",
"shell.execute_reply": "2026-06-10T19:53:19.159432Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"clean: max attention on any one key = 0.491\n",
"attack: max attention on any one key = 0.696 (position 3)\n",
"[ ok ] one anomalous token captured the majority of attention — a hijacking vulnerability\n"
]
}
],
"source": [
"# a high-salience key hijacks attention away from the rest of the input\n",
"torch.manual_seed(SEED)\n",
"q = torch.randn(1, 1, 32)\n",
"k = torch.randn(1, 8, 32)\n",
"v = torch.randn(1, 8, 32)\n",
"_, w_clean = scaled_dot_product_attention(q, k, v)\n",
"k_attack = k.clone()\n",
"k_attack[0, 3] = k_attack[0, 3] / k_attack[0, 3].norm() * 30.0 # inject a large-norm key at position 3\n",
"_, w_attack = scaled_dot_product_attention(q, k_attack, v)\n",
"print(f\"clean: max attention on any one key = {w_clean.max().item():.3f}\")\n",
"print(f\"attack: max attention on any one key = {w_attack.max().item():.3f} (position {int(w_attack.argmax())})\")\n",
"# the property: the attacked attention collapses onto the injected position\n",
"assert w_attack.max() > 0.5, \"a high-salience key should capture >50% of attention mass\"\n",
"print(\"[ ok ] one anomalous token captured the majority of attention — a hijacking vulnerability\")"
]
},
{
"cell_type": "markdown",
"id": "d6ce0bf3",
"metadata": {},
"source": [
"> **Interpretation.** A single injected key seized the majority of the attention mass. In a real system, an attacker who can place text in the input (a retrieved document, a quoted email, a zero-width-character payload) can do the same and make the model effectively ignore the legitimate content. The defenses are input sanitisation at the tokenizer level (strip zero-width and control characters, normalise homoglyphs) and monitoring attention concentration; no defense is universally deployed. Ch 24 builds a scored injection harness; the mechanism is the one you just watched.\n",
"\n",
"**Embeddings encode the biases of their training text.** Word2vec, GloVe, and the token embeddings inside today's LLMs all learn from text statistics, including the text's biases (`programmer - man + woman ≈ homemaker` is the canonical demonstration, Bolukbasi 2016). The detection (the WEAT test) is a 20-line cosine-similarity comparison; the mitigation is harder and mostly involves curating the training text. Audit embeddings for known bias axes before deploying.\n"
]
},
{
"cell_type": "markdown",
"id": "ec0b36bb",
"metadata": {},
"source": [
"## Test yourself\n",
"\n",
"Three parts: concept self-checks with folded answers, two auto-checked problems, and a capstone with a rubric and a folded reference. Every answer is in this notebook; if unsure, re-run that section.\n"
]
},
{
"cell_type": "markdown",
"id": "0c2f5463",
"metadata": {},
"source": [
"### Part A — Concepts\n",
"\n",
"1. What does the temperature parameter divide, and what happens to sampling as `T → 0`? AnswerIt divides the logits before the softmax. As `T → 0` the largest logit dominates entirely, so sampling becomes greedy argmax (deterministic). See Part 5.\n",
"2. Why does attention take the softmax over the *key* axis rather than the query axis? AnswerEach query needs a probability distribution over the source positions it could attend to, so each query's scores must sum to 1. That is a softmax over keys (the last axis of the `(B, T_q, T_k)` score tensor). Softmax over queries gives a meaningless normalisation. Exercise 14.5.\n",
"3. In the experiment-log table printed in Part 3, why is the `NB_FAST` expected val loss (~3.0-3.5) so much higher than the full-run loss (~1.7-2.1)? Is the FAST run broken? AnswerNo. `NB_FAST` runs 60 steps versus 1500, a few percent of the training, so the model is far less converged. Both are correct for their step budget; the gap is expected and documented. This is exactly why the log records both settings.\n",
"4. Why divide the dot product by `sqrt(d_k)`? Point to the entropy numbers you printed. AnswerThe dot product's variance grows with `d_k`, so at large `d_k` the unscaled scores spread out, the softmax saturates to near one-hot, and gradients vanish. In the deliberate-failure cell the unscaled entropy collapsed toward 0 at `d_k=1024`, while the scaled version stayed above 2 nats at every `d_k`. Part 4.\n",
"5. A model has a 256-byte vocabulary; after 50 BPE merges, what is the vocabulary size, and did the sequence get longer or shorter? Answer306 (256 + 50). Each merge adds exactly one new token and replaces every occurrence of a pair, so the sequence gets shorter (or stays equal if the pair vanished). You watched the byte count drop in Exercise 14.2.\n",
"6. What is teacher forcing, and why does it make the char-RNN loss a clean per-position cross-entropy? AnswerAt training time the decoder conditions on the *true* previous characters (present in `x`), not its own predictions. So every position is an independent supervised next-character prediction and the loss is the mean cross-entropy over all positions. At inference the model must instead consume its own predictions, which is the exposure-bias gap. Part 3.\n",
"7. `nn.Embedding(V, d)` is equivalent to which explicit matrix operation, and what does that tell you about its gradient? AnswerOne-hot encode the IDs to `(B, T, V)` and multiply by the `(V, d)` table. Because the one-hot selects single rows, only the looked-up rows of the table receive gradient on a given batch. Exercise 14.3.\n"
]
},
{
"cell_type": "markdown",
"id": "464c8286",
"metadata": {},
"source": [
"### Part B — Auto-checked problems\n",
"\n",
"Two problems that ask you to compute something new with the chapter's pieces. Write the body; the check asserts the property; the solution is folded below.\n"
]
},
{
"cell_type": "markdown",
"id": "a2f2bddf",
"metadata": {},
"source": [
"**Exercise 14.9 — Causal masking** · Difficulty 3/5 · ~12 min\n",
"\n",
"Self-attention for a language model must not let position `t` peek at positions after it. Implement `causal_attention(Q, K, V)`: scaled dot-product attention with an upper-triangular mask that sets scores for future keys to `-inf` *before* the softmax. Shapes: `Q, K, V` are `(B, T, d)` (so `T_q == T_k == T`). The check verifies a behavioural property: perturbing a *future* value must not change an earlier output (causality), which is the exact property Ch 15's masked attention relies on.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "24bdf5c8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:19.160916Z",
"iopub.status.busy": "2026-06-10T19:53:19.160815Z",
"iopub.status.idle": "2026-06-10T19:53:19.165871Z",
"shell.execute_reply": "2026-06-10T19:53:19.165559Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 14.9 causal attention: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def causal_attention(Q, K, V):\n",
" \"\"\"Q, K, V: (B, T, d). Scaled dot-product attention with a causal mask.\"\"\"\n",
" B, T, d = Q.shape\n",
" scores = Q @ K.transpose(-2, -1) / (d ** 0.5) # (B, T, T)\n",
" # TODO 1: build an upper-triangular boolean mask (True where key index > query index)\n",
" # torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)\n",
" mask = None\n",
" attempted(mask)\n",
" # TODO 2: set masked positions to -inf BEFORE softmax (why -inf? so they get 0 weight)\n",
" scores = scores.masked_fill(mask, float(\"-inf\"))\n",
" weights = torch.softmax(scores, dim=-1)\n",
" return weights @ V, weights\n",
"\n",
"def _check_causal():\n",
" torch.manual_seed(SEED)\n",
" Q = torch.randn(1, 6, 8); K = torch.randn(1, 6, 8); V = torch.randn(1, 6, 8)\n",
" out, w = causal_attention(Q, K, V)\n",
" check_shape(out, (1, 6, 8))\n",
" # weights above the diagonal must be exactly 0\n",
" upper = w[0].triu(diagonal=1)\n",
" assert float(upper.abs().sum()) < 1e-6, \"future keys received nonzero weight; the mask is not causal\"\n",
" # behavioural: perturb the LAST value; the FIRST output must be unchanged\n",
" V2 = V.clone(); V2[0, -1] += 100.0\n",
" out2, _ = causal_attention(Q, K, V2)\n",
" assert torch.allclose(out[0, 0], out2[0, 0], atol=1e-5), \\\n",
" \"changing a future token altered an earlier output — attention is leaking from the future\"\n",
"\n",
"check(\"14.9 causal attention\", _check_causal)"
]
},
{
"cell_type": "markdown",
"id": "2025a37c",
"metadata": {},
"source": [
"Hint 1 (conceptual)The mask is upper-triangular above the diagonal: for query `i`, keys `j > i` are the future and must be hidden. `torch.triu(..., diagonal=1)` gives exactly those positions as `True`.\n",
"\n",
"Hint 2 (why -inf)You mask *before* softmax with `-inf` so `exp(-inf) = 0`: the future keys contribute zero weight and the surviving weights still sum to 1. Masking *after* softmax (setting weights to 0) breaks the normalisation.\n",
"\n",
"Solution\n",
"\n",
"```python\n",
"mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)\n",
"```\n",
"The rest is already written for you in the stub.\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "61c5a1e6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:19.166956Z",
"iopub.status.busy": "2026-06-10T19:53:19.166860Z",
"iopub.status.idle": "2026-06-10T19:53:19.170408Z",
"shell.execute_reply": "2026-06-10T19:53:19.170098Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 14.9 causal attention\n",
"[ ok ] causal attention: earlier outputs are provably independent of future tokens\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines causal_attention; the check re-verifies it.\n",
"def causal_attention(Q, K, V):\n",
" B, T, d = Q.shape\n",
" scores = Q @ K.transpose(-2, -1) / (d ** 0.5)\n",
" mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)\n",
" scores = scores.masked_fill(mask, float(\"-inf\"))\n",
" weights = torch.softmax(scores, dim=-1)\n",
" return weights @ V, weights\n",
"\n",
"check(\"14.9 causal attention\", _check_causal, required=True)\n",
"print(\"[ ok ] causal attention: earlier outputs are provably independent of future tokens\")"
]
},
{
"cell_type": "markdown",
"id": "28bd0e06",
"metadata": {},
"source": [
"**Exercise 14.10 — Length-normalised sequence scoring** · Difficulty 2/5 · ~10 min\n",
"\n",
"Beam search ranks candidate sequences by total log-probability, but raw log-prob sums systematically prefer *short* sequences (every token adds a negative number). The standard fix (Wu et al. 2016) divides by `length ** alpha`. Implement `score_sequence(log_probs, alpha)` returning the length-normalised score `sum(log_probs) / (len ** alpha)`. The check verifies a behavioural property: with `alpha=0` (raw sum) a shorter, less-confident sequence outranks a longer, *more*-confident one, but with `alpha=1` (the per-token mean) the more-confident longer sequence wins. That reversal is exactly the short-sequence bias the normalisation corrects.\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "c078305a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:19.171487Z",
"iopub.status.busy": "2026-06-10T19:53:19.171389Z",
"iopub.status.idle": "2026-06-10T19:53:19.174750Z",
"shell.execute_reply": "2026-06-10T19:53:19.174445Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 14.10 length-normalised scoring: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def score_sequence(log_probs, alpha):\n",
" \"\"\"log_probs: list of per-token log-probabilities. Return length-normalised score.\"\"\"\n",
" # TODO: sum(log_probs) divided by len(log_probs) ** alpha; return -inf-safe for empty input\n",
" result = None\n",
" attempted(result)\n",
" return result\n",
"\n",
"def _check_beam():\n",
" short = [math.log(0.5), math.log(0.5)] # 2 tokens, lower per-token confidence\n",
" long_ = [math.log(0.7)] * 5 # 5 tokens, higher per-token confidence\n",
" # alpha=0: raw sum favours the shorter sequence (fewer negative terms) — the documented bias\n",
" assert score_sequence(short, 0.0) > score_sequence(long_, 0.0), \\\n",
" \"with alpha=0 the raw sum should favour the shorter sequence (the short-sequence bias)\"\n",
" # alpha=1: the per-token mean favours the more-confident longer sequence — bias corrected\n",
" assert score_sequence(long_, 1.0) > score_sequence(short, 1.0), \\\n",
" \"with alpha=1 (per-token mean) the more-confident longer sequence should win\"\n",
"\n",
"check(\"14.10 length-normalised scoring\", _check_beam)"
]
},
{
"cell_type": "markdown",
"id": "c55e5bc0",
"metadata": {},
"source": [
"Hint 1Two operations: sum the list, then divide by `len(log_probs) ** alpha`. With `alpha=0` the denominator is 1 (raw sum); larger `alpha` rewards length.\n",
"\n",
"Solution\n",
"\n",
"```python\n",
"def score_sequence(log_probs, alpha):\n",
" if not log_probs:\n",
" return float(\"-inf\")\n",
" return sum(log_probs) / (len(log_probs) ** alpha)\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "552d6b05",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:19.175735Z",
"iopub.status.busy": "2026-06-10T19:53:19.175651Z",
"iopub.status.idle": "2026-06-10T19:53:19.177866Z",
"shell.execute_reply": "2026-06-10T19:53:19.177493Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 14.10 length-normalised scoring\n",
"[ ok ] length normalisation: alpha trades off sequence length against confidence\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines score_sequence; the check re-verifies it.\n",
"def score_sequence(log_probs, alpha):\n",
" if not log_probs:\n",
" return float(\"-inf\")\n",
" return sum(log_probs) / (len(log_probs) ** alpha)\n",
"\n",
"check(\"14.10 length-normalised scoring\", _check_beam, required=True)\n",
"print(\"[ ok ] length normalisation: alpha trades off sequence length against confidence\")"
]
},
{
"cell_type": "markdown",
"id": "28f4cc0f",
"metadata": {},
"source": [
"### Part C — Capstone: a self-attention layer on the Shakespeare batch\n",
"\n",
"Assemble the chapter's pieces into one self-attention layer and run it on a real batch from the corpus. This is the bridge to Ch 15: there, you stack this twelve times into a transformer.\n",
"\n",
"**Deliverables:**\n",
"1. A `SelfAttention` module: project a batch `(B, T, d_model)` into `Q, K, V` with three `nn.Linear(d_model, d_k)` layers, apply *causal* scaled dot-product attention (reuse Exercise 14.9's masking logic), return `(B, T, d_k)`.\n",
"2. Run it on the embeddings of one real `get_batch(\"train\")` batch (embed with a fresh `nn.Embedding(vocab_size, d_model)`), and confirm the output shape and that attention weights are lower-triangular.\n",
"3. Report the per-head parameter count and one sentence on how this differs from the GRU you trained (no recurrence, all positions in parallel).\n",
"\n",
"**Self-assessment (pass / partial / fail):**\n",
"- (a) `SelfAttention` returns `(B, T, d_k)` on a real batch.\n",
"- (b) The attention weight matrix is lower-triangular (causal): no weight above the diagonal.\n",
"- (c) `Q, K, V` are three *separate* projections of the same input (self-attention), not the encoder/decoder split of cross-attention.\n",
"- (d) You divide by `sqrt(d_k)` and can state why (Part 4).\n",
"- (e) You can name what Ch 15 adds on top (multiple heads, stacking, positional encodings, an FFN).\n",
"\n",
"My solution (reference)\n",
"\n",
"```python\n",
"class SelfAttention(torch.nn.Module):\n",
" def __init__(self, d_model, d_k):\n",
" super().__init__()\n",
" self.to_q = torch.nn.Linear(d_model, d_k, bias=False)\n",
" self.to_k = torch.nn.Linear(d_model, d_k, bias=False)\n",
" self.to_v = torch.nn.Linear(d_model, d_k, bias=False)\n",
"\n",
" def forward(self, x): # x: (B, T, d_model)\n",
" Q, K, V = self.to_q(x), self.to_k(x), self.to_v(x) # each (B, T, d_k)\n",
" return causal_attention(Q, K, V) # reuse Exercise 14.9\n",
"\n",
"torch.manual_seed(SEED)\n",
"d_model, d_k = 32, 16\n",
"emb_c = torch.nn.Embedding(vocab_size, d_model)\n",
"sa = SelfAttention(d_model, d_k)\n",
"xb_c, _ = get_batch(\"train\", torch.Generator().manual_seed(SEED))\n",
"x = emb_c(xb_c[:, :16]) # (B, 16, d_model), trim T for a small demo\n",
"out, w = sa(x)\n",
"print(\"self-attention out:\", tuple(out.shape), \"weights:\", tuple(w.shape))\n",
"assert out.shape == (xb_c.shape[0], 16, d_k)\n",
"assert float(w[0].triu(diagonal=1).abs().sum()) < 1e-6 # lower-triangular\n",
"print(f\"params per self-attention head: {param_count(sa):,}\")\n",
"print(\"Unlike the GRU, this reads all T positions in one matmul with no recurrence; \"\n",
" \"Ch 15 stacks it with multiple heads, positional encodings, and an FFN.\")\n",
"```\n",
"The reference catches the lesson: self-attention is the same `softmax(QKᵀ/√d_k)V` you built, with `Q, K, V` all projections of one sequence and a causal mask. That is the entire core of a transformer block.\n"
]
},
{
"cell_type": "markdown",
"id": "31b9bba1",
"metadata": {},
"source": [
"## Reflection\n",
"\n",
"Write ~150 words, in the cell below or on paper, about the dumbest bug you hit in this notebook and how you found it. The strong candidates: softmaxing over the query axis instead of the keys (Exercise 14.5), forgetting `dtype=torch.long` so the embedding refused to index, masking *after* softmax so the weights stopped summing to 1, or sampling from logits instead of probabilities. State what symptom you saw, what you suspected, the one print or assert that confirmed it, and the fix. Nobody grades this. Writing it is the point: the named-bug muscle is what makes the next debugging session fast.\n"
]
},
{
"cell_type": "markdown",
"id": "0efc7860",
"metadata": {},
"source": [
"*Your reflection (double-click to edit):*\n",
"\n",
"...\n"
]
},
{
"cell_type": "markdown",
"id": "2c096d15",
"metadata": {},
"source": [
"## Going further\n",
"\n",
"- Jay Alammar, *Visualizing A Neural Machine Translation Model* — the canonical pre-transformer attention explainer, with the alignment-heatmap intuition this notebook's attention is built on.\n",
"- Lilian Weng, *Attention? Attention!* (2018) — a thorough survey from Bahdanau through self-attention and the variants we skipped.\n",
"- *Dive into Deep Learning*, the Bahdanau-attention and attention-scoring-functions sections — the same math with full runnable code; pair with this notebook.\n",
"- Karpathy, *minbpe* and *The spelled-out intro to language modeling* — the BPE and char-LM lineage this notebook compresses.\n",
"- Bahdanau, Cho, Bengio (2014) and Luong, Pham, Manning (2015) — the two original attention papers, both readable in an afternoon.\n",
"\n",
"## What this enables\n",
"\n",
"- **Ch 15 — Transformers from Scratch**: the `softmax(QKᵀ/√d_k)V` and the causal mask you built here *are* the transformer's core. Ch 15 drops the RNN entirely, stacks self-attention with FFNs and LayerNorms, and trains on this same tiny_shakespeare corpus. Concretely: our char-RNN reached val ~1.7-2.1; a small transformer on the same data reaches noticeably lower.\n",
"- **Ch 16 — Multimodal Transformers**: cross-attention (the Bahdanau form, `Q` from one stream, `K=V` from another) is what lets a model condition vision on text.\n",
"- **Ch 22 — Mech-Interp**: the alignment-as-interpretation intuition (read the attention weights to see what the model attends to) is where transformer interpretability starts.\n",
"\n",
"One concrete forward teaser: the char-RNN reads positions strictly left to right through a single hidden state, so a character 60 steps back must survive 60 GRU updates to influence the prediction. Self-attention lets position `t` read position `t-60` in one matmul. Run the next cell to feel the difference in path length.\n"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "6f039fd9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:53:19.178611Z",
"iopub.status.busy": "2026-06-10T19:53:19.178533Z",
"iopub.status.idle": "2026-06-10T19:53:19.180616Z",
"shell.execute_reply": "2026-06-10T19:53:19.180164Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"RNN: information from position 0 reaches position 63 through 63 sequential GRU updates\n",
"self-attention: position 63 reads position 0 in 1 matmul (constant path length)\n",
"That constant path length, plus parallelism across positions, is why Ch 15 drops the RNN.\n"
]
}
],
"source": [
"# the forward gap: path length from a distant token to the current prediction\n",
"T = 64\n",
"print(f\"RNN: information from position 0 reaches position {T-1} through {T-1} sequential GRU updates\")\n",
"print(f\"self-attention: position {T-1} reads position 0 in 1 matmul (constant path length)\")\n",
"print(\"That constant path length, plus parallelism across positions, is why Ch 15 drops the RNN.\")"
]
},
{
"cell_type": "markdown",
"id": "cc40c06a",
"metadata": {},
"source": [
"---\n",
"*Built top-to-bottom. If every check above printed `[ ok ]`, you reproduced the chapter. Total running time and last-verified stamp written by CI.*\n",
"\n",
"Total running time: (CI-written) · Verified on: numpy 2.x, torch 2.x, Python 3.12\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "obvix-nb",
"language": "python",
"name": "obvix-nb"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"obvix": {
"title": "Ch 14 — NLP with RNNs and Attention"
}
},
"nbformat": 4,
"nbformat_minor": 5
}