{
"cells": [
{
"cell_type": "markdown",
"id": "b1a7febc",
"metadata": {},
"source": [
"# Ch 16: Multimodal Transformers (notebook)\n",
"\n",
"`[<- 15 transformers-from-scratch]` · **this notebook** · `[17 efficient-inference ->]`\n",
"\n",
"Runs top-to-bottom in ~6 min on free Colab CPU. Last verified 2026-06-11.\n",
"\n",
"**What you'll build**\n",
"- A Vision Transformer from scratch: a `PatchEmbedding` that is one `Conv2d`, a learned `[CLS]` token, learned position embeddings, and the same encoder `Block` you wrote in Ch 15. You prove the conv-as-patch-projection equivalence with an assert, then run a `randn` smoke test after every module.\n",
"- A tiny ViT trained on FashionMNIST, complete with a deliberate broken training loop (gradients never zeroed) that you diagnose from the loss curve and fix.\n",
"- CLIP's contrastive loss from eight lines, derived as symmetric softmax cross-entropy on a similarity matrix, checked against `F.cross_entropy` and recovered on synthetic ground-truth-aligned embeddings.\n",
"- A zero-shot classifier built out of a frozen embedding space, and the cross-attention fusion primitive (`Q <- modality A, K,V <- modality B`) that wires text into every multimodal model.\n",
"\n",
"**How this notebook works.** Code cells with a `# TODO` are yours to fill in. Run the cell to grade yourself: `[ ok ]` passed, `[FAIL]` shows what went wrong, `[ -- ]` means not attempted yet. Every exercise has a hint ladder (open only as many as you need) and a folded solution below it. The notebook runs top-to-bottom even if you fill in nothing: the solution cells redefine the pieces so later cells work. See Ch 00 for the full protocol.\n"
]
},
{
"cell_type": "markdown",
"id": "671010f2",
"metadata": {},
"source": [
"## Before you start\n",
"\n",
"1. A ViT cuts a `(3, 224, 224)` image into non-overlapping `16x16` patches. How many patches, and what is each patch's flattened dimension? Answer`(224/16)^2 = 14^2 = 196` patches. Each patch is `3*16*16 = 768` numbers flattened. That is why ViT-B/16 has 196 tokens of `d_model=768`. \"An image is worth 16x16 words\", and the word count is 196.\n",
"2. CLIP trains an image encoder and a text encoder so their outputs agree. What is the loss, in one phrase? AnswerSymmetric softmax cross-entropy on the image-text similarity matrix: the diagonal (correct pairs) should be the largest entry in its row and its column. No caption generation, no masked LM. One dot product, one cross-entropy.\n",
"3. Predict before you run: a ViT block uses the exact same attention as a GPT block from Ch 15, with one thing removed. What is removed, and why? AnswerThe causal mask. Vision is not autoregressive: every patch may attend to every other patch in both directions. Drop the triangular mask and the GPT block becomes a ViT block.\n"
]
},
{
"cell_type": "markdown",
"id": "d7540ef9",
"metadata": {},
"source": [
"## Setup\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "d38a846f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:29.569936Z",
"iopub.status.busy": "2026-06-10T20:02:29.569848Z",
"iopub.status.idle": "2026-06-10T20:02:31.005830Z",
"shell.execute_reply": "2026-06-10T20:02:31.005275Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"numpy 2.2.6 . torch 2.12.0+cpu . torchvision 0.27.0+cpu\n"
]
}
],
"source": [
"import numpy as np\n",
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import matplotlib.pyplot as plt\n",
"import torchvision\n",
"print(f\"numpy {np.__version__} . torch {torch.__version__} . torchvision {torchvision.__version__}\")\n",
"if np.__version__ < \"2.0\":\n",
" print(\"WARN: written for NumPy 2.x; older versions may differ slightly\")\n",
"if torch.__version__ < \"2.0\":\n",
" print(\"WARN: written for modern torch (2.x); older versions may differ slightly\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a15938cc",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.007183Z",
"iopub.status.busy": "2026-06-10T20:02:31.007051Z",
"iopub.status.idle": "2026-06-10T20:02:31.012958Z",
"shell.execute_reply": "2026-06-10T20:02:31.012518Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"device cpu . FAST=False\n"
]
}
],
"source": [
"import os, math, random\n",
"SEED = 0\n",
"FAST = bool(os.environ.get('NB_FAST')) # CI smoke mode: ~10x fewer steps, same code paths\n",
"device = 'cuda' if torch.cuda.is_available() else 'cpu' # CPU is the canonical target\n",
"rng = np.random.default_rng(SEED)\n",
"torch.manual_seed(SEED); random.seed(SEED)\n",
"print(f'device {device} . FAST={FAST}')\n",
"\n",
"# ── house self-check harness (identical across all chapter notebooks) ──\n",
"import numpy as _np\n",
"\n",
"def check(label, test_fn, required=False):\n",
" \"\"\"Run one self-check. test_fn raises AssertionError (with a teaching\n",
" message) on failure, NotImplementedError if the stub is unfilled.\n",
" required=True is used only in solution cells; it is what CI grades.\"\"\"\n",
" try:\n",
" test_fn()\n",
" except NotImplementedError:\n",
" if required:\n",
" raise AssertionError(f\"{label}: reference solution incomplete\")\n",
" print(f\"[ -- ] {label}: not attempted yet — fill in the TODO above, then re-run.\")\n",
" return False\n",
" except AssertionError as e:\n",
" if required:\n",
" raise\n",
" print(f\"[FAIL] {label}: {e}\")\n",
" return False\n",
" print(f\"[ ok ] {label}\")\n",
" return True\n",
"\n",
"def attempted(*vals):\n",
" \"\"\"Treat None placeholders as 'not attempted'.\"\"\"\n",
" if any(v is None for v in vals):\n",
" raise NotImplementedError\n",
"\n",
"def check_shape(x, want):\n",
" assert tuple(x.shape) == tuple(want), \\\n",
" f\"shape {tuple(x.shape)}, expected {tuple(want)} — check your reshape/transpose order\"\n",
"\n",
"def check_close(got, want, atol=1e-5, rtol=1e-4, msg=\"\"):\n",
" g, w = _np.asarray(got, dtype=float), _np.asarray(want, dtype=float)\n",
" assert g.shape == w.shape, f\"shape {g.shape} vs expected {w.shape}. {msg}\"\n",
" bad = ~_np.isclose(g, w, atol=atol, rtol=rtol)\n",
" assert not bad.any(), \\\n",
" f\"{bad.mean():.2%} of values wrong (max diff {abs(g - w).max():.3g}). {msg}\"\n",
"\n",
"\n",
"def param_count(module):\n",
" \"Trainable parameter count, printed after every module init (spec tier-2 convention).\"\n",
" return sum(p.numel() for p in module.parameters() if p.requires_grad)\n",
"\n",
"\n",
"def layer_summary(module, x):\n",
" \"Dummy forward that prints the output shape, the d2l layer_summary habit.\"\n",
" with torch.no_grad():\n",
" y = module(x)\n",
" print(f\"{module.__class__.__name__:18} {tuple(x.shape)} -> {tuple(y.shape)}\")\n",
" return y"
]
},
{
"cell_type": "markdown",
"id": "7e37159b",
"metadata": {},
"source": [
"> **Note:** seeds make this notebook's printed numbers reproduce on CPU. Library versions and BLAS threading can shift the last digit or two; quoted numbers hold for the pinned environment. If your loss is 1.034 and the page says 1.031, you did nothing wrong. GPU is never the canonical path here: the one finetune cell that wants CUDA prints why it skipped and runs nothing on CPU.\n"
]
},
{
"cell_type": "markdown",
"id": "031c764b",
"metadata": {},
"source": [
"## The map\n",
"\n",
"> **Part 1 - Patches as tokens.** Build `PatchEmbedding` (one `Conv2d`), prove it equals flatten-then-linear, add the `[CLS]` token and position embeddings, and reuse the Ch 15 encoder block with the causal mask removed. A `randn` smoke test follows every module.\n",
"> **Part 2 - A tiny ViT, trained and broken.** Assemble the full ViT, train it on FashionMNIST, then watch a version that forgets to zero its gradients fail, diagnose it from the loss curve, and fix it. Experiment log included.\n",
"> **Part 3 - CLIP's shared space.** Derive the contrastive loss as symmetric softmax cross-entropy on a similarity matrix, check it against `F.cross_entropy`, and recover the diagonal on synthetic aligned embeddings.\n",
"> **Part 4 - Zero-shot and cross-attention.** Turn a frozen embedding space into an open-vocabulary classifier, then implement the `Q <- A, K,V <- B` cross-attention primitive that fuses every multimodal model. GPU-fenced finetune appendix closes the part.\n"
]
},
{
"cell_type": "markdown",
"id": "cb99ae1e",
"metadata": {},
"source": [
"## Part 1 - Patches as tokens\n",
"\n",
"> **Objectives**\n",
"> - Implement `PatchEmbedding` as a single strided `Conv2d` and prove it equals flatten-each-patch-then-linear.\n",
"> - Prepend a learned `[CLS]` token and add learned position embeddings.\n",
"> - Reuse the Ch 15 transformer encoder block (no causal mask) and confirm its shape with a `randn` smoke test.\n",
"\n",
"A Vision Transformer sees an image the way a language model sees a paragraph. Cut the image into non-overlapping patches, flatten each patch to a vector, add a position embedding, hand the sequence to the same `nn.Module` you trained on Tiny Shakespeare. The model does not know it is looking at pixels. It attends to tokens. That is the whole trick, and the rest of multimodal modeling is choosing how to project each modality into one shared residual stream.\n"
]
},
{
"cell_type": "markdown",
"id": "b93d177e",
"metadata": {},
"source": [
"### 1.1 The anchor dataset: FashionMNIST patches\n",
"\n",
"FashionMNIST is 70,000 grayscale `28x28` images in 10 clothing classes. It downloads to `data/` once and is a no-op on re-run (idempotent, the Géron cached-fetch pattern). We use a small patch size of `7`, which cuts each `28x28` image into `4x4 = 16` patches of `7x7` pixels. Sixteen tokens per image, not 196, so everything runs on CPU in seconds.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "193a4a6c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.013980Z",
"iopub.status.busy": "2026-06-10T20:02:31.013899Z",
"iopub.status.idle": "2026-06-10T20:02:31.061948Z",
"shell.execute_reply": "2026-06-10T20:02:31.061553Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"train 60000 . test 10000 . image shape (1, 28, 28)\n"
]
}
],
"source": [
"# Recap: FashionMNIST is the house vision dataset for Ch 10/12/16/18 (spec §5).\n",
"from torchvision import transforms\n",
"from torch.utils.data import DataLoader, Subset\n",
"\n",
"tfm = transforms.ToTensor() # uint8 [0,255] HxW -> float [0,1] (1,28,28)\n",
"train_full = torchvision.datasets.FashionMNIST(root=\"data\", train=True, download=True, transform=tfm)\n",
"test_full = torchvision.datasets.FashionMNIST(root=\"data\", train=False, download=True, transform=tfm)\n",
"CLASS_NAMES = [\"T-shirt\", \"Trouser\", \"Pullover\", \"Dress\", \"Coat\",\n",
" \"Sandal\", \"Shirt\", \"Sneaker\", \"Bag\", \"Ankle-boot\"]\n",
"print(f\"train {len(train_full)} . test {len(test_full)} . image shape {tuple(train_full[0][0].shape)}\")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "7904f19e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.062915Z",
"iopub.status.busy": "2026-06-10T20:02:31.062839Z",
"iopub.status.idle": "2026-06-10T20:02:31.331020Z",
"shell.execute_reply": "2026-06-10T20:02:31.330474Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3kAAAGBCAYAAADbrniIAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAXT1JREFUeJzt3XlgVeWd//FvIOyQBAhr2NcAIUQ2BZFdVBQXXLBaxXFfWrXjQttxOtpq62inarWjdrTouNeCilZEpbiVTQTKvovsayAJSQgJ3N8f/Rl1zveD98ANCYf36y/98HDuufc+59zz5OZ8SIrFYjEDAAAAAERCtcreAQAAAABA4rDIAwAAAIAIYZEHAAAAABHCIg8AAAAAIoRFHgAAAABECIs8AAAAAIgQFnkAAAAAECEs8gAAAAAgQljkAQAAAECEsMiLU05OjuXk5Fj37t2tevXq5f8/bty4uP7+vffea7fffrv7Z1OmTLGf/OQn8u+++eabNnv27CPZbZzgjnbeApWpXbt21rVr1/I5/Ic//OGw45977jk7//zzzczso48+spycnIrfSeAIfT2/e/XqZZ06dbLzzjvPZs6cWdm7BRyRsrIyu++++ywzM9OysrIsJyfHrr/+etu7d+8RbY9r36OXXNk7cLxYuHChmZmtX7/ecnJyyv8/Ec4991w799xz3T8rKyuzN99803JycuyUU05J2GPixBDPvC0rK7Pk5Mo5FVTmY+P48Nprr1lOTo599dVXlp2dbaeddpplZ2dX9m59B/MYR+rr+W1mNnnyZBs9erRNmzbNTj755PIxhw4dMjOzatX4uTyqrmuuucZyc3Nt1qxZ1rBhQ4vFYvaXv/zFcnNzLS0tLfT2uPY9epwxEmz16tV26qmnWq9evaxnz552zz33lP/Z1q1bbcyYMda9e3cbPny45ebmmlnwp889evSwa665xnJycuyll16yKVOm2MMPP2w5OTn2zDPPVMbTQsS0a9fOJkyYYP3797fx48fbvn377Oqrr7asrCzLysqy++67r3zs0KFD7c033yz//4suusiee+45MzN75plnrHv37paTk2M9e/a0OXPmmNk/j4Ozzz7b+vXrZ9nZ2fbEE0+U//2kpCT7j//4D+vXr5/97Gc/OybPF8e/tm3bWteuXe2yyy77zm9FPPHEE3bVVVd9799/4YUXLDs727Kzs+3ss8+2zZs3m5lZly5dbN68eeXjnnvuObvgggvMzGzbtm12ySWXWP/+/QPn8/97DAFHa+zYsXbjjTfab3/7W7v33nvtwgsvtDPOOMOysrJs69atNm3aNBs0aJD16dPH+vfvbzNmzDAzfd3x9ttvW3Z2tuXk5FhWVpa99dZblfn0EGFr1qyx119/3SZOnGgNGzY0s39+1l988cXWoUMHe/jhh61Hjx7Ws2dPu/zyyy0vL8/MzKZPn24DBgywk046yXr06GHPPvusmZm9++67XPsmAD96TLAnnnjCzjnnnPKL168XcmZmc+bMsS+++MIaN25sl156qT399NPuRe7y5cvtv//7v8sn+4wZMywnJ0f+uidwJHbv3m1z5syxpKQkmzBhgpWUlNiiRYusuLjYBg0aZJmZmd/7a5133HGHrVixwlq0aGGlpaVWUlJiBw8etB/84Af24osvWmZmphUVFdkpp5xiJ598svXr18/MzKpXr26ff/75sXiaiIjFixfbihUrbMyYMaH/7pIlS+yuu+6yL774wjIyMuyBBx6wa6+91qZOnWpXXXWVPffcc9a3b18zM5s4caLdeeedZmY2fvx4+/nPf25DhgyxsrIyO+ecc+z111+3iy++2My+ewwBiXDyySfblClTrEePHjZr1ixbsGCBNWvWzNatW2f33nuvTZs2zVJSUmzNmjV22mmn2fr16+V1xz333GNPP/20DRgwwA4dOmT5+fmV+dQQYfPnz7fOnTtbenp64M+mTp1qf/rTn2zWrFmWlpZm119/vf30pz+1J5980nr37m2fffaZVa9e3XJzc+2kk06yM844w0aPHm3nnnsu175HiUVegg0ePNjuuusu27dvnw0ZMsRGjhxZ/mdnnnmmNW7c2MzMBgwYYIsXL3a30aFDBxsyZMgx2V+cuK666qryi9MPP/zQ/uu//suqVatm9erVsyuvvNI++OCD713kjRgxwq644gobM2aMnXXWWdalSxdbtmyZLV261C699NLycQUFBbZs2bLyRd7VV19dcU8MkTJu3DirU6eO1a1b1/70pz/ZkiVLQt/jMWPGDDvzzDMtIyPDzMxuvvlm++Uvf2kHDx60K6+80k466ST7r//6L9u8ebOtWrXKzjrrLCssLLTp06fb9u3by7ezb98+W7lyZfn/f/sYAhIhFouV//fo0aOtWbNmZmb23nvv2Zo1a2zw4MHlf16tWjXbsGGDvO4YMWKE3XbbbXbRRRfZqFGjuEcVleLDDz+0cePGlf/K5k033fSdH5Rdc801tmrVKktOTrbdu3fbkiVLrFWrVpW4x9HBIu8o3XrrrfbJJ5+Y2T9/HejCCy+0gQMH2gcffGBPPPGEPfroo/buu++amVnt2rXL/1716tWtrKzM3Wb9+vUrfsdxwjvcPPv2hWtycrIdPHiw/P/3799f/t+TJk2yL774wj766CMbPXq03X///dazZ09r1KjRYe9bZY4jXt++Z8nMbMWKFXI+xuvb87tVq1bWt29fe+utt2zp0qX2wx/+0JKTk8u3O3v27O+cu7+NeYxE+/zzzy0rK8vMvju/YrGYnX766fbyyy8H/k7nzp3d647f/e53tnTpUpsxY4aNHz/eLr/8crv77ruP2XPBiaN37962evVq2717d/mXGcq3z7833nijjR492iZNmmRJSUnWu3fvIzqnw8c9eUfp97//vS1cuNAWLlxoPXv2tNWrV1uzZs3syiuvtIceeighzUApKSnlv78MVISRI0fas88+a7FYzAoLC+2FF16wUaNGmZlZp06dyu+1+/LLL+2zzz4zs3+WTaxdu9b69u1rd955p1100UU2d+5c69q1q6WkpNjEiRPLt79mzZrv/OoycKQ6depk8+bNs4MHD1pRUZFNmjTpe//OsGHD7L333rMtW7aYmdlTTz1lI0aMsOrVq5uZ2b/8y7/Yn/70J/vf//3f8m+Z69evb8OGDbMHH3ywfDtbtmyxTZs2VcCzAszeeuste/LJJ+2OO+4I/NkZZ5xhH374oS1atKg8mzt3rpmZvO5YsWKF9ejRw370ox/ZTTfdRFMhKkynTp3swgsvtGuuuab8Ny1isZhNmjTJOnToYH/+85/Lf1346aefLr++2LNnj7Vt29aSkpLsk08+sX/84x/l2+Ta9+jxTV6C/eUvf7EXX3zRatasaYcOHbKnnnrqqLd5xRVX2FVXXWVvvvmm3XLLLXbttdcmYE+Bb/z7v/+73XrrrdazZ08zM7v44ovtkksuMTOzu+++28aNG2c9e/a0Hj16lLe+HTx40K6++mrLzc215ORka9KkiU2cONGSk5PtnXfesdtvv90eeeQRO3jwoKWnp7s/gQbCGjt2rL3++uvWrVs3a9WqlZ100klWVFR02L+TlZVlDz/8sJ155plmZta6dWv7n//5n/I/P++88+ymm26yzp07W7du3crzl156yf71X//VsrKyLCkpyerVq2dPP/00v0qEhBk3bpzVrl3bCgsLrXv37vbuu+/aySefbFOnTv3OuE6dOtnLL79sN9xwgxUVFdmBAwfspJNOspdfflled/z85z+3lStXWs2aNa1u3br25JNPVsZTxAniT3/6k91///128sknW3Jysh06dMgGDx5s//mf/2lFRUU2YMAAq1atmmVnZ9t///d/m5nZgw8+aDfffLP96le/spycnO+0ynLte/SSYt/+BXAAAAAAwHGNX9cEAAAAgAhhkQcAAAAAEcIiDwAAAAAihEUeAAAAAEQIizwAAAAAiBAWeQAAAAAQIXH/O3nf/hfqjxX1mBX5rz5kZma6+RNPPBHIXn/9dXfsggUL3PzAgQNuXlpa6uZZWVmB7IILLnDHrl271s0ffvhhN//6H6s83hzNe18Zc7gi9e3b183Hjx/v5rt37w5kBQUF7tiysjI3T09Pd3P1vmzYsCGQ9erVyx3brFkzN2/SpImbDxs2zM2ruqo6hyvjfNu0aVM3Hz58uJt7/06SOpctX77czdV5OC0tzc0HDhwYyNQ/Kv3zn//czYuLi908jMp4f5SqOocrUrt27dx86NChbn7eeee5uXcefvHFF92x8+fPd3N1nXLhhRe6+YgRIwKZ+rcl1b788Y9/dPPjVZTncLVq/vc3hw4dCrWd+vXru3mPHj0CWffu3d2xixcvdvP9+/e7ecuWLd18+/btgezb/3B6PKrSOTQR4tlvvskDAAAAgAhhkQcAAAAAEcIiDwAAAAAihEUeAAAAAEQIizwAAAAAiJCkWJy1MoloE6rIZpucnBw3v/TSS91ctVAdPHjQzevVqxfI6tSp445t3LixmyfCqlWr3Fy1JnXt2tXNvaaiadOmuWN/+9vfuvmSJUvcvCJFuRErrLvuusvNR48e7ebeHGnfvr07tkGDBm6u2jVzc3PdPC8vL5CpNkSvdc7MrFOnTm6u9r2qq+w5nKjzsDcXbrvtNnfsyJEj3bxWrVpuXlhYGPd41TSo5rCiWo43bdoUyLZu3eqOVZ8J6vj45JNPAtnjjz/ujt2zZ4+bV4bKnsOJctZZZwWyn/zkJ+5Y1ZBas2ZNN1ftgd689Jq0zXTj8Pr1691ctSJ789U7N5vpYzIjI8PNp0+fHshuvfVWd2xVEpU5nAjqOlGdQ7t16xbI+vTp44799NNP3VydE1Wbtnc8ee3dZmYLFy5086ihXRMAAAAATjAs8gAAAAAgQljkAQAAAECEsMgDAAAAgAhhkQcAAAAAEXJM2zXDSklJcfP//d//DWTZ2dnu2GrV/HVsQUGBm6tGLK95TTVx1qhRw81TU1PdXDXJeW2IiWgiNTOrXbt2IFPNcKo9TLUmXXHFFUe+Y9+DRqxv3HvvvW7eunVrN/daXxs1auSODftaefNJbSdsu+agQYPc/NRTTw1kqnWuKqnsORy2XbNjx45u/vbbbwcyr7XXLNx51UyfW0tKSgKZammrX7/+UW/bzD//qQa45OTkuLeh8qKiInfsU0895eZvvPGGm1ekyp7DYak57J1D1RyuW7eum6trDNV47TVgqnO2oratcq9JUzVxqmNSHWde66Y6x995551uXhmOtzmcCOo4aNeunZt/9dVXbj527NhAptquX3rpJTdXn9VqH73rg7S0NHes+ryZN2+emx+vaNcEAAAAgBMMizwAAAAAiBAWeQAAAAAQISzyAAAAACBC/LvEq4jJkye7edu2bQPZjh073LHqRmR1g7y6Gdm70VZtQ92Uu2vXLjevXr26m3vUTd5hFRcXBzJ1s6q6uXPw4MFunpmZ6eYrVqyIc+8Qjy5duri5KoXwiijq1avnjlUlAzt37nRzNYe9EiJVqKTmtioy8ubf8VC8UtnCFg785je/cfNt27YFMlXOoN5DtS9hzsOqYEUVqajzXK1atdzcO0ZUOYXab/WY3pxXJS233HKLm3/wwQduvm/fPjc/Ed1xxx1urs5nHnV+UqVTai54+ZdffumO9QpTDveY6npHzW2PKiZS1zteOUdWVpY79uyzz3bzv/71r3HuHY6GKirxzuVm+hy6cePGQKYK9y644AI3V+/5hx9+6ObLly8PZKokyVsjmOlyQe96OCr4Jg8AAAAAIoRFHgAAAABECIs8AAAAAIgQFnkAAAAAECEs8gAAAAAgQqpEu2afPn3cXDXkeC2VqvlJtf6pdqqMjAw399oGVduWal5T+6jarLwmOdVSp5q8CgoK3HzTpk1xb0NR+33ttde6+Z133hlq+zi89PR0N2/QoIGbey2Bqamp7ljVkqiOJ3UsqPZOj2qAU4/ZsGHDuLeN79eiRQs3b968uZt7zX+qGVKdW1SLq5o33jxTjYLq/KRy9Zng7YvahnqearzXgKmaONVrMmbMGDd/5ZVX3PxE9Nxzz7n5T37yk0CmGjdVk58636rrAM+BAwfcXJ3jlfz8fDdPRHug2kfvM8RrXzSjRbMieOfEDh06uGNVE3FOTo6bq/dxy5Ytgaxjx47uWHUcqM8KdQ0+cODAQNamTRt3rNoX77rXzD9XqrHHG77JAwAAAIAIYZEHAAAAABHCIg8AAAAAIoRFHgAAAABECIs8AAAAAIiQKtGuOWzYMDdXbXterhrWVDNfSUmJm0+YMMHNvTYh1b7TsmVLN9+6daubq2ZCr81KvSaqNal3795u/uMf/ziQea2lZroVVL3mF110kZvTrplYqhlTzTOv4a9Hjx7uWNVcqZr/FDW3PUVFRW7utcyamXXv3j3UvuDw1Huu2jW9+aQa01QzpGqjVOc575yj5ofKFfVZ4W1HnfvUY6p2zSZNmgQydR5Wr+3pp5/u5rRrfmPu3LluPmvWrEB27rnnumPnzJnj5urzUTXH7t69O5Cp5ko1F9R5WD2mt4+qidObk4fjPeZPf/rTUNvAkfOaNFu3bu2OVS2ra9ascfPs7Gw3944n1T7brl07Nx88eLCbf/75527ev3//QKbaP//2t7+5uToPn3rqqYFs5cqV7tiFCxe6eVXFN3kAAAAAECEs8gAAAAAgQljkAQAAAECEsMgDAAAAgAhhkQcAAAAAEVIl2jVVG6NqXvNa0FRrTu3atd08Ly/Pzf/nf/7HzUeNGhXIVHPlxIkT3fyGG25w8yVLlrh5o0aNAplqgFPNRo888oib33zzzYFMtYSp11C1IWZmZrp5ly5dAtmqVavcsfiGahps0KCBm6v5VFpaGvfYtLQ0N2/VqpWbq/ZEr8FNzRvVJKdaH1u0aOHmODKqSU2dc7zWTdWmqnLVEui1GZuZrV27NpCtX7/eHVtYWBjqMdV477hRTZfqNTznnHPi3hd17KkGZXXs4fv9/ve/D2S33XabO3bDhg1uvnPnTjdX88k7/xUUFKhddKljUj2m99leo0YNd6zaF9XmPHXq1ECmmjuReN75YseOHXGPNTOLxWJu/v7777u59/6OGTPGHTtt2jQ3V58J06dPd3PvGl8dB40bN3ZzdXx4x4K6vlBNpPv27XPzysY3eQAAAAAQISzyAAAAACBCWOQBAAAAQISwyAMAAACACKkSxSu9evVy840bN7q5d8OmKqdQUlJSQo1/7733Apm6ibN79+5ufuedd7r5G2+84ebejayqHGX+/Plu3qdPHzf3Sm3UDfyq1ObQoUNurm5QHzBgQCCjeOX7eQU8ZvpGX1Vgkp6eHsjUcaPmgnrP69Sp4+YzZ86MexuqaEkVZSQlJbk5jsyrr77q5p9++qmbX3755YEsKyvLHfvrX//azVesWBHn3ml169Z1czUnVa7mvFc8pc79r7zyipv/7Gc/c/PPP/88kDVr1swdqwqLOnTo4Ob4hvrc9M45gwYNcsc+8MADoR5TvV/eY6o5WVxc7Obq+ai8pKQkkKniC0WNf/vtt0NtB0dGzRGvBEp9lqrzljqHNmnSxM29c+JXX33ljlUFP3PmzHFzVbrlXVer56nmqrpm8I4btQ1VPpeIz7KKwDd5AAAAABAhLPIAAAAAIEJY5AEAAABAhLDIAwAAAIAIYZEHAAAAABFyTNs1VfPazp073Vw151SvXj2QqdYc1Ui0e/duN1e8ffcaq8zMWrRo4eaqnUvte2lpadxjvebKw/EajDIyMtyxYds1VSPYaaedFsief/55tYv4/xo2bOjmav6p98Vr4VLb8I4xM7MePXq4+ebNm928TZs2gWz9+vXuWNWimZ+f7+be8YEj99BDD7m5mk8zZswIZAsWLHDHqjZj1UimznPeXFDn8r1797q5mjexWCzufUlNTXXHquNj7dq1bu41lKrWXPU81TGMb6hrCc/WrVvdXL2H7du3d3N1PisoKAhk6hhT21DNf2rueC2JYZsJVXsijg2vHdvMPz+peeO1YpqZ5ebmurlq3/auq9PS0tyx1157bajHVO3C3vNU574wbbpmfoP5gQMHQu0f7ZoAAAAAgArHIg8AAAAAIoRFHgAAAABECIs8AAAAAIgQFnkAAAAAECHHtF1zwoQJbq4aMFVTlNf2qLahWoZUy07fvn3dvHHjxoHMa+QxM6tRo4abq1Ye1fbm7bvXkGimm43GjRvn5l5jo2rFVE1yarzaR/Xa4vBUI1ZRUVGo7XjzskGDBu7YXbt2ublqIFRNht4cbtu2rTtWtQeqY1UdZzgy06ZNc/MRI0a4+YUXXhjIRo0a5Y5VLbo33XSTm6vzWadOnQJZ/fr13bFqrqrmWHXe8lrWVBviiy++6OZeo6KZ/5moWt327Nnj5mPHjnXzgQMHurlqtcPhqdZJdQ5Vc8RrLFQNwmpOqusaNXc8YRpHzcx27NgRajwSK0zTpTonqs/pevXqublqVvfmn7oeOffcc938448/dnPVvu19JqgWTXWOr1u3rpt7jfgLFy50xzZv3tzNqyq+yQMAAACACGGRBwAAAAARwiIPAAAAACKERR4AAAAARAiLPAAAAACIkGParjlz5kw3V201XpOamVlKSkogU+1Aq1evdnPVGjR79mw395qyVHuW2rZq/FENQUlJSXFvWzV/qVa3VatWBTLVPKT2Wz3mli1b3PzNN990cxyemmeq3VTx3q+8vDx3bLdu3UJtWzX/eQ256phs06aNm6tWMTW3cWQefPBBN1ftv95xvnz5cnfsmDFj3PwXv/hFnHun96WkpMQdq86VqnVTtQ165z/V7Kpa7dTxMXfu3EC2bds2d+yMGTPcXB1PtGh+P++cqM63mzZtcvPs7Oy4t23mz1c1J9U8U3NbNTF7nxWqoTM9Pd3NN2/e7OYedU0TttET31CNw95nrGp8VY316hyq5pNHNdxPnz7dzTdu3BjqMb35qsaqlll1LeE1g4Z9TbzrdTN9bB8rfJMHAAAAABHCIg8AAAAAIoRFHgAAAABECIs8AAAAAIgQFnkAAAAAECHHtF3zySefDJU3bNjQzTt37hzIbrrpJnfskCFD3Fw1jy1ZssTN9+7dG8hU85Vqo0wE1eCjmrxUg1ZqamogW7RokTv28ssvj3PvUBHUe6ta4BRvvGqbUu1cytq1a928V69egcxrdjUzKywsdHNvrprphjkcmcmTJ7v5iBEj3Lxv376BbOrUqe7YKVOmuHnTpk3dfMOGDW4epulStaCp5j/FawT02tjMdKub1whtZta2bdtAdvvtt8c91sxs6NChbr5gwQI3X7hwoZvj8NavX+/m6vxcs2ZNN/eua9S2VRtl48aN3Vy1uHrbUe2B6vnQjHlsqJZz9ZnsNQ536NDBHavOW971rVm4Zkh1HlYt2GqeqetnL1fncnVtpBpAvUZZ9dzV+6OOyV27drn5scI3eQAAAAAQISzyAAAAACBCWOQBAAAAQISwyAMAAACACDmmxSthqZuI586dG8jUTcTDhw93c3VTpbpZul69eoFM3SAathBDlal4udq2KtBQRQBeKcHMmTPVLqISqfdc3Qivbq4uLi4OZN4Nx4fbhqLKVAYOHBjIVBnQ9u3b3bxly5ZuXpEFRyei7t27u7k3b8zMtm3bFshmz57tjj311FPdPCsry83V+TnMe66OG7XtMOfhsOd+77UyM3v55ZcDmSpGWbdunZtv3LjRzdUxiSOjjoNEFGCp+aTKg9Rjqmsm7zwftlxLFWsgsdT5Sb3n3rWpKnpS18lheYUnar9V2Yk6npT69esHMnXO9spozMy6dOni5hkZGYFMzXd1bdSsWTM3p3gFAAAAAJAwLPIAAAAAIEJY5AEAAABAhLDIAwAAAIAIYZEHAAAAABFSJdo1VUOOarfxGiNVs09+fr6bqzargwcPurnavkc9nzDbSJQwbXR79+5NyLbDttrhyKjX02u+MvNbLdUxFnYuLF26NO6xqtFTHTc7d+50c+ZTYnXo0MHN1Xxq1apVIFMtkqqRTDXEFhQUuHm1asGfS6pthD3Hh+E12pnpVrcmTZq4ufe6qNZD7/U2M0tLS3Pz5s2bu7lq6TwRhWnGVPNMnZ9Us7VqwAwzVm1bNRnu2LEjkKk5uW/fvjj3DhVBfSYXFhbGPV41WO/evdvNGzdu7OZhrjHU57eaT6pdUz1/79yqPpsUdd72GjDVNZBqvFXHXmXjmzwAAAAAiBAWeQAAAAAQISzyAAAAACBCWOQBAAAAQISwyAMAAACACKkS7ZqqwUc1lXnWrl3r5qpdU7XyqNYqj9rvRLVrqu141H6rpiKPeq0Ur+nOLDHtdfiGep1r1qzp5qr9yZsjql0ubMPavHnz3Nzb97CtrLVq1XJz1diII6PmmWpq845z1YpZt25dN1fvuZojXq7Ok+r5qFxtx9vHsMekej5eq5vSqFEjN1efZS1btnRz2jW/4b2Pak6q1tOGDRu6uTo/qffRo+aHOp5SU1PdPMx1jToO2rZtG/c2VBMpvp96b9Vx7l1XqrZM9Vka9lrWe3/VOU6dK9VxoFpES0pKApl6rdS+qHnZrFmzQKbaiVVDqbruqmx8kwcAAAAAEcIiDwAAAAAihEUeAAAAAEQIizwAAAAAiBAWeQAAAAAQIVWiXVMJ095YXFzsjlWtUqplSLXveM1GYVs01fgw7XBq217zkJluH/Iek0asqilsG6Bq4fJa4NQ2li1bFufe/dPevXvjHqvmsGrECrsdHJlEtE7m5ua6Y+vUqRP3Ng63L2HeczU27PnZa3lWnx/q2FPPZ9u2bYEsTJupmT5uVBskvqHmn2fnzp1uvmTJEjffuHGjm3ufyeo991r/zPR1zfr1693c275q4ty6daubq7ZWJJZqxgzT3KsaKhV17afa2b1zkboGVcK2OXvnM3VOVPut2mq9z6ew+9e6dWs3r2x8kwcAAAAAEcIiDwAAAAAihEUeAAAAAEQIizwAAAAAiBAWeQAAAAAQIVW6XTNMk5pqyVLtO2Gb11TTTph9Cdse6LW9hW2dC9NeF7atkHbDY0O1aqnGQtUg5bWjqZZA1QynFBQUuLnX2hW2gVA1f6mGOSSWOm9555bt27e7Y9VcDcs7J6pzXNh5FqZdNGzTpRJmDodpmz6SfcHhnXbaaW6+bt06N//qq6/c3Gvny8/Pd8empKS4uWrGDNMy3qJFC3es0rx5czdv2rRpINuxY4c7Vs3hMC2nUaeaIdV1QOfOnQOZOva9Nl8zs6ysLDfft2+fm9euXdvNPWHfW9XS6V2/7Nmzxx3br18/N8/Ly3Nz73NLNduqFub09HQ3r2x8kwcAAAAAEcIiDwAAAAAihEUeAAAAAEQIizwAAAAAiJAqXbySCBkZGW6ubthUN6x6JSNhbtSvaGpfSktL3dzbR27UP76oG32Liorc3LtxW93kvWbNmiPfsW/xClnUDeSqNEDd5F1YWHjkO4aAsEVK3jlEnVfVPAtbGOU9pirmCVtSFeb5h31M9ZngFdLs3bvXHRum7OBIxkdZmMKP1q1bu2O7d+/u5qp4JS0tzc2987Y639arV8/N27dv7+Zq7qgClzBUCcdll10WyB599FF3LAUr30+dW9T1mfd5unv37rjHmuliLPWee+rXr+/mqlxKjVelQt521Hxv166dmy9btszN58yZE8jOOussd+zixYvdXJ3jMzMz3XzFihVunmh8kwcAAAAAEcIiDwAAAAAihEUeAAAAAEQIizwAAAAAiBAWeQAAAAAQIVW6XTNs25tHNRUpqn3o4MGDgUy16YTN1fP0xqt2KtVeV1JSEvdjqm0oiXh/8P28uWdmVrduXTdv1aqVm3vvr5rvK1eujHPvDi83NzeQqdY51eSViDZEVC7V9BimRdPMb0kM22Ycdt5449U2VJOcanf0Wu1U02JOTk6ox6yMlueqKkyr4xlnnOHmqplPze38/Hw395r/Nm/e7I5VzXzq+WzatMnNs7OzA9n27dvdsY0bN3Zz1ZzrNZh36tTJHZuo1uYoU/NJXQd44z/99FN3rJo3qpE7TOO6utZWj5mcHG754bVpq2uJsPPMayNVDaXqfKs+E1QL+rHCN3kAAAAAECEs8gAAAAAgQljkAQAAAECEsMgDAAAAgAhhkQcAAAAAEVKl2zUTQbVLqtYg1RDkjVetQaplRz1mmLYe1UikHlO1JnlUUxGOL/Xq1Yt7rGrgU01qYXltb926dXPHqmNVtb6q4wZHpqCgwM3VfFKNkR6vRdIsfFNZmJbEMK3Fh8u987badmlpaahte6/hhg0b3LF9+/Z187CfcTg8r4nSzGzRokVurl5n1Vxcq1atuPcl7Huojg8v379/vzu2devWbq7aQr3caxA1o10zHuocotqnvfdRXceGOWcfjjeH9+7d645Vz0e1iObl5bm51xquns+6devcvGXLlm6+c+fOQKY+99RxvXHjRjdX79uxwjd5AAAAABAhLPIAAAAAIEJY5AEAAABAhLDIAwAAAIAIYZEHAAAAABES+XbNMG1sh+O1o6mGNUU1AanmNU/Yxjg13mtfUg14YfcFx4aaT3Xr1o07V+2GiWrX3LFjRyDLzMx0x6p2V5Vv3rz5SHfrhKbawdTxrOaZatvzqIZU1bymePuons/BgwfdPMz51sxvNFbbDtu47G17/fr17lj1Gqp9UePxDa8FcuvWre5Y1Qao2vNUE3YiPntVe6Kaf2EaPVUjd7NmzdzcOw83adIk7sfDd4Vp+TXzz8NqTqrGyLDnSm/+qfmucjVX1fP0tqMaPdV8b9q0qZt7nyFz5851x6rXsLi42M1p1wQAAAAAJAyLPAAAAACIEBZ5AAAAABAhLPIAAAAAIEIiX7yiSgPCSkTJSEUWr6hthyleUYUdqJrUzfdhClkKCgrcsaqQJazdu3fHvW31fFSxhhqPw1PnhDDlIGbhim/UzfTqMcMUZqnzZ9hcPaZXShD2+ahigwYNGgSyVatWuWPDFhiELZg5EbVp0yaQqddTvf7q/KSKWry5oLatNGzY0M3VOdHbvnrML7/80s07d+7s5tu3bw9kqamp7thGjRq5eW5urpufiMK8h2b+/Nu1a5c7tm/fvke+Y99SUlISyNQ5Mey1hHdONDPbv39/IFMlKIoqQWndunUgU+fhwYMHu7n3mpjp4rhjhW/yAAAAACBCWOQBAAAAQISwyAMAAACACGGRBwAAAAARwiIPAAAAACKkSrdrJqLRUlFNQGGo/QvbahZmX8K+JqppMUxjHCqXamkrLCx0czX/vHbNLVu2HPmOxWH9+vWBrEaNGu5Yrz3rcEpLS49klyCEbe4N064Ztv1XzRFvO+q8Faah00wfN94+qrbMsOd+r4Vw6dKl7lj1GiaitflE5c0d9XoWFRW5uWqlVnPYaxtUc1UdH/Xr13dz1czoNf9lZGS4Y+fNm+fmqlVw69atgUw1QapWUNo1j1yYz83i4mI3V3NVvY/ePFPnRJWrx1Sf695xpto18/Ly3Dw/Pz/ufdm7d687NmyzctjrmkTjmzwAAAAAiBAWeQAAAAAQISzyAAAAACBCWOQBAAAAQISwyAMAAACACKnS7Zph2s4Ur8nKTDdihaEasVT7jmq+SsTzDCsR7ZoVuX/4hmqhUvOpZs2aceeqQSpRduzYEcjUvAnbtBi2PRGHF7Zdc8OGDXFv22v3MzPbuXOnmxcUFLi5mvOesA2YYVoq1TZq1arl5qoh12uHU62l6jHVcaCa8fCN9PT0QKbOn2quZmVlubl6z72GP/WYar43aNDAzdV2vIa/7Oxsd+xf//pXN1efFd5jqhZN5uSRU+cW7zyckpLiju3Ro4ebL1q0yM3VHPauFdV7q64rVYum+qyoU6dO3GPVuV89pvc8w3zWHG58Zc95vskDAAAAgAhhkQcAAAAAEcIiDwAAAAAihEUeAAAAAEQIizwAAAAAiJATtupINamFaWRT2wibq3Y01abmCduM5wnbroljI2y7puLNheLi4lDbCNsE67W6qcZbdex5bXRq2/h+6j0Mc74x0++LRzXDqVy1oDVq1CiQqXkTts1YCXPuV6+J16JpZtayZctApua1ak5U7W1qPL7htWuq93b37t1unpqa6ubqfdm6dWsgU+/Vnj173LywsNDNw3zeK/v27Qu1L971i9q/Fi1auPnKlSvj3LvoU82YrVu3dvOFCxcGsjZt2rhj27Vr5+b/+Mc/3FzNYe/cqq4f1fl5y5Ytbt64ceO4t6PmmTomi4qK3Lxp06aBTF3TqM8m71xipp//scI3eQAAAAAQISzyAAAAACBCWOQBAAAAQISwyAMAAACACGGRBwAAAAARUqXbNVW7TRiqwadLly5urhrZvAYp1Yqp2hDVeJV7z1819agWJMXbdth2zUS8Pzhyqu1M8ZqowrZrhm2l3bVrVyALc4yZhWvuxPdTx7lqPVXvV5gmv0mTJrm5apLbsWOHm3vnubAts+pcGaZ1VM1VtS95eXluPm/ePDcPs+1EvD8nqvr16wcy1cDXsGHDUNuuXbu2m3vHmZqTTZo0cfOdO3e6uWpx9baj2gA7duzo5mrOe/NMjW3QoIGb4xtLlixx8y+//NLNvXOLaqh866233LxOnTpx7t0/hTnnlpSUhMrT0tLcvKCgIJCp+a6uR9T1jvd6qcbbN954w83V3FZtnMcKnwIAAAAAECEs8gAAAAAgQljkAQAAAECEsMgDAAAAgAip0sUriaBu4lQ3bKoboL2blNWN7SpXhSxhqBtKVZnCxo0b3bxu3bqBTN1wrajnqW66xpFRN9+rfPfu3W7uFQGELS8JW7zi3aBdq1Ytd6wqWFE3QHulCfh+6iZ7VTyi3nN1bvX85je/iXssfOr4SMT7c6Lq3LlzIFMFF6pIRVHvi/fZq87DM2fOdPPLLrvMzdX1y/Tp0+Pev7DzySv0Uq/hjBkz3BzfyM/PD5V7evfuHeoxw14HqOtnj7o2UEUl6vrRe8ywx6S6ZvCOmzZt2rhj16xZ4+ZeMUxVwDd5AAAAABAhLPIAAAAAIEJY5AEAAABAhLDIAwAAAIAIYZEHAAAAABFSpds1VdubahnzLFiwwM2XLVvm5nv37nXzMM2Yqp1q3759bq6ej/f8vbZCM91IdODAATdv2LBhIJs7d647VqFF89hYtGiRm7/99tturuZqbm5uIAvbdhb2Pd+2bVsgW716tTvWm5NmZjt27HDzJUuWhNoX/JM3D8zMVq1a5eabNm1y8zlz5sT9mOpcroQ5x58oXnrpJTfv0KGDm8+fP78idycSbr755kCmPmPV5/prr73m5qqt+quvvgpkrVq1cseuX7/ezefNm+fmYUyaNCnU+Ndff/2oHxOJ5zVDqrZMlauWSjXeOz+r40Y1vqptq+00bdo0kKlrA9WiqRpKi4uL4x6rVNW2eb7JAwAAAIAIYZEHAAAAABHCIg8AAAAAIoRFHgAAAABECIs8AAAAAIiQpBg1ZgAAAAAQGXyTBwAAAAARwiIPAAAAACKERR4AAAAARAiLPAAAAACIEBZ5AAAAABAhLPIAAAAAIEJY5AEAAABAhLDIAwAAAIAIYZEHAAAAABHCIg8AAAAAIoRFXpwmT55sffr0sZycHMvMzLThw4fboUOHErb9vn372kcfffS949q1a2cLFy5M2OMCam4fbq5de+21NmPGDLnNe++91/bv319Be4wTRUWfd78tKSnJ9u7dWyHbBuLRrl0769q1q+Xk5Fi3bt3ssssus8LCwsreLZwACgoKrH79+nbNNdfENf6jjz6ynJwc98+GDh1qb775Zuh9ONK/p6xfv96eeuqphG3veJRc2TtwPNi6datdf/319sUXX1jbtm3NzGz+/PmWlJRUyXsGHJ0jndvPPPOMm5eVlVlycrLdd999dvvtt1vt2rUTvs84MRxv592v5z5wNF577TXLycmxQ4cO2ZgxY+y5556zW265pbJ3CxH32muvWZ8+fWzy5Mn22GOPWf369St7l47a14u8G2+8sbJ3pdLwTV4ctm/fbtWrV7dGjRqVZ71797akpCS78847rV+/fpaTk2ODBw+2lStXlo9JSkqyX//619a/f39r3769TZw4sfzPZs6caTk5OZaVlWX/8i//YmVlZeV/9rvf/a58m/369bNZs2YdmyeKE87h5rbZP79JGTBggLVv397uv//+8jHf/onbVVddZVdffbUNHjzYsrKyyk+op512muXk5NiOHTuO3RNCZBxubrZr185+8YtfuHNz27Ztdskll1j//v2tZ8+eds8995T/2eHO11+LxWI2YcIEO/fcc62oqMhWr15tZ599tvXr18+ys7PtiSeeKB+blJRk//Ef/2H9+vWzn/3sZxX0SuBEdODAASsqKrKGDRva4sWLbdCgQda7d2/r3r37d+Z7QUGBjRs3zjIzM+20006zG264wa666qrK23Ecl5599lmbMGGCDR482F577TUzM3vuueds5MiR9oMf/MB69uxpffv2tXXr1gX+bn5+vo0aNcp++ctfBv6soKDArrvuOuvfv79lZ2fb9ddfbwcOHJD7MX36dOvXr5916tTJ7rjjDovFYmZmtmbNGhs5cqRlZ2dbTk7Od77xmzZtmvXu3duys7NtyJAhtmzZMjMzu/HGG23lypWWk5Nj55577tG8PMevGL7XwYMHY2PHjo01bNgwdv7558ceeuih2KZNm2KxWCy2Y8eO8nGvvPJK7Iwzzij/fzOL/fa3v43FYrHY8uXLY/Xr14+VlpbGSkpKYq1atYp98MEHsVgsFps2bVrMzGIzZswIbHPWrFmxrl27lv9/27ZtYwsWLKiop4oTzOHmdtu2bWM//vGPY7FYLLZz585YSkpK+Z8NGTIk9sYbb8RisVhs/Pjxsezs7Fh+fn75ds0stmfPnmP6XBAtRzo3R40aFfvoo49isVgsVlpaGjvjjDNif/7zn2Ox2Pefr7dt2xYbN25c7Oabb46VlZXFysrKYn369IktX748FovFYoWFhbGePXvG5s6dW/537rvvvgp+JXCiaNu2baxLly6xXr16xVJTU2PDhw+PlZaWxvLz82P79++PxWKxWFFRUSwnJyc2a9asWCwWi915552x8ePHxw4dOhTLz8+PZWVlxcaPH1+JzwLHm6VLl8YyMjJiZWVlsbfeeis2YMCAWCwWi02cODGWkpISW7duXSwWi8UmTJgQu/7662OxWCw2Y8aMWK9evWIbNmyI9e7dO/b888+Xb+/b1wfXXXdd+Z8dOnQods0118Qeeughdz+GDBkSGz58eOzAgQOxwsLCWJ8+fWIvvfRSLBaLxfr37x976qmnYrFYLLZq1apYo0aNYuvXr49t37491qhRo9iiRYtisVgs9uKLL8a6desWO3ToUPk+nsj43ZI4VKtWzSZNmmQrVqywjz/+2KZOnWoPPPCAzZs3z+bOnWuPP/64FRQU2KFDhyw3N/c7f/fyyy83M7PMzExLTk62bdu2WW5uriUnJ9vIkSPNzGzUqFHWoUOH8r+zYMECe+CBB2z37t2WnJxsK1eutOLiYqtTp86xe9I4IRxubpuZXXbZZWZmlp6ebh06dLAvv/zSMjIyAtu5+OKLrUGDBsd03xFtRzI309LSbPr06bZ9+/by7ezbt6/8G7sPPvjgsOfrs88+28477zz793//dzMzW7ZsmS1dutQuvfTS8jEFBQW2bNky69evn5mZXX311RX3IuCE8/Wva5aVldkNN9xgEyZMsAkTJtjNN99sCxcutGrVqtnGjRtt4cKFdsopp9j06dPtkUcesaSkJGvQoIGNGzfO1qxZU9lPA8eRZ5991q688kqrXr26jR492m644QZbvny5mVn5b0t8/d+PP/54+d/bvn27DR482J555hkbMWKEu+0333zTZs2aZb/73e/MzKy4uNiqV68u9+XKK6+0GjVqWI0aNeyHP/yhffjhhzZmzBibP3++/f3vfzczs86dO9ugQYPs008/tdTUVOvZs6f17NnTzP55zX3LLbfY5s2bj/6FiQAWeSFkZmZaZmam3XDDDXbmmWfayy+/bI8++qh9/vnn1rFjR1u0aJENHjz4O3/n2/ckVa9e/Tu/lvltX/963IEDB2zs2LE2Y8YM69evn+Xn51tqaqqVlJSwyEOF+b9ze8qUKWYW//yNwu/vo2oKMzdj//9Xe2bPnh24H3TDhg32ox/96LDn6+HDh9sHH3xgt912m6WkpFgsFrNGjRodtuyKuY+KkJycbBdeeKHdddddlpeXZ+np6bZgwQJLTk62sWPHymKrqnrPKqqm0tJSe+GFF6xGjRr28ssvm5lZUVGRPfvss5aVlXXYa4C0tDTr1KmTvfPOOzZ8+HB37sViMZs0aZJ16dLlO/myZcvKf1B36qmn2h/+8Ad3/9R8Zp7Hh3vy4rB58+bynyCYme3Zs8e+/PJLS01NtRo1aliLFi0sFot9516Nw8nMzLSysrLydsIPP/zQ1q5da2Zm+/fvtwMHDlibNm3MzL7zUxMg0dTc7tix41Ftt0GDBpaXl3e0u4cT2JHMzfr169uwYcPswQcfLM+2bNlimzZtsry8vO89X//85z+3sWPH2siRI2337t3WtWtXS0lJ+c791GvWrAl8AwhUhL/97W/WtWtX27Nnj7Vq1ar8N3s++OCD8jHDhw+3559/3mKxmO3bt8/+/Oc/V+Ie43gzZcoU69Chg23evNnWr19v69evt9mzZ9sLL7xgpaWlh/27tWrVssmTJ9uWLVvsuuuuc5uPzz//fPvP//zP8sXhnj17bM2aNda9e3dbuHChLVy48DsLvBdffNFKS0utuLjYXn75ZRs5cqQ1aNDAevfuXX4eXrNmjX322Wc2ePBgO+WUU2zx4sW2ZMkSMzN79dVXLSMjwzIyMiwlJeWEvw7hm7w4lJWV2S9/+Uv78ssvrW7dulZWVmbjx4+32267zdatW2c9evSwxo0b2/nnnx/X9mrWrGmvvfaa3XzzzXbw4EHr16+f9erVy8zMUlJS7P7777f+/ftbenr6d35NCEg0NbfPO+88u+222454u3fccYedfvrpVrduXXv//fetadOmCdxrnAiOdG6+9NJL9q//+q+WlZVlSUlJVq9ePXv66aetV69edumll37v+fr222+3evXq2fDhw23atGn2zjvv2O23326PPPKIHTx40NLT08t/4g0k2rhx46xOnTpWVlZmbdu2taeeesp27dplV1xxhT3//PPWsWNHGz58ePn4X/ziF3bNNddYt27dLD093Xr16mVpaWmV9wRwXHn22WfLbyv6Wrdu3SwjI8MKCgq+9+9//Q3gtddea5dffrm98MIL3/nzRx55xH76059aTk6OVatWzZKTk+2hhx6yTp06udvr1q2bnXrqqZabm2vnnXde+TXwSy+9ZDfeeKM98cQTlpSUZM8880z5lyEvvfSSXXnllVZWVmYNGza0119/3ZKSkiw7O9t69OhhWVlZ1qFDh/LfAjmRJMW+/v0WAAAAHDdKS0vt4MGDVrt2bSssLLQzzjjDfvzjH9u4ceMqe9cAVDK+yQMAADgO7dmzx8466yw7ePCg7d+/38477zy75JJLKnu3AFQBfJMHAAAAABFC8QoAAAAARAiLPAAAAACIEBZ5AAAAABAhLPIAAAAAIELibtfkX5dHVXA0PUGJmMPVqvk/F/H+EdAjGe+pWbOmm3/9b8T8Xz169HDzOXPmBLJt27bFvR+J0rZtWzfv3r27m7/33ntunojOqES8P2FV9hwGjhZzGMc75jCOd/HMYb7JAwAAAIAIYZEHAAAAABHCIg8AAAAAIoRFHgAAAABECIs8AAAAAIiQpFicFUO0CaEqqOxGLLUNlYdpaXz66afdvFatWm5eUlLi5s2aNXPzBg0aBDL1eqpGzwULFrh5nTp13Ly0tDSQqfbPgoICN1+3bp2bp6WlBbIpU6a4YydNmuTmSkW2blb2HAaOFnMYxzvmMI53tGsCAAAAwAmGRR4AAAAARAiLPAAAAACIEBZ5AAAAABAhLPIAAAAAIEJo1zwK6jUJ28wXpuUp7PtwNA1S32fgwIFuPnPmTDfv2rVrIFu1apU7Vu13ZTdiJap18Te/+U0g69ixozt2y5Ytbq4aMA8ePOjmqampgaxFixbu2MmTJ7v5U0895eazZs1y8+3btweywsJCd+yuXbvcvHr16m7uvReNGjVyx86ePdvNH3nkkVCPqV7bMCp7DgNHizmM4x1zGMc72jUBAAAA4ATDIg8AAAAAIoRFHgAAAABECIs8AAAAAIiQ5MregRNJIkpQKrJIZejQoW7es2dPN+/cubOb//rXv3Zz72blUaNGuWNLSkrcvLKFLV7p0KGDm2dlZQWyDRs2uGNr1arl5mouqH3ZvHlz3Ntu27atm1988cVuXlRU5OY7d+4MZAUFBe5YVXaino9XgqJKarzX+3CPqQpWvPGJKGMBThSqtKIiP9sqUtjn441XY8OeExPxmBX5fBBdiTquGzRo4OaDBg0KZFOnTg21bbWP3nFWVlYWatthhSnvOZrjiW/yAAAAACBCWOQBAAAAQISwyAMAAACACGGRBwAAAAARwiIPAAAAACLkhG3XTEQTkBqbqLa9K6+8MpDNnj3bHXvaaae5+a233urmXgthdna2O3b16tVuPn/+fDe//fbb3XzhwoVufjwJ27g0YsQIN/fa0erVq+eO3b9/v5snJ4c7fOvXrx/Itm7d6o5NT0938zFjxrj5ggUL3NxryqpTp447VjXGlZaWurnXdKqO65o1a7q5Om4++ugjNw/TiAUgKGxTnNeMq87D3jnOzGzevHmhHjOMsM8nzPhEXUsk4romUeMRTap5XM3hTp06ufm1117r5sXFxYGssLDQHauumebOnevmYa7r1DWAev5qfJjHVC278eCbPAAAAACIEBZ5AAAAABAhLPIAAAAAIEJY5AEAAABAhLDIAwAAAIAIOWHbNStDZmamm6uWxKFDhwayvn37umMbNmzo5s8995ybf/LJJ4FMtWX26dPHzfv16+fmBw4ccHOvTWnNmjXu2Kjo3r27m3uNS6pdU72eYRtivfbKGjVquGNLSkrcXLVZqfZKbzvqMVULl2rKSk1NDWS1a9d2x6rXxGvuM9PtmmHbVQF8V926dd38kksucfNzzz03kC1atMgdqxp6VYvuxo0bA1laWpo71msKNtOfYaqheNeuXW7uUfuizs/q+XvtfGr/9u7dG/c2DrcvHnUeVp8JKq9Vq1YgU89n4sSJce4djoaaH+pzffjw4W4+cuRIN9+0aVMg8+aBmT7HnH766W7+zDPPBLLt27e7YxPVqu81Aavjt6ioKNS2v41v8gAAAAAgQljkAQAAAECEsMgDAAAAgAhhkQcAAAAAEcIiDwAAAAAi5IRt11QNOWGoBp+BAwe6+bZt29w8Pz/fzZ999tlA9pOf/MQdu2XLFjd/5JFH3Lxp06aBTL0mK1eudHPVuqkajLyWxKi3a3bs2NHNvZZG1SRWp04dN1etk6WlpW7utT+phk7VlKW2rdo1vcdUDZUqVw1aXhOVeq3U82zSpImbA6gYY8aMcfOcnBw3v+eeewKZass888wz3VydKxcuXBjI2rdv745V575TTjnFzVWLZvPmzQNZ48aN3bHFxcVuvnPnTjfv2rWrm+fm5sa9jezs7FD74rVxqsbNwYMHu7l6/t77Y2a2fPnyQOa1FZqZde7c2c2RWKoFXFHt7O3atXNz75qkWjX/e6pp06a5+UknneTmDz30UCCbN2+eO3bx4sVu7s1JM7P+/fu7uff8Z86c6Y6dNWuWm8eDb/IAAAAAIEJY5AEAAABAhLDIAwAAAIAIYZEHAAAAABFywhavqGIJr8zBzC8lUTf6qpu8s7Ky3Hzo0KFufsMNNwQydWO5utFU2bFjR9xjvZIWM/9mbjOzjIwMN7/66qsD2d///nd37JIlS+Lcu6pBlabs27fPzRs0aBDI1I396vXcuHGjm6v5592krI4DRZWgKF4hizrGwvL2pVGjRu5Y9Vp16NAhIfsCID6bN292c1W81Ldv30CmShvy8vJC5UOGDAlkH3/8sTu2ZcuWbn7FFVe4+XvvvefmXrGEOie++uqrbq4+k+vVq+fmXrGJKqnq1q2bm6vyh927dweyLl26uGMbNmzo5uqzT5XSec9/0KBB7tiJEye6OY6cV2SmivtUEZ93XJuZFRQUuLk3t9U8U/nnn3/u5l4BoLq+HzBggJuPHTvWzdXc9vbl2muvdceqIqN48E0eAAAAAEQIizwAAAAAiBAWeQAAAAAQISzyAAAAACBCWOQBAAAAQIScsO2aYVo0leLiYjf3WgzNzIYPH+7mL774opvfeOONce9LRfKauczMUlJS3HzevHlu7jUEqbZG9ZhVVYsWLdy8bt26bh6mrVU1Rq5cudLN1fwL066pjg81Xh03XguXoh5TNUv17t07kBUWFrpjVftpWlpafDuHChFmfpj58ywRTclmZsnJ/sehan0MQx2TiWqa9ag57z2fMJ97RyszM9PNW7Vq5eZt2rQJZKp9uWPHjm7uNVqamWVnZweyGTNmuGPVOX7t2rVunp6e7ubeOeqrr75yxyoHDhxwc9Ui7DVmqtdbfWYp27dvD2RjxoyJe6yZfj87derk5l4zo7oeUS2i+EbY83AYv/rVr9xcHU+KNy/VuVkdH6qB1ZtP6tw8f/58N/caOs30Pt5yyy2BTLV9X3TRRW4eD77JAwAAAIAIYZEHAAAAABHCIg8AAAAAIoRFHgAAAABEyAlbvAIAqFyJKPxQpQFht52IgpWbbrrJze+55x43z8jIOOrHVEpLSyts2wCAqu+EXeQl4uKioKDAzT/55JNQueK1Qu3fv98dG/b5eBdGahuqBSk3N9fN1esyderUQNayZUt3bNu2bd28qvKaHs10w533+qsWMNUeqC5K1WN6bVGJaJk9HG876jHVxfrBgwfd3Hv+qamp7tht27a5+e7du91ctfGtX7/ezQHERx1zTZo0cXPv2FUtmqrFVG3ba3tUDXfnnXeem3/xxRdurtorFy1aFMhU83b79u3dXLVR9uvXz81nzpwZyIYMGeKO3bt3r5urzzjv/KzeB3VeVe+P+kz09lE9pvo8xDcqsl13z549bq6uK1VrvdfErhqRVVO5un725pm6TjnttNPcfODAgW6u5mXTpk0D2XvvveeOPRr8uiYAAAAARAiLPAAAAACIEBZ5AAAAABAhLPIAAAAAIEJO2OIVAEDVFKYxMxGtmGZmP/jBD9z8pJNOCmQXX3yxO1aVBuzatcvNX3nllbj3I6yaNWu6+d133x3I7r///oQ8JgCg6mCRVwFUG6Jq61HtO2HGqgbCRFDNV/v27XNzdYHmvS6qBSlRF27HSrNmzdxcvRYlJSWBrHnz5u7Y/Px8N1etYao63Xv91f6peaZauNT888ar/VP7op6n9xqqZrxVq1aFesycnBw3p10TODrqnP/ll1+6+WeffRbIzjzzTHesamNcsWKFm3vnVnUefuyxx9x82LBhbq4+N0eMGBHIvOd4uFz90xvvvvuum2dnZweybt26uWNfffVVN1fNf15jptcgamZ2yimnuHmjRo3cXFm2bFkgU++x16CKY6du3bpurq4xVF5UVBTI8vLy3LFhW7O965Sw10bqeaprI2890Lp1a3fs0eDXNQEAAAAgQljkAQAAAECEsMgDAAAAgAhhkQcAAAAAEULxCgCgQoVpyzxc7unUqZObqwbMgQMHuvmoUaPcfO3atYFs06ZN7lhVkqRu+B89erSbJ8Kll17q5ieffHKFPSYAoOpgkVcBwjZdqvFeHbdq7lTCXlx56tWr5+bjx49383feecfNX3755UCmGjq9JqWqrGPHjm6umiH3798fyBo3buyOVc2Qqq1VPaYnbIummn9qnnnUfqttqznijVfbUM9HPf+uXbu6OYCjo5qIc3Nz3dxruk1JSXHHquZeNd7bl169erljp0+f7uaqCVqdQ+64445Apj7vfvjDH7p5q1at3HzixIlu/vHHHwcy1Qq6cuVKN1fNpRdddFEgS0tLc8euXr3azWvVquXmqkXU2xevcdPMrEGDBm6Ob4RtkvSuWVVrbsuWLd3ca8c+XO7NkQMHDrhj1fGk5qXXxqnaMtU/TVNQUODmqampbu410KrXsG/fvm4eD35dEwAAAAAihEUeAAAAAEQIizwAAAAAiBAWeQAAAAAQIZEpXklEwcjxQJW0hC1kCVMOs2vXLjdfsGCBm6ubRJ9++ulApgpLZs6cGefeVQ0tWrRw89q1a7v53r17A5m60dcraTEzS072D98wc17dWK2o40yVD4ShbrhWNzrv2bMnkKnSGfU8VamQej+jQL0WqhBHvf7qpndP2POwukH+gQceCGTjxo1zx6qb77du3ermc+fOdXNvTqkSihUrVri5Ksr41a9+5eaepk2burl6/r/73e/cPDMzM5D16dPHHfvFF1/EuXfxU9s8//zz3XzNmjWBTL2HQ4YMcfMmTZq4+WOPPRbIVDHM3Xff7ebqvHXXXXe5+fbt2wPZbbfd5o5VZVyqYGbAgAFuPmXKlED2+OOPu2OHDh3q5s2bN3fzf/zjH4FMlbecc845bt6mTRs3X7JkiZt7x6QqzJk1a5ab4xthi9a860d1HlLzZufOnW6uzq3e55P6/G7durWbq88sr9RFHWPqukvttzqG//CHPwQyr2TqcI8ZD77JAwAAAIAIYZEHAAAAABHCIg8AAAAAIoRFHgAAAABECIs8AAAAAIiQyLRrRq1FM6wwbZmKavbx2rPMzF599VU3Vw1aZ5xxRiBTzX0bN25086pKNSiptkePaqgsLi4OtS+qPdHLE9WuqZ6n14il2kLVXFDH9r59+9zco/YvJSXFzVu2bBn3tqsy7/1S76ESpkVTGTFihJtfeOGFbn7ZZZe5+e7duwPZsmXL3LHqeFLvuTqGveNPNXeqZuFt27a5ufc8VSujOg8sXrzYzb3GODO/8begoMAdWxEKCwvd/KyzznLzpUuXBrJXXnnFHavew0aNGrm59zmj5p6aN6oZcs6cOW6+du3aQPbCCy+4Y8eOHevm6rw9f/58N+/QoUMgU/OjYcOGbq7ad73XXDVvq/dBPebUqVPd/Kqrrgpkqt0w7PnuRKTaG8Oc+1UTqmqfVZ/JYRo9VeOwusbwPj/UvqhmdNXo6bV9m5lt2rTJzb3zzMMPP+yOnT17tpvHg2/yAAAAACBCWOQBAAAAQISwyAMAAACACGGRBwAAAAAREpniFQDAd3mlNYkoaTIzu/XWWwPZjTfe6I5t1qyZm6ub0lWZiLfvatuKKpBQBT9eyYXaxs6dO91clXZ4Zs6c6eYXXHBB3NswM7vnnnvc/Oabbw5kGzZscMf+8Ic/DPWYAICqg0XecSZM89DhTJgwIZCp5qsnn3zSza+44go3Vw1G7777biBr27atOzYRjX7HUthmL6+5KT093R2r2ujUXAhDXayq9jb1mKrNKsw21L6oFi6v4VDNG9WUpRo9w7aOAohP165d3Vw1Q3qfbd27d3fHfvrpp26u2gNPPfXUQLZo0SJ3bH5+vpt369bNzdXC+fLLLw9k6jV555133Fw1/A0aNMjNS0tLA9nChQvdsarFVf0AwzsPn3322e7YVatWufmjjz7q5l26dHFz7/1Unx+tW7d282PJuw5Qn4Pqs0ddS3jvrXotFNVEHIZ3fWemr1/UPAvTsq3mpHpt1XWA9xoqaqx6zdW+ZGdnB7K8vLy49yNeXMkAAAAAQISwyAMAAACACGGRBwAAAAARwiIPAAAAACKE4hUAOM717t3bzU8//fRApkoe1E3pLVu2dPP69esHsr1797pjN2/e7Oapqamh9sXLVSumVwhhpot8VLGBV/yhijzUzfeqZMArLOrfv787dsuWLW7uvQ9murl09erVgaxu3bru2Ouuu87NAQBVH4u844xq0WzXrp2b33vvvW7uNf6opqKLLrrIzb2LBTN9AeRdLIZpNaoKatWqFWq8ulht0qRJIFNtZ+rCWVXHl5SUuLl3ARq24Uu9X+o996gLXrUN9Zpv3749kKkmL9VcGubC3sxfIBxvcxioTOpzQzUUb9u2LZCtXLnSHasan5ctW+bmy5cvD2Tqn56YNWuWmzdv3tzNR48e7ebeub9NmzbuWLWAV23Gl112mZtPmTIlkKnPD9VGWVBQ4OYtWrSI6/HM9Llc/fMgc+bMcfMvvvgikJ133nnuWNXoWRHCtJ8notEyUQYPHuzmF154oZt7rbTqh2qqbV21aKrrAO81VI+p3gc1/xLxw0NFPc99+/YFsrFjx7pj33777VCP+W38uiYAAAAARAiLPAAAAACIEBZ5AAAAABAhLPIAAAAAIEIoXgGA48SPfvQjN1c3bHtlFqps5sCBA26u2ii9G9DVtlWBhGqjVAU6XglR2KZLVYak9t27WV/d2K/KQ9Rjeq9tfn6+O1YVNezZsyfUeG8fGzRo4I4FABy/qsQiL0wj0fFAPR91EaHad7yLqMzMTHfsww8/7Oaqycxr0LrjjjvcsaplSMnJyXHzDh06BDLVWFZVNWzYMNR41VLpXVSpi9IwzZVm+rjx3kc1J1Uelrcv6jVRx4FqC61Xr14gU4uDLl26uLlqNFX70rRp00Cm/nkAAEHqfPbpp5+6ubfIHjZsmDu2T58+bq7++QmvpXLdunXuWPVPjyjqc/Nvf/tbIFOLbK+J00yfE5csWeLmc+fODWTqBxKqgVDl3vXOxo0b3bGdO3d2c9WuqZ7/5MmTA5lqIFTbqAiJuGZt1KiRm6t/ysZ7TdVY9cNA9fmo5pn3Ga5aJxs3buzmYY5JM/8z2fs8NtM/sFT/VMzMmTMDmfrBpGoiVddveXl5bu61cp9yyinu2KPBr2sCAAAAQISwyAMAAACACGGRBwAAAAARwiIPAAAAACKkShSvhL1ZNUwpRNjSkERQz0cVsqgbVjMyMgKZKkfxbuY20zdyXnzxxW6eCOo1956/eu5VVVpampurm9JVyYhXGvLVV1+5Y1Uzn2rPU/PMuzFYvVdqv8OOVzcjh9m2em29m8KXLl3qjm3Tpo2bqxu01WvovW/H0gsvvODmn3/+uZsPHDgwkGVlZblj27Zt6+aqFMIrIVKlGuqcqOaNKkvwcjXH1HuoSnXUvqt99Ozbt8/NVSGQN//Uca32O0xRgdoXVbDw17/+1c3vvvtuN4+HKoVQraLeMee1rJrp4hG17SuuuCKQNWvWzB27e/duNy8uLnZz79gz89/fOXPmuGNVcZoqTXn88cfd3CukUYUYqoxKHZPt2rULZMOHD3fHTp061c2/+OILN1eft2HKXhJVIhYPdb31q1/9KpCp11M95zDXler4UOeWgoICN1efj95rqo4Dr9TEzOySSy5x83nz5rm59zmkzlvenDycnj17xvV4ZnqeqWtZdax6xS7qM/ho8E0eAAAAAEQIizwAAAAAiBAWeQAAAAAQISzyAAAAACBCWOQBAAAAQIRUiXbNsCqjMdOjWpvU/oVtEb333nsD2ZYtW9yxvXr1cvNx48aFesxEUM8zPT09kKn2pqpKNdaVlpa6uWpj9Boj33vvPXesem/VY4ZpA1SNgqrRU71fajte85dqQ1SPqZ6n9xqqNjrVJus1XJnp51m3bl03P1bUOUe1CqrWPo9qMW3fvr2bd+rUKZCpVjPVqKjec/U8vbmt5tOuXbvcXDVgqvZEr6lOtdepXDXPhWkXVueesO2B3uui2j8r4rNWNV16bdJmZi1atAhkqoFPfT527NjRzbdu3RrI1q9f745Vc1s1/H300Udu7r2PK1eudMc2atTIzXNzc91cNYPWqFEjkKn5rhr+1Pjt27cHMtUQeeqpp7q5ev7vvvuum3ft2jWQqbZQ7z0+Wqq59/e//72be3NYXSepPBHnCrVtdX5SUlNTA5maNw8++GCox7zpppvc3Du2VbPw9OnT3XzdunVu3rlz50Cm5pO6NvCOMTN9PeZd1+zcudMdezT4Jg8AAAAAIoRFHgAAAABECIs8AAAAAIgQFnkAAAAAECEs8gAAAAAgQqpEu2bYlkqvuUm1SnmtRma6+SqMRDWP3XfffW5eVlYWyLKzs92xF1xwwVHvh2pIVLz9O9x2vHbN4416zoqa2952wjZXqoY11ebktRCqbau2PdXOpZoZVe5RbYjqOGvdunUg++yzz9yxeXl5bq4asVQDo9cqdiyp9kbV4uqd/8K2Map55p1DwzakKqq9zpsLar6rfVHbVo103jGitq3aWps0aeLmKSkpgUzNSfUaqmNYNcEWFBTEve2vvvrKzY9G2HbdAQMGBDKvDc9MzwXV9vjGG28EMtWuOXDgQDdXzbaLFy92c++ceN1117lj1WeCarpU54Fp06YFMtVQOmHCBDfPyspy8z/+8Y+B7B//+Ic79mc/+5mbq/Zd7/gwM2vVqlUgU83KFXHOvvLKK91cNUyuXbs2kKlzhcpV06pHnUPUa7Fx40Y3V2213rnFa1k1M3v++efd/Pzzz3fzt99+2829dlv1WvXp08fNhw0b5ubeeUMde+qaRn1+KN61lHrfvGudePFNHgAAAABECIs8AAAAAIgQFnkAAAAAECEs8gAAAAAgQqpE8QoA4MipohyVh1GnTh03924SV8U86gZ5dRO7ugHdo4pUVAlH2PIktX2PV2pipgsMvBIcVaSiXpOwBVje+KKiInes2m8AQNVXJRZ5YVsqu3fvHshU+0x+fr6bq+Yx9WGXCBkZGW6uWru8trHTTjstofv0bep9UG1oYbfTpk2b0PtU1ah5o9rp9u/f7+bexa0aq1qbmjdv7uaqDdG7WG/cuLE7dseOHW7esGFDN1fP37voVY+p5keYRkk199RrpRrw1HuhFjwA4qNa+IqLi918+fLlgUz9cEC1aL777rtu7jXEnnTSSe7Y2bNnu7nXnGimPyu8fVeNnqo1XLVoqtfFa3dVbZmqLVQ1enrnVvXDjnXr1rm5+kGKatf0rklUI7JqbT4a6vNRtVQ2aNAgkJWUlITahvpBmXd9oF43dW2gWnTVY3rHqvrMVD+E8pptzfRnsteuqRpHVTOmupbwrl/UfqvrYfVDODXe+wGfutbr0qWLm8eDX9cEAAAAgAhhkQcAAAAAEcIiDwAAAAAihEUeAAAAAERI3MUr3k2CZuFLUxKx7ZkzZx71Y1aGP/7xj26ubqo8++yzK3J3AlQznnp/wm4nMzMz9D5VNerGWFUmkpqa6ubea+TdnG2mjwN1k726Ydi7uVjdLOzdqG+m30NVSuDdoK7KW1QbYpjXZdu2be7YrVu3uvmKFSvcvHPnzm6u3v8oU4UYKvfs2bMnUbuD41zXrl3d/NJLL3Vzr+FTFXvs3LnTzS+77DI379ixYyBTxQ/t27d381atWrn5+++/7+ZesYv6/FBlIoo6t3bq1CmQqSIVVcii9sXbTk5Ojjs2OzvbzVVBniqY8T4/1Tl7wIABbn40Nm/e7Obqs3rTpk2BTD239PR0N1elIV6xjDoOVONu2JZjrxRQfU6rz3VViNOtWzc395qiVUmN+rxRz9PbF1UmF+b6ykyXtXmFRXl5ee5YdTzFg2/yAAAAACBCWOQBAAAAQISwyAMAAACACGGRBwAAAAARwiIPAAAAACIk7nbNRLRoJmrbXtvju+++647NyMhw89/85jdu/sorr4TaF88vfvELNz/zzDPd/LHHHnPzJUuWHPW+VAbV4KSav44n9evXD5UrXmvVySef7I5VTVmtW7d28wMHDri51+almlCrV6/u5qoRSzWvea+Lmh+5ublu3qNHDzf32sZOP/10d6xq1VJzsqSkxM2bNWvm5gDio5oxVRul1+SnGiDVcTtnzpy4x9etW9cdq9oDVdtenz593Nw7b6mmRUWdb5cuXerm3jm3RYsWoR5TnfvatWsXyNTnx4YNG9y8UaNGbq7ez/Xr18eVmekG5aOxcOFCN588ebKbX3311YHMa401M1u3bp2b79+/3829z1jViqmaHlVrtHofvfdFXUuo6/uioiI3V03Y3nbUY6prjDCvobqOUi2nKg/T0qkafLdv3+7m8eCbPAAAAACIEBZ5AAAAABAhLPIAAAAAIEJY5AEAAABAhLDIAwAAAIAIibtdc+jQoW6uGmjy8/MD2Z49e9yxhYWFbq6albyGHNWa07FjRze/44473Hz69OluvmPHjkA2atQod+ytt97q5h9//LGb//SnP3XzqiJs+2m1av7PDtR7dDxp0qSJm69Zs8bNU1NT3dxratu2bZs71muXM9PHh2rQ8tqcvKbawz2manVTLVdey5dqo8vLy3Nz1VzqPX/VEqbOMZmZmW6u9rEiW4aBE0FKSoqbqxZhrylvxIgR7tgFCxa4+dy5c93cawseNGiQO9a7pjHTbZyqufeNN94IZKqJs02bNm5+6NAhN1eNjd6+q22rzxXV2Oi1Cqpz9sqVK91cnftVI7l3nabO/aqxsCKo1navjfPOO+90x3ptpWa62dp7/dXnnWrLVK+daqn0tqOuJdRnpppPKvf2UY1V+6J441WjpZrbqiFWHavNmzcPZIsWLXLHvvjii27+wgsvuPm38U0eAAAAAEQIizwAAAAAiBAWeQAAAAAQISzyAAAAACBCWOQBAAAAQITE3a6pGn9U7rUQqlat0tJSN8/NzXVzr61m48aN7tiXXnrJzVWLjWrtGjhwYCDLzs52x/797393c9XoqRpKa9WqFchU81VVUlRU5Obvv//+Md6TxFMtVCpX763XXqlaqNTr6c0Ps8S0mKalpbn5l19+GWo7XmuVej6q+ctrtjXzXy/V/llQUODmqhVUHWeqdRNAfJYuXermqqXSO0b/8pe/uGPVOaR79+5uvnXr1kCmWo7VNcM555zj5qottFmzZoFMNXcuXrzYzdW1kWob9BqXN2/e7I71XhMzf7/N/PfHa3w0M2vVqpWbq3P88uXL3TwjIyOQqRbNP//5z25+NFSDuGpSnDp1alyZmdmwYcPcXDV3tm3bNpCpVm+13+q4Ue2a6nPTE+bz20zPS+8zWX3eq+ejePui1iXq+kW9th988IGbe3N75syZahePGN/kAQAAAECEsMgDAAAAgAhhkQcAAAAAEcIiDwAAAAAihEUeAAAAAERIUkxV3PzfgU5LXqI0btzYzVUTU6NGjeIeq/bbayQyM+vWrZubN2jQIJB99tln7tiXX37ZzVUDaNSoxtX58+cHMu+9PJw4p6srEXN43Lhxbn7nnXe6+fr16928U6dOgWzNmjXu2Pr167t5YWGhm6sGSK8Rq169eu5Y1aSmWuBUI10YXgOcmVnDhg3d3JsLX331lTu2f//+bq6ez65du9z80UcfDWQff/yxO1ap7DkMHC3mMI53UZ7DmZmZbp6enu7mYdtQvesa1Ua5du1aN8fRi2cO800eAAAAAEQIizwAAAAAiBAWeQAAAAAQISzyAAAAACBCkit7B8zMdu/eHSpH1abKRv7whz8c2x2pAEuXLnXzoqIiN8/Oznbzf/u3fwtkqjBFFROpchBVYNK5c+dAdu6557pj1Xt46NAhN+/SpYub5+bmBrIaNWq4Y99//303r1bN/1lUampqIFOviTfWzKxPnz5urm5E//vf/+7mAABUBStWrEjIdpYsWZKQ7aDy8E0eAAAAAEQIizwAAAAAiBAWeQAAAAAQISzyAAAAACBCWOQBAAAAQIQkxWKxWFwDk5Iqel+A7xXndHVV5Bw+66yz3HzQoEFuft999wWyAwcOJHSf8A3VrvnYY4+5+WeffebmzzzzzFHvS1Wdw0C8mMM43jGHcbyLZw7zTR4AAAAARAiLPAAAAACIEBZ5AAAAABAhLPIAAAAAIEJY5AEAAABAhMTdrgkAAAAAqPr4Jg8AAAAAIoRFHgAAAABECIs8AAAAAIgQFnkAAAAAECEs8gAAAAAgQljkAQAAAECEsMgDAAAAgAhhkQcAAAAAEcIiDwAAAAAi5P8BIjhrS8IJLMAAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: one image per class, the dataset we will tokenize into patches\n",
"gen = torch.Generator().manual_seed(SEED)\n",
"fig, axes = plt.subplots(2, 5, figsize=(9, 4))\n",
"seen = {}\n",
"for img, lab in train_full:\n",
" if lab not in seen:\n",
" seen[lab] = img\n",
" if len(seen) == 10:\n",
" break\n",
"for k, ax in enumerate(axes.ravel()):\n",
" ax.imshow(seen[k].squeeze(), cmap=\"gray\"); ax.set_title(CLASS_NAMES[k], fontsize=8); ax.axis(\"off\")\n",
"plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "f4a0d514",
"metadata": {},
"source": [
"> **Interpretation.** Ten clothing categories, all `28x28` grayscale. The classifier we build never sees these as images: it sees 16 patch-vectors per image. The picture above is the last time pixels look like pixels.\n"
]
},
{
"cell_type": "markdown",
"id": "ead60bee",
"metadata": {},
"source": [
"### 1.2 Patch embedding is one convolution\n",
"\n",
"Take an image $x$ of shape $(B, C, H, W)$. Cut it into non-overlapping $P \\times P$ patches. Flatten each patch to a $C \\cdot P^2$ vector and apply one linear projection to $d_{model}$. That is *exactly* what a `Conv2d` with `kernel_size = stride = P` computes: each output position is a dot product of the kernel with one non-overlapping patch. So the entire patch embedding is\n",
"\n",
"$$ \\text{patch\\_embed}(x) = \\text{flatten}_{2}\\big(\\text{Conv2d}_{P,P}(x)\\big)^{\\top} \\in \\mathbb{R}^{B \\times N \\times d_{model}}, \\quad N = (H/P)(W/P). $$\n",
"\n",
"The `Conv2d` output is $(B, d_{model}, H/P, W/P)$; `flatten(2)` collapses the grid to $N$, and a transpose puts $d_{model}$ last. There is exactly one convolution in the whole network, and this is it.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "81171581",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.331999Z",
"iopub.status.busy": "2026-06-10T20:02:31.331900Z",
"iopub.status.idle": "2026-06-10T20:02:31.338962Z",
"shell.execute_reply": "2026-06-10T20:02:31.338645Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"PatchEmbedding params 3200 . n_patches 16\n",
"PatchEmbedding (2, 1, 28, 28) -> (2, 16, 64)\n"
]
},
{
"data": {
"text/plain": [
"tensor([[[ 0.7898, 0.4204, -0.2867, ..., 0.1992, -0.4541, -0.1304],\n",
" [-0.4734, -1.0486, -0.7415, ..., -0.8428, 0.5032, -0.5154],\n",
" [ 0.9195, 1.0711, -0.3529, ..., 0.4962, -1.0164, 0.3588],\n",
" ...,\n",
" [-0.3487, 0.0825, -0.5655, ..., -0.1979, -0.0720, -0.6119],\n",
" [ 0.0349, -0.6403, -0.7787, ..., 0.3056, 0.5033, -0.9362],\n",
" [-0.2745, -0.3448, -1.5200, ..., 0.7875, 0.4659, -0.3681]],\n",
"\n",
" [[ 0.0881, 0.0625, -0.6228, ..., 0.5780, 0.5368, -0.9291],\n",
" [ 1.1483, -0.4163, -0.8490, ..., 0.3119, -0.0111, -0.2605],\n",
" [-0.0895, -0.1463, -0.3671, ..., 0.1946, 0.4876, 0.1145],\n",
" ...,\n",
" [-0.9548, 0.0354, -0.3757, ..., 0.5884, 0.3628, 0.7647],\n",
" [ 0.5369, -0.8804, -1.1623, ..., 0.3330, 0.0269, -0.1967],\n",
" [-0.0954, -0.2290, -0.4089, ..., -0.1790, -0.4007, -0.4343]]])"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# the entire patch embedding, ~6 lines\n",
"class PatchEmbedding(nn.Module):\n",
" def __init__(self, image_size=28, patch_size=7, in_channels=1, d_model=64):\n",
" super().__init__()\n",
" assert image_size % patch_size == 0, \"patch_size must divide image_size\"\n",
" self.n_patches = (image_size // patch_size) ** 2 # (28/7)^2 = 16\n",
" self.proj = nn.Conv2d(in_channels, d_model, kernel_size=patch_size, stride=patch_size)\n",
"\n",
" def forward(self, x):\n",
" # x: (B, C, H, W) -> conv (B, d_model, H/P, W/P) -> (B, N, d_model)\n",
" x = self.proj(x) # (B, d_model, 4, 4)\n",
" return x.flatten(2).transpose(1, 2) # (B, 16, d_model)\n",
"\n",
"pe = PatchEmbedding()\n",
"print(f\"PatchEmbedding params {param_count(pe)} . n_patches {pe.n_patches}\")\n",
"layer_summary(pe, torch.randn(2, 1, 28, 28)) # randn smoke test"
]
},
{
"cell_type": "markdown",
"id": "18d99e8e",
"metadata": {},
"source": [
"> **Predict:** if I build the same projection as `unfold`-then-`Linear` using the conv's own weights, will the two outputs match to floating-point tolerance? AnswerYes, bitwise up to accumulation order. The conv kernel reshaped to `(d_model, C*P*P)` is the linear weight; the conv bias is the linear bias. They are the same linear map applied to the same patches.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "b01b3e64",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.339928Z",
"iopub.status.busy": "2026-06-10T20:02:31.339853Z",
"iopub.status.idle": "2026-06-10T20:02:31.343291Z",
"shell.execute_reply": "2026-06-10T20:02:31.343009Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] Conv2d patch embedding == flatten + linear (max diff 0.00e+00)\n"
]
}
],
"source": [
"# the equivalence, asserted (spec §2 implementation ladder: prove scratch == library path)\n",
"img = torch.randn(2, 1, 28, 28)\n",
"with torch.no_grad():\n",
" out_conv = pe(img) # (2, 16, 64) via Conv2d\n",
"\n",
"# manual: unfold into 7x7 patches, flatten, apply the SAME weights as a linear map\n",
"patches = img.unfold(2, 7, 7).unfold(3, 7, 7) # (2, 1, 4, 4, 7, 7)\n",
"patches = patches.permute(0, 2, 3, 1, 4, 5).reshape(2, 16, 1 * 7 * 7) # (2, 16, 49)\n",
"W = pe.proj.weight.detach().reshape(64, -1) # (64, 49)\n",
"b = pe.proj.bias.detach() # (64,)\n",
"out_linear = patches @ W.T + b # (2, 16, 64)\n",
"\n",
"check_close(out_conv, out_linear, atol=1e-5,\n",
" msg=\"conv-as-patch-embed should equal flatten-each-patch-then-linear\")\n",
"print(\"[ ok ] Conv2d patch embedding == flatten + linear (max diff \"\n",
" f\"{(out_conv - out_linear).abs().max().item():.2e})\")"
]
},
{
"cell_type": "markdown",
"id": "820e5927",
"metadata": {},
"source": [
"> **Interpretation.** The patch embedding is not a special vision operation. It is a linear projection of flattened pixel patches, packaged as a strided convolution because that is the fast way to slice a grid. Everything downstream is a plain transformer.\n"
]
},
{
"cell_type": "markdown",
"id": "8e48eec7",
"metadata": {},
"source": [
"### 1.3 The `[CLS]` token and position embeddings\n",
"\n",
"Two learned pieces turn the patch sequence into a transformer input. A `[CLS]` token (BERT's trick) is one extra learned vector prepended to the sequence; after the encoder, its final embedding is the image-level summary we classify from. A position embedding is one learned vector per position, added to the sequence so the model can tell patch 0 from patch 15. The original ViT uses a *1D* learned position embedding in row-major order and lets the model recover the 2D grid structure from data, which it does.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "e45452b6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.344250Z",
"iopub.status.busy": "2026-06-10T20:02:31.344175Z",
"iopub.status.idle": "2026-06-10T20:02:31.347450Z",
"shell.execute_reply": "2026-06-10T20:02:31.347174Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"sequence with CLS + positions: (2, 17, 64) (16 patches + 1 CLS = 17 tokens)\n"
]
}
],
"source": [
"# CLS + position: prepend one learned token, add one learned vector per position\n",
"d_model, n_patches = 64, pe.n_patches\n",
"cls_token = nn.Parameter(torch.zeros(1, 1, d_model))\n",
"pos_embed = nn.Parameter(torch.zeros(1, n_patches + 1, d_model)) # +1 for CLS\n",
"nn.init.trunc_normal_(cls_token, std=0.02) # std 0.02 is the BERT/ViT init constant\n",
"nn.init.trunc_normal_(pos_embed, std=0.02)\n",
"\n",
"x = pe(torch.randn(2, 1, 28, 28)) # (2, 16, 64)\n",
"cls = cls_token.expand(2, -1, -1) # (2, 1, 64), broadcast over batch\n",
"x = torch.cat([cls, x], dim=1) # (2, 17, 64), CLS at position 0\n",
"x = x + pos_embed # add learned positions, shape (2, 17, 64)\n",
"print(f\"sequence with CLS + positions: {tuple(x.shape)} (16 patches + 1 CLS = 17 tokens)\")"
]
},
{
"cell_type": "markdown",
"id": "e2eb21f9",
"metadata": {},
"source": [
"> **Common confusion:** the `[CLS]` token has no pixels behind it. It starts as a learned constant, then collects information from the patches through attention. Reading the final `[CLS]` embedding is reading \"what the whole image looked like to the attention stack\".\n"
]
},
{
"cell_type": "markdown",
"id": "ce670d05",
"metadata": {},
"source": [
"### 1.4 The encoder block: Ch 15, minus the causal mask\n",
"\n",
"The transformer block is the one you wrote in Ch 15: pre-norm, multi-head self-attention, residual, pre-norm, MLP, residual. The only change for vision is that there is *no causal mask*. Every patch attends to every patch, both directions. We build it once here as a tested standalone module, then assemble the full ViT in Part 2 (spec §3.1: build the method as a standalone tested module first, assemble the class once, never edit a class in place).\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "f143fff3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.348307Z",
"iopub.status.busy": "2026-06-10T20:02:31.348234Z",
"iopub.status.idle": "2026-06-10T20:02:31.355394Z",
"shell.execute_reply": "2026-06-10T20:02:31.355105Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Block params 49728\n",
"Block (2, 17, 64) -> (2, 17, 64)\n"
]
},
{
"data": {
"text/plain": [
"tensor([[[ 0.4150, 1.5213, 0.8685, ..., 1.9337, -1.1844, 0.9101],\n",
" [-0.6223, 0.2045, 0.4129, ..., 0.5570, 1.3243, 0.0257],\n",
" [ 0.3831, -1.3120, -0.2267, ..., -0.5348, 2.0213, -0.8667],\n",
" ...,\n",
" [ 0.5730, 0.0767, 0.6658, ..., -0.1013, -0.0594, -0.2884],\n",
" [ 0.2445, 0.3874, -1.0063, ..., 0.2443, 1.5174, 0.5737],\n",
" [-1.7899, -0.6879, -0.3670, ..., -1.0438, 0.6943, 0.7704]],\n",
"\n",
" [[-1.0487, -0.0044, -0.3122, ..., -0.0603, 0.4362, 0.9849],\n",
" [-0.8953, 1.2300, 2.5959, ..., -0.5046, -1.3233, -0.3245],\n",
" [ 0.0710, -0.7830, 1.0420, ..., 0.6524, 0.6996, -1.7590],\n",
" ...,\n",
" [-0.7293, -1.3094, 1.8351, ..., -0.5098, 0.2970, -0.9772],\n",
" [ 0.6445, 0.6863, 0.1769, ..., -0.0645, 1.0327, 1.5735],\n",
" [ 0.2801, -0.4873, -0.2818, ..., -0.5776, -2.2987, 1.1127]]])"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# the bidirectional encoder block (Ch 15 block, no causal mask)\n",
"class Block(nn.Module):\n",
" def __init__(self, d_model, n_heads):\n",
" super().__init__()\n",
" assert d_model % n_heads == 0, \"n_heads must divide d_model\"\n",
" self.n_heads, self.d_k = n_heads, d_model // n_heads\n",
" self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)\n",
" self.out = nn.Linear(d_model, d_model, bias=False)\n",
" self.ln1, self.ln2 = nn.LayerNorm(d_model), nn.LayerNorm(d_model)\n",
" self.mlp = nn.Sequential(nn.Linear(d_model, 4 * d_model), nn.GELU(),\n",
" nn.Linear(4 * d_model, d_model))\n",
"\n",
" def attn(self, x):\n",
" B, T, C = x.shape\n",
" q, k, v = self.qkv(x).split(C, dim=-1) # each (B, T, C)\n",
" q = q.view(B, T, self.n_heads, self.d_k).transpose(1, 2) # (B, h, T, d_k)\n",
" k = k.view(B, T, self.n_heads, self.d_k).transpose(1, 2) # (B, h, T, d_k)\n",
" v = v.view(B, T, self.n_heads, self.d_k).transpose(1, 2) # (B, h, T, d_k)\n",
" scores = q @ k.transpose(-2, -1) / math.sqrt(self.d_k) # (B, h, T, T); no mask\n",
" a = F.softmax(scores, dim=-1)\n",
" o = (a @ v).transpose(1, 2).reshape(B, T, C) # (B, T, C)\n",
" return self.out(o)\n",
"\n",
" def forward(self, x):\n",
" x = x + self.attn(self.ln1(x)) # pre-norm attention + residual\n",
" x = x + self.mlp(self.ln2(x)) # pre-norm MLP + residual\n",
" return x\n",
"\n",
"blk = Block(d_model=64, n_heads=4)\n",
"print(f\"Block params {param_count(blk)}\")\n",
"layer_summary(blk, torch.randn(2, 17, 64)) # randn smoke test: (2,17,64) -> (2,17,64)"
]
},
{
"cell_type": "markdown",
"id": "9e920720",
"metadata": {},
"source": [
"> **Interpretation.** The block preserves shape `(B, T, d_model)`, so blocks stack freely. No mask means `scores` is a full `(T, T)` matrix; row sums are 1 and the upper triangle is non-zero, the opposite of Ch 15's causal GPT.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "b02a4705",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.356243Z",
"iopub.status.busy": "2026-06-10T20:02:31.356169Z",
"iopub.status.idle": "2026-06-10T20:02:31.358984Z",
"shell.execute_reply": "2026-06-10T20:02:31.358702Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"rows sum to 1: True . upper triangle non-zero: True\n",
"[ ok ] bidirectional: a causal GPT would have a zero upper triangle here\n"
]
}
],
"source": [
"# falsification demo: ViT attention is bidirectional, NOT causal (assert it)\n",
"with torch.no_grad():\n",
" B2, h2, T2 = 1, 4, 17\n",
" q = k = torch.randn(B2, h2, T2, 16)\n",
" scores = q @ k.transpose(-2, -1) / math.sqrt(16)\n",
" a = F.softmax(scores, dim=-1) # ViT softmax over ALL positions\n",
"rows_sum_to_one = torch.allclose(a.sum(-1), torch.ones(B2, h2, T2), atol=1e-5)\n",
"upper_nonzero = (a[0, 0].triu(diagonal=1) > 0).any().item()\n",
"print(f\"rows sum to 1: {rows_sum_to_one} . upper triangle non-zero: {upper_nonzero}\")\n",
"assert rows_sum_to_one and upper_nonzero, \\\n",
" \"ViT attention must be bidirectional: every row sums to 1 over ALL keys, upper triangle > 0\"\n",
"print(\"[ ok ] bidirectional: a causal GPT would have a zero upper triangle here\")"
]
},
{
"cell_type": "markdown",
"id": "883f9df2",
"metadata": {},
"source": [
"### Exercise 16.1 - Patchify and count tokens\n",
"`Difficulty 1/5 · ~8 min`\n",
"\n",
"Write `seq_len(image_size, patch_size, has_cls)` returning the transformer sequence length: the number of patches, plus one if a `[CLS]` token is prepended. This is the single most common shape bug in ViT code, so isolate it before the composite (spec §3.4). Raise a clear error if `patch_size` does not divide `image_size`.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "fa59e635",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.359768Z",
"iopub.status.busy": "2026-06-10T20:02:31.359697Z",
"iopub.status.idle": "2026-06-10T20:02:31.362984Z",
"shell.execute_reply": "2026-06-10T20:02:31.362609Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 16.1 token counts: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 16.1 rejects bad patch size: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def seq_len(image_size, patch_size, has_cls=True):\n",
" \"\"\"Sequence length fed to the transformer. (image_size/patch_size)^2 patches, +1 if CLS.\"\"\"\n",
" # TODO 1: assert patch_size divides image_size with a helpful message\n",
" # TODO 2: n_patches = (image_size // patch_size) ** 2\n",
" n_patches = None\n",
" # TODO 3: return n_patches + 1 if has_cls else n_patches\n",
" result = None\n",
" attempted(n_patches, result)\n",
" return result\n",
"\n",
"def _toy_seqlen():\n",
" # 28x28 image, 7x7 patches -> 16 patches, +1 CLS = 17\n",
" assert seq_len(28, 7, True) == 17, f\"got {seq_len(28,7,True)}, expected 17 (16 patches + CLS)\"\n",
" # ViT-B/16 on 224x224: 196 patches + CLS = 197\n",
" assert seq_len(224, 16, True) == 197, f\"got {seq_len(224,16,True)}, expected 197\"\n",
" # no CLS: just the patch count\n",
" assert seq_len(28, 7, False) == 16, f\"got {seq_len(28,7,False)}, expected 16\"\n",
"\n",
"def _raises_on_bad_patch():\n",
" try:\n",
" seq_len(28, 5, True)\n",
" except AssertionError:\n",
" return\n",
" raise AssertionError(\"seq_len(28, 5) should raise: 5 does not divide 28\")\n",
"\n",
"check(\"16.1 token counts\", _toy_seqlen)\n",
"check(\"16.1 rejects bad patch size\", _raises_on_bad_patch)"
]
},
{
"cell_type": "markdown",
"id": "3d1904a4",
"metadata": {},
"source": [
"Hint 1 (conceptual)The patch grid is `image_size // patch_size` along each axis, squared for a 2D image. The `[CLS]` token is one extra position.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"assert image_size % patch_size == 0, \"patch_size must divide image_size\"\n",
"n_patches = (image_size // patch_size) ** 2\n",
"return n_patches + 1 if has_cls else n_patches\n",
"```\n",
"\n",
"\n",
"Help: \"expected 17, got 5\"You likely returned `image_size // patch_size` (one axis) instead of squaring it. The patch grid is 2D: `(28//7)**2 = 16`, not `28//7 = 4`.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "cb609367",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.363746Z",
"iopub.status.busy": "2026-06-10T20:02:31.363679Z",
"iopub.status.idle": "2026-06-10T20:02:31.365749Z",
"shell.execute_reply": "2026-06-10T20:02:31.365497Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 16.1 token counts\n",
"[ ok ] 16.1 rejects bad patch size\n",
"FashionMNIST ViT: 17 tokens . ViT-B/16: 197 tokens\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines seq_len; the checks below re-verify the reference.\n",
"def seq_len(image_size, patch_size, has_cls=True):\n",
" assert image_size % patch_size == 0, \\\n",
" f\"patch_size {patch_size} must divide image_size {image_size}\"\n",
" n_patches = (image_size // patch_size) ** 2\n",
" return n_patches + 1 if has_cls else n_patches\n",
"\n",
"check(\"16.1 token counts\", _toy_seqlen, required=True)\n",
"check(\"16.1 rejects bad patch size\", _raises_on_bad_patch, required=True)\n",
"print(f\"FashionMNIST ViT: {seq_len(28, 7)} tokens . ViT-B/16: {seq_len(224, 16)} tokens\")"
]
},
{
"cell_type": "markdown",
"id": "43357b20",
"metadata": {},
"source": [
"### Exercise 16.2 - Prepend `[CLS]`, add positions\n",
"`Difficulty 2/5 · ~12 min`\n",
"\n",
"Write `add_cls_and_pos(patches, cls_token, pos_embed)`. Given patch embeddings of shape `(B, N, d)`, a `cls_token` of shape `(1, 1, d)`, and a `pos_embed` of shape `(1, N+1, d)`, prepend the `[CLS]` token to every example in the batch and add the position embedding. Output shape `(B, N+1, d)`. The check verifies the shape, that `[CLS]` lands at position 0, and that positions were actually added (not silently dropped).\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "a4ae6ebf",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.366577Z",
"iopub.status.busy": "2026-06-10T20:02:31.366511Z",
"iopub.status.idle": "2026-06-10T20:02:31.370632Z",
"shell.execute_reply": "2026-06-10T20:02:31.370378Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 16.2 cls + positions: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def add_cls_and_pos(patches, cls_token, pos_embed):\n",
" \"\"\"patches: (B, N, d) . cls_token: (1, 1, d) . pos_embed: (1, N+1, d) -> (B, N+1, d).\"\"\"\n",
" B = patches.shape[0]\n",
" # TODO 1: expand cls_token to (B, 1, d) without copying memory (use .expand)\n",
" cls = None\n",
" # TODO 2: concatenate cls in FRONT of patches along the token axis (dim=1)\n",
" x = None\n",
" # TODO 3: add the position embedding (broadcasts over the batch)\n",
" x = None\n",
" attempted(cls, x)\n",
" return x\n",
"\n",
"def _cls_pos_checks():\n",
" torch.manual_seed(SEED)\n",
" B, N, d = 3, 16, 64\n",
" p = torch.randn(B, N, d)\n",
" cls = torch.randn(1, 1, d); pos = torch.zeros(1, N + 1, d) # zero pos isolates CLS placement\n",
" out = add_cls_and_pos(p, cls, pos)\n",
" check_shape(out, (B, N + 1, d))\n",
" # CLS must be broadcast to every batch row at position 0 (pos is zero, so equals cls)\n",
" assert torch.allclose(out[:, 0], cls.expand(B, -1, -1).squeeze(1), atol=1e-6), \\\n",
" \"position 0 should be the CLS token for every batch element\"\n",
" # with a non-zero pos, the output must differ from raw concat (positions were added)\n",
" pos2 = torch.randn(1, N + 1, d)\n",
" out2 = add_cls_and_pos(p, cls, pos2)\n",
" assert not torch.allclose(out2[:, 1:], p, atol=1e-4), \\\n",
" \"patch positions look unchanged: did you forget to add pos_embed?\"\n",
"\n",
"check(\"16.2 cls + positions\", _cls_pos_checks)"
]
},
{
"cell_type": "markdown",
"id": "abdb4c4e",
"metadata": {},
"source": [
"Hint 1 (conceptual)`expand` gives a batched view of the single CLS vector with no copy. `torch.cat([cls, patches], dim=1)` puts CLS first. Adding `pos_embed` broadcasts its leading batch dim of 1.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"cls = cls_token.expand(B, -1, -1) # (B, 1, d)\n",
"x = torch.cat([cls, patches], dim=1) # (B, N+1, d)\n",
"x = x + pos_embed # broadcast over batch\n",
"```\n",
"\n",
"\n",
"Help: \"shape (B, N+1, d) but CLS is at the end\"Order matters in `torch.cat`. `[cls, patches]` puts CLS at index 0; `[patches, cls]` puts it last. ViT reads position 0, so CLS goes first.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "f7f6d89f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.371401Z",
"iopub.status.busy": "2026-06-10T20:02:31.371335Z",
"iopub.status.idle": "2026-06-10T20:02:31.374183Z",
"shell.execute_reply": "2026-06-10T20:02:31.373872Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 16.2 cls + positions\n",
"[ ok ] CLS prepended at position 0, positions added\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines add_cls_and_pos; the checks below re-verify the reference.\n",
"def add_cls_and_pos(patches, cls_token, pos_embed):\n",
" B = patches.shape[0]\n",
" cls = cls_token.expand(B, -1, -1) # (B, 1, d), a view\n",
" x = torch.cat([cls, patches], dim=1) # (B, N+1, d)\n",
" x = x + pos_embed # learned positions, broadcast over batch\n",
" return x\n",
"\n",
"check(\"16.2 cls + positions\", _cls_pos_checks, required=True)\n",
"print(\"[ ok ] CLS prepended at position 0, positions added\")"
]
},
{
"cell_type": "markdown",
"id": "ad65548f",
"metadata": {},
"source": [
"### Exercise 16.3 - Split and merge attention heads\n",
"`Difficulty 3/5 · ~12 min`\n",
"\n",
"The single most error-prone line in any transformer is reshaping `(B, T, d)` into per-head `(B, h, T, d_k)` and back. Isolate it before it hides inside the `Block` (spec §3.4). Write `split_heads(x, n_heads)` taking `(B, T, d)` to `(B, h, T, d_k)` and `merge_heads(x)` taking `(B, h, T, d_k)` back to `(B, T, d)`. The check verifies both shapes and that `merge_heads(split_heads(x))` round-trips to the original tensor exactly, which catches the classic transpose-vs-reshape ordering bug.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "0ab0395a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.374796Z",
"iopub.status.busy": "2026-06-10T20:02:31.374732Z",
"iopub.status.idle": "2026-06-10T20:02:31.378436Z",
"shell.execute_reply": "2026-06-10T20:02:31.378217Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 16.3 split + merge heads: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def split_heads(x, n_heads):\n",
" \"\"\"(B, T, d) -> (B, n_heads, T, d_k). View into heads, then move the head axis up.\"\"\"\n",
" B, T, d = x.shape\n",
" d_k = d // n_heads\n",
" # TODO 1: view x as (B, T, n_heads, d_k)\n",
" # TODO 2: transpose the head axis to position 1 -> (B, n_heads, T, d_k)\n",
" out = None\n",
" attempted(out)\n",
" return out\n",
"\n",
"def merge_heads(x):\n",
" \"\"\"(B, n_heads, T, d_k) -> (B, T, d). The exact inverse of split_heads.\"\"\"\n",
" B, h, T, d_k = x.shape\n",
" # TODO 3: transpose back to (B, T, n_heads, d_k), then reshape to (B, T, h*d_k)\n",
" # use .contiguous() before reshape so the memory layout is valid\n",
" out = None\n",
" attempted(out)\n",
" return out\n",
"\n",
"def _heads_checks():\n",
" torch.manual_seed(SEED)\n",
" B, T, d, h = 2, 17, 64, 4\n",
" x = torch.randn(B, T, d)\n",
" s = split_heads(x, h)\n",
" check_shape(s, (B, h, T, d // h))\n",
" m = merge_heads(s)\n",
" check_shape(m, (B, T, d))\n",
" # round-trip must be exact: a transpose-vs-reshape mistake corrupts this\n",
" assert torch.equal(m, x), \\\n",
" \"merge_heads(split_heads(x)) must equal x; a reshape that skips .contiguous()/transpose corrupts the order\"\n",
"\n",
"check(\"16.3 split + merge heads\", _heads_checks)"
]
},
{
"cell_type": "markdown",
"id": "ebb24484",
"metadata": {},
"source": [
"Hint 1 (conceptual)Splitting is a `view` that factors `d` into `(n_heads, d_k)`, then a `transpose(1, 2)` to bring heads before the time axis. Merging reverses both steps; reshape needs contiguous memory after a transpose.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"# split\n",
"out = x.view(B, T, n_heads, d_k).transpose(1, 2) # (B, h, T, d_k)\n",
"# merge\n",
"out = x.transpose(1, 2).contiguous().reshape(B, T, h * d_k)\n",
"```\n",
"\n",
"\n",
"Help: round-trip is not exactYou probably reshaped without transposing back first, or skipped `.contiguous()`. After `transpose`, memory is not contiguous, so `reshape` either errors or silently reinterprets the bytes in the wrong order. Transpose first, make contiguous, then reshape.\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "d491e1a5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.379198Z",
"iopub.status.busy": "2026-06-10T20:02:31.379124Z",
"iopub.status.idle": "2026-06-10T20:02:31.381771Z",
"shell.execute_reply": "2026-06-10T20:02:31.381520Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 16.3 split + merge heads\n",
"[ ok ] head split/merge round-trips exactly: this is the reshape inside every Block\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines split_heads / merge_heads; the checks below re-verify the reference.\n",
"def split_heads(x, n_heads):\n",
" B, T, d = x.shape\n",
" d_k = d // n_heads\n",
" return x.view(B, T, n_heads, d_k).transpose(1, 2) # (B, h, T, d_k)\n",
"\n",
"def merge_heads(x):\n",
" B, h, T, d_k = x.shape\n",
" return x.transpose(1, 2).contiguous().reshape(B, T, h * d_k) # (B, T, d)\n",
"\n",
"check(\"16.3 split + merge heads\", _heads_checks, required=True)\n",
"print(\"[ ok ] head split/merge round-trips exactly: this is the reshape inside every Block\")"
]
},
{
"cell_type": "markdown",
"id": "194e37e8",
"metadata": {},
"source": [
"> **Key takeaways.** Patch embedding is one strided `Conv2d`, provably equal to flatten-then-linear. A learned `[CLS]` token and a learned 1D position embedding turn 16 patches into a 17-token sequence. Splitting and merging heads is a `view`-then-`transpose` round-trip that must be exact. The encoder block is the Ch 15 block with the causal mask deleted: bidirectional, shape-preserving, stackable.\n"
]
},
{
"cell_type": "markdown",
"id": "1cfd2265",
"metadata": {},
"source": [
"## Part 2 - A tiny ViT, trained and broken\n",
"\n",
"> **Objectives**\n",
"> - Assemble the full `ViT` in one cell from the tested pieces and confirm it with a `randn` smoke test.\n",
"> - Train it on FashionMNIST with the four-comment loop and read the loss curve.\n",
"> - Experience the canonical training bug (gradients never zeroed), diagnose it from the curve, and fix it.\n",
"\n",
"We have a patch embedder, a CLS-and-positions step, and an encoder block. Assemble them once, the Karpathy two-tempo rhythm: many tiny exploration cells, then one consolidated liftable cell.\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "bab832d1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.382635Z",
"iopub.status.busy": "2026-06-10T20:02:31.382566Z",
"iopub.status.idle": "2026-06-10T20:02:31.391461Z",
"shell.execute_reply": "2026-06-10T20:02:31.391162Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ViT params 204,042\n",
"ViT (2, 1, 28, 28) -> (2, 10)\n"
]
},
{
"data": {
"text/plain": [
"tensor([[-1.0719, -1.1520, -0.4687, 0.9351, -0.4871, 0.6005, -0.4680, -0.1833,\n",
" 0.7697, -0.4627],\n",
" [-0.3457, -0.1656, -0.0332, 0.0565, 0.4189, -0.1299, 0.2276, -0.6506,\n",
" -0.0749, -0.4740]])"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# the full ViT, assembled once from the Part 1 pieces (never edited in place)\n",
"class ViT(nn.Module):\n",
" def __init__(self, image_size=28, patch_size=7, in_channels=1, num_classes=10,\n",
" d_model=64, n_heads=4, n_layers=4):\n",
" super().__init__()\n",
" self.patch_embed = PatchEmbedding(image_size, patch_size, in_channels, d_model)\n",
" n_patches = self.patch_embed.n_patches\n",
" self.cls_token = nn.Parameter(torch.zeros(1, 1, d_model))\n",
" self.pos_embed = nn.Parameter(torch.zeros(1, n_patches + 1, d_model))\n",
" self.blocks = nn.ModuleList([Block(d_model, n_heads) for _ in range(n_layers)])\n",
" self.ln_f = nn.LayerNorm(d_model)\n",
" self.head = nn.Linear(d_model, num_classes)\n",
" nn.init.trunc_normal_(self.cls_token, std=0.02)\n",
" nn.init.trunc_normal_(self.pos_embed, std=0.02)\n",
"\n",
" def forward(self, x):\n",
" x = self.patch_embed(x) # (B, N, d)\n",
" cls = self.cls_token.expand(x.shape[0], -1, -1)\n",
" x = torch.cat([cls, x], dim=1) + self.pos_embed # (B, N+1, d)\n",
" for blk in self.blocks:\n",
" x = blk(x)\n",
" return self.head(self.ln_f(x[:, 0])) # classify from the CLS token\n",
"\n",
"torch.manual_seed(SEED)\n",
"vit = ViT().to(device)\n",
"print(f\"ViT params {param_count(vit):,}\")\n",
"layer_summary(vit, torch.randn(2, 1, 28, 28).to(device)) # smoke test: (2,1,28,28) -> (2,10)"
]
},
{
"cell_type": "markdown",
"id": "f350c4e3",
"metadata": {},
"source": [
"> **Interpretation.** About 215k parameters, output `(2, 10)`: two images in, ten class logits each out. The `randn` smoke test passes before we touch any data. This is the lucidrains discipline: `preds = v(torch.randn(...))` after every architecture, so a shape bug surfaces in one second, not after a five-minute training run.\n"
]
},
{
"cell_type": "markdown",
"id": "2b900d63",
"metadata": {},
"source": [
"### 2.1 Loaders, sized for the CPU budget\n",
"\n",
"We subset to keep the full run under the budget and state the sizes in prose (spec §8 honest trimming). `FAST` mode shrinks both the data and the step count about tenfold so CI smoke runs exercise the same code paths in seconds.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "27b42583",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.392283Z",
"iopub.status.busy": "2026-06-10T20:02:31.392212Z",
"iopub.status.idle": "2026-06-10T20:02:31.395444Z",
"shell.execute_reply": "2026-06-10T20:02:31.395162Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"train 6000 . test 2000 . 47 steps/epoch . 3 epochs . FAST=False\n"
]
}
],
"source": [
"# subset sizes are stated, not hidden; FAST cuts them ~10x for CI\n",
"N_TRAIN = 600 if FAST else 6000 # of 60000 available\n",
"N_TEST = 400 if FAST else 2000 # of 10000 available\n",
"BATCH = 128\n",
"EPOCHS = 1 if FAST else 3\n",
"LR = 3e-4 # AdamW default-ish; small ViT, small data\n",
"\n",
"g = torch.Generator().manual_seed(SEED)\n",
"train_idx = torch.randperm(len(train_full), generator=g)[:N_TRAIN]\n",
"test_idx = torch.randperm(len(test_full), generator=torch.Generator().manual_seed(SEED + 1))[:N_TEST]\n",
"train_ds, test_ds = Subset(train_full, train_idx.tolist()), Subset(test_full, test_idx.tolist())\n",
"train_loader = DataLoader(train_ds, batch_size=BATCH, shuffle=True,\n",
" generator=torch.Generator().manual_seed(SEED))\n",
"test_loader = DataLoader(test_ds, batch_size=256, shuffle=False)\n",
"steps_per_epoch = math.ceil(N_TRAIN / BATCH)\n",
"print(f\"train {N_TRAIN} . test {N_TEST} . {steps_per_epoch} steps/epoch . {EPOCHS} epochs . FAST={FAST}\")"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "ff92a761",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.396164Z",
"iopub.status.busy": "2026-06-10T20:02:31.396095Z",
"iopub.status.idle": "2026-06-10T20:02:31.398312Z",
"shell.execute_reply": "2026-06-10T20:02:31.397936Z"
}
},
"outputs": [],
"source": [
"# @torch.no_grad eval helper: split-loss + accuracy (spec tier-2 convention)\n",
"@torch.no_grad()\n",
"def evaluate(model, loader):\n",
" model.eval()\n",
" correct, total, loss_sum = 0, 0, 0.0\n",
" for xb, yb in loader:\n",
" xb, yb = xb.to(device), yb.to(device)\n",
" logits = model(xb)\n",
" loss_sum += F.cross_entropy(logits, yb, reduction=\"sum\").item()\n",
" correct += (logits.argmax(-1) == yb).sum().item()\n",
" total += yb.numel()\n",
" model.train()\n",
" return loss_sum / total, correct / total"
]
},
{
"cell_type": "markdown",
"id": "7791839f",
"metadata": {},
"source": [
"### 2.2 The training loop (four comments, the same every chapter)\n",
"\n",
"The loop is the four-comment skeleton you have written in every deep-learning chapter: forward, backward, update, track. The one line that matters most for this part is `opt.zero_grad()`. We seed at the top so a learner re-running this cell in isolation reproduces the printed curve.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "22958ad6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:31.399038Z",
"iopub.status.busy": "2026-06-10T20:02:31.398968Z",
"iopub.status.idle": "2026-06-10T20:02:34.187560Z",
"shell.execute_reply": "2026-06-10T20:02:34.187244Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"final train loss 0.712 . test loss 0.826 . test acc 69.9%\n"
]
}
],
"source": [
"# > **Runtime:** ~2-4 min on CPU at full fidelity, seconds in FAST mode.\n",
"torch.manual_seed(SEED)\n",
"vit = ViT().to(device); vit.train()\n",
"opt = torch.optim.AdamW(vit.parameters(), lr=LR)\n",
"good_losses = []\n",
"for epoch in range(EPOCHS):\n",
" for xb, yb in train_loader:\n",
" xb, yb = xb.to(device), yb.to(device)\n",
" opt.zero_grad() # zero the gradient buffers FIRST\n",
" logits = vit(xb) # forward\n",
" loss = F.cross_entropy(logits, yb) # forward (loss)\n",
" loss.backward() # backward\n",
" opt.step() # update\n",
" good_losses.append(loss.item()) # track stats\n",
"te_loss, te_acc = evaluate(vit, test_loader)\n",
"print(f\"final train loss {good_losses[-1]:.3f} . test loss {te_loss:.3f} . test acc {te_acc:.1%}\")"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "d9c422d1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:34.188586Z",
"iopub.status.busy": "2026-06-10T20:02:34.188508Z",
"iopub.status.idle": "2026-06-10T20:02:34.247077Z",
"shell.execute_reply": "2026-06-10T20:02:34.246641Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAqkAAAEiCAYAAADauUtBAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAdMtJREFUeJzt3Xd4VEUXwOHfkh5CEhJaKIHQe++9I72oKKAUUUDpCCL6KVjoglJFLDRBqoggLfQivfcaehdIgBASkvn+GHaTTTZlIclu4LzPkyfsvXPvzk6W5OyUMwallEIIIYQQQgg7ks7WFRBCCCGEECI2CVKFEEIIIYTdkSBVCCGEEELYHQlShRBCCCGE3ZEgVQghhBBC2B0JUoUQQgghhN2RIFUIIYQQQtgdCVKFEEIIIYTdkSBVCCGEEELYHQlShUgDZs6cicFg4MKFC7auit3q3LkzHh4eSSprMBgYNmxYylboFXL58mVcXV3Zvn27ratiFy5cuIDBYGDmzJm2rgoAlStX5pNPPrF1NYSwmgSpQthAixYtcHd358GDB/GW6dChA87Ozvz3339mx40Ba2JfefLkSeFXkbiE6vrpp5/aunopqnbt2hgMBgoUKGDxfGBgoKktFi9ebDpubDNXV1euXr1q8b7Fixc3O5YnTx6aNWtmduzhw4cMHTqU4sWLkz59enx9fSldujR9+/bl2rVrpkAqKV+JfTj6+uuvqVSpEtWqVUti66Q9I0aM4K+//rJ1NZ7L4MGDmTJlCjdu3LB1VYSwiqOtKyDEq6hDhw4sX76cpUuX0rFjxzjnQ0NDWbZsGa+99hq+vr68++67vP3227i4uFCzZk3mzJljVv7999+nYsWKdOvWzXQsqb2KqeHrr78mICDA7FjsQCs1PX78GEfHlP/15+rqytmzZ9m9ezcVK1Y0Ozd37lxcXV0JCwuzeO2TJ08YNWoUkyZNsvp5IyIiqFmzJidPnqRTp0707t2bhw8fcuzYMebNm0fr1q2pUKFCnPfRuHHjuHLlCt9//73Z8cyZM8f7XLdv32bWrFnMmjXL6nqmJSNGjOCNN96gVatWtq6K1Vq2bImnpydTp07l66+/tnV1hEgyCVKFsIEWLVqQIUMG5s2bZzFIXbZsGY8ePaJDhw4AODg44ODgAEDevHnJmzevWfkePXqQN29e3nnnnZSv/HNo3Lgx5cuXt3U1TFxdXVPlefLly8fTp0/5448/zILUsLAwli5dStOmTVmyZInFa0uXLs3PP//MkCFDyJ49u1XP+9dff3HgwAHmzp1L+/btzc6FhYURHh5O+vTp47xf5s+fz71796x6H/3+++84OjrSvHlzq+r4oh49ekT69OnjHFdKERYWhpubW6rWx56lS5eON954g9mzZ/PVV19hMBhsXSUhkkSG+4WwATc3N9q0acP69eu5detWnPPz5s0jQ4YMtGjRAkiZOannz5/nzTffxMfHB3d3dypXrsw///xjVmbTpk0YDAYWLlzI8OHDyZkzJ66urtSrV4+zZ8++cB0uXrzIRx99RKFChXBzc8PX15c333wzzuuMiIjgq6++okCBAri6uuLr60v16tUJDAyMc8+rV6/SqlUrPDw8yJw5MwMHDiQyMtKsjKU5qQcOHKBx48Z4enri4eFBvXr12Llzp1kZ489h+/btDBgwgMyZM5M+fXpat27N7du3Lb7Gdu3asWDBAqKiokzHli9fTmhoKG3bto23bT777DMiIyMZNWpUvGXic+7cOQCLw++urq54enpafc/4/PXXX1SqVMliz/2uXbto0qQJGTNmJH369JQsWZIJEyaYldmwYQM1atQgffr0eHt707JlS06cOGFWZtiwYRgMBo4fP0779u3JmDEj1atXB6KnOqxZs4by5cvj5ubGTz/9BMD9+/fp168fuXLlwsXFhfz58zN69GiznwVAVFQUEyZMoESJEri6upI5c2Zee+019u7dC+j3y6NHj5g1a5ZpCkTnzp2tbqukvFaw7r24ZcsWunfvjq+vL56ennTs2JF79+7FuWeDBg24ePEiBw8etLreQtiKBKlC2EiHDh14+vQpCxcuNDt+9+5d1qxZQ+vWrVOsN+jmzZtUrVqVNWvW8NFHHzF8+HDCwsJo0aIFS5cujVN+1KhRLF26lIEDBzJkyBB27txp6uVNiuDgYO7cuWP2BbBnzx7+/fdf3n77bSZOnEiPHj1Yv349tWvXJjQ01HT9sGHD+Oqrr6hTpw6TJ0/m888/x9/fn/3795s9T2RkJI0aNcLX15fvvvuOWrVqMW7cOKZPn55g/Y4dO0aNGjU4dOgQn3zyCV988QVBQUHUrl2bXbt2xSnfu3dvDh06xNChQ/nwww9Zvnw5vXr1snjv9u3bc/36dTZt2mQ6Nm/ePOrVq0eWLFnirVNAQAAdO3bk559/5tq1awnWP7bcuXMDMHv2bJRSVl1rjYiICPbs2UPZsmXjnAsMDKRmzZocP36cvn37Mm7cOOrUqcOKFStMZdatW0ejRo24desWw4YNY8CAAfz7779Uq1bN4geyN998k9DQUEaMGMEHH3xgOn7q1CnatWtHgwYNmDBhAqVLlyY0NJRatWrx+++/07FjRyZOnEi1atUYMmQIAwYMMLtv165dTcHs6NGj+fTTT3F1dTUFhnPmzMHFxYUaNWowZ84c5syZQ/fu3a1qq6S+Vmvfi7169eLEiRMMGzaMjh07MnfuXFq1ahXn516uXDkAWdwm0hYlhLCJp0+fKj8/P1WlShWz49OmTVOAWrNmjenYjBkzFKCCgoIs3it9+vSqU6dOSX7ufv36KUBt3brVdOzBgwcqICBA5cmTR0VGRiqllNq4caMCVJEiRdSTJ09MZSdMmKAAdeTIkQSfx1hvS19KKRUaGhrnmh07dihAzZ4923SsVKlSqmnTpgk+V6dOnRSgvv76a7PjZcqUUeXKlTM7BqihQ4eaHrdq1Uo5Ozurc+fOmY5du3ZNZciQQdWsWTPO66lfv76KiooyHe/fv79ycHBQ9+/fNx2rVauWKlasmFJKqfLly6uuXbsqpZS6d++ecnZ2VrNmzTK176JFi+I8x549e9S5c+eUo6Oj6tOnj8X7GuXOndusfUJDQ1WhQoUUoHLnzq06d+6sfv31V3Xz5s0E27Bp06Yqd+7cCZaJ6ezZswpQkyZNMjv+9OlTFRAQoHLnzq3u3btndi5mu5UuXVplyZJF/ffff6Zjhw4dUunSpVMdO3Y0HRs6dKgCVLt27eLUIXfu3ApQq1evNjv+zTffqPTp06vTp0+bHf/000+Vg4ODunTpklJKqQ0bNijArI0t1dWa/2NBQUEKUDNmzLD6tVr7XixXrpwKDw83HR8zZowC1LJly+LUy9nZWX344YdJeg1C2APpSRXCRhwcHHj77bfZsWOHWU/KvHnzyJo1K/Xq1Uux5165ciUVK1Y0DZmCXmjVrVs3Lly4wPHjx83Kd+nSBWdnZ9PjGjVqAHrKQFJMmTKFwMBAsy/ArKc4IiKC//77j/z58+Pt7W3WS+rt7c2xY8c4c+ZMos/Vo0cPs8c1atRIsJ6RkZGsXbuWVq1amc319fPzo3379mzbto2QkBCza7p162Y2r69GjRpERkZy8eJFi8/Rvn17/vzzT8LDw1m8eDEODg60bt060deSN29e3n33XaZPn87169cTLW/k5ubGrl27GDRoEKCHhrt27Yqfnx+9e/fmyZMnSb5XQoyZJzJmzGh2/MCBAwQFBdGvXz+8vb3Nzhnb7fr16xw8eJDOnTvj4+NjOl+yZEkaNGjAypUr4zxf7J+tUUBAAI0aNTI7tmjRImrUqEHGjBnNevDr169PZGQkW7ZsAWDJkiUYDAaGDh0a577JNXczqa/1ed+LTk5Opscffvghjo6OFtvP2BZCpBUSpAphQ8Yh83nz5gFw5coVtm7dyttvv21aKJUSLl68SKFCheIcL1KkiOl8TP7+/maPjUGJpblvllSsWJH69eubfYFeZf/ll1+a5gxmypSJzJkzc//+fYKDg03Xf/3119y/f5+CBQtSokQJBg0axOHDh+M8j3E+Yey6JlTP27dvExoaGm97REVFcfnyZbPj1rbH22+/TXBwMKtWrWLu3Lk0a9aMDBkyxFunmP73v//x9OlTq+emenl5MWbMGC5cuMCFCxf49ddfKVSoEJMnT+abb76x6l6JUbGGlo1zYhPK4GB8j8XX7nfu3OHRo0dmx2NniEjo+JkzZ1i9ejWZM2c2+zK+94xzwc+dO0f27NnNgsfkltTX+jzvxdgpzjw8PPDz87M4XUIpJYumRJoiQaoQNlSuXDkKFy7MH3/8AcAff/yBUsqq+Z6pIb6AOXZwYq3evXszfPhw2rZty8KFC1m7di2BgYH4+vqaLW6pWbMm586d47fffqN48eL88ssvlC1bll9++SVJ9Uxu1raHn58ftWvXZty4cWzZsiXOivuEGLM2WNubGlPu3Ll577332L59O97e3sydO/e57hObr68vkPQPKy8qvjnalo5HRUXRoEGDOD34xq/XX389patrd+7fv0+mTJlsXQ0hkkxSUAlhYx06dOCLL77g8OHDzJs3jwIFClChQoUUfc7cuXNz6tSpOMdPnjxpOp8aFi9eTKdOnRg3bpzpWFhYGPfv349T1sfHhy5dutClSxcePnxIzZo1GTZsGO+///4L1SFz5sy4u7vH2x7p0qUjV65cL/QcoIf833//fby9vWnSpIlV1/7vf//j999/Z/To0S9Uh4wZM5IvXz6OHj36Qvcx8vf3x83NjaCgILPj+fLlA+Do0aOmnsvYjO+x+No9U6ZMFlNMJVW+fPl4+PBhvM8fs9yaNWu4e/dugr2pL9IDmdTX6urqavV78cyZM9SpU8f0+OHDh1y/fj3Oe+zq1auEh4ebRkuESAukJ1UIGzP2mn755ZccPHgwVXpRmzRpwu7du9mxY4fp2KNHj5g+fTp58uShaNGiKV4H0D2SsXsfJ02aFCdlVOxdtzw8PMifP3+yzK10cHCgYcOGLFu2zGyI9ObNm8ybN4/q1asnS8qmN954g6FDhzJ16lSz+b1JkS9fPt555x1++umnJO0adOjQIYtzDy9evMjx48ctDic/DycnJ8qXL29K1WRUtmxZAgIC+OGHH+J84DD+vP38/ChdujSzZs0yK3P06FHWrl1rdSAfW9u2bdmxYwdr1qyJc+7+/fs8ffoUgNdffx2lFF999VWccjHfm+nTp7f44Skpkvpan+e9OH36dCIiIkyPf/zxR54+fUrjxo3Nyu3btw+AqlWrPtdrEMIWpCdVCBsLCAigatWqLFu2DCBVgtRPP/2UP/74g8aNG9OnTx98fHyYNWsWQUFBLFmyhHTpUufza7NmzZgzZw5eXl4ULVqUHTt2sG7dOtMwslHRokWpXbs25cqVw8fHh71797J48eJ40z5Z69tvvyUwMJDq1avz0Ucf4ejoyE8//cSTJ08YM2ZMsjyHl5dXnNys1vj888+ZM2cOp06dolixYgmWDQwMZOjQobRo0YLKlSvj4eHB+fPn+e2333jy5MkL1SO2li1b8vnnnxMSEmIKoNKlS8ePP/5I8+bNKV26NF26dMHPz4+TJ09y7NgxU+A4duxYGjduTJUqVejatSuPHz9m0qRJL9xWAIMGDeLvv/+mWbNmdO7cmXLlyvHo0SOOHDnC4sWLuXDhApkyZaJOnTq8++67TJw4kTNnzvDaa68RFRXF1q1bqVOnjuk9Vq5cOdatW8f48ePJnj07AQEBVKpUKcn1Seprtfa9GB4eTr169Wjbti2nTp1i6tSpVK9e3ZRj2SgwMBB/f3/KlCnzfA0qhC3YLK+AEMJkypQpClAVK1a0eD65U1AppdS5c+fUG2+8oby9vZWrq6uqWLGiWrFihVkZSymSlLKcYieheu/Zs8fi+Xv37qkuXbqoTJkyKQ8PD9WoUSN18uRJlTt3brPX8+2336qKFSsqb29v5ebmpgoXLqyGDx9ulnqnU6dOKn369HGew5i+KCZipaBSSqn9+/erRo0aKQ8PD+Xu7q7q1Kmj/v333yS9HmM7bdy40XTMUqqo2BJLQRWbMc1WYimozp8/r7788ktVuXJllSVLFuXo6KgyZ86smjZtqjZs2BBvfaxNQaWUUjdv3lSOjo5qzpw5cc5t27ZNNWjQQGXIkEGlT59elSxZMk66qnXr1qlq1aopNzc35enpqZo3b66OHz9uVsb4M7x9+3ac54j92mN68OCBGjJkiMqfP79ydnZWmTJlUlWrVlXfffed2Xvn6dOnauzYsapw4cLK2dlZZc6cWTVu3Fjt27fPVObkyZOqZs2ays3NTQEJ/n+L7/9HUl6rUta9Fzdv3qy6deumMmbMqDw8PFSHDh3M0lwppVRkZKTy8/NT//vf/+KtsxD2yKBUCmZ6FkII8dLr2rUrp0+fZuvWrbauyitj5syZdOnShT179iS65fBff/1F+/btOXfuHH5+fqlUQyFenMxJFUII8UKGDh3Knj17ZDcjOzV69Gh69eolAapIc2ROqhBCiBfi7+9PWFiYrash4hFzgaQQaYn0pAohhBBCCLsjc1KFEEIIIYTdkZ5UIYQQQghhdyRIFUIIIYQQdueVWzgVFRXFtWvXyJAhwwttcyeEEEIIIayjlOLBgwdkz5490Y1jXrkg9dq1a8myD7cQQgghhHg+ly9fJmfOnAmWeeWC1AwZMgC6cZJjP24hhBBCCJE0ISEh5MqVyxSPJeSVC1KNQ/yenp4SpAohhBBC2EBSplzKwikhhBBCCGF3JEgVQgghhBB2R4JUIYQQQghhdyRIFUIIIYQQdkeCVCGEEEIIYXckSE1h94IhPMLWtRBCCCGESFteuRRUqW3wd7B2O2TPArn8wN8P3nwNKpa0dc2EEEIIIeyXBKkprGcHqFsZLl2DS9dh0264eBUWTrB1zYQQQggh7JcEqSmsVGH9ZfTTAhg/A55GgqOD7eolhBBCCGHPXtkgNTw8HKWUaceDyMhIIiMjSZcuHY6OjmblAJycnJKlbNmiEPYEjp6KoGh+haOjI+nS6anBUVFRPH36FIPBgJOTk+m+ERERKGXbsk+fPiUqKgoHBwccHBysLquUIiJCT851dnZOkbKW2t2asin9s0+o3a0pm9beJ6nxs3/R90lq/exf9H2S1n729vY+kd8Rti8rvyPkd0RkZCRJ9coGqVOmTGHQoEG4u7sDsGvXLrZu3UrJkiVp3LixqdzkyZOJiIigR48eeHl5AbB//342bNhA0aJFad68uanstGnTePz4Me+99x6ZM2cG4MiRI6xZs4YCBQrQpk0bShTUPaiBq35hzdMQOnbsiJ+fHwAnTpxgxYoV5M6dm7ffftt031mzZvHff//Rrl07/P39ATh79ixLly4lR44cvPPOO6ay8+bN48aNG7zxxhvky5cPgIsXL7Jw4UKyZMlCly5dTGUXLVrE5cuXadmyJYUL6+7ea9euMXfuXDJmzEi3bt1MZZcuXcr58+dp0qQJJUqUAOD27dvMnDkTDw8PevbsaSq7YsUKTp06RYMGDShbtiwA9+7d4+eff8bFxYV+/fqZyq5Zs4ajR49Su3ZtKlWqBMDDhw+ZOnUq6dKlY9CgQaayGzZs4MCBA1SrVo3q1asD8OTJEyZM0HMnBg4caPqFs2XLFnbv3k3FihWpU6cOoP/jfP/99wD07dsXV1dXAHbs2MH27dspU6YMDRs2ND3fhAkTiIqK4qOPPjLtMbx37142bdpE8eLFadq0qans1KlTefLkCR988AE+Pj4AHDp0iMDAQAoVKkSrVq1MZadPn87Dhw/p3LkzWbNmBeD48eOsXLmSvHnz8uabb5rKzpgxg3v37tGhQwdy5swJwOnTp1m2bBm5cuWiffv2prK///47t27dom3btgQEBAAQFBTE4sWLyZYtG506dTKVXbBgAVevXqV169YULFgQgCtXrvDHH3/g6+vL+++/byq7ZMkSLl68SLNmzShWrBgAN2/eZPbs2Xh6evLhhx+ayv7999+cOXOGRo0aUbp0aQDu3LnDb7/9hpubG3369DGVXbVqFcePH6du3bpUqFAB0Hs6T5s2DScnJwYMGGAqGxgYyOHDh6lRowZVq1YF4PHjx0yaNAmAwYMHm8pu2rSJffv2UaVKFWrWrAnoX6TGn33//v1Nf7C2b9/Ojh07KFeuHPXr1zfdw1i2d+/eqf47wuiXX34hJER+R8jvCPkdIb8jXq7fEcY2TwpZ3Z/KXF2gWAFZ8S+EEEIIkRCDUkrZuhKpKSQkBC8vL27fvo2vr69NuumHToTNuyNY+1va66aXoTwZyrPHn70M5dm+rPyOkN8R8jvC9u+TtPCzf/ToET4+PgQHB+Pp6UlCXtkgNSmNk1KWrYc+38L+peDrbZMqCCGEEEKkOmviMBnut4GyesoOB47bth5CCCGEEPZKglQbyJkVMmeE/RKkCiGEEEJYJEGqDRgMujdVglQhhBBCCMskSLWRMkXh0AmwIl2YEEIIIcQrQ4JUGylbFELD4FSQrWsihBBCCGF/JEi1kRIFwSGdDPkLIYQQQlgiQaqNuLtBkXyywl8IIYQQwhIJUm1IFk8JIYQQQlgmQaoNlSkK5y/DvWBb10QIIYQQwr5IkGpD5Yrq7wdO2LYeQgghhBD2RoJUG/LPDj5esPuwrWsihBBCCGFfJEi1IYMBWtSFXxfDniO2ro0QQgghhP2QINXGPv9QL6D64H9w4aqtayOEEEIIYR8kSLUxZyf46SvI6AVdhsD9EFvXSAghhBDC9iRItQPenjBzpF7l3+0LeBJu6xoJIYQQQtiWBKl2IncO+PlbvdJ/wixb10YIIYQQwrYkSLUjFUpAh+awcDVERtq6NkIIIYQQtiNBqp1p0xBu34Xt+21dEyGEEEII25Eg1c6UKAj5csHSQFvXRAghhBDCdiRItTMGA7RqAKu3QuhjW9dGCCGEEMI2JEi1Q63qQ2gYrN1u65oIIYQQQtiGBKl2yN9PL6Jaus7WNRFCCCGEsA0JUu1U6/qwdY9eRCWEEEII8aqRINVONa0N6dLB8g22rokQQgghROqTINVOeXtC3coy5C+EEEKIV5NNg9SRI0dSoUIFMmTIQJYsWWjVqhWnTp1K9LpFixZRuHBhXF1dKVGiBCtXrkyF2qa+1g3g8Ck4e8nWNRFCCCGESF02DVI3b95Mz5492blzJ4GBgURERNCwYUMePXoU7zX//vsv7dq1o2vXrhw4cIBWrVrRqlUrjh49moo1Tx11K4OnB7zWFUq3hGpv639v3m3rmgkhhBBCpCyDUkrZuhJGt2/fJkuWLGzevJmaNWtaLPPWW2/x6NEjVqxYYTpWuXJlSpcuzbRp0xJ9jpCQELy8vAgODsbT0zPZ6p5S9h2DI6fhUSg8egybdsOTcFg3Q+dUFUIIIYRIK6yJwxxTqU5JEhwcDICPj0+8ZXbs2MGAAQPMjjVq1Ii//vrLYvknT57w5MkT0+OQkJAXr2gqKldMfxnVrghv9oXNe/S/hRBCCCFeRnazcCoqKop+/fpRrVo1ihcvHm+5GzdukDVrVrNjWbNm5caNGxbLjxw5Ei8vL9NXrly5krXeqa1CCShZCH5ZZOuaCCGEEEKkHLsJUnv27MnRo0eZP39+st53yJAhBAcHm74uX76crPdPbQYDvP8mbN0LJ8/bujZCCCGEECnDLoLUXr16sWLFCjZu3EjOnDkTLJstWzZu3rxpduzmzZtky5bNYnkXFxc8PT3NvtK6JrXALzP8utjWNRFCCCGESBk2DVKVUvTq1YulS5eyYcMGAgICEr2mSpUqrF+/3uxYYGAgVapUSalq2h0nR+jcGv5aJztSCSGEEOLlZNMgtWfPnvz+++/MmzePDBkycOPGDW7cuMHjx49NZTp27MiQIUNMj/v27cvq1asZN24cJ0+eZNiwYezdu5devXrZ4iXYTLtm4OgAc5bZuiZCCCGEEMnPpkHqjz/+SHBwMLVr18bPz8/0tWDBAlOZS5cucf36ddPjqlWrMm/ePKZPn06pUqVYvHgxf/31V4KLrV5GXhmgbWMdpAY/hFNBsGoL/LwQbv5n69oJIYQQQrwYu8qTmhrSWp7UhFy8CrXehdg/we5vwWc9bFMnIYQQQoj4pNk8qcI6uXPAlC8h5BHkywV5c8Go6bBhpwSpQgghhEjbJEhN45rWNn9ctzIsWg2Xb0AuywkPhBBCCCHsnl2koBLJp0Z5vaBq405b10QIIYQQ4vlJkPqSyZAeKpTUQ/5CCCGEEGmVBKkvobqV4d8DEPbE1jURQgghhHg+EqS+hOpWhifhOlAVQgghhEiLJEh9CeXLBbn8ZMhfCCGEEGmXBKkvIYNB96Zu3GWeQ/XRY/jfDzrxvxBCCCGEPZMg9SVVtzJcuQFnLurHUVHQf6Teoeqjr2S+qhBCCCHsmwSpL6nKpcDVJXrI//uZsHYbDOkOl6/DyJ9sWj0hhBBCiARJMv+XlKsLVCur86XmyAIT58DgD6DH2/rc0IlQpzLUrmjrmgohhBBCxCU9qS+xupVhzxH4eDS0qg8fttPHO7WCWhVh4Gj4774tayiEEEIIYZkEqS+xOpUgMgqK5IXRg/SCKtDfvxsMkZHw6Xfmi6uEEEIIIeyBBKkvsRxZYdZomDEKXJ3Nz2XxgdEDYe12WL3VNvUTQgghhIiPBKkvudoVwcfL8rmG1aFSSfh1cerWSQghhBAiMRKkvuK6vK7nrR45ZeuaCCGEEEJEkyD1FdewGuTMBr8tefF77T4M1dvDzf9e/F5CCCGEeLVJkPqKc3CAzq1h+cYXCy6jouDrKToH6x8rkq9+QgghhHg1SZAqaNsEnBzh92UJl7t0HUb/DPdD4p77ZxMcOQ2lCsPc5RDxNEWqKoQQQohXhNVB6urVq9m2bZvp8ZQpUyhdujTt27fn3r17yVo5kTq8PODNxjq4DAu3XOavddDkA5g6D/oM1+mrjMIjYOyvOi/r6IFw6z+dNUAIIYQQ4nlZHaQOGjSIkBDdlXbkyBE+/vhjmjRpQlBQEAMGDEj2CorU0aWNTuz/93rz4w9DYcBI6DtcB6E/DoOte2HcjOgyf6zQvayfdoMi+aBCCZj9VypWXgghhBAvHau3RQ0KCqJo0aIALFmyhGbNmjFixAj2799PkyZNkr2CInUE5NRB6IwlUDgvHDsLx87Axl1wLxjGfwptGuqNAAa/DyOnQ4mCUKM8TJgNbzSCQgH6Xh1bQe9v4HQQFAyIfo6dB2HZevi2n54LK4QQQggRH6uDVGdnZ0JDQwFYt24dHTt2BMDHx8fUwyrSpq5vQIeB0LwHpEsH+XJBxRLQp6MOYo26vw2HT8HHo3Su1YePoH/n6POv1YDMGWH2Mh2Qgi7/3mfw6DG80xKK5U/NVyaEEEKItMbqILV69eoMGDCAatWqsXv3bhYsWADA6dOnyZkzZyJXC3tWrSz8NgJ8vXWvqJur5XIGA4wdDK17wtJA6P6W3t3KyNkJ2jXTmwQM/kBnDeg0GArm0T20uw9JkCqEEEKIhFk9J3Xy5Mk4OjqyePFifvzxR3LkyAHAqlWreO2115K9giL1GAxQrwqULhJ/gGqU3g1+/gbebgo9O8Q936E5hD3RC63eHQSZMsLMUXr1/+4jKVN/IYQQQrw8DEopZetKpKaQkBC8vLwIDg7G09PT1tV5qfUYCqu26M0ClkyEbJlhzC+wYCXsXaKDYiGEEEK8OqyJw6zuSd2/fz9HjkR3hS1btoxWrVrx2WefER4eT/4i8Ur6qD2UKwa/j9UBKkDFknDnHgRdsW3dhBBCCGHfrA5Su3fvzunTpwE4f/48b7/9Nu7u7ixatIhPPvkk2Sso0q6SheDPyeaLrsoV04uydh22Xb2EEEIIYf+sDlJPnz5N6dKlAVi0aBE1a9Zk3rx5zJw5kyVLkmEDePFSy5BeL5raI0GqEEIIIRJgdZCqlCIqKgrQKaiMuVFz5crFnTt3krd24qVUsWTSelI37IROn8KDRylfJyGEEELYF6uD1PLly/Ptt98yZ84cNm/eTNOmTQGd5D9r1qyJXC2EDlKv3IBrtxIut3AVbNoFvb6Bp5EJlxVCCCHEy8XqIPWHH35g//799OrVi88//5z8+XXCy8WLF1O1atVkr6B4+VQoob/vTqA3VSl9vkoZ2LoHvp2aOnUTQgghhH2wOpl/yZIlzVb3G40dOxYH2etSJIGvN+TPDXuOQKv6lsucvQT/3YeP2kHTWvC/HyCfP7zbMhUrKoQQQgibsTpINdq3bx8nTpwAoGjRopQtWzbZKiVefhVLwK5D8Z/fdQgc0kG54lCzApy7BEMnQp4cUKN86tVTCCGEELZhdZB669Yt3nrrLTZv3oy3tzcA9+/fp06dOsyfP5/MmTMndx3FS6hiSZi3Au4Gg49X3PO7DukUVund9OP/fQRBV/UGAbPH6FRWQgghhHh5WT0ntXfv3jx8+JBjx45x9+5d7t69y9GjRwkJCaFPnz4pUUfxEqpYUn/fY2GLVKVg5yGoVCr6mKMDTB0KRfJBx09g//HUqacQQgghbMPqIHX16tVMnTqVIkWKmI4VLVqUKVOmsGrVqmStnHh55cgKObNaXjx14Src+s88SAXdqzpzFBTOqwPVgydSp65CCCGESH1WB6lRUVE4OTnFOe7k5GTKnypEUlQoaTlI3XVI70plzAIQk4c7zBoNBfPAu4Pg0Mnnf/4Zf8If/zz/9UIIIYRIOVYHqXXr1qVv375cu3bNdOzq1av079+fevXqJWvlxMutahk4egbOXzY/vuuQ3pUqQ3rL1xkD1Xy5oevn8DDU+ueOjISJs2Hu39ZfK4QQQoiUZ3WQOnnyZEJCQsiTJw/58uUjX758BAQEEBISwqRJk1KijuIl1aIeZPWF8TPMj+86HHeoP7YM6WHKlxDyEKbMtf659x3Ti7ZOX5CNAoQQQgh7ZPXq/ly5crF//37WrVvHyZN6rLVIkSLUrx9Pwksh4uHqDH07waffQY92ULwAXL4BV29CpZKJX58jK3R/C6bNh3ZNwT970p97zTYwGOBJuJ4Dm9//+V+HEEIIIZKf1T2pAAaDgQYNGtC7d2969+4tAap4bm++Bnlzwdhf9eNdB3XwWDEJQSrAh+10Cqvh05L+nErB2u3QuKZ+fPKcVVUWQgghRCpIUk/qxIkTk3xDa9JQbdmyhbFjx7Jv3z6uX7/O0qVLadWqVbzlN23aRJ06deIcv379OtmyZUvy8wr74egAH3eBnl/rRVS7DkPhAPD2TNr17m4wuBv0HwE7DkKV0olfc/oCXLoG3/TVw/4nzkOzuG8rIYQQQthQkoLU77//Pkk3MxgMVgWpjx49olSpUrz33nu0adMmydedOnUKT8/oKCZLlixJvlbYnya1oFgBGP0z3L4LdSpZd32rejD7L/h6Mqz4CRLbnXfNNr34qkppnc7q5PnnrbkQQgghUkqSgtSgoKAUefLGjRvTuHFjq6/LkiWLabcrkfalSwefvA+dBuvHiS2asnT9sF7Q8iNYsBLaN0+4/NptULsiuDhDkbywfOPz1VsIIYQQKee55qTaWunSpfHz86NBgwZs377d1tURyaBWheh5qNYGqQCli8AbjeCbqQnvRnXtFhw5DQ2r68eF8+qFWsEPrX9OIYQQQqScNBWk+vn5MW3aNJYsWcKSJUvIlSsXtWvXZv/+/fFe8+TJE0JCQsy+hP0xGGDEAPj8Q/D1fr57fNsPihfUPbIn4lkMFbgdnByjpxQUyae/n5IhfyGEEMKupKkgtVChQnTv3p1y5cpRtWpVfvvtN6pWrZrgnNmRI0fi5eVl+sqVK1cq1lhYo0Bu6Nb2+a93c4Vfh4O/H7wzKO4mAaBX9VcuDZ4e+nHeXDpojS+oFUIIIYRtpKkg1ZKKFSty9uzZeM8PGTKE4OBg09flyxYiF/HS8PSAOWN1doAOA+HKjehzwQ9h50FoWC36mLOTDo5l8ZQQQghhX6xO5m9vDh48iJ+fX7znXVxccHFxScUaCVvz8YK5Y+GNvlDrHShbDKqXhcgovbtUg2rm5Qvn1WmohBBCCGE/nitIvX//Prt37+bWrVtERUWZnevYsWOS7/Pw4UOzXtCgoCAOHjyIj48P/v7+DBkyhKtXrzJ79mwAfvjhBwICAihWrBhhYWH88ssvbNiwgbVr1z7PyxAvsWyZ4a8psGoLbN0HvyyCkEdQujD4ZTYvWzgfrN4KUVE6U4AQQgghbM/qIHX58uV06NCBhw8f4unpicFgMJ0zGAxWBal79+41S84/YMAAADp16sTMmTO5fv06ly5dMp0PDw/n448/5urVq7i7u1OyZEnWrVtnMcG/EJkywrst9VdkpF7Vnylj3HJF8kJoGFy6DnlypH49hRBCCBGXQSmlrLmgYMGCNGnShBEjRuDu7p5S9UoxISEheHl5ERwcbLYhgHh13boLFV6HaV9Fb5UaXzmHdM+ffUAIIYR41VkTh1k9uHn16lX69OmTJgNUISzJnFEHngktnnoaCa0+0sHsu4P0pgHBD1KtikIIIcQrx+ogtVGjRuzduzcl6iKETRgMiS+e2rBDJ/3v9Q6EP4XB30G5NvCnTIcWQgghUoTVc1KbNm3KoEGDOH78OCVKlMDJycnsfIsWLZKtckKkliL5dA7V+Pz+t150NaCLfnzzDnw6Dr6fBa3qy4IrIYQQIrlZHaR+8MEHAHz99ddxzhkMBiIjI1+8VkKksiJ5dQaAh6HgEWsmy6VrsHkPjP0k+ljWTND7XWjdEzbthrqVU7e+QgghxMvO6v6fqKioeL8kQBVpVWHj9qhBcc/NW6E3CWgeK4lEmSJQvADM/ivFq0dYOHw9RebBCiGEeHXIIKUQQP7ceuX+yVjboz4J14uk3mikt12NyWDQ6a027da9rSlp50H4dTFs25eyzyOEEELYiyQN90+cOJFu3brh6urKxIkTEyzbp0+fZKmYEKnJ1Rny5oINu+CtpuDooI+v3gp3g6F9c8vXtawHI36COX/D5z1Srn67D+vvZy8lXE4IIYR4WSQpT2pAQAB79+7F19eXgICA+G9mMHD+vH3vLyl5UkV8/gyEgaOgejmY9CV4eUDbvmBIBwu+j/+6b6bC4jWwayG4ptAOvG/21YFqy3ow8X8p8xxCCCFESrMmDktST2pQUJDFfwvxMmnTALL4wEdf6QVRn34Auw7D5C8Svu7dlnrR1d8boG3j5K9XWDgcOgHOTnBOelKFEEK8ImROqhAxVC8Hy6YCCj74Qm+j2qhGwtfkyQG1KuoFVNbt35Y0h0/Ckwh4rQacuwxRUcn/HEIIIYS9sToFFcCVK1f4+++/uXTpEuHh4Wbnxo8fnywVE8JWAnLC0qnwv++hYkndg5mYTq3gvc/g4AkoUzR567P7sE6L1byO7q29cQeyZ0ne5xBCCCHsjdVB6vr162nRogV58+bl5MmTFC9enAsXLqCUomzZsilRRyFSnZcHTEpkmD+m2hV1j+oPs2HWqOSty+4jUK44FHw2HfzcJQlShRBCvPysHu4fMmQIAwcO5MiRI7i6urJkyRIuX75MrVq1ePPNN1OijkLYPQcH+LQbbNoFG3dZLvM4zPr7RkbC/mNQsQTkzPZsXurlF6urEEIIkRZYHaSeOHGCjh07AuDo6Mjjx4/x8PDg66+/ZvTo0cleQSHSitdqQOXSerV/xFPzc0dOQbk28PNC6+554jw8eKSnHTg66N5aWTwlhBDiVWB1kJo+fXrTPFQ/Pz/OnYvOfn7nzp3kq5kQaYzBAF/2hPOX4fdl0cev3YL3PoenT2HiHAh+mPR77j6ke09LFtaP8/lLkCqEEOLVYHWQWrlyZbZt2wZAkyZN+Pjjjxk+fDjvvfcelSvLBubi1VYsP7zdBL6fBfeC4WEodP0MnB31YqzwCJi+IOn3230EShXWmw1A0oPUS9fgw2G6F1YIIYRIi6wOUsePH0+lSpUA+Oqrr6hXrx4LFiwgT548/Prrr8leQSHSmoFd9VzS736D3t/Apevw20gdwL73Ovy2GG7fTfw+SsGeI3qo3yifv17d/zA04WtH/AQrN8Pf61/stQghhBC2YlWQGhkZyZUrV/D39wf00P+0adM4fPgwS5YsIXfu3ClSSSHSkkwZofe78PvfsHk3TB0KhZ6tzO/+lp5bOmVu4vcJugJ37pkHqfn1fz3OJ7B4at8xWLUFfLxg4arnfx1CCCGELVkVpDo4ONCwYUPu3buXUvUR4qXQpY1O8D/yY/3dyNsTur0Nc5fDlRsJ32PXYUiXDsoViz6WN5f+fjaeIX+lYMQ0KJofvu0HB0/C6SRsEhcVBXeDEy8nhBBCpBarh/uLFy/O+fPnU6IuQrw0XJxh9mh4q0ncc++9Dp4eMGF2wvfYcxiK5IMM6aOPebhDtkzxz0tdux32HoXPukODas96U1cnXt9l66Ha2xBixaIuIYQQIiVZHaR+++23DBw4kBUrVnD9+nVCQkLMvoQQCUvvBr06wOI1sP94/OV2H4FKJeMej2/x1NNIGD0dapaHGuV1VoBW9WFpYNyUWLFt2wehYfq7EEIIYQ+sDlKbNGnCoUOHaNGiBTlz5iRjxoxkzJgRb29vMmbMmBJ1FOKl0745lCgIb/aBSb/rANPo9l0Y/B1cvg5VSse9Np+/5TmpC/6B81fg0+7Rx958Tc9r3bgz4frsPaq/x7cRgRBCCJHarN4WdePGjSlRDyFeKS7OsHgi/DALxs+A9Ttg9EDYsBMm/w6OjjCstx6yjy2fP8z/R2cQcHDQxx49hu9nQuv6OouAUdH8OhhetBoaVrdcl9t34cJVvaPVpt16XqvBkOwvWQghhLCK1UFqQEAAuXLlwhDrr5hSisuXZb9GIZLK2Qk+eR/qVYEBI6Hhe+CQDjq2gn6d9CIrS/L763yrV25A7hz62M8L9XzSj9+LW/7N1+CryXDrLmTxiXt+3zH9vX8n+Hg0HD9nHugKIYQQtmD1cH9AQAC3b9+Oc/zu3bsEBAQkS6WEeJWUKwarfta7Va35Tfegxheggu5JhegV/rfuwk/zoXMb3RsaW8t6Ou3V0kDL99t7FLJngeZ19XzZTc8x5D99IbzV3/rrhBBCiPhYHaQqpeL0ogI8fPgQV1fXZKmUEK8adzfo+gYUSEKq4WyZwN0Vzj0buJgwC5ycoGcHy+W9PfVQ/6JVeig/tr1HoXxxPQWhWlnr56UqBXOWwc6DksZKCCFE8knycP+AAQMAMBgMfPHFF7i7u5vORUZGsmvXLkqXLp3sFRRCmDMYdL7Uc5f01x8r9GIprwzxX9O2Mbz7CRw8AWWKRh8PewJHT+ssAAC1K8IXEyD4IXh5JK0+h07qbVgB9h21PI82Mf/d1999va2/VgghxMspyUHqgQMHAN2TeuTIEZydnU3nnJ2dKVWqFAMHDkz+Ggoh4jCmoRr9C/hl1vNYE1KtrB7SX7TaPEg9fEqnpypfXD+uXQkio2DbXmhaO7pceITOEpA9S9x7L1sPmX30xgN7jlgfpCoFnQbrIHvud9ZdK4QQ4uWV5CDVuKq/S5cuTJgwAU/PBCbNCSFSVD5/vfVpeAT88Bm4Oidc3sEBXm8EM/+ELz4Ct2czc/Ye1fNQC+fVj3NkhYJ59JC/MUiNjITuX8KOg7B5DmTNFH3fyEhYvhGa19FZAvYctf61rN8BR06DkyOEPtZTH4QQQgir56TOmDFDAlQhbMy4wr9YAb0wKinebAQPHsGqrdHH9h7VPauODtHH6lTSqaiiovTjcTP043QGmDjH/J47DurgtGU9KF8CjpzSUwiSSimdhisgp+7R3Xko6dcKIYR4uVkdpAohbK94AZ3C6vMeepg9KXLngMql9QIq0EHovmPRQ/1GtSvpwPPYWVixEabMhcEfQJ93dX5W4/xTgL83QO7sUKowVCihA81DJ5P+Ojbu0r2o3/bTvbhb9yb9WiGEEC83CVKFSINy54Ajy/VcU2u0fQ3+PQCXruvsAPdD4gap5YvrKQDTF8DAMdCiLnR/Czq1hoxeuucT4Em4nnLQvK5ezFU4ADzc9bzUpFBKZyYoX1y/jurlJEgVQggRTYJUIdIoVxfrr2lcUweSS9boof506cwXUoHuoa1eXveS5s0FYwbpINTNFXq/A38Gwukg2LxbbyBgnG7g4KBzviZ1XurmPXDwpN64wGCAmuXhzEW4HjcNs3gB85ZDj6G2roUQQlhPglQhXiHubtCsjl7lv+ewXjDl4R633BsNdYD68zfRi6wA2jXTw/LjZuggtkhevdDKqHwJ2H9UL6hKiHEuapmiugcVdG+qwQDb9r3wyxQx7DgIa7fB4zBb10QIIawjQaoQr5i2jeHqTb0qP/ZQv1HD6rBhlg5IY3J20j2fq7fCmm3QItairQolIOQRnL6QcB027IQDx6N7UUFPJShRELbsea6XJeJx9aZOK3bsrK1rIoQQ1pEgVYhXTNmiOoVVeET8QSpEB4+xtW4QfX3zuubnShfWqaQszUs9f1kvwmrWHd77TD93rQrmZWqU1z2pxswC4sVdvam/W7OgTQgh7IEEqUK8YgwG3ZsKCQep8XF0gJEDdC9ormzm59xcoXhB8yBVKfhkLNTpCJN+B38/mPwFzBkTNxCuUV5vrXr8nPX1EnGFR8DN//S/JUgVQqQ1SU7mL4R4eXRpAyULxR3OT6pKpfSXJRVK6NRVRtMXwoKVMKw3vN3EfI5rbGWLgrurXuVfvMDz1U1Eu3Fbf0gokFuCVCFE2iM9qUK8glycoWqZlLl3heJw7ZYeZt60G0ZNh4/a68A4oQDVWK9Kpaybl3rpGrzeW/fSCnNXng31N6kFF67qlGNCCJFWSJAqhEhW5Z5NIViwEnp/DbUrwsD3kn59zfI6PVbM1eihj3Ve1tg274ZmPXRKrO9+1Y9FNON81Ndq6O+HT9muLkIIYS0JUoUQycrXWy+smjAbMvvChM91DtWkqlFBz6Xcvh/W/Qs9v4LSraBsa+g7XB8LC9eLsDp9qtNYbZ2nF2H1Hxk9B1PoIDVzRp1qzDM9HJIgVQiRhsicVCFEsqtWVm+t+utw8PSw7tr8/pAtE7z/Pz2fskheGNBZ96Su2AR/rQMXJ3gSAb3fhf6ddBA8fgg0/kAHsnPHWhcYx6QUrNwMNStAhvTPdw97ceWGnnecLh2UKASHZV6qECINkSBVCJHsPuuuV//7elt/rcEAgz+AU0HQqj4UyRd9rm8nPbS/ZrvOqVq7YvS5TBlh4v+g/ccweS707fh8dd+6Fz76Cnp2gE/ef7572IurN6MXx5UqDIvX2LY+QghhDZsO92/ZsoXmzZuTPXt2DAYDf/31V6LXbNq0ibJly+Li4kL+/PmZOXNmitdTCGEdN9fnC1CN2jSEId3NA1SjggF6e9aYAapRldI6OP1hluVcrYlRCsb8ogPl+f9Yngeblly9CTmepQkrVRhu/adX/AshUpZSku85Odg0SH306BGlSpViypQpSSofFBRE06ZNqVOnDgcPHqRfv368//77rFkj3QNCCK33O1A4AH5eZP21q7bAkdMwvB/8dx/+2ZzctUs9UVFw/bZ5TyrAQSuH/COewtNEtrkVQpjr/Q0MnWTrWqR9Nh3ub9y4MY0bN05y+WnTphEQEMC4ceMAKFKkCNu2beP777+nUaNGKVVNIUQa4uCge2LH/gIPHiV9XunTSPjuN51doEMLWLkFZv8FbRqkaHVTzO27egFazmdBarZMkMVXz0s1rvZPiu5fgqszTB2WItUU4qV0KgiilK1rkfalqdX9O3bsoH79+mbHGjVqxI4dO2xUIyGEPWpaWy+sCtye9Gv+XAvnLkXPQ+3UCg4chyOJrIi/cRtafghnLz1vbVOGMUeqsSfVYIBShazrSb14FdbvgDXbdM9yUkRG6qFOIV5l/93Xv09CHyet/JY9+v+bMJemgtQbN26QNav5FjlZs2YlJCSEx48tvxOePHlCSEiI2ZcQ4uWWPYve8nX5xsTLgp57+sMsnfS+RCF9rF4VHeDN+ivha6cv1IHf9zNeqMrJ7mqsIBWgVBEddCd1rtwf/+ieaIMB/t6QtGte7wMTZ1tXVyFeJk8j9fbOSsGJ84mXV0pnJfnut5SvW1qTpoLU5zFy5Ei8vLxMX7ly5bJ1lYQQqaB5Xd07kZRdluYt1/M3P46x6YCDA7zTQgdn94ItX3c/BP5YAQXz6Pmrp4KSperJ4upNnRs1ZgqwUoUg5BEEXUn8+vAIWLQaXm8IdSrpnubEnArSvc/7jj9/vW3p/GV4Z5D5RhJCWOtecPRowtHTiZe/c08HtZt2y/zv2NJUkJotWzZu3rxpduzmzZt4enri5uZm8ZohQ4YQHBxs+rp8+XJqVFUIYWNNauk5Yau3JlxuzxH4YTa80UjnaI3prSb6j82CVZavnb0MIqNgzhjIkcW+ehBjpp8yKvmslzgpSf0Dt+s/nh2aQ+sGereqMxcTvsbYc30ukXL2auVmnYLM2sVlQsR0557+7uQIR88kXt744TbkIew7mnL1SovSVJBapUoV1q9fb3YsMDCQKlWqxHuNi4sLnp6eZl9CiJdfFh+oVEpvAGBJVBT8+Ae81U/3hH7aLW4ZX29oXgfmLNNzLWN6HAYz/4S2jSFbZuj1juXe1EvX4YsJelrAvmOpl9YqZvopI29PyJMDDp1I/Pq5y/WUiYIBeuqDpwcsTaA3VSlYvkGXu3Iz6XPx7MmuQ/p7UtpHiPgYg9TyJeDY2cTLn7kAzk56d7j1ssTGjE2D1IcPH3Lw4EEOHjwI6BRTBw8e5NIlvQJhyJAhdOwYnZG7R48enD9/nk8++YSTJ08ydepUFi5cSP/+/W1RfSGEnWtRB/7dH/1Hw+heMLz3GYyaDj3ehj/Gx5/XtWMrvXPT1D/Mjy9cBfdCoFtb/fj1RnF7U08Hweu9dfA27jdo0wuKN9PHZi6NfxpBcrDUkwo6FdXWfQkPK164qrel7dBcP3Zx1sH60nXxz2c9elpf9/6b+vH5JEwpsCcRT2Hvs16sAxKkihdg/H1Tu4L+HZDYB9PTF/QoTt0qsH5nilcvTbFpkLp3717KlClDmTJlABgwYABlypThyy+/BOD69eumgBUgICCAf/75h8DAQEqVKsW4ceP45ZdfJP2UEMKi12rq76u2RB/be1Rvn3rwBMwcBZ98AI4JbKFauojeIOC7X2H4NN1j+DRS94w2qw3+2XU5Zyfz3tQDx+HNfuDrBWtnwNEVsHwafNZD92h+MwUqvAHdvtRzZ5OTUvEHqe+9ruek/pJAHtk/VoBXBj1lwqhNQ7h2C3YesnzN3xvBxws6ttSPz9rBkP/Fq1CqpZ5rmpgjpyE0DCqXhkOv6HB/lyHm/1fE87lzD9xdoUJJ/eHn9IWEy5++AAXyQN3K+v+NrPKPZtM8qbVr10YlkKvE0m5StWvX5sCBAylYKyHEy8LHC6qXgxUbda/g9IUw5mcoUxQmfwl+mZN2nwFdwDsDfDVF935WLaN7V3/62rzc641g8u/wyRj9h6dIPpgxCryeLV4qWUh/dWmj/5AtWw+LV8O7n8DfP0Yn3E/I5Rtw6jzUrxp/mZCH8DA0OkdqTKWL6OcfP0PnS82Tw/x8zAVTri7Rx8sVg9zZ9QKqqmXMr4mK0m3cpBZk9NL5WM/ZQUqulVv04radByFvImtmdx3SgcU7zaHXN3DzDmTNlCrVtAsPQ2HDTj1do3FNW9cmbbtzDzL5QJG8OjPGsTN6G2dLlNK9rXUq6d9Vzk7659Dl9dSts71KU3NShRDCWs3qwK7D0GkwjPwJur0F879PeoBq9N4b8P1nOkgbNEYn/S9ewLyMsTf14EmoUAJ+HxsdoMaWKSN0fQNW/AQBOWHK3MTrsOcINO8OXT9PuLfFmCM1exbL5we+pwPJwd/FzWm6ZqvO8di+uflxg0EvoFq5Oe7q933HdC9ri7r6cX5/+8gbu+5f/f1IElZY7zyk5+CWL6Efv2q9qUHPepsPJ2FRnUjYf/f1/293N8jnn/DiqZt3dMaNQgHg4Q6VS8mQf0wSpAohXmqNauhVtodPw4yReoGU03OOIbVpAD9/q/8A9elouUzbxvDrcPhluP4jlRgHB/iwnU6Yn1AKqz8Dof3H+o+ZZ3qYvzL+sldv6O+xF04ZubvBqI91D+P8f/SxiKd6CsCn46BKGSiQO+51bRrAo8e6rjGt2Kh7HSs8C/Dy+dt+uP9uMOw/rvO8JhZ4PY2EvUf0UL9xZ65XbYX/uWdB6vnLEPzQtnVJ627fg0ze+t/FCyQcpBqnAhTMo7/XraL/Xz4MTbn6pSUSpAohXmpeHrrndM0ves7Xi6pXBXYtig7IYnNw0EPxzk5Jv2frBrrX01JvalSUTvLdfwS0rAdzxkLrhnrhVniE5ftdvQkuTtF/KC2pXg7efE3Ps/1nEzTtBt/+CK3qw7Rhlq/JnUMHcl9OjE7uHxmpr29WG9I9+4uSP7deRJWUnI93g1MmN+TGnbrturTRwX9Ci1eOndFBQaVSuse4dGE9Z/lVcu5S9NzsxHZZEwn7757+IAtQrACcOBc3O4jRqSA9rSaXn35cr7L+wLh1b+rU1d5JkCqEeOmVK6bTRNkrZyfo/rbOM3ohxjC+UvC/H2DSHBj8AYz9RJft0FzPewv81/L9rtyE7Fmjg8b4fPERuDrDR19Bene9sGt4f72wKz4/fQW1KkDvb6DnVzoP7e17evW/UX5/HUBfvp7w898PgVrvwPQFCZd7Hut26GCz7rM/+gn1Uu88pAMFYx7ZUoV172tSd+Z6GZy7pP+fZEj/6k11SG537oHvsyC1eAEIexLdUx3b6Qt61ML4f9U/u34sqag0CVKFEMIOvN1EZwL48VmqK6Vg2CSdr3TsJ/BRe93LB3rIv1wxvVOWJfGt7I/NK4OeljD5C1gyMf7FHTF5e8KkL/Q12/brADdnNr0gy8i4KUJiQ/4zl+pFXotXx50b+yKehMPm3VCvql685uiQ8JD/rkO6PY2936WLwINHaS+N1os4f1n3gBcvmLTNHoRlSkXPSQXdkwrxD/mfuRA91G9Urwps3PVqfUiKjwSpQghhB1xddI7RJWv0IqTh03QQN6K/nucaW/tmsG2f5QVUSQ1SQQdkzesm3usaW/O6sPY3vViqV4foABr0/FQP94RX+D8Mhd+WQNH8upcpKTvzJNWuQ3rubIOqul0L5ol/8VRkJOw5rIf6jUw7c70iQ/5RUTogz+eve58Px9OTum2fzJVMTMgjPYqQ+VmQ6uWhh/KPWXh/K6V7Ui0FqXfuySI2kCBVCCHsxjst9aKmt/vDzwthWG/o0MJy2WZ1dLqgP/6Je86aIPVFZPXVvartmpkfNxieLZ5KIEidu1zvSvXT13ojhWXr4y9rrXU79OsvnFc/LlEo/j/4J87rwKJK6ehjnh66/q/K4qmrt3Tvc95cOkC/fhtu3TUvc+UGdBgIvyaQXzctOxWkM1e8qDvP2i3m5iDxLZ66elN/mCoYYH68bDH9Hty8+8Xrk9ZJkCqEEHbCw10v9Ll4TSf979Im/rKuLjqX6aLV5guoHofp4UZLOVJTU0JpqMLCdRDepiH4++mA++8N8S8usYZSsP5f3Rtl7N0tUVDnogyzsHhq50G9o1bsHLUpsXjq0rXk37ghORh7vPP5R7dD7N5U4/bCa7enWrVS1dR58MnYF38P/ndff8/kE32seAE4fibulJbYK/uNHB30h4XkHF1IqyRIFUIIO9LrHb2AqftbiZdt3+zZAqoYgcPVW/p7avSkJiSfP5y7aHmu6cKV+o/5h+3041b1db7IXYdf/HlPBemFY/WrRB8rWUhnEDh5Lm75XYf05g4uzubHSxXWq7ItBbbPQyn4eDT0GKoXctmT85f068+RRWeZyJQxbi/y8g16DvPRM3o6SlIEXUneucYp6eAJPQ85oQV2SXH72ZaoxuF+0PN8Qx7FXUh4OgjSu1n+v1osPxw7+2J1eRlIkCqEEHbEyTF6TmRiCgboBPTT5us/aErBtWeJ/OPLkZpa8ufWf5hjDxtHPIWf5uuUVQE59bEyRfSq5r/WvfjzrvtX/+GvXDr6WOG8z3Llxhryj4qC3Ud0AvXYyhTVdT2RTIHC9v2w+7Ae3rW31fPnLuufhYOD7n0uWci8J/X8sznDQ7rrXr7AJPSm7joEtd/V2R/s3b3g6Kwae4682L3u3NXvNc8Ym3gUy6+/H4nVM2qcjxpzPrfpmgJ6OsC94LjnIiOhXuekb2EbHgGjpsPtu4mXtTcSpAohRBrWv7OeL9jkA/2H65dFehFUNhtv6Wlc4R978dSydbqns2eH6GMGg84Bu2qzeT7Tu8Hw8Sg4bkWguO5fqFnBvGfUxVkH9LEXTx07o9NgVbIQpBbOq1f7J8e8VKXg+5l62oGnB2zf9+L3TE5nL0G+GNvGli6sV/gbe0GXb9SBf6t6OviPL/WZkVJ6dzfQGz3YO+PPOGsm2Hv0xe713309HzVm4JnZR9879vSR0xegQB7L9yn6LLA9bqH3//QFnTkjqR/qFqzUWUPmrUhaeXsiQaoQQqRh1cvB7sUwcxSUKqT/yObN9fy7aiUX/+y61y1mGqqIpzBlnl51b1zUZNSynu553bhLP775H7TtC4vXwFeTEx82DnsC0xfqgCPmUL9RyYJxk9RPmafn7pYvHre8s5PuAUuOXs9t+/TPZUAXvZvXVhsFqSs3w8nzcY+fvwx5/aMflyysg3fj8PSKjdCgGri56u87D+rUYfFZvRUOnND5dDfsjLuNbnLYdwxqdtAfYlZufrGsA4dOgo+XzlSx+/CLTVGImSM1psY1YM6y6A9tUVFw5mLc+ahGeXPq9rY05L//mP6+bV/8G3oYhYXD5N910PzPpqS+Cvth419j9kkpxdOnT4lMjln8QoiXloODA46OjhgsjdelIidHqFNJfz0Og3A7mPPo5Ah5cpovnpr9lx5WnfJl3PIFcushzmXr9UKT9h/rXtXPP4ThP+rh8url4l73JFxnOJgyF+7e1/N0m9eNW65EQb1LV9gTvejs4Ak9XDr+0/h3BytVGDa/4EInpWD8TD2loU4l3ev91WQdVHm4v9i9rXHnHvQdDjXKw28joo8/eAS3/jPvSTUunjp4Eh4/0T13n3bTxxpUhaETYdNuHdTF9jQSxvyiA9RhvaFOR122cc3kfT3rd+hey8On9AcZJ0doWA1++Ny63d5AB9SlCkPFEnpB39WbOvfv87gTY7epmAZ/AFv26p/Bn5Ph+i39XowvSHVw0B/kLKWu2ndM98iHPNT/jpmZIrY/luspN0O6w4hpllNe2TMJUmMJDw/n+vXrhIZKMjghROLc3d3x8/PD2dk58cKpwM0V3GxdiWfy+0f3HP13H36YqYNI41BmbC3rwbhfdQDp4ACLJkKubPDPRr01bLWy5sOol2/odF3XbkGbBtC3o+7BtaRkIYiM0sOnZYrAqJ/1pgit6sdf/woldK7a0T9Dv05xF1ddvamD7mpl47/Hlj2652vWaF336uV0ILfrkM5AkFrmrdC9blv36sA0Q3p9/PyznZDyxehJ9fHSQdqhk3ohkaeHDm5BL/IpVkDPS7UUpC5cqe85+Uvdo18kr+7pTCxI3XMEZv4JIz7WuUUTc/iUDs5+GQ6XrsPabTq3cMkl0OPtxK83Ukrnw+3cBsoVj67LiwSpxrnWMbm7wYTPoXVPPfWjbFF9vFBA3LJGxfLr90ls+4/r9+2qzXr73/iC1Mdh+sNbmwbQubXeue6fTVCws3WvyZYkSI0hKiqKoKAgHBwcyJ49O87OzjbvIRFC2CelFOHh4dy+fZugoCAKFChAOmsz4r/k8vnrzQkAxv4CGODj9+Iv37Kunsvo5grzvoveynZgV3hnkB46NgZ2jx7D+5/r+beBM6LnwManYED04qmHj2DHAfh1uA6G49O4JgzqCj/M0j134z7VPbIXruqURUvW6IBzzhg9DzY241zUMkV1zyLoACZ7Ft0znFpBasRTPdRcuxJs2qWnVBgDTOOHiLy5zK8p9Sy37M07uh1i9k42rKbnPodHmB8PfQzfz9IBlHGxUONaeqFcWLjegteS+f/o7X8jnupg+O2mCb8epXQA/UFb/djfT2+EcfWm/lm1qKvbOCkuXoN7IXpTC19v/Z7dexRaN0ja9bHduWd5+gjoD0oDusDYX6F2RfBMr+eqxqd4Af3hwtj7D/rDXtAVPRf9cZjupf6sh+Xr5/yt53X37ag/YDWsplOJ9etkebGWPZIgNYbw8HCioqLIlSsX7u6pOA4jhEiT3NzccHJy4uLFi4SHh+Pq6mrrKtmV/Lnhxh3dGzR/JQztpXvp4pMtMyyaoAPOjDHKVS8HlUrCuBl6yByg33A9dL50cuIBKug/0oXz6lXri1bpQCKxINHBQacEq18VBoyCVh9B1TJ6O1hfLxjcTQd9g8bo3be8Mphfv3a7HkqePTo6KDD2pm5LxXmpq7boIf25Y/WUiFVbYgSpl6N3CIupVBH9gUEpGDHA/FyDajr43nUouocV9A5i90P0hwqjprVh/AzYshsaVje/z9NI+GaK7q1u30zP0Vy5JfEg9cJVPdRdOlZu2/6d9SKvr6fAtK+ijxsXcu0+orf/jfnB5MBx/d14rwolXmyF/3/3zXOkxtbjbZ2kf+MuvRVvQsFisQJ67urJ89HbDhvrW7YYOKTTeZKv3YoblD96DD/OgzcbR48uNK0NS9bq3vHYc8LtlXzst0B6Q4QQSSW/L+JnDB57f6vnnL7bMvFrKpQwD1BB/yH/+D09P2/VVh2sBv4LE/8Xd7eehJQspIOYo2f0HMuk9iYVzgvLpuqA9W6wDra3/QHd2sJ3g3UP4pcTza/ZdUjPP6xTKW4va/VyOlCInZ4rpcz4U09JKBgAr9XQAZJxMdO5WCv7jUoV0sFdpozm6bwAiubTC86Mif2jouDPtbp3+d2WeoqGUX5/PQdyZax0SQ9DoeMn8Pvf8G0/GPkxNK+jMx8EP0j49RgXs8VO1ebpAV98pIPwTc92a4qMhMHfwU8LdIBnPG508ITu3fb21I8rFNc/m9h1uHBVL0D6cqLOdft6b5j2h3mZsCf6dVmak2rk4ADff6Z7UeOb9mJUMEAHojHnpe47Bll8dfvXKK/Pb9oV99pZS/W0jt7vRB+rUV63kXFjhrRAfrsKIYRIEcZ5jjfv6MDOMYGh9cRUKgU1y8P/vtfBwqfdrB8uL1FQD1HXrayDYWs4Oeqeun+m6/l9xuHX7Fngq746HdA/z7bV3H0YOn+q5x3+OCxuMFy1jP6eGqmoDp3Uc2KNu5c1rqkDVOOCsHOXzOejGhUvqKdSNKkV9+dmMOje5XX/6ikBr/eB/iN1QN6vU9x7NamlyxrTi4WFwwf/09f+/l30h5fXaup5w4ntanXoJOTOHh1YxtSirs6g8OUEHTD2Ha57G78fonsm/1gR917GXkqIfl/ETEUVHqF/nlPn6WkiDx7pe8/52/xexkT+mbwTrn+OrPDPz3oqSUJcnfVoRMwV/vuP6feVwaB77ssWi86IYfTffR2Uv9XEfG6tsxM0qq7npaaVTRYkSBXPpXPnzrRq1crW1XglpGRbv/vuu4wYMSLxglY4fvw4OXPm5NGjR8l6X5H2pHfTwUSjGpZX5lvr4/d0T2abBknbkSu2KmV0zsrBH7x4XWJqXV/3UH4+Xi8S6jRYD5f/OlzPr40ts49eULR9f/LWw5IZf0IuPx2Yg557WihAp4mKjNQ9hJZ6Uj3c9eKnXu/EPQd6yP/aLWjeAx6FwvzvYcpQ8yT2Rk1r6cBu2z79nP2G6x7BX4ebL/rJ6quDxJWbE35Nh07G3cbWyGCAb/rqutXvrF/n1KF6C972zWD9TrhxW5d9Eq4DwJj38s+ufz4xg9TflugtbZdO0fOf534Hfd7V003u3Isu958xSE2gJ9X0PH5xp4dYUix/dJ7giKc640K5GHNe61TS7yPjBwCl4NPvIJ0B+lr4wNCsjl7YZin/qj2SIFWIV9ShQ4dYuXIlffr0MR37888/adiwIb6+vhgMBg4ePBjnurCwMHr27Imvry8eHh68/vrr3Lx503S+aNGiVK5cmfHjx6do/f/8808aNGhA5syZ8fT0pEqVKqxZsybBa06dOkWdOnXImjUrrq6u5M2bl//9739ERJgnG1y0aBGFCxfG1dWVEiVKsHLlypR8KS+1eePghyHJc6/SRSDwNxjzyfMt/MiTA/YuSf75eAaDnrfp4AAfDtN5Rn+LJ0A1qlZOr7RPyR6tW3d1jtOOrcznYTauCev/hfNXdC+hpZ5U0MFlVl/L5yqV0ruGDesNK39OOA1SgTz6Of7ZpBdIrd2m05BZ2kShSU3dLvHlYY14qqdrxBekgp5a0qOd/kDzy/DozAIt6+m5yQtW6ccnzunXXyZGT6rBYD4v9eZ/MHG2bsOYK/FNabpiJOg3BqyW8qQ+r2IF4MR5HdyfPKenFBgzA4BeDPfocXR956/UPdGjBkIWC3Njq5XVPdD/pIFNFkCC1JdWeHgybTgtXlqTJk3izTffxMMjuuvj0aNHVK9endGjR8d7Xf/+/Vm+fDmLFi1i8+bNXLt2jTZt2piV6dKlCz/++CNPn6Zcws4tW7bQoEEDVq5cyb59+6hTpw7NmzfnwIED8V7j5OREx44dWbt2LadOneKHH37g559/ZujQoaYy//77L+3ataNr164cOHCAVq1a0apVK44efcGtaF5RObPp9DvJxbhK3974esOkL/SinxkjEn/N1cvpRWXnLqdcneYtB0dHeKux+fHGNfXGCXOW6cexV/YnhZOj7jnt0ibxaRwGgx7y/zNQr1YfPUj3xFrSuKYORNfFs6vV6Qu61zChIBXg4y76A0ntitHHMqTX0wEWrNRB34ETegi8SD7zaysU1721YeEweroObPt3Ni+TIytkzmi+I9nte/q1+nonXDdrFCugA9Nzl2HvMV3f4gWjzxfNp+eobtqte8W/nqyH+RtVt3w/J0fd679iUxoZ8levmODgYAWo4ODgOOceP36sjh8/rh4/fmyDmr2YWrVqqZ49e6q+ffsqX19fVbt2baWUUuPGjVPFixdX7u7uKmfOnOrDDz9UDx48MF03Y8YM5eXlpVavXq0KFy6s0qdPrxo1aqSuXbtmKvP06VPVv39/5eXlpXx8fNSgQYNUx44dVcuWLU1lwsLCVO/evVXmzJmVi4uLqlatmtq9e7fp/MaNGxWgVq9erUqXLq1cXV1VnTp11M2bN9XKlStV4cKFVYYMGVS7du3Uo0ePEnyt06dPVzlz5lRubm6qVatWaty4ccrLy8t0vlOnTmZ1U0qpvn37qlq1apkeR0ZGqhEjRqg8efIoV1dXVbJkSbVo0SLT+bt376r27durTJkyKVdXV5U/f37122+/KaWUevLkierZs6fKli2bcnFxUf7+/mrEiBGJ/oyeV+zXk1hbK6XUsmXLVP78+ZWLi4uqXbu2mjlzpgLUvXv3lFL6Z+rl5aVWrFhh8TmDgoIUoA4cOGB2/P79+8rJycmsrU6cOKEAtWPHDtOxJ0+eKBcXF7Vu3boXe/FWKlq0qPrqq6+suqZ///6qevXqpsdt27ZVTZs2NStTqVIl1b17d4vXp+XfG8J2HoYqla++UjP+fL5rf1+mVNNuSn09xXKZx0+UKtdaqc/Gxz0XFaVUrXeUKtBQqYKNlIqMtL4O1joVpFT+BkpNm5942dY9ler6meVz85YrlaeuUo9Cn68eB44r5V9bqQ07lerzrVItP4xb5vBJXWbafP193nLL9+r6mVIdBkY/njRHqdItn69e8bkfouvw51qlen2tVMuP4pYZOFqpOu/qc9XbK/Ug4T+hassefc8f/1Dq6dPkrW9SJBSHxSY9qUkUHh5OeHg4KsZHj8jISMLDw+P0FiVH2ecxa9YsnJ2d2b59O9OmTQP0yuOJEydy7NgxZs2axYYNG/jkk0/MrgsNDeW7775jzpw5bNmyhUuXLjFw4EDT+XHjxjFz5kx+++03tm3bxt27d1m6dKnZPT755BOWLFnCrFmz2L9/P/nz56dRo0bcvWu+fHXYsGFMnjyZf//9l8uXL9O2bVt++OEH5s2bxz///MPatWuZNGlSvK9x+/bt9OjRg759+3Lw4EEaNGjA8OHDrW6rkSNHMnv2bKZNm8axY8fo378/77zzDps368lQX3zxBcePH2fVqlWcOHGCH3/8kUyZdEK7iRMn8vfff7Nw4UJOnTrF3LlzyZMnT7zPNXfuXDw8PBL82rp1a5LrnlhbBwUF8cYbb9CqVSsOHTpE9+7d+fzzz83ucfjwYYKDgylfvrylp4jXvn37iIiIoH796AzohQsXxt/fnx07dpiOOTs7U7p06QRf19atWxNtl7lz5ya5blFRUTx48AAfnwTyv8Ry9uxZVq9eTa1atUzHduzYYfb6ABo1amT2+oR4Uend9LDt+n+T3qN1/KweLq/4Bvxvgu5RnPmnzg8a21+BcOc+dH0j7jmDQfdYPgnX22+mRoKKgnng4LKkzSVuUktvgvDAwrT2gyehUJ7n750vVVj3Ps5brofqYy6aMiqSH9xdYdR0nau0beO4ZUDPOz50Qmc3gGdbono/X73i45VBj0YcP6cXTZUrFrdMnUq6p/XQSfjhs8R3MqtWFt57XafleqOv+a5wdielI2Z787w9qaNGjVKjRo0y6+Xbvn27GjVqlFq5cqVZ2XHjxqlRo0ap+/fvm47t3r1bjRo1Sv39999mZSdMmKBGjRqlbt26ZToWu/cqKWrVqqXKlCmTaLlFixYpX19f0+MZM2YoQJ09e9Z0bMqUKSpr1qymx35+fmrMmDGmxxERESpnzpym3r2HDx8qJycnNXfuXFOZ8PBwlT17dtN1xp7UmD1rI0eOVIA6d+6c6Vj37t1Vo0aN4q3/W2+9FaeXq0OHDlb1pIaFhSl3d3f177//mpXp2rWrateunVJKqebNm6suXbpYrEPv3r1V3bp1VVRUVLz1jCkkJESdOXMmwa/Q0Pi7BWK+nqS09eDBg1Xx4sXN7vH555+b9aQuXbpUOTg4xPsa4utJnTt3rnJ2do5TvkKFCuqTTz4xO9a6dWvVuXPneF9XaGhoou0SEhIS7/WxjR49WmXMmFHdvHkz0bJVqlRRLi4uClDdunVTkTG6kpycnNS8efPMyk+ZMkVlyZLF4r2kJ1U8ryVrdY/WgJFKPQm3XObqTaWmzlOqQRddtlxrpcb+qtSVG7pHtWQLpb6cYH5NZKRSdTsp9f7n8T/3wRP6fj2tG3hIFVdu6LotDYx77rWuSg0aE/e4NWb/pXtjjT2UlrT/WJ/ffTj++2zdq8ucvagf9/paqbf6v1jdLPngC6XqddLPtWJT3PPBD5Qq/JpS42dYd9/dh5Wq/a5SBRooNfl3pcIjkqO2ibOmJ9UOZ/aI51WuXNzls+vWrWPkyJGcPHmSkJAQnj59SlhYGKGhoaYNC9zd3cmXL3pSjp+fH7du3QIgODiY69evU6lSJdN5R0dHypcvb+r9PXfuHBEREVSrFj3JyMnJiYoVK3LiRIxZ5UDJkiVN/86aNSvu7u7kzZvX7Nju3bES2cVw6tQpWrdubXasYsWKrFixIp4r4jp79iyhoaE0aGC+pUh4eDhlyujcMB9++CGvv/46+/fvp2HDhrRq1YqqVasCerV9gwYNKFSoEK+99hrNmjWjYcOG8T5fhgwZyJAhCcs4kyApbX3q1CkqVDBPzFixYkWzx48fP8bFxSVFd1Rzc3NLcHthNzc38udPJFFgEs2bN4+vvvqKZcuWkSVL4lvNLFiwgAcPHnDo0CEGDRrEd999F2eEQYiU1qaBXoU9cDRcv60T0Ht66J7VLXvg18V6v3dj6qDBH+icqzHn5HZuDdPmQ5+O0b14G3fB2Ysw6uP4n7tkId27WaZo/GVsJUdWvZhp5RbzbWsfh+kcpu+0eLH7t6ynt1B9HBb/63/vdT1vOKFUZcY8rQdP6IVht+/pearJrVh+WPNsUKqchfp6esC/8y2n5EpIhRKw6me9S9e43/QGF5YWs9mSBKlJ1L9/f0AHBEaVKlWifPnycZJ59+rVK07ZsmXLUqpUqThle/ToEadsiRJWJvB7Jn369GaPL1y4QLNmzfjwww8ZPnw4Pj4+bNu2ja5duxIeHm4KUmM+N4DBYDCbfpCcYj6XwWCw+NxRxrGT55QuXbo49Y+5evvhQ71s9J9//iFHjhxm5VxcdPLDxo0bc/HiRVauXElgYCD16tWjZ8+efPfdd5QtW5agoCBWrVrFunXraNu2LfXr12fx4sUW6zN37ly6d++eYJ1XrVpFjRo1rH6tzytTpkyEhoYSHh5u1Z7z2bJlIzw8nPv37+Pt7W06fvPmTbJlM9/s+u7du2YffmLbunUrjRvHM472zE8//USHDh0SLDN//nzef/99Fi1aFGeYPj65cumVIkWLFiUyMpJu3brx8ccf4+DgQLZs2cyyFYDl1ydEcmhVX+/41O0LeKMPdGiuE9yfvqCHmscM0kPzGdJbvr5za5i+QKdJMubd/HmRDvLi254T9JD/6l9SZ6j/eTSupQOn23d1SijQq/ojo3T2hBfh6aEXUK3foVOkWVKvSuJ5eD09dHB64AS83gju3IXCVmwukVTFC+jvObJGbxUcW+wNMJLK1UXnHH6nhXlOVXshQWoSWfpD7uDggIOFjZ+To2xy2LdvH1FRUYwbN84UHC9cuNCqe3h5eeHn58euXbuoWVPn8Xj69Cn79u2jbNmyAOTLl880FzZ37tyADgr37NlDv379kuW1GBUqVIg9e/aYHYv9OHPmzHFWYh88eNAUEBctWhQXFxcuXbpkNhcxtsyZM9OpUyc6depEjRo1TD1uAJ6enrz11lu89dZbvPHGG7z22mvcvXvX4nzIFi1amPVEWxI7WI5PUtq6UKFCcVImxW6j0qVLAzqnqfHfSVGuXDmcnJxYv349r7/+OqB7bi9dukSVKua/0Y8ePcobb1iYEPdM+fLlLaa4iilr1qwJnv/jjz947733mD9/Pk2bJrKXYjyioqKIiIggKioKBwcHqlSpwvr1683eu4GBgXFenxDJpUppWDIJunwKQydBg6p6F6aKJRNPtZXRC9o3h9lL9ZabF67ohPNThyZ+bTL9qUkRbRrAzwt0Sq9543Rv8uFT4OKUPCnE/vchfND2xfewL1Mkeges/+4nLUeqtYo9G3Aqm4K93vYYoIIEqS+1/PnzExERwaRJk2jevLnZgipr9O3bl1GjRlGgQAEKFy7M+PHjuX//vul8+vTp+fDDDxk0aBA+Pj74+/szZswYQkND6do1kS01rNS7d29q1qzJ+PHjad68ORs2bGDVqlVmw9Z169Zl7NixzJ49mypVqvD7779z9OhR01B+hgwZGDhwIP379ycqKorq1asTHBzM9u3b8fT0pFOnTnz55ZeUK1eOYsWK8eTJE1asWEGRInqG/fjx4/Hz86NMmTKkS5eORYsWkS1bNrOexZiSc7g/KW3dvXt3xo8fz+DBg+natSsHDx5k5syZAKZ2ypw5M2XLlmXbtm1mQerdu3e5dOkS165dA3QACroHNVu2bHh5edG1a1cGDBiAj48Pnp6e9O7dmypVqlC5cmXTfS5cuMDVq1cT7Nl80eH+efPm0alTJyZMmEClSpW4ceOG6b5eXrpbYfLkySxdupT169cDulfbycmJEiVK4OLiwt69exkyZAhvvfWW6UNM3759qVWrFuPGjaNp06bMnz+fvXv3Mn369OeuqxCJKZhH5xt98Mj6gOGDN/U2mHP/1snpc/npDRTSssw+8NM38HZ/vXvUyI91MFi0QPKkIPP0sLzxgLVKF4Fl63Wu0nshyZsj1ShrJt2bWr9q8t/b7qX0BFl78zKnoOrbt2+c4+PHj1d+fn7Kzc1NNWrUSM2ePdtsAY0xBVVMS5cuVTHfGhEREapv377K09NTeXt7qwEDBsRJQfX48WPVu3dvlSlTpgRTUBmfN77nHjp0qCpVqlSCr3X69OkqR44cphRU3377rcqWLZtZmS+//FJlzZpVeXl5qf79+6tevXqZpaCKiopSP/zwgypUqJBycnJSmTNnVo0aNVKbN29WSin1zTffqCJFiig3Nzfl4+OjWrZsqc6fP296/tKlS6v06dMrT09PVa9ePbV///4E6/wiYi8ES6ytlYqbgurHH39UgNl7e+rUqapy5cpm1xkX0sX+Gjp0qNnzf/TRRypjxozK3d1dtW7dWl2/ft3sPiNGjEhwAVxyqFWrlsW6durUyVRm6NChKnfu3KbH8+fPV2XLllUeHh4qffr0qmjRomrEiBFx/s8vXLhQFSxYUDk7O6tixYqpf/75J956pOXfG+Ll8clYnf4ooK5Svy2xdW2Sz4KVesHQzD+Vqtkh7iIxWzOmq1q1RX8P3G7rGtk/axZOGZRKE+lck01ISAheXl4EBwfj6Wk+yzgsLIygoCACAgJwdU1gqxBhVz744ANOnjxpVRqnV83w4cOZNm0aly9HZw5//PgxhQoVYsGCBck6lB0eHk6BAgWYN2+e2QKvl5X83hD2IOgK1O2k0w/tXKhTXL0svp6iU21FRsH3n+mpAPYi4ikUa6rnuC5aDcumWk5rJaIlFIfFJsP9Is357rvvaNCgAenTp2fVqlXMmjWLqVOn2rpadmXq1KlUqFABX19ftm/fztixY00L+ozc3NyYPXs2d+7cSdbnvnTpEp999tkrEaAKYS8CckLPDpAt08sVoAJ81kMvJNu6F0oVsnVtzDk56qH4jbv045QY7n+VSZAq0pzdu3czZswYHjx4QN68eZk4cSLvv/++ratlV86cOcO3337L3bt38ff35+OPP2bIkLgbqNeuXTvZnzt//vzJllpKCJF0A9+zdQ1ShqODXgi2YefzbeGa0koXgX3H9L8zedu0Ki8dCVJFmmNthoJX0ffff8/3339v62oIIUSy8PQwz5lqT4zD++ndwE1m/CQrO82QJoQQQghh/4xBakqkn3rVSZAqhBBCCPGccmUDHy8JUlOCDPdb8IolPBBCvAD5fSHEq81g0FuourrYuiYvHwlSYzAm8w4NDcXN7SVbHimESBGhoaFA3O2FhRCvjvFDXnz3KhGXBKkxODg44O3tza1btwBwd3c328lICCGMlFKEhoZy69YtvL29k207YyFE2pMcu2CJuKRZY8mWTe9HZwxUhRAiId7e3qbfG0IIIZKPBKmxGAwG/Pz8yJIlCxEREbaujhDCjjk5OUkPqhBCpBAJUuPh4OAgf3yEEEIIIWxEUlAJIYQQQgi7I0GqEEIIIYSwOxKkCiGEEEIIu/PKzUk1Jt4OCQmxcU2EEEIIIV4txvgrKRuhvHJB6oMHDwDIlSuXjWsihBBCCPFqevDgAV5eXgmWMahXbE+/qKgorl27RoYMGVIlUX9ISAi5cuXi8uXLeHp6pvjzvUyk7Z6ftN2LkfZ7ftJ2z0/a7vlJ272Y1Gw/pRQPHjwge/bspEuX8KzTV64nNV26dOTMmTPVn9fT01P+4zwnabvnJ233YqT9np+03fOTtnt+0nYvJrXaL7EeVCNZOCWEEEIIIeyOBKlCCCGEEMLuSJCawlxcXBg6dCguLi62rkqaI233/KTtXoy03/OTtnt+0nbPT9ruxdhr+71yC6eEEEIIIYT9k55UIYQQQghhdyRIFUIIIYQQdkeCVCGEEEIIYXckSE1hU6ZMIU+ePLi6ulKpUiV2795t6yrZlZEjR1KhQgUyZMhAlixZaNWqFadOnTIrExYWRs+ePfH19cXDw4PXX3+dmzdv2qjG9mvUqFEYDAb69etnOiZtl7CrV6/yzjvv4Ovri5ubGyVKlGDv3r2m80opvvzyS/z8/HBzc6N+/fqcOXPGhjW2D5GRkXzxxRcEBATg5uZGvnz5+Oabb8y2OZS207Zs2ULz5s3Jnj07BoOBv/76y+x8Utrp7t27dOjQAU9PT7y9venatSsPHz5MxVdhOwm1X0REBIMHD6ZEiRKkT5+e7Nmz07FjR65du2Z2j1e1/RJ778XUo0cPDAYDP/zwg9lxW7edBKkpaMGCBQwYMIChQ4eyf/9+SpUqRaNGjbh165atq2Y3Nm/eTM+ePdm5cyeBgYFERETQsGFDHj16ZCrTv39/li9fzqJFi9i8eTPXrl2jTZs2Nqy1/dmzZw8//fQTJUuWNDsubRe/e/fuUa1aNZycnFi1ahXHjx9n3LhxZMyY0VRmzJgxTJw4kWnTprFr1y7Sp09Po0aNCAsLs2HNbW/06NH8+OOPTJ48mRMnTjB69GjGjBnDpEmTTGWk7bRHjx5RqlQppkyZYvF8UtqpQ4cOHDt2jMDAQFasWMGWLVvo1q1bar0Em0qo/UJDQ9m/fz9ffPEF+/fv588//+TUqVO0aNHCrNyr2n6JvfeMli5dys6dO8mePXucczZvOyVSTMWKFVXPnj1NjyMjI1X27NnVyJEjbVgr+3br1i0FqM2bNyullLp//75ycnJSixYtMpU5ceKEAtSOHTtsVU278uDBA1WgQAEVGBioatWqpfr27auUkrZLzODBg1X16tXjPR8VFaWyZcumxo4dazp2//595eLiov7444/UqKLdatq0qXrvvffMjrVp00Z16NBBKSVtFx9ALV261PQ4Ke10/PhxBag9e/aYyqxatUoZDAZ19erVVKu7PYjdfpbs3r1bAerixYtKKWk/o/ja7sqVKypHjhzq6NGjKnfu3Or77783nbOHtpOe1BQSHh7Ovn37qF+/vulYunTpqF+/Pjt27LBhzexbcHAwAD4+PgDs27ePiIgIs3YsXLgw/v7+0o7P9OzZk6ZNm5q1EUjbJebvv/+mfPnyvPnmm2TJkoUyZcrw888/m84HBQVx48YNs/bz8vKiUqVKr3z7Va1alfXr13P69GkADh06xLZt22jcuDEgbZdUSWmnHTt24O3tTfny5U1l6tevT7p06di1a1eq19neBQcHYzAY8Pb2BqT9EhIVFcW7777LoEGDKFasWJzz9tB2jqnyLK+gO3fuEBkZSdasWc2OZ82alZMnT9qoVvYtKiqKfv36Ua1aNYoXLw7AjRs3cHZ2Nv3CMcqaNSs3btywQS3ty/z589m/fz979uyJc07aLmHnz5/nxx9/ZMCAAXz22Wfs2bOHPn364OzsTKdOnUxtZOn/8Kvefp9++ikhISEULlwYBwcHIiMjGT58OB06dACQtkuipLTTjRs3yJIli9l5R0dHfHx8pC1jCQsLY/DgwbRr1860/7y0X/xGjx6No6Mjffr0sXjeHtpOglRhN3r27MnRo0fZtm2brauSJly+fJm+ffsSGBiIq6urrauT5kRFRVG+fHlGjBgBQJkyZTh69CjTpk2jU6dONq6dfVu4cCFz585l3rx5FCtWjIMHD9KvXz+yZ88ubSdsIiIigrZt26KU4scff7R1dezevn37mDBhAvv378dgMNi6OvGS4f4UkilTJhwcHOKspL558ybZsmWzUa3sV69evVixYgUbN24kZ86cpuPZsmUjPDyc+/fvm5WXdtS/ZG7dukXZsmVxdHTE0dGRzZs3M3HiRBwdHcmaNau0XQL8/PwoWrSo2bEiRYpw6dIlAFMbyf/huAYNGsSnn37K22+/TYkSJXj33Xfp378/I0eOBKTtkiop7ZQtW7Y4i22fPn3K3bt3pS2fMQaoFy9eJDAw0NSLCtJ+8dm6dSu3bt3C39/f9Pfj4sWLfPzxx+TJkwewj7aTIDWFODs7U65cOdavX286FhUVxfr166lSpYoNa2ZflFL06tWLpUuXsmHDBgICAszOlytXDicnJ7N2PHXqFJcuXXrl27FevXocOXKEgwcPmr7Kly9Phw4dTP+WtotftWrV4qQ7O336NLlz5wYgICCAbNmymbVfSEgIu3bteuXbLzQ0lHTpzP98ODg4EBUVBUjbJVVS2qlKlSrcv3+fffv2mcps2LCBqKgoKlWqlOp1tjfGAPXMmTOsW7cOX19fs/PSfpa9++67HD582OzvR/bs2Rk0aBBr1qwB7KTtUmV51itq/vz5ysXFRc2cOVMdP35cdevWTXl7e6sbN27Yump248MPP1ReXl5q06ZN6vr166av0NBQU5kePXoof39/tWHDBrV3715VpUoVVaVKFRvW2n7FXN2vlLRdQnbv3q0cHR3V8OHD1ZkzZ9TcuXOVu7u7+v33301lRo0apby9vdWyZcvU4cOHVcuWLVVAQIB6/PixDWtue506dVI5cuRQK1asUEFBQerPP/9UmTJlUp988ompjLSd9uDBA3XgwAF14MABBajx48erAwcOmFafJ6WdXnvtNVWmTBm1a9cutW3bNlWgQAHVrl07W72kVJVQ+4WHh6sWLVqonDlzqoMHD5r9DXny5InpHq9q+yX23ost9up+pWzfdhKkprBJkyYpf39/5ezsrCpWrKh27txp6yrZFcDi14wZM0xlHj9+rD766COVMWNG5e7urlq3bq2uX79uu0rbsdhBqrRdwpYvX66KFy+uXFxcVOHChdX06dPNzkdFRakvvvhCZc2aVbm4uKh69eqpU6dO2ai29iMkJET17dtX+fv7K1dXV5U3b171+eefmwUG0nbaxo0bLf6O69Spk1Iqae3033//qXbt2ikPDw/l6empunTpoh48eGCDV5P6Emq/oKCgeP+GbNy40XSPV7X9EnvvxWYpSLV12xmUirFFiBBCCCGEEHZA5qQKIYQQQgi7I0GqEEIIIYSwOxKkCiGEEEIIuyNBqhBCCCGEsDsSpAohhBBCCLsjQaoQQgghhLA7EqQKIYQQQgi7I0GqEEIIIYSwOxKkCiGEEEIIuyNBqhBC2KHOnTvTqlUrW1dDCCFsRoJUIYQQQghhdyRIFUIIG1q8eDElSpTAzc0NX19f6tevz6BBg5g1axbLli3DYDBgMBjYtGkTAJcvX6Zt27Z4e3vj4+NDy5YtuXDhgul+xh7Yr776isyZM+Pp6UmPHj0IDw+3zQsUQojn5GjrCgghxKvq+vXrtGvXjjFjxtC6dWsePHjA1q1b6dixI5cuXSIkJIQZM2YA4OPjQ0REBI0aNaJKlSps3boVR0dHvv32W1577TUOHz6Ms7MzAOvXr8fV1ZVNmzZx4cIFunTpgq+vL8OHD7flyxVCCKtIkCqEEDZy/fp1nj59Sps2bcidOzcAJUqUAMDNzY0nT56QLVs2U/nff/+dqKgofvnlFwwGAwAzZszA29ubTZs20bBhQwCcnZ357bffcHd3p1ixYnz99dcMGjSIb775hnTpZABNCJE2yG8rIYSwkVKlSlGvXj1KlCjBm2++yc8//8y9e/fiLX/o0CHOnj1LhgwZ8PDwwMPDAx8fH8LCwjh37pzZfd3d3U2Pq1SpwsOHD7l8+XKKvh4hhEhO0pMqhBA24uDgQGBgIP/++y9r165l0qRJfP755+zatcti+YcPH1KuXDnmzp0b51zmzJlTurpCCJGqJEgVQggbMhgMVKtWjWrVqvHll1+SO3duli5dirOzM5GRkWZly5Yty4IFC8iSJQuenp7x3vPQoUM8fvwYNzc3AHbu3ImHhwe5cuVK0dcihBDJSYb7hRDCRnbt2sWIESPYu3cvly5d4s8//+T27dsUKVKEPHnycPjwYU6dOsWdO3eIiIigQ4cOZMqUiZYtW7J161aCgoLYtGkTffr04cqVK6b7hoeH07VrV44fP87KlSsZOnQovXr1kvmoQog0RXpShRDCRjw9PdmyZQs//PADISEh5M6dm3HjxtG4cWPKly/Ppk2bKF++PA8fPmTjxo3Url2bLVu2MHjwYNq0acODBw/IkSMH9erVM+tZrVevHgUKFKBmzZo8efKEdu3aMWzYMNu9UCGEeA4GpZSydSWEEEIkj86dO3P//n3++usvW1dFCCFeiIz9CCGEEEIIuyNBqhBCCCGEsDsy3C+EEEIIIeyO9KQKIYQQQgi7I0GqEEIIIYSwOxKkCiGEEEIIuyNBqhBCCCGEsDsSpAohhBBCCLsjQaoQQgghhLA7EqQKIYQQQgi7I0GqEEIIIYSwOxKkCiGEEEIIu/N/FMSvNeBlwVwAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: the loss curve we will compare the broken run against\n",
"plt.figure(figsize=(7, 3))\n",
"plt.plot(good_losses, color=\"#1E40FF\", lw=1)\n",
"plt.xlabel(\"step\"); plt.ylabel(\"train loss\"); plt.title(\"ViT on FashionMNIST (correct loop)\")\n",
"plt.axhline(math.log(10), ls=\":\", c=\"#888\", label=f\"random guess = log(10) = {math.log(10):.2f}\")\n",
"plt.legend(); plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "f9384579",
"metadata": {},
"source": [
"> **Interpretation.** Loss starts near `log(10) = 2.30` (the cross-entropy of uniform guessing over 10 classes) and falls. Test accuracy lands well above the 10% chance line. A 215k-param ViT on a few thousand images is no ResNet, but it learns. The `log(10)` reference line is the single most useful sanity check for any 10-class loss curve.\n"
]
},
{
"cell_type": "markdown",
"id": "3ae769aa",
"metadata": {},
"source": [
"### 2.3 A deliberate failure: forgetting to zero the gradients\n",
"\n",
"Here is the canonical PyTorch bug, staged on purpose (spec §2 deliberate-failure demo). PyTorch *accumulates* gradients into `.grad` by default. If you never call `zero_grad()`, every step adds the new gradient on top of all the previous ones, so the effective gradient keeps growing. We run the same loop with that one line removed, watch the loss limp, then prove the cause with a gradient-norm probe.\n",
"\n",
"> **Predict:** with `zero_grad()` removed at this learning rate, will the broken run blow up, or just learn worse? AnswerAt `lr=3e-4` it does not blow up: the optimizer (AdamW here) partly rescales the swollen gradient, so the loss still falls, but far more slowly and noisily, and ends well above the correct run. That quiet degradation is the dangerous case, because nothing crashes. The unambiguous proof of the bug is not the curve but the gradient norm: a second backward without zeroing roughly doubles it.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "72244e60",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:34.248236Z",
"iopub.status.busy": "2026-06-10T20:02:34.248153Z",
"iopub.status.idle": "2026-06-10T20:02:35.137367Z",
"shell.execute_reply": "2026-06-10T20:02:35.136916Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"broken-run first loss 2.368 . last loss 1.967\n",
"correct run reached 1.107 by the same step: the broken run lags badly\n"
]
}
],
"source": [
"# the bug: zero_grad() is missing, so gradients accumulate across ALL steps\n",
"torch.manual_seed(SEED)\n",
"broken = ViT().to(device); broken.train()\n",
"opt_b = torch.optim.AdamW(broken.parameters(), lr=LR)\n",
"bad_losses = []\n",
"n_bug_steps = steps_per_epoch # one epoch is enough to see it\n",
"it = iter(train_loader)\n",
"for _ in range(n_bug_steps):\n",
" try:\n",
" xb, yb = next(it)\n",
" except StopIteration:\n",
" it = iter(train_loader); xb, yb = next(it)\n",
" xb, yb = xb.to(device), yb.to(device)\n",
" logits = broken(xb) # forward\n",
" loss = F.cross_entropy(logits, yb) # forward (loss)\n",
" loss.backward() # backward -- BUG: no opt_b.zero_grad() above\n",
" opt_b.step() # update on the ACCUMULATED gradient\n",
" bad_losses.append(loss.item())\n",
"print(f\"broken-run first loss {bad_losses[0]:.3f} . last loss {bad_losses[-1]:.3f}\")\n",
"print(f\"correct run reached {good_losses[len(bad_losses)-1]:.3f} by the same step: the broken run lags badly\")"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "8eaa8d8f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.138324Z",
"iopub.status.busy": "2026-06-10T20:02:35.138222Z",
"iopub.status.idle": "2026-06-10T20:02:35.201200Z",
"shell.execute_reply": "2026-06-10T20:02:35.200779Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAEiCAYAAAAF9zFeAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAApC9JREFUeJzs3XV4U9cbwPFv6q60tECRYcOd4cXdBhtDhg4Yw32wwXDbcGcwXDZkyHB3G+7WUrSUCnVPzu+PO/KjVEhqaeF8nicP9N5zz3mTJumbm3PfoxJCCCRJkiRJkiQpmzEydACSJEmSJEmSlBoykZUkSZIkSZKyJZnISpIkSZIkSdmSTGQlSZIkSZKkbEkmspIkSZIkSVK2JBNZSZIkSZIkKVuSiawkSZIkSZKULclEVpIkSZIkScqWZCIrSZIkSZIkZUsykZU+aiqViv79+xs6jA/q1q0b+fPnT/d+x48fj0qlSvd+pYxx/PhxVCoVx48fT7SvadOm9OrVS/vz/v37sbGxwd/fP8PjUqlUjB8/PsPH+Vik9HtMSf78+enWrVuGxCRJHyuZyErZ3tmzZxk/fjzBwcGGDkWSMsSZM2c4ePAgP/74o3Zb48aNKVSoENOmTTNgZJIkSYYlE1kp2zt79iwTJkzI1ons8uXLuX//frr3O2bMGKKiotK9Xylz/fbbb9SrV49ChQol2P7999+zbNkywsLCDBSZlJRatWoRFRVFrVq19Dru/v37LF++PIOikqSPk0xkJSkLMDU1xdzcPN37NTExwcLCIt37zQoiIyMNHYJWdHQ0Go0mQ/p+/fo1e/bsoV27don2tW3blpiYGLZs2ZIhY2dHERERhg4BIyMjLCwsMDLS70+subk5pqam6RJDfHw8sbGx6dJXRhFCyA/aUprJRFbK1saPH8+IESMAKFCgACqVCpVKhY+PT4J2O3bsoGTJkpibm1OiRAn279+fqK8XL17Qo0cPcubMqW23cuVKneJ4Oxd3y5YtFC9eHEtLS6pWrcrNmzcBWLZsGYUKFcLCwoLatWsnii+pObJ//vknFSpUwNbWFjs7O0qVKsW8efO0++Pi4pgwYQKFCxfGwsICZ2dnatSowaFDhxI8Pu/PkX0bqy6PyfHjx6lYsSIWFhYULFiQZcuW6TTvdvXq1drfxfu32rVrJ2i7fv16KlSogKWlJU5OTrRv355nz54laFO7dm1KlizJ5cuXqVWrFlZWVvz000+Akuh999135MyZEwsLC8qUKcOaNWtSjC85ixYt4rPPPsPS0pLKlStz6tQpateunSDmt/Mf//zzT8aMGUPu3LmxsrIiNDSUoKAghg8fTqlSpbCxscHOzo4mTZpw/fr1RGM9f/6c1q1bY21tjaurK0OGDCEmJiZRuz179hAfH0/9+vUT7XN1daV06dLs3LkzwfaQkBDu3btHSEjIB+/zpUuXaNSoETly5MDS0pICBQrQo0ePDx539epVmjRpgp2dHTY2NtSrV4/z589r9wcHB2NsbMz8+fO12wICAjAyMsLZ2RkhhHb7Dz/8gJubW4L+L1y4QOPGjbG3t8fKygpPT0/OnDmToM3b5+KdO3fo2LEjjo6O1KhR44Oxv0+f18SH7jckPUf24cOHtG3bFjc3NywsLMiTJw/t27dP8DtKao5scHAwgwcPxsPDA3NzcwoVKsSMGTMSfHDy8fFBpVIxc+ZM5s6dS8GCBTE3N+fOnTs6Pwa7du1CpVJx48YN7bZt27ahUqlo06ZNgrbFihXjm2++0f4cHx/PpEmTtOPmz5+fn376KdHzOX/+/DRv3pwDBw5QsWJFLC0tWbZsGQCHDh2iRo0aODg4YGNjQ9GiRbWv8bdiYmIYN24chQoVwtzcHA8PD0aOHJnk60b6dJgYOgBJSos2bdrw4MEDNm3axJw5c8iRIwcALi4u2janT5/m77//pm/fvtja2jJ//nzatm3L06dPcXZ2BsDPz48qVapo/6C5uLiwb98+vvvuO0JDQxk8ePAHYzl16hS7du2iX79+AEybNo3mzZszcuRIFi9eTN++fXnz5g2//vorPXr04OjRo8n2dejQITp06EC9evWYMWMGAHfv3uXMmTMMGjQIUP6IT5s2jZ49e1K5cmVCQ0O5dOkSV65coUGDBinGqstjcvXqVRo3boy7uzsTJkxArVYzceLEBI9tcmrVqsW6desSbHvy5AljxozB1dVVu23KlCmMHTuWdu3a0bNnT/z9/VmwYAG1atXi6tWrODg4aNsGBgbSpEkT2rdvz7fffkvOnDmJioqidu3aPHr0iP79+1OgQAG2bNlCt27dCA4O1j5WuliyZAn9+/enZs2aDBkyBB8fH1q3bo2joyN58uRJ1H7SpEmYmZkxfPhwYmJiMDMz486dO+zYsYOvv/6aAgUK4Ofnx7Jly/D09OTOnTvkypULgKioKOrVq8fTp08ZOHAguXLlYt26dUk+J86ePYuzszP58uVLMu4KFSqwY8eOBNu2b99O9+7dWbVqVYoXD71+/ZqGDRvi4uLCqFGjcHBwwMfHh7///jvFx+r27dvUrFkTOzs7Ro4ciampKcuWLaN27dqcOHGCL774AgcHB0qWLMnJkycZOHAgoDzvVCoVQUFB3LlzhxIlSgDKa6dmzZra/o8ePUqTJk2oUKEC48aNw8jIiFWrVlG3bl1OnTpF5cqVE8Tz9ddfU7hwYaZOnZogQdaHLq8JXe53UmJjY2nUqBExMTEMGDAANzc3Xrx4we7duwkODsbe3j7J4yIjI/H09OTFixd8//335M2bl7NnzzJ69Gh8fX2ZO3dugvarVq0iOjqa3r17Y25ujpOTk873v0aNGqhUKk6ePEnp0qUB5fdiZGTE6dOnte38/f25d+9egotoe/bsyZo1a/jqq68YNmwYFy5cYNq0ady9e5ft27cnGOf+/ft06NCB77//nl69elG0aFFu375N8+bNKV26NBMnTsTc3JxHjx4l+OCi0Who2bIlp0+fpnfv3hQrVoybN28yZ84cHjx4kOg1IH1ChCRlc7/99psAxOPHjxPtA4SZmZl49OiRdtv169cFIBYsWKDd9t133wl3d3cREBCQ4Pj27dsLe3t7ERkZmWIMgDA3N08Qw7JlywQg3NzcRGhoqHb76NGjE8XbtWtXkS9fPu3PgwYNEnZ2diI+Pj7ZMcuUKSOaNWuWYlzjxo0T77/MdX1MWrRoIaysrMSLFy+02x4+fChMTEwS9fkhUVFRokKFCiJXrlzC19dXCCGEj4+PMDY2FlOmTEnQ9ubNm8LExCTBdk9PTwGIpUuXJmg7d+5cAYj169drt8XGxoqqVasKGxubBI97SmJiYoSzs7OoVKmSiIuL025fvXq1AISnp6d227FjxwQgPvvss0TPi+joaKFWqxNse/z4sTA3NxcTJ05MFPfmzZu12yIiIkShQoUEII4dO6bdXqNGDVGhQoVkY586daoAhJ+fn3bbqlWrBCBWrVqV4v3evn27AMS///6bYjtAjBs3Tvtz69athZmZmfDy8tJue/nypbC1tRW1atXSbuvXr5/ImTOn9uehQ4eKWrVqCVdXV7FkyRIhhBCBgYFCpVKJefPmCSGE0Gg0onDhwqJRo0ZCo9Foj42MjBQFChQQDRo00G57+/zu0KFDivF/iK6vCV3v99vnyNvf49WrVwUgtmzZkmIc+fLlE127dtX+PGnSJGFtbS0ePHiQoN2oUaOEsbGxePr0qRBCeY4Bws7OTrx+/Vrv+/9WiRIlRLt27bQ/ly9fXnz99dcCEHfv3hVCCPH3338LQFy/fl0IIcS1a9cEIHr27Jmgr+HDhwtAHD16NMH9A8T+/fsTtJ0zZ44AhL+/f7KxrVu3ThgZGYlTp04l2L506VIBiDNnzqTuTkvZnpxaIH306tevT8GCBbU/ly5dGjs7O7y9vQFlnta2bdto0aIFQggCAgK0t0aNGhESEsKVK1c+OE69evUSTA94e3ambdu22NraJtr+dvykODg4EBERkWCaQFJtbt++zcOHDz8Y2/s+9Jio1WoOHz5M69attWcRAQoVKkSTJk30Hq9v377cvHmTbdu2ab9C/vvvv9FoNLRr1y7BY+7m5kbhwoU5duxYgj7Mzc3p3r17gm179+7Fzc2NDh06aLeZmpoycOBAwsPDOXHihE7xXbp0icDAQHr16oWJyf+/qOrUqROOjo5JHtO1a1csLS0Txfh2XqRarSYwMFD7Nem7z6G9e/fi7u7OV199pd1mZWVF7969E40TGBiYbAyAdl9AQIB2W7du3RBCfLCU09sz3rt37yYuLi7Ftm+p1WoOHjxI69at+eyzz7Tb3d3d6dixI6dPnyY0NBSAmjVr4ufnp72Q8dSpU9SqVYuaNWty6tQpQDkTKoTQnpG9du0aDx8+pGPHjgQGBmqfFxEREdSrV4+TJ08mmo/cp08fnWJPiS6vCV3v9/vennE9cOCAXnO7t2zZQs2aNXF0dEzwGqlfvz5qtZqTJ08maN+2bVudvjFJzru/l7CwMK5fv07v3r3JkSOHdvupU6e0Z9tBeS4DDB06NEFfw4YNA5SpMe8qUKAAjRo1SrDt7fNw586dyc4137JlC8WKFePzzz9P8FjUrVsXINH7hfTpkIms9NHLmzdvom2Ojo68efMGUL4qCw4O5vfff8fFxSXB7W3i9Pr1a73HefvHy8PDI8ntb8dPSt++fSlSpAhNmjQhT5489OjRI9F8vYkTJxIcHEyRIkUoVaoUI0aMSDC/TZ9YIeFj8vr1a6KiohJdJQ8kuS0ly5YtY9WqVSxYsIAqVapotz98+BAhBIULF070uN+9ezfRY547d27MzMwSbHvy5AmFCxdOdFFNsWLFtPt18bbd+/fNxMQk2fq+BQoUSLRNo9EwZ84cChcujLm5OTly5MDFxYUbN24kmAv55MkTChUqlGiucdGiRZMcS6TwdfnbfampF+zp6Unbtm2ZMGECOXLkoFWrVqxatSrFOYf+/v5ERkYmGWuxYsXQaDTaOc5vk9NTp04RERHB1atXqVmzJrVq1UqQGNnZ2VGmTBkA7Qezrl27JnperFixgpiYmERzf5P6XehLl/cJXe/3+woUKMDQoUNZsWIFOXLkoFGjRixatOiDc5gfPnzI/v37Ez0Ob+dLv/8aSevjULNmTXx9fXn06BFnz55FpVJRtWrVBAnuqVOnqF69uvY19+TJE4yMjBK9dtzc3HBwcEj0Gkwqxm+++Ybq1avTs2dPcubMSfv27dm8eXOCpPbhw4fcvn070WNRpEiRJB8L6dMh58hKHz1jY+Mkt79NAN6+WX777bd07do1ybZv54ylZpwPjZ8UV1dXrl27xoEDB9i3bx/79u1j1apVdOnSRXshU61atfDy8mLnzp0cPHiQFStWMGfOHJYuXUrPnj1TFWtKMaXGxYsXGTRoED179kx0tlGj0aBSqdi3b1+S8djY2CT4+f2zn4aWVDxTp05l7Nix9OjRg0mTJuHk5ISRkRGDBw9OdVUDZ2fnFD/0vN33dn64PlQqFVu3buX8+fP8888/HDhwgB49ejBr1izOnz+f6Hegr1y5clGgQAFOnjxJ/vz5EUJQtWpVXFxcGDRoEE+ePOHUqVNUq1ZNmxi9fZx+++03ypYtm2S/GfHcyOjXxKxZs+jWrZv29Tpw4ECmTZvG+fPnk5yDDcpj0aBBA0aOHJnk/rdJ3FtpfRzeXih38uRJvL29KV++PNbW1tSsWZP58+cTHh7O1atXmTJlSqJjdf0glVSMlpaWnDx5kmPHjrFnzx7279/PX3/9Rd26dTl48CDGxsZoNBpKlSrF7Nmzk+z3/RMG0qdDJrJStpfWlatcXFywtbVFrVYneWW4oZiZmdGiRQtatGiBRqOhb9++LFu2jLFjx2rPfjg5OdG9e3e6d+9OeHg4tWrVYvz48R9MZD/E1dUVCwsLHj16lGhfUtuS4u/vz1dffUXZsmVZtGhRov0FCxZECEGBAgUS/UHWVb58+bhx4wYajSbBWdl79+5p9+vaDyj3rU6dOtrt8fHx+Pj46PRBBmDr1q3UqVOHP/74I8H24ODgBIlmvnz5uHXrFkKIBM/fpGoJf/7552zbti3ZMR8/fqw985taVapUoUqVKkyZMoWNGzfSqVMn/vzzzySfRy4uLlhZWSUZ67179zAyMkqQVNSsWZOTJ09SoEABypYti62tLWXKlMHe3p79+/dz5coVJkyYoG3/9ut9Ozu7LPV61Pd+J6VUqVKUKlWKMWPGcPbsWapXr87SpUuZPHlyku0LFixIeHh4pj0OefPmJW/evJw6dQpvb2/tGfVatWoxdOhQtmzZglqtTlAfN1++fGg0Gh4+fKj9JgSUC2iDg4N1fg0aGRlRr1496tWrx+zZs5k6dSo///wzx44d0077uH79OvXq1ZOrFUoJyKkFUrZnbW0NkOoFEYyNjWnbti3btm3j1q1bifZnxhKg7wsMDEzws5GRkTaZevu17/ttbGxsKFSoULqUojE2NqZ+/frs2LGDly9farc/evSIffv2ffB4tVpN+/btiY2NZdu2bYmmBIBSccLY2JgJEyYkOuslhEh0/5LStGlTXr16xV9//aXdFh8fz4IFC7CxscHT0/ODfQBUrFgRZ2dnli9fTnx8vHb7hg0bUjwb+j5jY+NE92XLli28ePEiUdwvX75k69at2m2RkZH8/vvvifqsWrUqb968SXZO9eXLl6latWqCbbqW33rz5k2ieN+eBU3ueWRsbEzDhg3ZuXNngjJyfn5+bNy4kRo1amBnZ6fdXrNmTXx8fPjrr7+0iZGRkRHVqlVj9uzZxMXFJahYUKFCBQoWLMjMmTMJDw9PNL4hXo+g//1+V2hoaILnFShJrZGRUYqv13bt2nHu3DkOHDiQaF9wcHCiPtNDzZo1OXr0KBcvXtT+Xt5+AJk+fTqWlpZUqFBB275p06YAiSoovD1z2qxZsw+OGRQUlGjb+8/Ddu3a8eLFiyQXjIiKisoS9YMlw5BnZKVs7+2b6s8//0z79u0xNTWlRYsW2gRXF9OnT+fYsWN88cUX9OrVi+LFixMUFMSVK1c4fPhwkm+0Galnz54EBQVRt25d8uTJw5MnT1iwYAFly5bVnvUoXrw4tWvXpkKFCjg5OXHp0iW2bt2aoCxOWowfP56DBw9SvXp1fvjhB9RqNQsXLqRkyZJcu3YtxWOXLl3K0aNH6dOnT6KLMHLmzEmDBg0oWLAgkydPZvTo0dpSV7a2tjx+/Jjt27fTu3dvhg8fnuI4vXv3ZtmyZXTr1o3Lly+TP39+tm7dypkzZ5g7d26Ci+xSYmZmxvjx4xkwYAB169alXbt2+Pj4sHr1agoWLKjzGaDmzZszceJEunfvTrVq1bh58yYbNmxIcHEQQK9evVi4cCFdunTh8uXLuLu7s27dOqysrBL12axZM0xMTDh8+HCi6RmvX7/mxo0b2pJvb+lafmvNmjUsXryYL7/8koIFCxIWFsby5cuxs7PTJihJmTx5srbuZ9++fTExMWHZsmXExMTw66+/Jmj7Nhm6f/8+U6dO1W6vVasW+/btw9zcnEqVKmm3GxkZsWLFCpo0aUKJEiXo3r07uXPn5sWLFxw7dgw7Ozv++eefZGN7y8fHhwIFCtC1a1dWr179wfa60Od+v+vo0aP079+fr7/+miJFihAfH8+6deu0H6KTM2LECHbt2kXz5s3p1q0bFSpUICIigps3b7J161Z8fHx0mlLytjyYLtMkatasyYYNG1CpVNqpBsbGxlSrVo0DBw5Qu3btBB9My5QpQ9euXfn9998JDg7G09OTixcvsmbNGlq3bp3gG47kTJw4kZMnT9KsWTPy5cvH69evWbx4MXny5NHG0LlzZzZv3qx9T6levTpqtZp79+6xefNmbW1a6ROUyVUSJClDTJo0SeTOnVsYGRklKG0FiH79+iVq/36ZGyGE8PPzE/369RMeHh7C1NRUuLm5iXr16onff//9g+MnNc7bkji//fZbgu1vS/O8W4rn/fJbW7duFQ0bNhSurq7CzMxM5M2bV3z//ffa0lVCCDF58mRRuXJl4eDgICwtLcXnn38upkyZImJjY7Vtkiu/petjcuTIEVGuXDlhZmYmChYsKFasWCGGDRsmLCwsUnw83o6b1O3dUlZCCLFt2zZRo0YNYW1tLaytrcXnn38u+vXrJ+7fv69t4+npKUqUKJHkWH5+fqJ79+4iR44cwszMTJQqVeqDZaeSM3/+fJEvXz5hbm4uKleuLM6cOSMqVKggGjdurG2T1O/vrejoaDFs2DDh7u4uLC0tRfXq1cW5c+eEp6dnovv95MkT0bJlS2FlZSVy5MghBg0aJPbv35+o/JYQQrRs2VLUq1cv0XhLliwRVlZWicqM6Vp+68qVK6JDhw4ib968wtzcXLi6uormzZuLS5cuJWjHe+W33h7bqFEjYWNjI6ysrESdOnXE2bNnkxzH1dU1UYmw06dPC0DUrFkzyWOuXr0q2rRpI5ydnYW5ubnIly+faNeunThy5Ii2zdvnWVJlm27evCkAMWrUqBQfg7f3T9fXhC73+/3yW97e3qJHjx6iYMGCwsLCQjg5OYk6deqIw4cPf3C8sLAwMXr0aFGoUCFhZmYmcuTIIapVqyZmzpypfa0n917zVoUKFYSbm9sHHwchhLh9+7YARLFixRJsnzx5sgDE2LFjEx0TFxcnJkyYIAoUKCBMTU2Fh4eHGD16tIiOjk50/5IqGXjkyBHRqlUrkStXLmFmZiZy5colOnTokKjsWGxsrJgxY4YoUaKEMDc3F46OjqJChQpiwoQJIiQkRKf7J318VEKk89UdkiR91Fq3bp3qsl/ZjUajwcXFhTZt2iT5lWZmebvC2L179yhcuLB2e7ly5ahduzZz5swxWGxZ1eLFixk5ciReXl7kzJnT0OEYTFhYGE5OTsydOzfRmXtJ+hjIObKSJCXr/XXQHz58yN69exMtM/sxiI6OTvTV69q1awkKCjL4/a1ZsyYNGzZM8PX1/v37efjwIaNHjzZgZFnXsWPHGDhw4CedxIJSgSB37tz06tXL0KFIUoaQZ2QlSUqWu7s73bp147PPPuPJkycsWbKEmJgYrl69muDMYFYWFBREbGxssvuNjY1xcXHh+PHjDBkyhK+//hpnZ2euXLnCH3/8QbFixbh8+XKSF6xJkiRJhiUv9pIkKVmNGzdm06ZNvHr1CnNzc6pWrcrUqVOzTRILSnWElFb4ypcvHz4+PuTPnx8PDw/mz59PUFAQTk5OdOnShenTp8skVpIkKYuSZ2QlSfqoXb58OcUSWpaWllSvXj0TI5IkSZLSi0xkJUmSJEmSpGxJXuwlSZIkSZIkZUuf3BxZjUbDy5cvsbW1lcvcSZIkSZIkZTFCCMLCwsiVK1eC5ceTa2wwU6dOFRUrVhQ2NjbCxcVFtGrVSty7d0/n4zdt2iQA0apVK52PefbsWbKF2uVN3uRN3uRN3uRN3uQta9yePXv2wbzOoGdkT5w4Qb9+/ahUqRLx8fH89NNPNGzYkDt37nxweVEfHx+GDx+eYI1uXbxdsvLZs2fJrostSZIkSZIkGUZoaCgeHh46LTOepS728vf3x9XVlRMnTlCrVq1k26nVamrVqkWPHj04deoUwcHB7NixQ6cxQkNDsbe3JyQkRCaykiRJkiRJWYw+uVqWutgrJCQEACcnpxTbTZw4EVdXV7777rvMCEuSJEmSJEnKgrLMxV4ajYbBgwdTvXp1SpYsmWy706dP88cff3Dt2jWd+o2JiSEmJkb7c2hoaFpDlSRJkiRJkrKALHNGtl+/fty6dYs///wz2TZhYWF07tyZ5cuXkyNHDp36nTZtGvb29tqbh4dHeoUsSZIkSZIkGVCWmCPbv39/du7cycmTJylQoECy7a5du0a5cuUwNjbWbtNoNAAYGRlx//59ChYsmOCYpM7Ienh4yDmykiRJkiRJWZA+c2QNOrVACMGAAQPYvn07x48fTzGJBfj888+5efNmgm1jxowhLCyMefPmJXm21dzcHHNz83SNW5IkSZIkSTI8gyay/fr1Y+PGjezcuRNbW1tevXoFgL29PZaWlgB06dKF3LlzM23aNCwsLBLNn3VwcABIcV6tocU9fYpp3ryGDkOSJEmSJOmjYtA5skuWLCEkJITatWvj7u6uvf3111/aNk+fPsXX19eAUaZN9OXLPCtfnqgzZwwdiiRJkiRJ0kclS8yRzUyZXUdWaDS8bNgQhCDXoUOoPrTUmiRJkiRJ0ics29aR/RipjIxwmjCBmKtXidBx0QZJkiRJkiTpw7JMHdmPlX8QLLlWHaPqv2PxqxcepnHY2ZtiZwN21mBrA7bWys3M1NDRSpIkSZIkZR8ykc1gIeFw/CKEGbci1DyW6NnJZ6vmZmBnAw62MGEgVC+fiYFKkiRJkiRlM3KObCbyHzyYkH/24XD8XyKM7AiNgLC3t3AIDVf+f+AMxMXBgT9ATqmVJEmSJOlTkm3qyH5qHEeNInzrVlQr55Bv3Lhk29WoCG36w76T0Kx25sUnSZIkSZKUncjzfZnIxM0N+379CF26lPjnz5NtV6EE1KoIc9fAfwuXSZIkSZIkSe/5ZBPZ2NhY3p1VoVariY2NJT4+PlG79Gzr0L8/RnZ2vJ42jdjYWO0Su6AstxsbG0tcXByDusIDH9h3CuLi4lJs+66MahsfH09sbCxqtTpVbYUQ2scno9om9bjr0zajf/dvJfW469M2s3/3aX2eZMbvXp+2hvzdp/V5kt1+91nteSLfIwzfVr5HpF/brPD7zMj3CF19slMLFi1axIgRI7CysgLgwoULnDp1itKlS9OkSRNtu4ULFxIXF0efPn2wt7cH4MqVKxw9epTixYvTokULbdulS5cSFRVFjx49cHFxAeDmzZscOHCAwoUL06ZNG4xsbXH88UfWPHpE5Jw5dOnSBXd3dwDu3r3L7t27yZcvH+3bt6dmRZi3BnwfriEwMJAOHTqQ978Vwh49esT27dvJnTs33377rTaGjRs38urVK7766isKFiwIwJMnT9i8eTOurq50795d23bLli08e/aMVq1a8fnnnwPw8uVLNmzYgKOjI71799a23b59O97e3jRt2pRSpUoB4O/vz+rVq7GxsaFfv37atrt37+b+/fs0aNCA8uWVK9bevHnD8uXLMTc3Z/Dgwdq2Bw4c4NatW9SuXZsvvvgCgPDwcBYvXoyRkREjRozQtj169ChXr16levXq1KhRA4CYmBjmzZsHwPDhwzE2Ngbg5MmTXLx4kcqVK1OnTh1AecHMmTMHgEGDBmFhYQHAuXPnOHPmDOXKlaNhw4ba8ebNm4dGo6Fv377Y2toCcOnSJY4fP07JkiVp1qyZtu3ixYuJiYmhV69eODk5AXD9+nUOHTpE0aJFad26tbbt77//Tnh4ON26dSNnzpwA3Llzh7179/LZZ5/x9ddfa9uuWrWKN2/e0KlTJ/LkyQPAgwcP2LlzJx4eHnTs2FHbdv369bx+/Zp27dppl3t+/PgxW7duxc3Nja5du2rb/vXXX7x48YIvv/ySIkWKAPD8+XM2bdqEs7MzPXv21Lbdtm0bT548oXnz5pQoUQIAPz8/1q5di52dHT/88IO27a5du3j48CGNGjWibNmyAAQEBLBy5UosLS0ZOHCgtu2+ffu4c+cOdevWpVKlSoAyL2rp0qWYmpoydOhQbdtDhw5x48YNatasSbVq1QCIiopiwYIFAPz444/atsePH+fy5ctUrVqVWrVqAcqb7dvf/ZAhQzAzMwPgzJkznDt3jgoVKlC/fn1tH2/bDhgwINPfI95asWIFoaGhKb5HvLVmjXyPAPkeId8j5HvEx/IesX//fnT1yZ6RNSTbzp1RmXz4M8SgLnD/MYRHZkJQkiRJkiRJ2cwnW7XA398fZ2dnVCoVoJy6V6vVGBkZYfJOkvn2KwlTU9N0bRv8zz+87tmTXBs3YlOvHqCcDYiPj0elUmFqqpTp6jQcgoLj2L5QYGZmgtF/ZQySagvKp0ohBCYm6ds2Pj4ejUaDsbGx9oyGPm2FENqvDt5+0k3vtkk97vq0zazffVKPuz5tM/t3n9bnSWb87tP6PMms331anyfZ7Xef1Z4n8j3C8G3le4R8j9DlPSIoKAhnZ2edqhZ8somsIcpvvUsIwcumTREREeQ+dgzVf7/U9/17E74aCEsnQJNamRykJEmSJElSJpNL1GYDKpUK5wkTiL11i/AtW5JtV6mUsjDCPFnBQJIkSZIkKQGZyBqQReXKWLdsSdCUKWiiopJtN7gr3PWGg2cyMThJkiRJkqQsTiayBuY0dixqPz9Cf/892TaVS0O1cvKsrCRJkiRJ0rtkImtgpgULYte9O29mz0YdGJhsu8Fd4Y4XHDqbicFJkiRJkiRlYTKRzQIcR4wAIXgza1aybb4oA1XLKat9fVqX50mSJEmSJCVNJrJZgHGOHDgMHkzoH38Q9/hxsu0Gd4E7j+CQnCsrSZIkSZIkE9mswr5PH4xz5CBo8uRk21Qpq9zmrpVnZSVJkiRJkmQim0UYWVnhOHo0Edu3E335crLtBneF2w/hsJwrK0mSJEnSJ86giey0adOoVKkStra2uLq60rp1a+7fv5/iMcuXL6dmzZo4Ojri6OhI/fr1uXjxYiZFnLFsO3TAtFgxgsaNI7l1KqqWhSplYI6cKytJ0n/UISHJvmdIkiR9zAyayJ44cYJ+/fpx/vx5Dh06RFxcHA0bNiQiIiLZY44fP06HDh04duwY586dw8PDg4YNG/LixYtMjDxjqIyNcR4/nuizZwlbty7ZdoO7KWdlj5zLvNgkSVLE3LxJzPXrhg5DSxMaytPSpVN8z5AkSfpYZaklav39/XF1deXEiRPUqqXbeqxqtRpHR0cWLlxIly5dPtg+qyxRmxwhBAGDBhG2fj0Ow4bhOHo0KqPEnzfaDYKIKNi9DP5bjlmSpAwWe/8+Lxo0wNjVFY9//9WuhW5IYRs24D9gABbVqpFr925DhyNJkpRm2XaJ2pCQEACcnJx0PiYyMpK4uLhkj4mJiSE0NDTBLStTqVTkmDcPp/HjCZ49G79u3dCEhydqN7gb3HoIR89nfoyS9ClSh4Tg9+23qExMiPf2JvbOHUOHBED41q2oLCyIPneOeF9fQ4cjSZKUqbJMIqvRaBg8eDDVq1enZMmSOh/3448/kitXLurXr5/k/mnTpmFvb6+9eXh4pFfIGUalUuEwcCA5N2wg6vhxXjZtStyzZwnaVC2rrPg1Z7X+c2XVAQFEX7iAiItLt5gl6WMm1Gpe9+6NOiCAXHv3YmRnR8SuXYYOi3hfX6JOnsTxp5/AxCRLxCRJkpSZskwi269fP27dusWff/6p8zHTp0/nzz//ZPv27VhYWCTZZvTo0YSEhGhvz95LCLMy68aNyb1/P5rQUF7Wr0/0hQvafSqVUsHg5gPdz8pqIiJ4M3MmT8uX52WTJvgUKoRf9+6E/fkn6oCADLoXuol7+pT4ly8NGoMkJefNtGlEHTmC64oVmH3+OVZNmmSJpDH877/B1BTbzp2xqluX8B07DB2SJElSpsoSiWz//v3ZvXs3x44dI0+ePDodM3PmTKZPn87BgwcpXbp0su3Mzc2xs7NLcMtOzIoXJ/fhw5gWKsTLVq0I27RJu69aOahUCuZ9oK6siI8ndO1anlWqxJuZM7Hr0oVc+/bh0L8/8c+e4d+3L0+KFuVFo0a8mT2bmNu3M+UKaE14OGEbNvCyaVOelS3L05IleVqmDK+//57QVauIvXMHodFkeBySlJLwHTsInj0bp19+wapePQCsW7Qg7v59Yj9QZSXDY9u6FauGDTG2t8e6dWtiLlwg/iO48FWSJElXBr3YSwjBgAED2L59O8ePH6dw4cI6Hffrr78yZcoUDhw4QJUqVfQaM6tf7JUcERtLwPDhhK1fj/2AATj98gsqY2NOX4ZOw2HlVKhX9b1jhCDy4EGCxo8n7v59bL76Cseff8Y0X74E7eL9/Ig6fJiIAweIOn4cER6Oce7cWDdqhFWjRljUqIGRpWX63A8hiD5/nrANG4jYuRMRGYmlpye2HTuiMjcn+vx5oi9cUK4KV6sxsrfH4osvsKhSBYsqVTArWxajZM6+S1J6i7l9m5eNGmHVuDGuy5drL+7SREXxpGhRHAYOxHH4cIPEFvvgAc+rVCHnmjVYt2iBJjQUnyJFcB43DvsffjBITJIkSelBn1zNoIls37592bhxIzt37qRo0aLa7fb29lj+lzh16dKF3LlzM23aNABmzJjBL7/8wsaNG6levbr2GBsbG2xsbD44ZnZNZEFJAkOXLSNwzBis6tfH9fffUdna0X4oXLkNHZpDv06Q0xmir1whaNw4os+cwaJmTZwnTMC8bNkPjxETQ9TZs0QeOEDkgQPEP3mCytISi+rVMS9TBrPPP8eseHFMCxVCZWqqc+zxL18S9tdfhG/cSJyXFyb58mHbsSO2HTpgksRZeE1EBDGXLyuJ7fnzRF+6hAgPBzMzzMuWxaJKFWzatME8hbPxkpQW6qAgXtSrh5GdHbn27cPIyirBfr+ePYl78IA8J08aJL6gKVMIXb6cvPfuaT/cverUCXVAALkPHDBITJIkSekh2ySyyZWuWbVqFd26dQOgdu3a5M+fn9WrVwOQP39+njx5kuiYcePGMX78+A+OmZ0T2bciDx/G77vvMMmdG7eNG4nNmZ9V22DZXxAbq+ErywN8daY/OYq44Tx+PJb166eqTJAQgrgHD4g8cICoEyeIvXsX9atXyk5TU8wKF8a0WDHMihXDrHhxzIoVw8TDQ1suTMTEELFvH2EbNxJ19Cgqc3OsW7bEtmNHLKpXT7KsWLKxxMcTe+fO/xPbc+fQRESQ5/RpTPPm1fu+SVJKRHw8r77+mphbt8h99CimSVwkGr5rF6+7dcPj0iVMP/ssc+MTgmfly2NZsyYu8+drt4dt2YL/99+T98aNJD8gSpIkZQfZJpE1hI8hkQWlnuWrTp3QBAeTc80azD7/nKfTF/LHPlu2On+PkbkZPdsb07OdEXYfPlGtM3VQELH37hF79y6xd+4Q99+/mv/KmqlsbDArWhQTDw+iTpxA8+YN5hUrYtupEzZffolROj3mmtBQntesiUmePLjv2oXK2Dhd+pUkgMAxYwhZtgz37duxrFEjyTaaiAieFCmC48iROAwalKnxRV+4wMsmTXDfuRPLmjX/H1NoKE+KFsVxzBgc+vXL1JikT4+Ijyd01SpsWrfG2MXF0OFIHxGZyKbgY0lkAdRv3uDXvTvRZ8+isrQEIXAYPJj4Dn1YtsOKtTvA0gL6tIeurcEqfaa5JiKEQO3rS+ydO0qCe+8ecV5eWHzxBbYdO2L2zrSR9BR19iy+LVrgNG4cDgMHZsgY0qcnbPNm/Pv0wXn6dOx7906xrV/XrsQ/f07uI0cyKTpFwIgRROzdS96bNxN9s/Gqc2fUr16R+9ChTI1J+vS8mTmTN1OnYtW0KTnXrcsSC4RIHweZyKbgY0pkAURcHEFTpiBiY3EcOhTjHDm0+/wCYMF6+HMPONgq82c7tgBzMwMGnM4Cx48nZMkSch8+jHmpUoYOR8rmYq5d42XTplh/+SUuCxd+8A9z+LZtvO7VC49r1zJtiouIi+NJ8eLYduiA88SJWSIm6dMTfekSL5s0waJ6daJPnsR19WpsWrY0dFjSR0Imsin42BJZXTx7BfPWwLaD4JYDBnaBrxqBqYmhI0s7ERPDiwYNEPHx5D56VFY0kFIt/vVrXtSti4mbG+67d+v0XNKEhSnTC37+GYf+/TMhSog8eJBX7duT+8SJJD+8acLClOkFo0fjMGBApsQkfVo04eE8r10bYwcHcu3bh1+PHsRcukSec+cwdnAwdHjSRyDbLlErZQwPN5j5IxxeBRVLwqiZ0PIHCIswdGRppzI3x3XpUuIfP+bNpEmGDkfKpkRsLK+7d4f4eHKuXavzByIjW1ss69Yl4p9/MjjC/wvbsgXTokUxS2YFRCNbWyzr1ydi585Mi0n6tAT+9BNqPz9cly1DZWpKjhkz0ERGEqTDBdeSlN5kIvsJKZgXFoyFf5bCM18YMhU+hvUGzIoXx3HsWEKWLCHqxAlDhyNlQ4E//0z0pUvkXLMGk1y59DrWumVLYv79N1MWItCEhxO5bx827dqlOO3BpnVrYq5cIS6JCi+SlBYRu3cTtn49zlOmYFqwIAAmuXLhPH48YWvXEnXmjIEjlD41MpH9BJUuCvPHwOFzMGe1oaNJH/Z9+mBRqxav+/VDHRxs6HCkbCR07VpC//iDHL/+isUXX+h9vFXjxmBqSsTu3RkQXUIRe/ciIiOx+eqrlGNq1AiVpaU8Kyulq3hfX/wHDcKqeXNsO3dOsM+2a1csqlQhYMgQNNHRBopQ+hTJRPYTVbcKjOwJ89fBno/gJKbKyAjXhQsR4eEEjBhh6HCkbCLez4/An37CtksX7Lp2TVUfxvb2WHp6Zsr0gvAtW7CoUiXJurbvMrK2xqphQ8K3b8/wmLKLmBs3iD5/3tBhZFtCo8G/b19U5ua4zJ2b6BsBlZEROebMIe7pU4JnzTJQlNKnSCayn7AfOkCLOjBsOtz1MnQ0aWeSJw85Zs4kYts2wrdtM3Q4UjYQPHcuKhMTnNI4t8+6ZUuiz50j3s8vfQJLgtrfn6jjx7H5+mvdYmrdmtjr14nz9s6wmLIT/0GD8B861NBhZFshS5cSdeIELosWYezklGQbs6JFcRw6lOB584i5fTuTI5Q+VTKR/YSpVPDbSPjMA3qOgaAQQ0eUdjZffYV1mzYEDB9O/PPnhg5HSqOwv/7iZcuW2gU30lP8ixeErV6Nfb9+ab7S2rppUzAyInLPnvQJLgnh27eDSoV1q1Y6tbdq0ACVlRXhcnqBsojL9evE3b+fIc+lj13M7dsETZyIXZ8+WNWpk2Jbh8GDMS1YkIBBgxBqdSZFKH3KZCL7ibO0gN8nQmQU9J0AcfGGjijtcsycicrKCv8BAxAfw9VsnyAhBMFz5+L/ww9EnznDm5kz032M4DlzUFlZYf/992nuy9jJCcuaNTN0ekH4li1Y1a+f7Nmw9xlZWWHVqBERO3ZkWEzZRfiWLWBqCkIQc/WqocPJVjRRUbzu3RuzQoVw+uWXD7ZXmZnhMncuMVevErpiRSZEKH3qZCIrkccNloyHf2/AlCWGjibtjB0ccF28mKgTJwhdtszQ4Uh6Emo1gaNGETRxIo4//ojjqFGELFuWrl+Rxz19Sui6dTgMHJhuyyZbt2xJ1OnTqAMC0qW/d8V5exNz+fIHL/J6n03r1sTevEnso0fpHlN2ITQawrdswbZDB4zs7Ii+fNnQIWUrQRMmEO/tjevvv+tcls7iiy+w69GDoMmTiXv2LIMjlD51MpGVAKhSFsYPgFV/w+Z9ho4m7Sw9PbHr04egiROJvXPH0OFIOtJER/P6u++UKgKzZ+P444/Y9++PsasrgTqcDdJV8G+/YezggF3PnunWp1XTpiAEEXv3plufb4Vv3YrKxkapkKAHy/r1UVlbf9LVC6LPnSP++XNs27fHvHx5Yi5dMnRI2UbkkSOE/v47TuPGYVa8uF7HOo0di5GdHQHDh/OJrbskZTKZyEpa37aEDs3g5zlw5SPI/ZzGjsUkf35e9+mDiIkxdDjSB6hDQnj11VdEHjxIznXrsOvWDQAjS0ucx48ncu/edKkTHOflRdiff+IweDBG1tZp7u8tE1dXLKpVI2LXrnTrE5RpFuFbt2LdvDlGVlZ6HWtkaYlV48af9PSC8L/+wiRvXswrV8a8QgViLl+WiZUO1AEB+Pfvj2WdOtj17q338UZ2duSYOZOoQ4eI+PvvDIhQkhQykZW0VCqYOEipM/v9L+CX/t+QZiojS0tcly0j9v59gqZPN3Q4UgriX7zgZdOmxN69i/v27Vg3aZJgv3WbNphXrkzgzz+n+QKSN7/+irGrK7bdu6epn6RYt2hB1MmT6VrLOPbaNeIePdJ7WsFbNq1bE3v7NrEPHqRbTNmFJjqa8J07sfn6a1RGRlhUqoTa3594+XV3ioQQ+A8ejIiLw2XRIlRGqUsVrJs0wbplSwJGj0YdFJTOUUqSQiayUgJmprB0AhgbQe+xEB1r6IjSxrx0aZxGjyZk/nyizp0zdDhSEmLv3uVFo0aIsDBy7duX5KIEKpUK56lTib1zh7B161I/1r17hG/diuOwYTrP99OHdfPmEB9P5L70m58TtmULxq6uWNaqlarjLevVQ2Vj80lOL4jcvx8RFoZNu3YAmJcvD0DMv/8aMqwsL2zNGiL37sVl3jxM3NzS1JfzjBkQF0fg2LHpFJ0kJSQTWSkRFyf4fRLc9YafZ0N2/xbOfsAALKpUwb9Pnwy5EEdKvejz53nZtCnGjo7kOnAAsyJFkm1rUb48Nu3bEzRlSqpLKL2ZMQOTPHmw/fbb1IacIhN3d8y/+CLdphcItZqI7dux/vJLVCYmqerDyMIC66ZNCf8EpxeEb9mCefnymBUuDIBxjhyYFCggL/hKQeyjRwSOGYNt585YN2uW5v5McubEacIEwjdtIvL48bQHKEnvkYmslKTSRWHGcNh6AFZl87UFVMbGuCxdioiO5lWnTmiiogwdkgRE/PMPvl9+iVmpUuTaswcTd/cPHuM0ZgwiKipV5bhibt4kYudOHEaMQGVmlpqQdWLTsiWRx46lS73SqJMnUfv56bwIQnKsW7cm7u5dYu/dS3NM2YU6MJDIQ4e0Z2PfsqhQQV7wlQx1cDD+33+Pibs7zlOmpFu/tp07Y1G9OgFDh6KJjEy3fiUJZCIrpeDLBtD7G5i8BE5n8xMYph4e5Ny4kdhbt/D/4QdZX9bAQleuxK9bN6yaNMF9yxadS2CZ5MqFw+DBSjkuL/2Wo3szfTomn32Gbfv2qQlZZ9YtWkBsLJEHD6a5r/CtWzEtWBDzcuXS1I9VnTqobG1TNb1AExrK6++/58nnn+P75ZcE/vILYVu2EHv3LiI+6xaeDt++HYTA5ssvE2w3r1CB2Js3EbHZfN5UOol/9YrQlSvxbdOGJ0WKEHvnDi7LlmFkY5NuY6hUKnLMmYPa15c3v/6abv1KEoBKGPDyzWnTpvH3339z7949LC0tqVatGjNmzKBo0aIpHrdlyxbGjh2Lj48PhQsXZsaMGTRt2lSnMUNDQ7G3tyckJAS7dKof+TFTq6HbKLhwA9xyKDXFzUzAxARMTZSfTd/9v7Hyr5UF9GoHBfIY+h4kFLFnD35dumDfvz/OEyYYOpxPjhCCN1OnEjxrFnZ9+uA8ebLeF5JooqJ4Vrky5mXK4LZ+vU7HRF++zMsGDXBZuhTb987QZYQX9etjnCsXbmvXproPTVQUT4oWxaFfPxx//DHNMb3u25eYq1fx0GOueMyNG/j16IH69WvsunUj7vFjYm/eJP7pUwBUFhaYFS+OWcmSmJcujVnJkpiVKJFiNQghBCI6GhEejiYsDE14OJrwcFRGRph/8QUqlSrN9xXgRcOGGDs54fbnnwm2v30u5Dp8GIv/5sx+auK8vYnYvZuIPXuUs9NGRljWqIFV8+ZYN22q07cjqfFm9mzeTJtG7oMH0/zhTPq46ZOrpW7SVTo5ceIE/fr1o1KlSsTHx/PTTz/RsGFD7ty5g3Uyb4Rnz56lQ4cOTJs2jebNm7Nx40Zat27NlStXKFmyZCbfg4+fsTEsHAfrdkBYhLLyV1w8xMVBvBpi4yD+v20xsRDx3z6fl3DoLPw1R1kCN6uwbtYM5ylTCPzpJ0zz5cOuRw9Dh/RJCZ47l+BZs3CaMAH7/v1TlbS8Lcf1ulcvok6cwNLT84PHvJk2DdMiRbBp2zY1YevNumVL3kyfjiY8PNVntiL370eEh6d5WsFbNq1bE/7nn8TeufPBmqBCCMJWrSLgp58wK1YM982bMf3sM+1+dXAwsbduEXvzJjE3bxJz+TJhGzcqbwYqFaaFCmFasCAiOlqbqGrCwpTkNTxc+YScBKfx43EYODDN9zXOy4uYS5dwXb480T7zkiXBzIyYS5fSNZHVREURPHs2Ju7uSkJfvHi6ntVMCyEEsbdvE7F7N5G7dxN75w4qCwss69bFZdEirBo1wtjRMcPjcBgwQJlS1KYNOVev1um1K0kfYtAzsu/z9/fH1dWVEydOUCuZK3S/+eYbIiIi2L17t3ZblSpVKFu2LEuXLv3gGPKMbObwD4IOQyEkHP6cDQXzGjqihAJGjSJ0xQrcNm3CqkEDQ4eTZQiNJtWldj7Yd2wsT0uVwrplS3L89lva+hKCl02aIMLDyX38eIoXQr29oMx15UpsWrdO07i6ivPx4Vn58mka81WnTqj9/Mh9+HC6xCRiY3lSpAh2vXvj9NNPybbThIbiP3gwETt2YNerF84TJ6IyN/9w/zExxN67R8zNm8TeukWctzdGlpaobG0xsrHB6J1/VTY2yv9tbJT/29oSsngx4X//Td5LlzB2cUnTfQ2aPp2QxYvJd/8+RpaWifa/aNgQ0wIFcE3Hlf/Cd+7kdffuytdVbxP6ggUxK1ECs1KlMC9VCrOSJTF2c0u3s84fEv/yJSFLlhCxZw/xPj4Y2dlh1agR1s2bY1m3brrWUdaVOiSE1z16EHXyJDl++01bL1qS3qVXriaykIcPHwpA3Lx5M9k2Hh4eYs6cOQm2/fLLL6J06dI6jRESEiIAERISkpZQJR28DhSiXlchKrYV4tETQ0eTkCY+Xvh27Ci88+QR0devGzqcLCHm/n3hU7SoCNuxI0P6D9u6VXg5OoqYu3fTpb+oy5eFl6OjCFm1KsV2L1q0EM9q1hQatTpdxtXVM09P8ap791QdGx8UJLxcXUXw0qXpGpNf377iaeXKQqPRJLk/+vp18aRCBeHt4SHCtm9P17E/JD4gQDzOl0+8Hjo0Tf1oNBrxpHx54devX7Jt/EeNEk/Kl0/TOO97PWCAeFqlitBER4vo69dF6IYNwn/UKPGieXPxOF8+4eXoKLwcHcXjwoXFyy+/FAG//CLCtm4V6qiodI3jLU1cnHher554/Nln4vWQISLi8GGhiYnJkLH0pYmLE/4jRwovR0fhP2qU0MTFGSyWuOfPhe8334ioy5cNFoOUmD65WpZJZNVqtWjWrJmoXr16iu1MTU3Fxo0bE2xbtGiRcHV1TbJ9dHS0CAkJ0d6ePXsmE9lM5B+UdZNZdXi4eF63rvApVkzEPXtm6HAMSh0WJp5+8YXwcnRUkr5kEp20eNGsmXjRokW69un3ww/icaFCQp3M6znyxAnh5egowvfuTddxdRE0a5bwzpNHqCMj9T42ZNUq4ZUjh4jz80vXmCIOHhRejo4i+tatBNs1Go0I+eMP4ZUzp3jm6SlivbzSdVxdvVm4UHg5O4uYO3dS3UfUhQvCy9FRRJ44kWybtx+q4gMCUj3OuzQajfApXlwE/PxzsvtjnzwR4bt3i8Bp04Rvp07iSenSwsvRUbzq3j1DXm9BM2cKL2dnEfXvv+ned3oJXrFCeOXIIV5+9VWyr+GMpNFohG/HjsoHjHz5RNSlS5keg5Q0fRLZLFO1oF+/fty6dYs/35uYn1bTpk3D3t5ee/PwyEITNj8BORzhzzngYAvfDIFHTw0d0f8ZWVuTc+NGVKam+H7zTZrLJWlCQ7NlNQQhBP6DBhH/8iXOM2YQe+sW0WfOpOsYsXfvEn32LHbpvJqWthxXElMVhBAETZ2KefnyWDVunK7j6sK6ZUtERARRR4/qfWz4li1Yenpi4uqarjFZenpi5OCQYMlaTWgor7/7joDhw7Hr0oVc+/cnmA+bmex79cIkf/40Fc8P37wZ41y5sKhRI9k25hUrAhCTTvVk4+7dQ+3ri2W9eknuV6lUmObNi3WzZjiNGoXb+vXkvX4d1xUriNixg/CNG9Mljrdibt3iza+/4jBoEBb/3desyP6773DbvJmYf//lRaNGxPn4ZOr4kbt3E7lvHy4LFmBarBi+bdoQLUuzZTtZIpHt378/u3fv5tixY+TJk/Jl7m5ubvj5+SXY5ufnh1syq4+MHj2akJAQ7e2ZXJow0+VwhE2zwdEO2mexZNYkZ07c/voL9YsX+HXrhoiL07uPmKtXef399/gUKpTqPgwp9Pffidi+HZcFC7Dr2RPTzz8nRIf55nqNsWoVxq6u6VJg/V3acly//56oHFfU4cPEXLyI4+jRmTYn8V1mhQphWqyYzosjCI2G6MuXCZoyhehz59LtIq93qczMsG7WjIgdOxBCEHPjBs/r1iXy8GFcV64kx6+/ZsiKZ/rE5zxhAlFHjxJ56JDex4vYWMK3b8fmq69SnOttkjcvRjlypFvSEnnkCCpLSyyqVtXrOJs2bbDp2JGAUaOIffQoXWIRsbH49+2LWeHCOI4cmS59ZiSrOnXIdfAgIi6OFw0aZNoKjJrQUAJGjcKqcWNsOnbEffNmzEuUUJLZixczJQYpnWT06eGUaDQa0a9fP5ErVy7x4MEDnY5p166daN68eYJtVatWFd9//71Ox8s5sobjHyREg+5CVGgjxMMsNs0g8sQJ4eXqKl4PGKDT13ya+HgRvmuXeNGkifBydBRPypYVAWPHCq+cOYVv585ZZi7ah0SdOye8XFwSfCUasmaN8HJySrevl9VhYcLbw0METp6cLv0l6j8yUviULCl8O3XSbtNoNOJ53brieePGGfK1ra6Cpk8X3nnzCk10dJL71eHhInz3bvG6f3/hU7So8hVngQLi9YABqZqSoIuII0eEl6OjCPj5Z4NPJUiKRqMRL1q0EE+/+EJoYmP1OjZ8zx5lHvbt2x9s69uhg3jZpk1qw0zgZevW4uXXX6fqWHVYmHhasaJ4VqdOurxvBE6ZIrxcXLLd3P/4wEDxokUL4eXqKkI3bMjw8fxHjBDeefIkmFamDgsTL5o2Fd4eHiLq3LkMj0FKXraZI/vDDz8Ie3t7cfz4ceHr66u9Rb7zBt65c2cxatQo7c9nzpwRJiYmYubMmeLu3bti3LhxwtTUNMULxN4lE1nD0iazX2a9ZDZ00ybh5egogmbNSraNOiREBC9eLJ6ULSu8HB3FiyZNRPiuXUITHy+EECLiwAElmf322yyfzMb5+Qmf4sXFiyZNEiQM6shI8bhQIeH/44/pMk7IqlXCy9k5Q+chv53zGHn8uBDi/wlN5KlTGTamLmLu3BFejo4i4sAB7ba4Z89EyB9/iJdffy283dyEl6OjeFq5sggYO1ZEnjmT4Re+aGJjxeOCBZULbUaMyLCLjdIi+vp14eXkJIKXL9fruFdduohnNWvq1DZo5kzxOF++NF8EqA4PF145c6bpwrzoq1eFl6urCPjllzTFEn3livDKkUMETZ+epn4MRRMTI14PGqR80Bo3Tvu+mt6iLl5Unl+LFyfapw4LEy+aN//ok1l1SIiIffJExHp5iZj790XM7dsi+sYNEX3lioi6eFFEnTsnIk+fFpHHj4uIw4dFxIEDOn1ATC/ZJpEFkryteucqZE9PT9G1a9cEx23evFkUKVJEmJmZiRIlSog9e/boPKZMZA0v4M3/k9kHPoaOJqGg6dOFl6OjCNuyJcH2WB8fEfDTT8Lbw0N4ubgIv969RfSVK0n2EXHwoJLMduqUZZNZTVyceNGihfApWlTE+fom2h84ebLw9vBI8wUYGo1GPKtZU/h27JimfnQZ53mjRuJptWpCExsrnlWvLl60apWhY+oa19PKlcXLdu1E4OTJ4lmtWsrV6zlyiBctW4o3CxeK2EePMj2u8L17Rfi+fZk+rj78+vUTjwsWFPFv3ujUPj44WHjlzCneLFigU/uIY8eUs7f376chyv8+vDo6ipiHD9PUz5t585QPPceOpep4dVSUeFqlinhWu7beZ7KzEo1GI4IXLxZezs7Ct2NHoQ4LS9/+/3t/eFanTrKJsjo8XLxo0UJ458kjIs+eTdfxs4KYu3eVv2X/VdLQ9eY/YkSmxZhtEllDkIls1vBuMnv/saGj+T+NRiP8fvhBeOXMKSLPnBFR584J386dhZezs3j82WcicOJEEffixQf7iTh4UHi7uQnfjh2zZDIbMG6c8MqRQ0SeOZPk/jhfX+Hl6ireLFyYpnHeXkEecehQmvrRaaz/ynH5fvON8HJ0FFHnz2f4mLoInDJFO2XAr1cvEbZ1q4gPDjZ0WFlenK+v8M6TJ9lKAO8LWb1aOfP/8qVO7dUhIcLLySnNX2P7jxwpnpQtm+YpLBq1Wrxs3Vr4FCsm4v399T4+YNw44ZUzZ6aeNctIEQcOCG8PD/GsZs10/TbnzZw5wsvZWURfu5ZiO3VEhHjRqpWSzJ4+nW7jG5omNlY8q1NHPK1cWUQcPSoiT50SUefOiaiLF0X0lSsi+sYNEXP7toi5f1/EenmJ2CdPRNzz5yLu1SuhDg3NtDhlIpsCmchmHQFvhGjYQ/dkVq0WIjhMiCcvhbhxX4jgDHpNaWJixIuWLYVXjhzar31DVq0S6ogIvfqJOHRISWY7dEh2jqQhhP/zj/BydPzgmSu/Pn3Ek9Kl0/RVt1+fPuJJuXKZVsPV74cfhJejY6rnK2YEdWSkiLp82aC1MrOroN9+E16urjrN4X3RrJl4+eWXevX/tEqVNNetfVKhgvAfNixNfbwV5+srHhcqpLxn6JEYR124ILycncWbuXPTJY6sIub2bfGkTBnhU7x4ms+cCyFErLe38HZ3FwFjxujUXh0RIV62bi28c+c2+DSl9BI0fbrwypEj2W8Uswp9crUstbJXZpAre2UtgcHQcRgEvIGeX0NYOASHQUiYsipYyH//Dw5Tlsh9t7qVWw44uArsM2AVSHVICG9mzMCqTh0s69VL9WpXkUeO4Pftt1h6epJzzRqdVkjKSHFeXjyvWxer2rVxXb06xav5Y65d40XduriuXo1Ny5Z6j6UODORpyZI4jh6dLsuO6iL+5Ute//ADzlOnYl6iRKaMKWUcTVQUzypXxrxcOdzWrk22XdzTpzwrWxaXJUuw/eYbnfv3HzCAmJs3yXP8eKrii/P25lnFiuTcsAHrJk1S1cf7Ivbvx69jR5x//RX7nj0/2F4TGckLT0+MHB3JtW8fKmPjdIkjq4j388P3yy9RBwTgvn17ql/XQghetW1LnJcXec6e1XlVM01UFH6dOhF94QJumzZhmcyqo9lBzNWrvGjYEIehQ3EaPdrQ4aRIn1xN77/O+/fv5/Tp09qfFy1aRNmyZenYsSNv3rzRP1rpk+bsABtnQd5csHgD7DwK1+7Bm1CwtYYShaBJLejTHiYPgsXjYMNMpZxXeCRMWZIxcRnb25Nj6lSsGjRI05KtVvXqkXPDBqJOnuRVly5ooqPTMUr9aCIieNWlCyZubrgsWPDBklTmZctiUbUqoaksxRW2YQMAtp06per41DDJlYtcO3fKJPYjYWRpifO4cUTu3k3UO3933he+dSsqKyu9y7uZV6hA7O3baCIjUxVf5NGjYGKCZQo1a/Vl3bgxdr16ETR2LLF37nywfdCkScS/eIHLokUfXRILSonEXP/8g0muXPi2bEnM1aup6id861aijh8nx8yZei3Na2RpSc4NG7CoUoVX7dsTmcoPPYamiYri9Q8/YFayJI7Dhxs6nPSl7+nekiVLai+uunHjhjA3NxejR48WVapUEd26ddO3u0wnpxZ8PDbsEiJvbSFOXDR0JB8WcfSo8HZ3Fy+//togV4lrNBrh9/33wjt3br1WTXo7DUHfr6E0arWyTKiOZfEkKTkajUY8r19fPPP0TPLinLcX1Pn17q1339G3binVLVJ5QY9vhw7pvlqdEMp0lKfVqomnVaumWIYt8vRp4eXomOTV9x+b+OBg8bxBA+GdN6/e89/jAwPF48KFU71ktBDKxXQvv/pKeLu7i4ijR1Pdj6EE/Pyz8HZzS9OqeZkpQ+fIWltbi8ePHwshhBg3bpxo27atEEKIy5cvi5w5c+rbXaaTiezHQ6MRov0QIap+I0SYftNXDSLi2DElmf3qq0xPZkP++EOpxrB1q17HaeLjxZMyZfROEiIOH85SF1xJ2VvU+fPCy9FRhK5fn2hf9JUrygWFhw/r3a8mPl5458kj3sybp/+x0dHKsRk0LzXm9m3h7e6e7JXi6rAw8aRsWfGiWbNMm4NuaOrQUKU0Vp48KS5B/D6/fv3E43z5RNyrV2kbPypKvGzXTni7uYmoCxfS1Fdmijx9Wng5OYk38+cbOhSdZegStWZmZkT+9zXM4cOHadiwIQBOTk6EpnGJT0nSh0oFM0ZAUAhM/93Q0XyYVe3auG3aRPTZs/h9+22mTTOIvnSJgNGjsevdG5u2bfU6VmVsjF3v3oRv3068r6/Ox4WuXIlZyZKYV66sb7iSlIjFF19g/eWXBE2ejCY8PMG+sM2bMXZ1xdLTU+9+VcbGmJctm6qlaqMvXEBERCS7LG1amRUvjtOkSYSuWEHE/v2J9geOG4c6IACXhQvTNP0pOzGytcXtr7+wqFxZ+Zpfh9Xfok6fJnzjRpzGj8ckZ860jW9hgdvatZjkzUvoqlVp6iuzaMLC8O/XD4sqVbDv29fQ4WQIvZ/9NWrUYOjQoUyaNImLFy/S7L85SQ8ePPjg8rKSlN7yusOPvWDdTjh/zdDRfJilp6eSzJ47h1+nTmiiojJ0PHVAAH7dumFetizOEyemqg+7b79FZWFB6MqVOrWPf/6cyAMHsOvRwyBLw0ofJ6fx49EEBxM8b552m4iPJ+Lvv7Fp2xaViUmq+jWvWDFVS9VGHjmCcc6cmGXgfGy7Hj2watwY//79E3yQjDx2jLBVq3CeMAHT/PkzbPysyMjKCreNG7GsU4dX335LxO7dybbVREcTMHQo5l98gW2XLukyvsrcHOsWLYg8dAgRH58ufWakwLFjUQcGfrRzqCEViezChQsxMTFh69atLFmyhNy5cwOwb98+GjdunO4BStKHdG0NFUvCyN8gynDXUunMslYtJZk9f57nVasSMHo0kceOIWJi0nWceD8/XvfujYiNJefKlajMzFLVj5GdHbadOhG6apVOiXfomjWorKyw+eqrVI0nSUkx9fDAvm9fQhYtIv75cwCijh1D7e+PTbt2qe7XokIF1L6+xL98qddxUUeOYFm3boZ+WFOpVMqFmWZm+Pfti9Bo0ISG4j9gAJaenth2755hY2dlKnNzcq5ejXWzZvh17074tm1JtgueM4e4J09wmTMnXc9aWzVqhCYoiOh//023PjNC5KFDhK1di/OkSR/3B55MmOqQpcg5sh8nr6dCFG4oxIS01e/PVNHXronXQ4cKn5IlhZejo/DOk0f4fvutCFmzRuei7m9pNBoR++iRCFm3Tvj17SuelC+vrMbi6qrXXLLkxHp7Cy8nJxGyenXKccTECJ/PPxf+w4eneUxJep86NFT4FC2qnbP9qmdP8bRKlTQtRhD38qXwcnQU4bt26X7MixfKnPNt21I9rj4ijx9X5jjOmyf8+vUT3nnzZuiSz9mFJj5e+PXtq7w3rVuXYF/MvXvCy9VVBE6enP7jqtXCp0iRNC8pnJHig4KET7Fi4uXXX6d5sQ5D0CdX0/u7mCtXrmBqakqpUqUA2LlzJ6tWraJ48eKMHz8es1Se9ZGktPjMA4b1gGnLoFltqJANqi+ZlymDy6xZCCGIvXOHyIMHiTx4kIChQ0Gjwax0aawaNMCqYUPMy5dP8LWQUKuJvX2b6HPnlNuFC6j9/EClwqxECazq18eiShUsqlVL87wwANMCBbBq2pSQpUux7dIl2bNQEXv2oPbzw+4TPVMkZSwjW1scf/qJgMGDse3Ykci9e3EYPjxNZ0VN3N0xzp2b6EuXsG7RQqdjIo8eBSMjLGvXTvW4+rD09MR+wACCJk0CtZoc8+djIqfyoTI2Vs5YW1gQMHAgIjoa+549ERoNAUOGYJo3Lw7DhqX/uEZGWDZoQOT+/ThPmJDu/aeHgBEjENHRuMyb99FP8dJ7QYRKlSoxatQo2rZti7e3NyVKlODLL7/k33//pVmzZsydOzeDQk0fckGEj5daDW36Q1gk7F0OFtn0M5U6KIioo0eJOHCAqCNH0AQHY+TsjFX9+pgWKED0v/8SffEiIiwMzMwwL1cOy6pVsahaFfPKlTG2t8+QuKLOnMG3RQvctm7Fqm7dJNu8bNkS1Gpy7dmTITFIklCreVG7NnFPniDCw8l740aakzq/bt1QBwSQK4X5lgnad+9O/PPn5NbhYqP0ImJjedmqFcYuLsriKh95cqIPIQSBP/9M6NKlOE2ciJGtLQFDhuC+cyeWNWtmyJgRu3fj16ULHpcuYfrZZxkyRmqFb9/O6+++w3X5cr0v8M0q9MnV9E5k7e3tuXLlCgULFmTGjBkcPXqUAwcOcObMGdq3b8+zZ8/SFHxGk4nsx+3BY2j2vbJK2I+9DB1N2on4eGIuXdKerY1//hzzihWxeJu4li+PkYVF5sQiBC9q18Y4Z07cN29OtD/2/n2eV62ard88pewh6sQJfL/8EosaNci1a1ea+wteuJA306eT38fngxeNifh4nhQpgl3v3jiNGpXmsfUhNBpQqWQSmwQhBG+mTCF49mzlgqw2bXBdtCjDxtOEh/OkcGEcx47FIQtVA4h/9Yrn1atjWasWritXZtvnij65mt5TC4QQaP5bJ/Tw4cM0b94cAA8PDwICAlIRriSlnyIFYGAXmLNKWRGsdNHU9SMEvA4CVyelzJehqExMlCkCVarg9MsvhgsE5cIT+z598O/Xj9gHDzArUiTB/tBVqzDKkQPr/94TJCmjWHp64jh6NBZVq6ZLfxYVKyIiI4m9exfz/6bNJSfmyhU0wcFYZVDZrZR8KmW2UkOlUuE0ZgxGNjaEbd6M86RJGTqekY0NFjVrErl/f5ZJZIUQBAwejMrUlBwzZ2bbJFZfer8qKlasyOTJk1m3bh0nTpzQlt96/PgxOdNhLp4kpVWf9lD0MxjxK8TG6X/89XvQcRhU/go6j4QnL9I/xuzKpk0bjF1dCVm2LMF2TUQEYZs2Yde5MypzcwNFJ31KHEeMSLelYc1KlwYTE53qyUYeOYKRgwPm5cuny9hS+nIYPBiPs2cxdnLK8LGsGzcm+tw51MHBGT6WLsLWrSPy4EFyzJuHsbOzocPJNHonsnPnzuXKlSv079+fn3/+mUKFCgGwdetWqlWrlu4BSpK+TE1g5o/w6Aks3qj7cY+fQ9/x0PIH8A+CsX3h8TNo0EPpJy7rlwzMcCpzc+x69CD8zz9Rv3mj3R6+bRsiPBzbrl0NGJ0kpY6RlRVmJUroVE4p6sgRLOvU+Whrckq6s2rYENRqoo4cMXQoxD15QuCYMdh26oR1o0aGDidT6T1HNjnR0dEYGxtjamqaHt1lGDlH9tMxa6WSgO5eBsUKJt/OLxDmr4VNu8HVGYZ2h7YNwdgYIqNg9mpYuRUK54dpw6B88cy6B1mT2t+fp6VL4zhqFA6DBilzZ+vUwcTNDbc//zR0eJKUKgHDhxN1+jQe588n20YdGMiTIkVwWbAA244dMzE6Kat6Xrs2poULk3P5coPFIDQafFu1Iv7pU/KcOoXRR5Db6JOrpXrCzeXLl1m/fj3r16/nypUrWFhYZPkkVvq09P9WKcs1fAbEqxPvD4uAmSvB81v45xj82BuOr4N2TZQkFsDKEsb8ALuWKmd62/SHMXMhNDxxf58KYxcXbL76ipDff0fExRFz+TKxN25g16OHoUOTpFQzr1iRuAcPUIeEJNsm6vhxEALLZKp2SJ8eq0aNiDp8GBGXinls6SRs3Tqiz5zBZeHCjyKJ1Zfeiezr16+pU6cOlSpVYuDAgQwcOJCKFStSr149/P39MyJGSUoVczNlisEdL/j9r/9vj4mFP7ZCzY7K9u5t4NRG+P4bsEhmemfJwrBzMfzSF7YdgPrdYd9J5aKwT5Fdnz6ofX2J2LWL0FWrMMmbN8PWnJekzGBeoQKgXMyVnMgjRzArUQITN7fMCkvK4qwbN0YTEkJ0CmfyM5ImNJSgKVOw+eabDCs1ltXpncgOGDCA8PBwbt++TVBQEEFBQdy6dYvQ0FAGDhyoV18nT56kRYsW5MqVC5VKxY4dOz54zIYNGyhTpgxWVla4u7vTo0cPAgMD9b0b0ieizOfQux3MWa2U5vr7INTtApOXQOOacHK9UqbL3ubDfRkbQ4+v4PBqKFUE+oyDXmPg5euMvhdZj3mJEljUqkXwnDlEbN+OXbducs6glK2ZFiyIkYMDMZcuJblfaDREHT0qP7BJCZiVKYOxmxuR+/cbZPw3s2cjIiNxGjPGIONnBXonsvv372fx4sUUK1ZMu6148eIsWrSIffv26dVXREQEZcqUYZGOtd7OnDlDly5d+O6777h9+zZbtmzh4sWL9Or1ERQMlTLMkG6Qx02pLztkGpQsAodWwfTh4Oaif3+5c8KKybBkPFy/D/W7KXNo1UlMX/iY2f/wA7F37iDUamw7dTJ0OJKUJiojI8zLl0+2ckHs7duoX782SNktKetSGRlh1bAhEQcOkE6XHOks7vFjQpYuxWHgQExy587UsbMSvevIajSaJOfCmpqaauvL6qpJkyY0adJE5/bnzp0jf/782jO/BQoU4Pvvv2fGjBl6jSt9WizMYd7PyjSCHl+lz8VaKhU09YQaFeDXFTBxMWw/DHN/goJ5095/dmDVoAGmRYpgXq4cxi6p+EQgSVmMeYUKhK5ciRAiUQ3OqCNHUFlbY/HFFwaKTsqqrJo0IWztWuIePkxUXzsjBY4bh7GLC/b9+2famFmR3mdk69aty6BBg3j58qV224sXLxgyZAj1MviTatWqVXn27Bl79+5FCIGfnx9bt26ladOmGTqulP2VLgoLf0n/igN2NjB5MGxbABGR8M0Q8HqavmNkVSojI3IfPEiOOXMMHYokpQuLihXRBAYS7+OTaF/kkSNY1qqFyiybrn0tZRjLWrVQWVpm6vSCqFOniNy9G+fx4zGyssq0cbMivRPZhQsXEhoaSv78+SlYsCAFCxakQIEChIaGsmDBgoyIUat69eps2LCBb775BjMzM9zc3LC3t09xakJMTAyhoaEJbpKU3iqUgM3zwMEWOgwDn09kEQUjO7tMWyJXkjKa9oKv96YXaEJDib5wQU4rkJJkZGmJpadnpiWyQq0m8OefMa9UCes2bTJlzKxM70TWw8ODK1eusGfPHgYPHszgwYPZu3cvV65cIU+ePBkRo9adO3cYNGgQv/zyC5cvX2b//v34+PjQp0+fZI+ZNm0a9vb22puHh0eGxih9unI4wsbZYGMJ7YfA05cfPkaSpKzD2MkJ04IFiX7vgq+oU6cgPl6W3ZKSZdWoEdEXL6IOCsrwscLWryf21i2cp079ZJahTUm6LYiQViqViu3bt9O6detk23Tu3Jno6Gi2bNmi3Xb69Glq1qzJy5cvcXd3T3RMTEwMMTEx2p9DQ0Px8PCQCyJIGcYvQJliEBsHf80Fj4+4Uo9/EFhbKvV2Jelj8LpPH+IePSL34cPabf5DhxJ9+jQeFy8aMDIpK4t/9YqnxYvjsmQJtt98k2HjaEJDeVqxIlb16uG6ZEmGjWNo+iyIoNPFXvPnz9d5cH1LcOkjMjISE5OEIRv/V/InuXzc3Nwcc7n2u5SJcuaATbOh3WDoMBQ2z4VcroaOKv0dOQcDJyuJ+l9zwd7W0BFJUtqZV6hA+I4diJgYVObmCCGIOnIEKz0uTJY+PSZubpiXL0/k/v0Zmsi+mTVLKbc1dmyGjZHd6JTIztHxYg6VSqVXIhseHs6jR4+0Pz9+/Jhr167h5ORE3rx5GT16NC9evGDt2rUAtGjRgl69erFkyRIaNWqEr68vgwcPpnLlyuTKlUvncSUpo7m7wJ//JbPthyjJbGpKfWVFQsDSP2HGcvCsDNfvQtdRsGGmcnZWkrIzi4oVITaWmBs3sKhUibiHD4l/9kzOj5U+yKpRI4IXLEDExmbIRYFx3t6ELF2K4/DhmMic5/+EAR07dkwAiW5du3YVQgjRtWtX4enpmeCY+fPni+LFiwtLS0vh7u4uOnXqJJ4/f67zmCEhIQIQISEh6XhPJClpT32FqPqNEJ7fCvEqwNDRpF1UtBADJgmRt7YQv/0hhFotxI17QhRvKkT7IUJExRg6QklKG01MjPB2cxPBS5YIIYQIXrxYeLu5CXV4uIEjk7K66Bs3hJejo4g4ejRD+vft3Fn4lCwp1BERGdJ/VqJPrpZl5shmFn3mXUhSenjyQjkza20Ff80BF6eMHc/XHw6chv2n4M4j6NQC+nYEW+u09fvKH3qNhQc+MHMktHjnupcL16HzSKhVEZZMAFO9K1RLUtbxonFjTPLkIeeKFfh+/TWo1bj//behw5KyOCEET0uXxrpZM3JMn56ufUedOoVvq1a4Ll+OTdu26dp3VqRPrqZ31QJJkvSTLzf8OQfCIpQ5s4HB6T+GzwtYugla94Uq7WDyYjAzhS/rw8pt4PktrNsJ8alcfezqHWjxg3Jx19b5CZNYgC/KwLKJcPwiDJ8Beq6NIklZikWFCsRcvowmKoroM2fksrSSTlQqFdaNGxO5f3+6rvIl1GoCf/pJlttKhkxkJSkTFMijzJkNDoWOwyAoJG39CQH3vGHuGmj8nZKozlkDrjlgzk9weTusnQETBsLxdVDnCxg7Dxr2UC7S0uc99u+D8M1gZZnfXUuhVDIL19T5QllBbddRZaxP67se6WNiXqEC8U+eELFrFyI6Ws6PlXRm1agR8U+fEnf3brr1GbZ+PbG3b8tyW8mQUwskKRM98FEu/sqZAzbNAgc9noLRMXDXS5kysP+UchbW1hrqVYXGNcGzUsplsG49hMlL4NxVqFYOxvSFEoWSb69WKxd0LfsLvm4MU4aAuQ7XL/y1F0b+pkxn+LGX7vcvKUJATKyyzLAkZZa4Z894VqYMpoUKoYmKIu+NGzKBkHSiiY7mSeHCOAwdiuOQIWnv7xMpt/U+fXI1mchKUia7/1g5w5k7JywYC5HREPgGAkOUf4NCIOC/f9/dHhGlHO9kDw1rKMlrtXK6JZdvCQFHz8OUpeD9DL5qBMN7JK6oEBoOAybDyX9hzA/Qoy3o83d8xRaYtFhJZPt21P249+NcsB4ePIaDK5UzwpKUGYQQPC1WDPXr19h27ozLvHmGDknKRl516YL61StyHzyY5r4Cx40j9I8/8Lh48ZOqVJDudWTfFxwczMWLF3n9+jWa9ybDdenSJTVdStIno2gB2DhLWcq2znsvFysLcHKAHA7Kv4XywRcO4OwIzvaQNxeUKw4mxqkbW6VSzuDWqgSbdsOc1bD7OPRuB9+3V8pnPX4O3/2szIddM11pq6+eXytzgmcsBxsr6NJat+PUath3EhZtgDteULGk8pjMWqlMmZCkzKBSqTCvWJHIvXvl/FhJb9aNGuE/cCBqf3+MXVJfd1GW29KN3mdk//nnHzp16kR4eDh2dnYJvm5RqVQEZcLybGkhz8hKWcXL18o8V2eH/98sLTI3htBwWLwRVm4FO1ulwsHKbeDiCCumwGdpWNFZCOWs7B9blSS0TYPk28bFw47DSizez6BmRejfSbmIbMM/MGYu7P0diqcwFUKS0lPw/PkETZ5MvocPMba3N3Q4Ujai9vfnyeef4zJ/PradOqW6n1dduhBz9SoeFy5gZGWVjhFmfRk6taBIkSI0bdqUqVOnYpUNH1iZyEpSYs9ewa/LlQu1PCsrUx7sbdLerxDKfNltB2DpBGVKxLuiY2HzXlj2Jzz3g4bVoV8nKFvs/23i4pWL1DzclQvYJCkzaCIiiHvwAPNy5QwdipQNvWjUCOOcOXH7b0EnfX1q5bbel6GJrLW1NTdv3uSzzz5LU5CGIhNZSUrey9eQ0xmMUzl1ISlqNfSfBIfPwqppUKOCMt93/S5YvlkpR9a8tpLAfp7M28reE/DDeGVKRvXy6RebJElSRngzZw7Bs2eT7+FDjCz0+6pNqNW8qF0blZUVufbv/yQvNMzQOrKNGjXi0qVLqQ5OkqSsK5dr+iaxoPQ372eoWg56jVGmG1RrD7+tgLpV4Oga5QxwckksQJNaUK4YTFsma9RKkpT1WTdqhIiIIPrUKb2O00RGEvjTT7Lclh70vtirWbNmjBgxgjt37lCqVClMTU0T7G/ZsmW6BSdJ0sfBzBSWTYAuPyoLM3RoBr2/USo36EKlglG94ZshsOd44gUZJEmSshLTYsUwyZuXyAMHsGqQwgUC74g8coSA4cNRv3qF0+TJWFSokMFRfhz0nlpgZJT8SVyVSoVancqlgzKJnFogSYYTG6fUw7VL5fzb7qPB6ykcXq0kx5IkSVlVwKhRROzZ88E6xPGvXxP4889EbNuGRc2a5Jg1C7NCn/aVrRk6tUCj0SR7y+pJrCRJhmVmmvokFpSzss9ewcbd6ReTJElSRrBq3Bj1ixfE3rqV5H6h0RC6di3Pq1Qh6tgxXBYtwn3Hjk8+idWXXKJWkqRso2gBaNsQ5q+F8EhDRyNJkpQ8y2rVUNnaErl/f6J9sffv49uiBQGDB2PVpAkeFy5g26GDnBObCjrNkZ0/fz69e/fGwsKC+fPnp9h24MCB6RKYJElSUoZ2V8qE/f6X8v/M5hegrMZWIE/mjy1JUvahMjPDqm5dIvfvx3HECEBZwjZ4zhyC587FxMMD9x07sKxVy8CRZm86zZEtUKAAly5dwtnZmQIFCiTfmUqFt7d3ugaY3uQcWUnK/qYtg7U74MQGcHXKvHHj1dCkJzx8Al/WhyHdIa975o0vSVL2ErZ5M/59+pD39m3ivLwIGDqUuCdPcBg4EIdhw/QuzfWpyNA6stmdTGQlKfsLCYOanaBFHZgyJPPG3bQbRs2Cfh1h834IDoUOzaH/t0r9XUmSpHepg4J4UqQIZiVLEnvjBuZffIHL7NmYFSv24YM/YRl6sZckSZKh2dsqCyhs2q0saZsZIqJg1ipoVQ9G9oKT62FYD2Vp3VqdYMZyJcGWJEl6y9jJCcuaNYn38SHH7Nnk2rNHJrHpLFVnZJ8/f86uXbt4+vQpsbGxCfbNnj073YLLCPKMrCR9HKJjoU5nZTnbJeMzfrw5q2HJRjiyFjzc/r89JFxZYnflNjA1gT7toXsbsLLM+JgkScr61CEhABjb2xs4kuxDn1xN7wURjhw5QsuWLfnss8+4d+8eJUuWxMfHByEE5cvLtSMlScocFmYwrDsMmwFX70C54hk3ll8gLPsLurVJmMQC2NvAyJ7KvoXrlYR31TYY0FmZdiDr3UrSp00msBlL76kFo0ePZvjw4dy8eRMLCwu2bdvGs2fP8PT05Ouvv9arr5MnT9KiRQty5cqFSqVix44dHzwmJiaGn3/+mXz58mFubk7+/PlZuXKlvndDkqSPwJcNlKVtpy2DjJztP3sVmJtBv2+Tb+PqBBMHwtG1UKsSjFsAdbrAtgMgS2xLkiRlDL0T2bt379KlSxcATExMiIqKwsbGhokTJzJjxgy9+oqIiKBMmTIsWrRI52PatWvHkSNH+OOPP7h//z6bNm2iaNGieo0rSdLHwdgYfuwFF27A0fMZM8b9x7B5Hwzuopx9/ZC87jB7NBz8A0oUhqHToeF38OceZTqEJEmSlH70nlpgbW2tnRfr7u6Ol5cXJUqUACAgIECvvpo0aUKTJk10br9//35OnDiBt7c3Tk5KzZ38+fPrNaYkSR+XOl9AlTLKxVa1KyvJbXqaukxJTju11O+4IgXg94lw7a4y5WDULPjtD+jSCr5tBc4O6RunJEnSp0jvM7JVqlTh9OnTADRt2pRhw4YxZcoUevToQZUqVdI9wHft2rWLihUr8uuvv5I7d26KFCnC8OHDiYqKSvaYmJgYQkNDE9wkSfp4qFQw+nvlzOnfh9K379OX4fgF5axvaue6li0GK6bA0TXQuCYs3gRVv4HRs+DR0/SNV5Ik6VOj9xnZ2bNnEx4eDsCECRMIDw/nr7/+onDhwhlescDb25vTp09jYWHB9u3bCQgIoG/fvgQGBrJq1aokj5k2bRoTJkzI0LgkSTKsssWgmSfMWqnUlrUwT3ufajVMWQIVSkCTdFh45zMPpebt8B6w/h9Ysx027oa6VaBXO6haVknKJUmSJN3pVX5LrVZz5swZSpcujYODQ/oGolKxfft2WrdunWybhg0bcurUKV69eoX9f1cB/v3333z11VdERERgaZm43k1MTAwxMTHan0NDQ/Hw8JDltyTpI/P4OdTvBiN6KiWw0mrrfqUiwt8LlWQ2vcXEKkvtrtgC97yV+bQ9v4bmtT+OSgdvQsBRXqwtSVIqZNiCCMbGxjRs2JA3b96kKcDUcnd3J3fu3NokFqBYsWIIIXj+/HmSx5ibm2NnZ5fgJknSx6dAHqXc1aIN8DoobX1FRSvzWZt5ZkwSC0oVhK8bw/4VsP43yOEIQ6ZCzY6wdBPExmXMuJlh4Xqo/DU8eWHoSCRJ+tjpPUe2ZMmSeHt7Z0QsH1S9enVevnypndoA8ODBA4yMjMiTJ49BYpIkKesY1AXMTeHLfvDgcer7+WMrBAYrc2MzmkoFNSvC2hlwaCV4VoaZK6HdYHj5OuPHT2837iu1dGPjlKkTkiRJGUnvRHby5MkMHz6c3bt34+vrm6YLqcLDw7l27RrXrl0D4PHjx1y7do2nT5UrIEaPHq0t9QXQsWNHnJ2d6d69O3fu3OHkyZOMGDGCHj16JDmtQJKkT4uLE+xYDDZW8GV/OH5R/z4C3sCSTdClNeTLne4hpqhIAfh1BGydD68DoVlv5YKz7CIyCgZNgWIFoXMr2LJfmUIhSZKUUfROZJs2bcr169dp2bIlefLkwdHREUdHRxwcHHB0dNSrr0uXLlGuXDnKlSsHwNChQylXrhy//PILAL6+vtqkFsDGxoZDhw4RHBxMxYoV6dSpEy1atGD+/Pn63g1Jkj5Sedxg2wKoXBq6j4ZVf+u3WMLcNWBkBANSWPwgo5UtBruXQcnC8O0ImL8ONBrDxaOrKUuVs8hzf1ZWOgsMhgOnDR2VJEkfM70u9gI4ceJEivs9PT3TFFBG02cCsSRJ2ZdardSAXbFFOTs4fgCYfKDG7KOn0LA7/Ngbvv8mc+JMiVqtJLHz1io1cuf+BA5Z9G3r6Hnlg8OkQcrZbIBvhgAC/pprwMAkScp29MnV9C6/VaBAATw8PFC9VydGCMGzZ8/07U6SJClDGBvD2L5Q0APGzlMuPFo4LuXVuaYvA3dX6Ppl5sWZEmNjGNINyhVXvrJv9j0sHQ+lsthihoHBMPJXqP2F8qHhrW9bQP9J8PAJFM5nsPAkSfqI6X1G1tjYGF9fX1xdXRNsDwwMxNXVFXUWX1RclyxfCEF8fHyWvy+SlNGMjY0xMTFJ9ME1uzl9GX4Yr8yhXTU16bmv568pZxDnj4FW9TI7wg97/kq5D/e8YeJAaN8sa9SdFQJ6j4VLt+DASnB1+v++2Dio0k55PMf1N1yMkiRlLxl6RlYIkeQftfDwcCwsLPTtLsuJjY3F19eXyMhIQ4ciSVmClZUV7u7umJmZGTqUVKtRAXYsgh4/Qau+8PskZQ7tWxqNMr+zzOfKggpZUR435SKwiYuU5W4v3YLJg8HSwG+7f+6Fg2eUx/TdJBaUerjfNIX1u2BkT8PHKknSx0fnRHbo0KGAsnDB2LFjsbKy0u5Tq9VcuHCBsmXLpnuAmUmj0fD48WOMjY3JlSsXZmZm2f5MlCSllhCC2NhY/P39efz4MYULF8bISO/rQ7OMgnmVZLbPeOg4DKYNU+q4AvxzTCkbtXmucqFXVmVupqwOVqEEjJ4Ntx/C0omQP5OrK7zl8wImLlSS1UY1km7ToZlSBWL38f8/3pIkSelF50T26tWrgPLH7ebNmwnOzpiZmVGmTBmGDx+e/hFmotjYWDQaDR4eHgkSdUn6VFlaWmJqasqTJ0+IjY3N9t+6ONrDul/h5zkwfAZ4P4OBneHX5dCwOnxRxtAR6qZNQyheCPqMg+bfw+xR0DCZRDKjxKth8BRlukZK0wby5oJalWDDLpnISpKU/nROZI8dOwZA9+7dmTdv3kd9xX92PuskSentY3s9mJkqtVoL5YNpy2DPcfD1h7W/Gjoy/Xz+GexaAiN+hV5joakn9O0IpYpkzviL1sP1+8p0B+sPlPH+toUS462HSkkxSZKk9KL3X6hVq1Z91EmsJEkfP5VKKa+1fBL4BynlogrmNXRU+rOzgaUTlMT89kPl7GznEXD2qn61c/V19Y5SEmzAt7ot4Vu3KrjlgI3/ZFxMkiR9mj6uUy2fsNq1azN48OAM6Tt//vzMnTs3Q/qWDO/48eOoVCqCg4MNHUqma1Adzm9RynRlVyqVMkf16FpYOFZZmazDUGWZ3gOn038hhYgoGDxVOfM7oLNux5gYQ4fmsOMwhMvraCVJSkcykZUk6ZNmb6PUa83uTIyhRV3YuxzWzFCmUPQeCw16wNb9EBefPuNMXgx+gcrqXaZ61L1p3xSiY5RkVpIkKb3IRPYTFhsrF0FPb4Z6TOXvUnpLpVJWAds8T1mqN39uGDYDanWCVdsgMir1fR8+Cxt3wy99oUAe/Y51c4H61ZSLvjJy2oMkSZ8Wmch+ROLj4+nfvz/29vbkyJGDsWPH8u56F/nz52fSpEl06dIFOzs7evfuDcC2bdsoUaIE5ubm5M+fn1mzZqU4zooVK3BwcODIkSMA3Lp1iyZNmmBjY0POnDnp3LkzAQEB2va1a9dm4MCBjBw5EicnJ9zc3Bg/frze90+lUrFixQq+/PJLrKysKFy4MLt27UrQ5sSJE1SuXBlzc3Pc3d0ZNWoU8fHJn4qqXbs2KpUq0c3HxweA4OBgevbsiYuLC3Z2dtStW5fr169rjx8/fjxly5ZlxYoVFChQQHtV/9OnT2nVqhU2NjbY2dnRrl07/Pz8dL6vkydPxtXVFVtbW3r27MmoUaMSlLfr1q0brVu3ZsqUKeTKlYuiRZWlntatW0fFihWxtbXFzc2Njh078vr16wR97927lyJFimBpaUmdOnW091X6+FQsCX9MgYMrlYoMkxZDtfbK/NZHTyEqWve+/INg5G9Qr6oyTSA1OrWAO15w9W7qjpckSUpEfGJCQkIEIEJCQhLti4qKEnfu3BFRUVEGiCxtPD09hY2NjRg0aJC4d++eWL9+vbCyshK///67tk2+fPmEnZ2dmDlzpnj06JF49OiRuHTpkjAyMhITJ04U9+/fF6tWrRKWlpZi1apVCY6bM2eOEEKIGTNmCGdnZ3HhwgUhhBBv3rwRLi4uYvTo0eLu3bviypUrokGDBqJOnToJYrOzsxPjx48XDx48EGvWrBEqlUocPHhQ26Zr167C09MzxfsIiDx58oiNGzeKhw8fioEDBwobGxsRGBgohBDi+fPnwsrKSvTt21fcvXtXbN++XeTIkUOMGzcu2T4DAwOFr6+v9tamTRtRtGhRERkZKYQQon79+qJFixbi33//FQ8ePBDDhg0Tzs7O2jHHjRsnrK2tRePGjcWVK1fE9evXhVqtFmXLlhU1atQQly5dEufPnxcVKlT44P17a/369cLCwkKsXLlS3L9/X0yYMEHY2dmJMmXKJHi8bGxsROfOncWtW7fErVu3hBBC/PHHH2Lv3r3Cy8tLnDt3TlStWlU0adJEe9zTp0+Fubm5GDp0qPZ5kjNnTgGIN2/eJBlPdn5dSAk99RVizFwhCjcUIm9t5Va2lRDNegvRa6wQ4xcI8ftfQuw5LsS1u0K8DhRCo1Fu3UcLUf5LIfyDUj++Wi1E9Q5CDJ2WbndJkqSPUEq52vtkIvuO5P5gqyMiRPS1a5l+U0dE6Hy/PD09RbFixYRGo9Fu+/HHH0WxYsW0P+fLl0+0bt06wXEdO3YUDRo0SLBtxIgRonjx4gmOmzNnjhg5cqRwd3fXJk1CCDFp0iTRsGHDBMc/e/ZMAOL+/fva2GrUqJGgTaVKlcSPP/6o/XnUqFGic+fOKd5HQIwZM0b7c3h4uADEvn37hBBC/PTTT6Jo0aIJHoNFixYJGxsboVarU+xbCCFmz54tHBwctHGfOnVK2NnZiejo6ATtChYsKJYtWyaEUBJZU1NT8fr1a+3+gwcPCmNjY/H06VPtttu3bwtAXLx48YNxfPHFF6Jfv34JtlWvXj1RIpszZ04RExOTYl///vuvAERYWJgQQojRo0cn+N0KoTxPZCL7aQkKFuLcVSG27hdi3lohfvxNiG+HC1GnixBFG/8/yc1bW4jCDYSo9o3y/8Nn0z724o1KIv3mw3+fJEn6ROmTyOq9RO2nKO7hQ17Uyfx1K3MfO4Z5Gd0rtFepUiXBSmRVq1Zl1qxZqNVqjP+7mqVixYoJjrl79y6tWrVKsK169erMnTs3wXGzZs0iIiKCS5cu8dlnn2nbXr9+nWPHjmFjY5MoHi8vL4oUUYpali5dOsE+d3f3BF95T5s2Taf7+G4/1tbW2NnZafu5e/cuVatWTfAYVK9enfDwcJ4/f07evMnXV9q3bx+jRo3in3/+0cZ8/fp1wsPDcXZ2TtA2KioKLy8v7c/58uXDxcVF+/Pdu3fx8PDAw8NDu6148eI4ODhw9+5dKlWqlOJ9vH//Pn37JryMvnLlyhw9ejTBtlKlSiVaNvby5cuMHz+e69ev8+bNGzT/XbL+9OlTihcvzt27d/niiy8SHFO1atUU45E+Po72UKVs0vuEgOBQeO4HL1/DSz948RoK5VWmFaTV141h1krYdhC++yrt/UmS9GmTiawOTAsXJvd/C0Jk9rjpzdraOlXH1axZkz179rB582ZGjRql3R4eHk6LFi2YMWNGomPc3d21/zc1NU2wT6VSaZMsfaRXP++6c+cO7du3Z/r06TRs2FC7PTw8HHd3d44fP57oGAcHB+3/U/uYptX740ZERNCoUSMaNWrEhg0bcHFx4enTpzRq1EheDCbpTKVSEl1H+4xZXCGHIzSuqVz01aOtMp4kSVJqyURWB0ZWVnqdGTWUCxcuJPj5/PnzFC5cWHtWNSnFihXjzJkzCbadOXOGIkWKJDiucuXK9O/fn8aNG2NiYqJdjrh8+fJs27aN/PnzY2Ji2KdTsWLF2LZtG0II7VnZM2fOYGtrS548SV9iHRAQQIsWLWjbti1DhgxJsK98+fK8evUKExMT8ufPr1ccz54949mzZ9qzsnfu3CE4OJjixYt/8PiiRYvy77//0qVLF+22f//994PH3bt3j8DAQKZPn64d99KlS4lie/8CufPnz3+wb0lKT51aQvshcP46VC1r6GgkScrOZNWCj8jTp08ZOnQo9+/fZ9OmTSxYsIBBgwaleMywYcM4cuQIkyZN4sGDB6xZs4aFCxdqE9V3VatWjb179zJhwgTtAgn9+vUjKCiIDh068O+//+Ll5cWBAwfo3r07arVa59hHjx6dIHFLjb59+/Ls2TMGDBjAvXv32LlzJ+PGjWPo0KHJLrPatm1brKysGD9+PK9evdLe1Go19evXp2rVqrRu3ZqDBw/i4+PD2bNn+fnnnxMliO+qX78+pUqVolOnTly5coWLFy/SpUsXPD09E03tSMqAAQP4448/WLNmDQ8fPmTy5MncuHEjwZSJpOTNmxczMzMWLFiAt7c3u3btYtKkSQna9OnTh4cPHzJixAju37/Pxo0bWb169QdjkqT0VKUMFPRQzspmBiFg4iKo2wV6/ASTl8Cm3XDhulKNQZYDk6TsS56R/Yh06dKFqKgoKleujLGxMYMGDdKW2EpO+fLl2bx5M7/88guTJk3C3d2diRMn0q1btyTb16hRgz179tC0aVOMjY0ZMGAAZ86c4ccff6Rhw4bExMSQL18+GjdunGzymBRfX1+ePn2qz91NJHfu3Ozdu5cRI0ZQpkwZnJyc+O677xgzZkyyx5w8eRJQ5rm+6/Hjx+TPn5+9e/fy888/0717d/z9/XFzc6NWrVrkzJkz2T5VKhU7d+5kwIAB1KpVCyMjIxo3bsyCBQt0uh+dOnXC29ub4cOHEx0dTbt27ejWrRsXL15M8TgXFxdWr17NTz/9xPz58ylfvjwzZ86kZcuW2jZ58+Zl27ZtDBkyhAULFlC5cmWmTp1Kjx49dIpNktKDSqWclZ22TEkkXZwydrypy2DlNviqEfi/UVY8e/7q/6ue2VlDAQ/47L9bwf/+LeABFmYp9y1JkmGphDDcZ9GTJ0/y22+/cfnyZXx9fdm+fTutW7fW6dgzZ87g6elJyZIluXbtms5jhoaGYm9vT0hICHZ2dgn2RUdH8/jx4wT1QCUpK2jQoAFubm6sW7cu08eWrwspIwSHQuWvYXBX6Nsx48ZZugmm/Q4TBkK3L/+/PSYWnrwE72f/v3k9g8fP4E2o0sbJHjbNhs8/S7pvSZIyRkq52vsMekY2IiKCMmXK0KNHD9q0aaPzccHBwXTp0oV69erpVWRekrKDyMhIli5dSqNGjTA2NmbTpk0cPnyYQ4cOGTo0SUo3DnbQog5s+Af6tAc9vsDR2Z97lCR2UJeESSyAuRkUya/c3vcmRFkw4pf58O1w2PrfCmmSJGU9Bp0j26RJEyZPnsyXX3754cbv6NOnDx07dpRlg6RsqUSJEtjY2CR527BhAyqVir1791KrVi0qVKjAP//8w7Zt26hfv76hQ5ekdNWppfIV/8nkp5yn2v5TMHo2dG4FQ7rpd6yjPVQqBet+BTsb6DRMKUUmSVLWk+3myK5atQpvb2/Wr1/P5MmTDR2OJOlt7969xMXFJbkvZ86cWFpacvjw4UyOSpIyX7liULygctFX7crp1++5azBwEjTzhAkDUl/iK4cjrJ8JXw1UzsxunqdskyQp68hWiezDhw8ZNWoUp06d0rnUU0xMDDExMdqfQ0NDMyo8SdLJ+xeWSdKn6u1FX2Pnga8/uLt8+JgPufkAev4MlcvA7NGQQvVBneRyhQ0z4euB0Hkk/DkH7BOv/yLpaNU2KF4Ivsj6FS2lbCLblN9Sq9V07NiRCRMmaFde0sW0adOwt7fX3t5dbUmSJEkyrNb1wdJcmc+aVo+fQ9cflVXIlk0EM9MPH6OLAnmUM7Mv/KD7KIiMSp9+PzU3H8D4hcrKbpKUXrJNIhsWFsalS5fo378/JiYmmJiYMHHiRK5fv46JiUmi5TvfGj16NCEhIdrbs2fPMjlySZIkKTk2Vkoyu2kPxOteejoRvwD4doRyEdmq6WBtmX4xglK5YM10uOsFvX9Rqh5I+pmxXPlwceGGnHMspZ9sk8ja2dlx8+ZNrl27pr316dOHokWLcu3atUTrx79lbm6OnZ1dgpskSZKUdXRqqSSizXrD75vBL1C/40PClK/9NRpY/5tSNisjlCsOf0yFi9dhwOS0Jd6fmlOXlNv0YWBuCruSPvckSXozaCIbHh6uTUpBKUJ/7do1bWH8d1d7MjIyomTJkglurq6uWFhYULJkSYOtdy9JkiSlTYlCyjzUQnlh5gqo0g66jlKSneiYlI+Niobuo+F1IKz7TZnTmpGqlYPF4+HIWRj56/8XVZCSp9HA9N+hfAlo0xDqVYOdRwwdlfSxMOjFXpcuXaJOnTran4cOHQpA165dWb16dbqs9iRJkiRlfTUqKLeQMNhzHLYegAGTwNYamtWGtg2VkljvViCIi4c+45Wv+zfNVhLhzFC/mnIh2aApYGOdtsoIn4Ldx+HWQ9gyT3mcWtWD73+BBz5J1/GVJH0YdGUvQ5Are0mSfuTrQjIUnxew7QD8fRCe+4GHu5LQtmmg/H/INCXpXT1dSYIz28Z/lFq1/b+FEd9l/vjZQWwc1OumJKx/TFG2RcdCpTbQtQ0Ml6tjS0nQZ2WvbDNHVvp4jB8/nrJlyxo6DEmSsrj8uWFYDzi1Ef6aA1XLwootUOtbqNNF+Xp63s+GSWIBOraA0d/DwvWw9E/DxJDVbdytLHoxsuf/t1mYQeNayu/v0zqVJmWEbFVHVso8sbGxmJmZJdoeFxeHqWk61bSRJEnSgZERVCmr3CYOhAOn4Z9j0K+TMu3AkPq0h7AImLYMbK2UC9ckRXgkzF+rnEUvWiDhvlb1YPM+uH4PyhYzTHzSx0Gekf2IaDQafv31VwoVKoS5uTl58+ZlyhTlu5ybN29St25dLC0tcXZ2pnfv3oSHh2uP7datG61bt2bKlCnkypWLokWL4uPjg0ql4q+//sLT0xMLCws2bNgAwIoVKyhWrBgWFhZ8/vnnLF68OEEsz58/p0OHDjg5OWFtbU3FihW5cOECq1evZsKECVy/fh2VSoVKpWL16tWZ9hhJkpS9WVoo5br+mALtmhg6GsXwHtDtS/h5LsxfJy8Ae2v5ZgiPgKHdE++rWhZcnORFX1LayTOyH5HRo0ezfPly5syZQ40aNfD19eXevXtERETQqFEjqlatyr///svr16/p2bMn/fv3T5BEHjlyBDs7Ow4dOpSg31GjRjFr1izKlSunTWZ/+eUXFi5cSLly5bh69Sq9evXC2tqarl27Eh4ejqenJ7lz52bXrl24ublx5coVNBoN33zzDbdu3WL//v3aZVjt7TOoVo4kSVImUKlgXH+wt1WK/V+7C3NGKz9/qvyD4Pe/oFubpCtJGBtDy7qw6wiM+SHtK7BJny6ZyOogKhoeGaB4QqG8ytkHXYSFhTFv3jwWLlxI165dAShYsCA1atRg+fLlREdHs3btWm2ZsoULF9KiRQtmzJhBzpw5AbC2tmbFihXaKQU+Pj4ADB48mDZt2mjHGjduHLNmzdJuK1CgAHfu3GHZsmV07dqVjRs34u/vz7///ouTk5NyXwoV0h5vY2ODiYkJbm5uqX9wJEmSshAjI+XMY9liMHgqNPselk6AkoUNHZlhzF8HJibQt2PybVrVgz+2wrlrhpvnLGV/MpHVwaOn0Pz7zB939zIopeNqvHfv3iUmJoZ69eolua9MmTIJau1Wr14djUbD/fv3tYlsqVKlkpwXW7FiRe3/IyIi8PLy4rvvvqNXr17a7fHx8dozq9euXaNcuXLaJFaSJOlTUbeK8t79wzho0x8mD077FAgh4Mg5mLsG3F2U0l+2Wbh0us8LpaLDiO+UldaSU7qockHfjsMykZVSTyayOiiUV3ljMsS4urK0TPt6jMktKvHu9rfzapcvX55oNTXj/74bSo9YJEmSsqu87rBtIYybDyN+hSu3YfxA5Wp9fZ29Cr+ugKt3oGJJ5exl2wGwcirkyaJfas38A3I4KtMKUqJSKdMLVv0Nk4ek7vGRJJnI6sDSQvczo4ZSuHBhLC0tOXLkCD179kywr1ixYqxevZqIiAhtUnrmzBmMjIwoWrSoXuPkzJmTXLly4e3tTadOnZJsU7p0aVasWEFQUFCSZ2XNzMxQq+XajpIkfbwszGDGcChfHMbOVRYEWDIBPHRMPq/egd/+gDNXlDOX636FmhXh4RPo8RO07gvLJyvL5mYlN+4rFSV+HQEW5h9u37KeMg3h2HloUivj45M+PrJqwUfCwsKCH3/8kZEjR7J27Vq8vLw4f/48f/zxB506dcLCwoKuXbty69Ytjh07xoABA+jcubN2WoE+JkyYwLRp05g/fz4PHjzg5s2brFq1itmzZwPQoUMH3NzcaN26NWfOnMHb25tt27Zx7tw5APLnz69djjggIICYmA+sQSlJkpRNfdMU/l4EwWHQrDccu5By+7te0PNnaN1PuWBq2UTYtQRqVVLOYBbJDzsXQ77c8M1g+OdoZtwL3QihlCErnA/aNtLtmML5oERhw1UvePQUhk6Dp76GGV9KO5nIfkTGjh3LsGHD+OWXXyhWrBjffPMNr1+/xsrKigMHDhAUFESlSpX46quvqFevHgsXLkzVOD179mTFihWsWrWKUqVK4enpyerVqylQQCkUaGZmxsGDB3F1daVp06aUKlWK6dOna6cetG3blsaNG1OnTh1cXFzYtGlTuj0GkiRJWU3JwrBnGVQoAd1Hw5zViUt0PX6uLMnbpBfc94E5P8H+FdC4ZuLlb50dYMMsaFob+k+CeWuyxsICpy4pUyF+7AUmelQhaFUPjp6D0PAPt01PR88rZ7a3HYRuP0JwaOaOL6UPuUTtO+RSnJKUmHxdSFL60Ghg0UalRFetSjDvJ4iKgXlrYcs+pa7qoC7wdRMw1WHinxDK1/KzVym1dWeMMNw8U41GqdRgZQFb5ydOvlPi6w9Vv4HfRsLXjTMuxreEgGV/wvTlUK8qDO4CnUcqizas/RXM5Vxdg9NniVo5R1aSJEmSMoGREQz4Fsp+rpx9bdADQsLAxgp+6gPfttRtXulbKpWS+H6WB4bNgGe+8Psk5UKrzLbrKNx5pH8SC0olhi9KK9MLMjqRjY6Bkb8pY/XrpCxmYWSkzDfuNAx+nKnUANb3PkiGI6cWSJIkSVImqlkR9vyuXAjW/1s4tRF6fq1fEvuuFnXhrznw9CW06gsPfNI13A+KjYOZK6FBNahUKnV9tKqnXNjmH5S+sb3L1x+++m+J44VjYWRPJYkFJe5Zo2D7IWXqh5R9yERWkiRJkjJZ7pzK2dNBXZQzsmlVrjjsWKz01aY/nLiY9j51teEfeOGnJIap1aQWGP+vvXuPq6rK/z/+OoAIyE0UBRUvk46ZKYmkw1dTEwz9OgxqjWWmaNqkgQpO2jhlWl4gQzLNtCxvk4ypk5qpeSnxmteisszv6GBaqUwXuaog7N8f++ep4wUFwcOB9/PxOI9He+919v4cV6c+rrPWZznBB2nlFpaNg4ch6kn46Rz8a46Z/F8pqrs5v/fVpbDyw4qJ42ak7Ye/p4CK+9wcJbIiIiJVQKMA86f90LvNRWVL11T8M3PyYPZSeCgSft+s7Pep7QNdO1RM9YLl6+GRBGjaCNbNL3m3tZEDYEBv+Fsy7DpU/rHcyMUCeDbF/MvB0rW3//mOSImsiIhIFeFVC96eBjF9YeKrkDDdrCRwqYJG9xasgLzzkDDk1u8VHW7Wzz35w63fC6DwkrkpxTPJ5tzb1Jk3nj9sscCUeOgUYu7O9n8Z5RPLzVq+Hr7PNHeIm7EAvjtze5/viJTIioiIVCHOzjApDhLHwu7PYMBYCOkLY6bBB9vMUdSyulQE6Udg3j9h8Hh4PRWG9oMG9W497ogws+rB2nKojftLlhnfO++b2wQn/hVca9zce2u4wNxJ0KA+DJkAmRU4b/e3zl+AOe+YFShmPwc+XvD3VypHabXKTFULREREqqBHo+CR3uauYpt3w9bdsGarmaiF3QM9OkHE/5SchBYVmdUI9qTD3nTY/wXk5psJ571t4OlhMKRv+cTr4Q4PdDZjjBtY9soBRzNg2LOQlw/LkuEP95T+Hl61YFGiuTHF4xNgxSwzvoq0dK2ZgCfEmM+flmDu4rZ6C/R7oGKf7cjsOiK7Y8cOoqKiaNCgARaLhTVr1pTY/r333qNHjx74+/vj7e1NWFgYmzZtuj3BioiIOBgnJ3OL26cfhw/fhl3/hGdHQFExvPCaWb/1f/8Cs5aYCW9xsZm4vr3K3GHsnj7wxxFm7duLBTBigLlY6ot1Zs3VEY+UvdrCtUSHw7FvzR3OymLTLugbC57u8P78siWxlzWoZyaz/zkFo6dW7OKrnDyYlwr9e5m7toFZ4zY6HF6YCz/+UnHPdnR2TWTz8vIIDg5m7ty5N9V+x44d9OjRgw0bNnDo0CHuv/9+oqKi+Oyzzyo4UhEREccXFABDHzTni366xvwJ+44geGuluYXunT3N3cVeehNy8mH4Q7DyVfhynfmeUY+Zi8luZsOGsrgvFGp7l37RV3GxmYz/ZaK5aOy918zPeqtaNzenGXy8F6a8fuv3u56FqyD/PIwebHt+UhxYgMlzKu7Zjs6uUwt69epFr169brr9rFmzbI6nT5/O2rVrWbduHe3atSvn6KQkQ4YM4dy5czccRRcRkcrJx9Mc8YsON2vB7v/CHJVt2xJCWttnl7AaLtC7m7nBwjNP/FrntSS5+TA20Zw+8fSwW5uWcC33d4QXx8Czr0DjQHj8ofK7N5hb4y5YCY9Fm5tD/FYdXzOZjZ9uzp2N+J/yfXZV4NBzZIuLi8nJycHPz8/eoYiIiDgs1xrQub35srfocHOR1sHD0KFtyW2//R6GPwc/ZMJbUysu0XvsT2Y1hRdfN8ucPdC5/O49f7k5beGpR699vU+EOUL97Cvmn4e3Z/k9uypw6KoFycnJ5Obm0r9//+u2uXjxItnZ2Tav6qCgoMDeIYiIiJRa6N3m/NQbTS/YeRCiRppltta8XvGjlX/7i7lxw6ip8Pk35XPPzJ9h0XvmdI/rlQazWMyFXzl5kPRm+Ty3KnHYRDY1NZUXXniBFStWUK/e9ZdcJiYm4uPjY30FBQXdxihvn27duhEXF0d8fDx169YlMjISgJSUFNq0aUOtWrUICgriqaeeIjc31/q+xYsX4+vry6ZNm2jVqhWenp707NmT06dPW9sUFRUxduxYfH19qVOnDuPHj8e4oh7IxYsXGT16NPXq1cPNzY3OnTtz4MAB6/W0tDQsFgubNm2iXbt2uLu70717dzIzM9m4cSOtWrXC29ubRx99lPz8/BI/64IFCwgKCsLDw4O+ffuSkpKCr6+v9fqQIUPo06ePzXvi4+Pp1q2b9bi4uJjExESaNWuGu7s7wcHBrFq1ynr9l19+YeDAgfj7++Pu7k6LFi1YtGgRYP4lIS4ujsDAQNzc3GjSpAmJiYkld5CIiNwUJyf4U3dYn2YmqVcyDLN+7eBn4J5WsHYetGhye+J6ZQLc1RyG/d3czexWzX0HXF3gyYdLbtewvplIL1tnVo+QXzlkIrt8+XKGDx/OihUriIiIKLHthAkTyMrKsr5OnTpVpmcWFBRQUFBgk8AVFRVRUFDApUuXyr1tWSxZsgRXV1d2797N/PnzAXBycmL27Nl89dVXLFmyhI8//pjx48fbvC8/P5/k5GT+8Y9/sGPHDk6ePMnTTz9tvT5z5kwWL17MwoUL2bVrFz///DOrV6+2ucf48eP517/+xZIlS/j0009p3rw5kZGR/PyzbQG+yZMn89prr7Fnzx5OnTpF//79mTVrFqmpqaxfv57NmzczZ871Z7Xv3r2bESNGMGbMGNLT0+nRowfTpk0r9Z9VYmIiS5cuZf78+Xz11VckJCTw2GOPsX37dgAmTpzI119/zcaNGzly5Ajz5s2jbt26AMyePZv333+fFStWcPToUZYtW0bTpk1LHYOIiFxbdDj8km2Ouv7WhYuQkAhT55nJ36Lp5lzf28WtJiyYAjVrmqWxbqUm7/dnIfUD+MsjZs3YG3nsT2bJs2eSzT8H+f+MSgIwVq9efcN2qamphpubm7FmzZoyPScrK8sAjKysrKuunT9/3vj666+N8+fPX3UtKSnJSEpKMvLy8qzndu/ebSQlJRkbNmywaTtz5kwjKSnJOHfunPXc/v37jaSkJOP999+3afvqq68aSUlJRmZmpvXcZ599VurP1bVrV6Ndu3Y3bLdy5UqjTp061uNFixYZgHHs2DHrublz5xr169e3HgcGBhozZsywHhcWFhqNGjUyoqOjDcMwjNzcXKNGjRrGsmXLrG0KCgqMBg0aWN+3bds2AzC2bt1qbZOYmGgAxvHjx63nnnzySSMyMvK68T/88MNG7969bc4NHDjQ8PHxsR7HxMRYY7tszJgxRteuXQ3DMIwLFy4YHh4exp49e2zaDBs2zBgwYIBhGIYRFRVlDB069JoxjBo1yujevbtRXFx83TirkpK+FyIiFaG42DDCYwxj9NRfz/2QaRh/fNIwfh9pGGu2Xvett8XRDMNo3dswBj9jGIWXynaPcTMMo10fw8jNv/n3/Ptbw2jewzAS3yjbMw3DMM7+ZBizFhvGsW/Lfo+KVlKudiW7jsjm5uaSnp5Oeno6ABkZGaSnp3Py5EnAHE0dPPjXWhSpqakMHjyYmTNn0rFjR86cOcOZM2fIysqyR/iVTvv2V8/S37p1K+Hh4TRs2BAvLy8GDRrETz/9ZPPzvYeHB3fccYf1ODAwkMzMTACysrI4ffo0HTt2tF53cXEhNDTUenz8+HEKCwvp1KmT9VyNGjXo0KEDR44csYmnbdtfZ+7Xr18fDw8Pfve739mcu/zsazl69CgdOnSwOXfl8Y0cO3aM/Px8evTogaenp/W1dOlSjh83ixeOHDmS5cuXc8899zB+/Hj27Nljff+QIUNIT0+nZcuWjB49ms2bN5fq+SIiUjKLBaIjYPMusyzVgS8h6kn46ZxZxzY63L7x/b4pzJsMOw/AizdXQdRGxnew6kNzgVetUmy00LwxjBkMb74LX/5f6Z6Zdx5eWQxdB0LKYnOzhz1VoHqpXasWHDx4kPvvv996PHbsWABiYmJYvHgxp0+ftia1AG+++SaXLl0iNjaW2NhY6/nL7StSQkICYCZol3Xs2JHQ0FCcrqgPEhcXd1XbkJAQgoODr2o7YsSIq9q2adOmTDHWqlXL5vjEiRP88Y9/ZOTIkUybNg0/Pz927drFsGHDKCgowMPD46pnA1gslqvmwJaX3z7LYrFc89nFxcW39AwnJ6er4i8sLLT+8+U5wuvXr6dhw4Y27WrWNCt79+rVi2+//ZYNGzawZcsWwsPDiY2NJTk5mZCQEDIyMti4cSNbt26lf//+RERE2MyxFRGRWxPdHZLfhvEvw4c7IeQueH3y9RdF3W73hZrb305IgWaNzK16b1bKIvD3M0tuldaTj5jzh8e/DO/Pu3FN30tF8O56M4nNzjUXlg2Khr/NhEHjzIVkj/QufRyVhV0T2W7dupWYMF2ZnKalpVVsQCVwdb26oJ6zszPOzs4V0rY8HDp0iOLiYmbOnGlNoFesWFGqe/j4+BAYGMi+ffvo0qULAJcuXeLQoUOEhIQAcMcdd1jn5jZpYs64Lyws5MCBA8THx5fLZ7msZcuWNovIgKuO/f39OXz4sM259PR0a9J81113UbNmTU6ePEnXrl2v+yx/f39iYmKIiYnhvvvuY9y4cSQnJwPg7e3Nww8/zMMPP8xDDz1Ez549+fnnn1UKTkSknDRuAO3ugnXbzMRrUlzFbcRQVo9GQcb35qhs40BzN64b+eY/5meaFl+2Wr01XOClcRD9lLno7XpluwwDtuyGpAXm7mT9esBfHzcXjoG5a9mk2eac24zvbr5ub2VTyf6VkPLUvHlzCgsLmTNnDlFRUTaLwEpjzJgxJCUl0aJFC+68805SUlI4d+6c9XqtWrUYOXIk48aNw8/Pj8aNGzNjxgzy8/MZNmxYOX4iGDVqFF26dCElJYWoqCg+/vhjNm7ciOU31a+7d+/Oyy+/zNKlSwkLC+Odd97h8OHD1k0zvLy8ePrpp0lISKC4uJjOnTuTlZXF7t278fb2JiYmhueff5727dvTunVrLl68yAcffECrVq0AsxJEYGAg7dq1w8nJiZUrVxIQEGBTOUFERG7dlDHmoqie99k7kuv72xNmPdu4F81pD3c1L7n9zIUQFAj9/7fsz2zbEp74M8xaDJGd4Y7Gttc/+xqmv2FuctG5vbmD290tbNvUcDFHY38XZC6eO/E9zPo7uLuVPS57cMDcW25WcHAwKSkpvPTSS9x9990sW7asTGWi/vrXvzJo0CBiYmIICwvDy8uLvn372rRJSkriwQcfZNCgQYSEhHDs2DE2bdpE7drl+xtQp06dmD9/PikpKQQHB/Phhx+SkJCAm9uv37zIyEgmTpzI+PHjuffee8nJybGZaw0wZcoUJk6cSGJiIq1ataJnz56sX7+eZs2aAeZI+YQJE2jbti1dunTB2dmZ5cuXA2YiPGPGDEJDQ7n33ns5ceIEGzZsuGraiIiI3Jo2v6/cSSyAs7OZAP4uyKxkcPbH67dNP2LuQJYQc+ujywlDIMDfnCJweUbeie/hqcnm/NecXPjHDFiWfHUSe5nFAsP/bFZi2HEA+sfD2Z9uLa7bzWJU1GTISio7OxsfHx+ysrLw9va2uXbhwgUyMjJo1qyZTWIkldsTTzzBN998w86dO+0dSpWk74WIyI2d/RH+NNKc+7piFnhcYxHXY+PgzH9h09tmAnyr9nwGA8bC+OHw35/NHdHq1oanH4e+PUr3jK+OweMTzOR24fQbjyxXpJJytStpCEkcTnJyMp9//jnHjh1jzpw5LFmyhJiYGHuHJSIi1Vj9urAwEY6fhPjpv46SXrY33ayLO3Zo+SSxAP/TDgb0hhlvwapN5r3T/gEP9Sz9M1o3NxeP1fGFB0fBR5+UT4wVTYmsOJz9+/fTo0cP2rRpw/z585k9ezbDhw+3d1giIlLNtW4Orz0PW/ZA4m+2kzUMePlt8yf+Xl3K95nPPWXOJd6xzFz45Vaz7PeqXxdWvmpWZBj+HCxcZcZemWmxlzic0lZeEBERuV3Cw2DiSHhhLjRraFY2SNsPBw/D4iTzp/vy5OkBg/uU3/083GH+C5D0pvkZ/vMdTB4FLuU0ilzelMiKiIiIlKOhD5pluZ6bZVYoSH4bQu+GbqXbv8dunJzg7yOgWRA89wqc/MEcafa+jdsB3yxNLRAREREpRxaLWfe2cygMnQCH/w3jhpX/aGxFG9AblrxklvOa8rq9o7k2jcheQzUr5CBSIn0fRERKz8UZ5j4P/cdAYD34wz32jqhsOreH1a9D7ZKLB9iNEtnfuLzzU35+Pu7updj8WKQKy8/PB67eylhERErmVQvWvVH5F0zdSPPGN25jL0pkf8PZ2RlfX18yMzMB8PDwsNkxSqQ6MQyD/Px8MjMz8fX1Lbetk0VEqpPKukiqqlAie4WAgAAAazIrUt35+vpavxciIiKViRLZK1gsFgIDA6lXrx6FhYX2DkfErmrUqKGRWBERqbSUyF6Hs7Oz/gcuIiIiUomp/JaIiIiIOCQlsiIiIiLikJTIioiIiIhDqnZzZC8Xd8/OzrZzJCIiIiJypcs52s1syFPtEtmcnBwAgoKC7ByJiIiIiFxPTk4OPj4+JbaxGNVs/8ni4mJ++OEHvLy8bttmB9nZ2QQFBXHq1Cm8vSvpHm9yy9TP1YP6uXpQP1cf6uvKxzAMcnJyaNCgAU5OJc+CrXYjsk5OTjRq1Mguz/b29taXpBpQP1cP6ufqQf1cfaivK5cbjcRepsVeIiIiIuKQlMiKiIiIiENSInsb1KxZk0mTJlGzZk17hyIVSP1cPaifqwf1c/WhvnZs1W6xl4iIiIhUDRqRFRERERGHpERWRERERBySElkRERERcUhKZCvY3Llzadq0KW5ubnTs2JH9+/fbOyS5RTt27CAqKooGDRpgsVhYs2aNzXXDMHj++ecJDAzE3d2diIgI/v3vf9snWCmTxMRE7r33Xry8vKhXrx59+vTh6NGjNm0uXLhAbGwsderUwdPTkwcffJCzZ8/aKWIpq3nz5tG2bVtrDdGwsDA2btxova5+rnqSkpKwWCzEx8dbz6mfHZcS2Qr07rvvMnbsWCZNmsSnn35KcHAwkZGRZGZm2js0uQV5eXkEBwczd+7ca16fMWMGs2fPZv78+ezbt49atWoRGRnJhQsXbnOkUlbbt28nNjaWvXv3smXLFgoLC3nggQfIy8uztklISGDdunWsXLmS7du388MPP9CvXz87Ri1l0ahRI5KSkjh06BAHDx6ke/fuREdH89VXXwHq56rmwIEDvPHGG7Rt29bmvPrZgRlSYTp06GDExsZaj4uKiowGDRoYiYmJdoxKyhNgrF692npcXFxsBAQEGC+//LL13Llz54yaNWsa//znP+0QoZSHzMxMAzC2b99uGIbZpzVq1DBWrlxpbXPkyBEDMD755BN7hSnlpHbt2sZbb72lfq5icnJyjBYtWhhbtmwxunbtaowZM8YwDH2fHZ1GZCtIQUEBhw4dIiIiwnrOycmJiIgIPvnkEztGJhUpIyODM2fO2PS7j48PHTt2VL87sKysLAD8/PwAOHToEIWFhTb9fOedd9K4cWP1swMrKipi+fLl5OXlERYWpn6uYmJjY+ndu7dNf4K+z47Oxd4BVFU//vgjRUVF1K9f3+Z8/fr1+eabb+wUlVS0M2fOAFyz3y9fE8dSXFxMfHw8nTp14u677wbMfnZ1dcXX19emrfrZMX355ZeEhYVx4cIFPD09Wb16NXfddRfp6enq5ypi+fLlfPrppxw4cOCqa/o+OzYlsiIiJYiNjeXw4cPs2rXL3qFIBWnZsiXp6elkZWWxatUqYmJi2L59u73DknJy6tQpxowZw5YtW3Bzc7N3OFLONLWggtStWxdnZ+erVj2ePXuWgIAAO0UlFe1y36rfq4a4uDg++OADtm3bRqNGjaznAwICKCgo4Ny5czbt1c+OydXVlebNm9O+fXsSExMJDg7m1VdfVT9XEYcOHSIzM5OQkBBcXFxwcXFh+/btzJ49GxcXF+rXr69+dmBKZCuIq6sr7du356OPPrKeKy4u5qOPPiIsLMyOkUlFatasGQEBATb9np2dzb59+9TvDsQwDOLi4li9ejUff/wxzZo1s7nevn17atSoYdPPR48e5eTJk+rnKqC4uJiLFy+qn6uI8PBwvvzyS9LT062v0NBQBg4caP1n9bPj0tSCCjR27FhiYmIIDQ2lQ4cOzJo1i7y8PIYOHWrv0OQW5ObmcuzYMetxRkYG6enp+Pn50bhxY+Lj45k6dSotWrSgWbNmTJw4kQYNGtCnTx/7BS2lEhsbS2pqKmvXrsXLy8s6T87Hxwd3d3d8fHwYNmwYY8eOxc/PD29vb0aNGkVYWBh/+MMf7By9lMaECRPo1asXjRs3Jicnh9TUVNLS0ti0aZP6uYrw8vKyzm+/rFatWtSpU8d6Xv3swOxdNqGqmzNnjtG4cWPD1dXV6NChg7F37157hyS3aNu2bQZw1SsmJsYwDLME18SJE4369esbNWvWNMLDw42jR4/aN2gplWv1L2AsWrTI2ub8+fPGU089ZdSuXdvw8PAw+vbta5w+fdp+QUuZPP7440aTJk0MV1dXw9/f3wgPDzc2b95sva5+rpp+W37LMNTPjsxiGIZhpxxaRERERKTMNEdWRERERBySElkRERERcUhKZEVERETEISmRFRERERGHpERWRERERBySElkRERERcUhKZEVERETEISmRFRERERGHpERWRERERBySElkREQcwZMgQ+vTpY+8wREQqFSWyIiIiIuKQlMiKiFQiq1atok2bNri7u1OnTh0iIiIYN24cS5YsYe3atVgsFiwWC2lpaQCcOnWK/v374+vri5+fH9HR0Zw4ccJ6v8sjuS+88AL+/v54e3szYsQICgoK7PMBRUTKkYu9AxAREdPp06cZMGAAM2bMoG/fvuTk5LBz504GDx7MyZMnyc7OZtGiRQD4+flRWFhIZGQkYWFh7Ny5ExcXF6ZOnUrPnj354osvcHV1BeCjjz7Czc2NtLQ0Tpw4wdChQ6lTpw7Tpk2z58cVEbllSmRFRCqJ06dPc+nSJfr160eTJk0AaNOmDQDu7u5cvHiRgIAAa/t33nmH4uJi3nrrLSwWCwCLFi3C19eXtLQ0HnjgAQBcXV1ZuHAhHh4etG7dmhdffJFx48YxZcoUnJz0w5yIOC79F0xEpJIIDg4mPDycNm3a8Oc//5kFCxbwyy+/XLf9559/zrFjx/Dy8sLT0xNPT0/8/Py4cOECx48ft7mvh4eH9TgsLIzc3FxOnTpVoZ9HRKSiaURWRKSScHZ2ZsuWLezZs4fNmzczZ84cnn32Wfbt23fN9rm5ubRv355ly5Zddc3f37+iwxURsTslsiIilYjFYqFTp0506tSJ559/niZNmrB69WpcXV0pKiqyaRsSEsK7775LvXr18Pb2vu49P//8c86fP4+7uzsAe/fuxdPTk6CgoAr9LCIiFU1TC0REKol9+/Yxffp0Dh48yMmTJ3nvvff473//S6tWrWjatClffPEFR48e5ccff6SwsJCBAwdSt25doqOj2blzJxkZGaSlpTF69Gi+++47630LCgoYNmwYX3/9NRs2bGDSpEnExcVpfqyIODyNyIqIVBLe3t7s2LGDWbNmkZ2dTZMmTZg5cya9evUiNDSUtLQ0QkNDyc3NZdu2bXTr1o0dO3bwzDPP0K9fP3JycmjYsCHh4eE2I7Th4eG0aNGCLl26cPHiRQYMGMDkyZPt90FFRMqJxTAMw95BiIhIxRgyZAjnzp1jzZo19g5FRKTc6XclEREREXFISmRFRERExCFpaoGIiIiIOCSNyIqIiIiIQ1IiKyIiIiIOSYmsiIiIiDgkJbIiIiIi4pCUyIqIiIiIQ1IiKyIiIiIOSYmsiIiIiDgkJbIiIiIi4pCUyIqIiIiIQ/p/Z9V2frF6cQEAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: broken vs correct, side by side\n",
"plt.figure(figsize=(7, 3))\n",
"plt.plot(bad_losses, color=\"#D11\", lw=1, label=\"broken: no zero_grad\")\n",
"plt.plot(good_losses[:len(bad_losses)], color=\"#1E40FF\", lw=1, label=\"correct\")\n",
"plt.axhline(math.log(10), ls=\":\", c=\"#888\", label=\"random guess\")\n",
"plt.xlabel(\"step\"); plt.ylabel(\"train loss\"); plt.title(\"the missing zero_grad(): slower, noisier, worse\")\n",
"plt.legend(); plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "a946ee9c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.202447Z",
"iopub.status.busy": "2026-06-10T20:02:35.202366Z",
"iopub.status.idle": "2026-06-10T20:02:35.243616Z",
"shell.execute_reply": "2026-06-10T20:02:35.243247Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"grad norm after 1 backward: 1.300\n",
"grad norm after 2 backwards, no zero_grad: 2.599 (~2x: gradients accumulated)\n",
"[ ok ] diagnosis confirmed: zero_grad() resets .grad to stop this accumulation\n"
]
}
],
"source": [
"# confirm the diagnosis numerically: accumulated grads are far larger than per-step grads\n",
"torch.manual_seed(SEED)\n",
"probe = ViT().to(device); probe.train()\n",
"xb, yb = next(iter(train_loader)); xb, yb = xb.to(device), yb.to(device)\n",
"# one clean backward\n",
"F.cross_entropy(probe(xb), yb).backward()\n",
"g1 = probe.head.weight.grad.norm().item()\n",
"# a second backward WITHOUT zeroing: grads accumulate\n",
"F.cross_entropy(probe(xb), yb).backward()\n",
"g2 = probe.head.weight.grad.norm().item()\n",
"print(f\"grad norm after 1 backward: {g1:.3f}\")\n",
"print(f\"grad norm after 2 backwards, no zero_grad: {g2:.3f} (~2x: gradients accumulated)\")\n",
"assert g2 > 1.5 * g1, \"without zero_grad, the second backward should roughly double the grad norm\"\n",
"print(\"[ ok ] diagnosis confirmed: zero_grad() resets .grad to stop this accumulation\")"
]
},
{
"cell_type": "markdown",
"id": "01631239",
"metadata": {},
"source": [
"> **Common confusion:** accumulation is not always a bug. It is exactly how you simulate a large batch on small memory: skip `zero_grad()` for K micro-batches, then step once. The bug is accumulating *unintentionally*. The fix is one line, `opt.zero_grad()` at the top of the loop, which is why the correct loop in 2.2 puts it first.\n"
]
},
{
"cell_type": "markdown",
"id": "6989c3d2",
"metadata": {},
"source": [
"### 2.4 Experiment log\n",
"\n",
"Every named change with its measured numbers, the Karpathy loss-log convention. It doubles as your expected-value reference: if your full-fidelity run is far from these, something is wrong. FAST numbers are noisier because they see a tenth of the data.\n",
"\n",
"| run | data | epochs | train loss | test acc | params | note |\n",
"|---|---|---|---|---|---|---|\n",
"| ViT (correct) | 6000 | 3 | ~0.7 | ~70% | 215k | the headline run |\n",
"| ViT (FAST) | 600 | 1 | ~1.7 | ~55% | 215k | CI smoke; fewer images, one pass |\n",
"| ViT (broken) | 6000 | 1 | lags the correct run | n/a | 215k | no `zero_grad`; gradients accumulate |\n",
"\n",
"> **Key takeaways.** A ViT is a patch embedder feeding a stack of bidirectional encoder blocks, classified from the `[CLS]` token. It trains with the same four-line loop as any classifier. The single most common training bug is a missing `zero_grad()`. At a moderate learning rate it does not crash; it quietly learns worse, which is harder to notice than a blowup. The unambiguous diagnostic is the gradient norm: a second backward without zeroing roughly doubles it.\n"
]
},
{
"cell_type": "markdown",
"id": "093fd3a6",
"metadata": {},
"source": [
"## Part 3 - CLIP's shared space\n",
"\n",
"> **Objectives**\n",
"> - Derive CLIP's contrastive loss as symmetric softmax cross-entropy on a similarity matrix.\n",
"> - Implement `clip_loss` in eight lines and check it against `F.cross_entropy`.\n",
"> - See why L2-normalization is mandatory, and recover the diagonal on synthetic aligned embeddings.\n",
"\n",
"CLIP projects images and captions into one shared embedding space and trains the projections to agree. The recipe: an image encoder, a text encoder, L2-normalize both outputs, and a single symmetric cross-entropy that says \"the matching image-caption pair should be the most similar pair in the batch\". No caption generation, no masked LM. We work in the toy regime here: real CLIP needs hundreds of millions of pairs and is impossible to reproduce on CPU, so we use synthetic embeddings with known structure and assert the loss behaves.\n"
]
},
{
"cell_type": "markdown",
"id": "550b1867",
"metadata": {},
"source": [
"### 3.1 The similarity matrix and the contrastive loss\n",
"\n",
"Given $N$ image-caption pairs, encode them to L2-normalized vectors $\\mathbf{I} \\in \\mathbb{R}^{N \\times d}$ and $\\mathbf{T} \\in \\mathbb{R}^{N \\times d}$. Form the similarity matrix scaled by a learned temperature:\n",
"\n",
"$$ S = s \\cdot \\mathbf{I}\\,\\mathbf{T}^{\\top} \\in \\mathbb{R}^{N \\times N}, \\quad s = \\exp(\\text{logit\\_scale}). $$\n",
"\n",
"Entry $S_{ij}$ is the cosine similarity of image $i$ with caption $j$, times $s$. The correct match for row $i$ is column $i$, the diagonal. The loss is symmetric cross-entropy: treat each row as a classification over captions (label $= i$), each column as a classification over images (label $= i$), average the two.\n",
"\n",
"$$ \\mathcal{L} = \\tfrac{1}{2}\\big(\\text{CE}(S, \\text{arange}(N)) + \\text{CE}(S^{\\top}, \\text{arange}(N))\\big). $$\n",
"\n",
"This is exactly InfoNCE: the numerator is the diagonal entry, the denominator is the row (or column) sum over all the in-batch negatives. Bigger batch means more negatives means a sharper gradient, which is why real CLIP trains at batch size 32,768.\n"
]
},
{
"cell_type": "markdown",
"id": "928847fc",
"metadata": {},
"source": [
"### Exercise 16.4 - The CLIP loss in eight lines\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"Implement `clip_loss(image_features, text_features, logit_scale)`. Inputs are already L2-normalized `(N, d)` tensors and a scalar `logit_scale`. Build the scaled similarity matrix, form the `arange(N)` labels (the diagonal is the correct match), and return the average of the image-to-text and text-to-image cross-entropies. Isolate this before the encoder so the hard sub-skill is tested alone (spec §3.4).\n",
"\n",
"Harder: after it passes, set `logit_scale = log(100)` and observe the loss drop toward zero on perfectly aligned features. That clamp at `log(100)` is exactly what CLIP does to stop the temperature collapsing.\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "8df3297f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.244422Z",
"iopub.status.busy": "2026-06-10T20:02:35.244346Z",
"iopub.status.idle": "2026-06-10T20:02:35.248573Z",
"shell.execute_reply": "2026-06-10T20:02:35.248310Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 16.4 clip_loss: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def clip_loss(image_features, text_features, logit_scale):\n",
" \"\"\"image_features, text_features: (N, d), L2-normalized. logit_scale: scalar tensor.\n",
" Returns the symmetric contrastive loss (scalar).\"\"\"\n",
" N = image_features.shape[0]\n",
" # TODO 1: logits = exp(logit_scale) * image_features @ text_features.T -> (N, N)\n",
" logits = None\n",
" # TODO 2: labels = torch.arange(N) on the same device as logits (the diagonal is correct)\n",
" labels = None\n",
" # TODO 3: image->text loss = F.cross_entropy(logits, labels)\n",
" # TODO 4: text->image loss = F.cross_entropy(logits.T, labels)\n",
" # TODO 5: return the average of the two\n",
" loss = None\n",
" attempted(logits, labels, loss)\n",
" return loss\n",
"\n",
"def _clip_loss_checks():\n",
" torch.manual_seed(SEED)\n",
" N, d = 8, 16\n",
" I = F.normalize(torch.randn(N, d), dim=-1)\n",
" T = F.normalize(torch.randn(N, d), dim=-1)\n",
" ls = torch.tensor(0.0) # exp(0) = 1, so logits are raw cosine sims\n",
" # random normalized features: loss should sit near log(N) = log(8) = 2.079\n",
" L = clip_loss(I, T, ls)\n",
" assert abs(L.item() - math.log(N)) < 0.6, \\\n",
" f\"random-feature loss {L.item():.3f} should be near log(8)={math.log(N):.3f}\"\n",
" # perfectly aligned features (T = I) with a hot temperature -> loss near 0\n",
" hot = torch.tensor(math.log(100.0))\n",
" L_aligned = clip_loss(I, I.clone(), hot)\n",
" assert L_aligned.item() < 0.05, \\\n",
" f\"aligned features at high temperature should give ~0 loss, got {L_aligned.item():.3f}\"\n",
"\n",
"check(\"16.4 clip_loss\", _clip_loss_checks)"
]
},
{
"cell_type": "markdown",
"id": "edd78293",
"metadata": {},
"source": [
"Hint 1 (conceptual)`F.cross_entropy(logits, labels)` already does softmax + negative-log-likelihood. You need it once on `logits` (image-to-text) and once on `logits.T` (text-to-image), with the same `arange(N)` labels because the correct match is always the diagonal.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"logits = logit_scale.exp() * image_features @ text_features.T # (N, N)\n",
"labels = torch.arange(N, device=logits.device)\n",
"loss = 0.5 * (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels))\n",
"```\n",
"\n",
"\n",
"Help: \"loss is near log(8) and will not drop on aligned features\"Check that you scaled by `logit_scale.exp()`, not by `logit_scale` itself. At `logit_scale=0` the scale is `exp(0)=1`; the aligned-features test uses `log(100)` so the scale is 100, which sharpens the softmax onto the diagonal.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "d784eca6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.249535Z",
"iopub.status.busy": "2026-06-10T20:02:35.249414Z",
"iopub.status.idle": "2026-06-10T20:02:35.253983Z",
"shell.execute_reply": "2026-06-10T20:02:35.253726Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 16.4 clip_loss\n",
"aligned-feature loss at logit_scale=log(100): 0.0000\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines clip_loss; the checks below re-verify the reference.\n",
"def clip_loss(image_features, text_features, logit_scale):\n",
" N = image_features.shape[0]\n",
" logits = logit_scale.exp() * image_features @ text_features.T # (N, N)\n",
" labels = torch.arange(N, device=logits.device)\n",
" loss_i2t = F.cross_entropy(logits, labels)\n",
" loss_t2i = F.cross_entropy(logits.T, labels)\n",
" return 0.5 * (loss_i2t + loss_t2i)\n",
"\n",
"check(\"16.4 clip_loss\", _clip_loss_checks, required=True)\n",
"# the Harder check: aligned features, temperature clamped at log(100)\n",
"torch.manual_seed(SEED)\n",
"I = F.normalize(torch.randn(8, 16), dim=-1)\n",
"print(f\"aligned-feature loss at logit_scale=log(100): \"\n",
" f\"{clip_loss(I, I.clone(), torch.tensor(math.log(100.0))).item():.4f}\")"
]
},
{
"cell_type": "markdown",
"id": "dd823918",
"metadata": {},
"source": [
"### 3.2 Why L2-normalization is not optional\n",
"\n",
"Cosine similarity is a direction-only measure. If you skip normalization, the dot product is dominated by vector *magnitude*, not direction, and the loss measures the wrong thing. The single most common reason a contrastive model refuses to learn is forgetting to normalize. We verify the property directly: two vectors pointing the same way have cosine 1 regardless of length.\n"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "b960bc5c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.254859Z",
"iopub.status.busy": "2026-06-10T20:02:35.254785Z",
"iopub.status.idle": "2026-06-10T20:02:35.257443Z",
"shell.execute_reply": "2026-06-10T20:02:35.257163Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"raw dot 50.0 . cosine 1.0000\n",
"[ ok ] this is why CLIP normalizes both encoders' outputs before the similarity matrix\n"
]
}
],
"source": [
"# property check: cosine is direction-only, the raw dot product is not\n",
"v1 = torch.tensor([3.0, 4.0])\n",
"v2 = torch.tensor([6.0, 8.0]) # exactly 2 * v1, same direction\n",
"raw_dot = (v1 * v2).sum() # 18 + 32 = 50, grows with magnitude\n",
"cosine = F.normalize(v1, dim=0) @ F.normalize(v2, dim=0) # 1.0, magnitude divided out\n",
"print(f\"raw dot {raw_dot.item():.1f} . cosine {cosine.item():.4f}\")\n",
"assert raw_dot.item() == 50.0, \"raw dot product scales with magnitude\"\n",
"check_close(cosine, 1.0, msg=\"same-direction vectors have cosine 1 after normalization\")\n",
"print(\"[ ok ] this is why CLIP normalizes both encoders' outputs before the similarity matrix\")"
]
},
{
"cell_type": "markdown",
"id": "b51b2971",
"metadata": {},
"source": [
"### Exercise 16.5 - Recover the diagonal (ground-truth synthesis)\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"This is a ground-truth-recovery check (spec §3.3 tier 4). Build a batch where image $i$ and caption $i$ share a known latent and differ only by small noise, so the *correct* answer for `clip_loss`'s row-argmax is the identity. Write `aligned_batch(N, d, noise, rng_seed)` returning normalized `(I, T)` such that `I[i]` and `T[i]` are near-copies of a shared random latent. The check asserts that the similarity matrix's row-argmax is exactly `arange(N)`: each image's most-similar caption is its own.\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "11f3c352",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.258056Z",
"iopub.status.busy": "2026-06-10T20:02:35.257991Z",
"iopub.status.idle": "2026-06-10T20:02:35.261748Z",
"shell.execute_reply": "2026-06-10T20:02:35.261406Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 16.5 diagonal recovery: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def aligned_batch(N, d, noise=0.05, rng_seed=SEED):\n",
" \"\"\"Return (I, T), both (N, d) L2-normalized, where I[i] and T[i] share a latent.\n",
" Small noise keeps them near each other but not identical.\"\"\"\n",
" g = torch.Generator().manual_seed(rng_seed)\n",
" latent = torch.randn(N, d, generator=g) # the shared concept per pair\n",
" # TODO 1: image features = latent + noise * randn, then L2-normalize (dim=-1)\n",
" I = None\n",
" # TODO 2: text features = latent + noise * randn (fresh noise), then L2-normalize\n",
" T = None\n",
" attempted(I, T)\n",
" return I, T\n",
"\n",
"def _recovery_check():\n",
" I, T = aligned_batch(N=12, d=32, noise=0.05, rng_seed=SEED)\n",
" # both must be unit-norm\n",
" assert torch.allclose(I.norm(dim=-1), torch.ones(12), atol=1e-5), \"I must be L2-normalized\"\n",
" assert torch.allclose(T.norm(dim=-1), torch.ones(12), atol=1e-5), \"T must be L2-normalized\"\n",
" sims = I @ T.T # (12, 12)\n",
" recovered = sims.argmax(dim=-1)\n",
" assert torch.equal(recovered, torch.arange(12)), \\\n",
" f\"each image's nearest caption should be its own pair; got argmax {recovered.tolist()}\"\n",
"\n",
"check(\"16.5 diagonal recovery\", _recovery_check)"
]
},
{
"cell_type": "markdown",
"id": "b80c9593",
"metadata": {},
"source": [
"Hint 1 (conceptual)Add independent small Gaussian noise to two copies of the same latent, then normalize each. Because the noise is small relative to the latent, `I[i]` stays closest to `T[i]`.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"I = F.normalize(latent + noise * torch.randn(N, d, generator=g), dim=-1)\n",
"T = F.normalize(latent + noise * torch.randn(N, d, generator=g), dim=-1)\n",
"```\n",
"Reuse the same generator `g` so both noise draws are seeded.\n",
"\n",
"Help: \"argmax is not arange; some images match the wrong caption\"Your `noise` is too large relative to the latent scale (the latent has unit-ish entries). Keep `noise` around 0.05; at `noise=1.0` the shared latent is swamped and pairs stop being recoverable.\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "98742434",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.262733Z",
"iopub.status.busy": "2026-06-10T20:02:35.262670Z",
"iopub.status.idle": "2026-06-10T20:02:35.265553Z",
"shell.execute_reply": "2026-06-10T20:02:35.265269Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 16.5 diagonal recovery\n",
"aligned batch contrastive loss: 0.0000 (low: pairs are recoverable)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines aligned_batch; the checks below re-verify the reference.\n",
"def aligned_batch(N, d, noise=0.05, rng_seed=SEED):\n",
" g = torch.Generator().manual_seed(rng_seed)\n",
" latent = torch.randn(N, d, generator=g)\n",
" I = F.normalize(latent + noise * torch.randn(N, d, generator=g), dim=-1)\n",
" T = F.normalize(latent + noise * torch.randn(N, d, generator=g), dim=-1)\n",
" return I, T\n",
"\n",
"check(\"16.5 diagonal recovery\", _recovery_check, required=True)\n",
"I, T = aligned_batch(N=12, d=32, noise=0.05, rng_seed=SEED)\n",
"print(f\"aligned batch contrastive loss: \"\n",
" f\"{clip_loss(I, T, torch.tensor(math.log(100.0))).item():.4f} (low: pairs are recoverable)\")"
]
},
{
"cell_type": "markdown",
"id": "345a4dc1",
"metadata": {},
"source": [
"### 3.3 Temperature shapes the softmax\n",
"\n",
"The learned temperature $s = \\exp(\\text{logit\\_scale})$ controls how peaky the per-row softmax is. Low scale (high temperature) spreads probability across all captions; high scale (low temperature) concentrates it on the diagonal. We sweep it on the aligned batch and watch the loss.\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "a7fdb3d8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.266305Z",
"iopub.status.busy": "2026-06-10T20:02:35.266239Z",
"iopub.status.idle": "2026-06-10T20:02:35.424913Z",
"shell.execute_reply": "2026-06-10T20:02:35.424483Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAEiCAYAAAAF9zFeAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAWbBJREFUeJzt3Xd0FFUbx/HvJpBOGiW0EHqXKlUgICAigoBUC0WKBUWKNAtFwAiKoFhAUFARQaqFIkhvKl0BBYFQBEIJJRAgSHLfP/bNypJCNiTZJPw+5+whe+fOzDNlZx/u3rljMcYYRERERESyGBdnByAiIiIikhpKZEVEREQkS1IiKyIiIiJZkhJZEREREcmSlMiKiIiISJakRFZEREREsiQlsiIiIiKSJSmRFREREZEsSYmsiIiIiGRJSmSzkYYNG1KxYkVnh5GpjBw5EovFkqbL7NatG0WLFrUrs1gsjBw5Mk3XM3PmTCwWC0eOHLlj3bVr12KxWFi7dm2axpAS48ePp2zZssTFxQFw5MgRLBYL77777h3nvZvjEz/vuXPnUjV/VnQ3+6tTp0506NAhjSO6Ow0bNqRhw4bODiNNpMc1ILtJ7PwtWrQo3bp1c05AKRR/TZs5c2aGrzux7xuxp0RWJAv5+OOPnXIxTUpUVBTjxo1jyJAhuLjocpKZDRkyhAULFrB7925nhyIikmb0zSPZ2uuvv861a9fSdJnTpk1j//79abrMxDz99NNcu3aNkJAQW1lSiWyDBg24du0aDRo0SPe4bvX5559z8+ZNOnfunKr50+P4SOKqVq3K/fffz4QJE5wdiojN/v37mTZtmrPDyLQy6vsmK1MiKw65efMmN27cyLD1RUdH39X8OXLkwMPDI42iscqZMyfu7u5pusxbxW+zq6srHh4eKfop2cXFBQ8PjwxvFZ0xYwatWrVK9T5Oj+OT1q5eversENJMhw4dWLhwIVeuXEn1Mv744480jCjruH79uq37jKQdd3d3cubM6ewwMq2UfN9k9PdyZqNENou4fPky/fr1o2jRori7u5MvXz6aNm3Kjh07EtTdt28fjRo1wsvLi0KFCjF+/Hi76Tdu3GD48OFUr14dPz8/vL29qV+/PmvWrLGrd2tfx0mTJlGiRAnc3d3Zt2+frU/m3LlzefXVV8mfPz/e3t60atWK48ePJ4jp119/5eGHH8bPzw8vLy9CQ0PZtGmTXZ34/lP79u3jiSeeICAggHr16iW5T/79919GjRpFqVKl8PDwIHfu3NSrV4+VK1cmWOatLBYLL774IvPmzaN8+fJ4enpSp04d2xf01KlTKVmyJB4eHjRs2DBBH9WU9Fk6evQoL7zwAmXKlMHT05PcuXPTvn37BMuK7we7bt06XnjhBfLly0fhwoXtpsXPU7RoUfbu3cu6deuwWCxYLBZb/8Kk+simZL87cm7dKjw8nN9//50mTZokWefTTz+1nTc1atRg69atdtMTOz7Xrl2jb9++5MmTh1y5ctGqVStOnDiRZB/Eixcv0q1bN/z9/fHz86N79+6JJp+zZs2ievXqeHp6EhgYSKdOnRKcq/H9zLdv306DBg3w8vLi1VdfTXL7IiIi6N69O4ULF8bd3Z0CBQrw2GOPJTjOy5YtIzQ0lFy5cuHr60uNGjWYPXu2bfqGDRto3749RYoUwd3dneDgYPr375/i1uqUbBtA06ZNiY6OtvuMOKply5aUL1+eCRMmcObMmVQvJylnzpyhR48eBAUF4eHhQeXKlfniiy/s6lSrVo22bdvald13331YLBZ+//13W9ncuXOxWCz8+eeftrITJ07wzDPPEBQUhLu7OxUqVODzzz+3W1b852nOnDm8/vrrFCpUCC8vL6Kiohzalp07d9K8eXN8fX3x8fGhcePG/PLLL3Z1UnIdS+l5lpjVq1dTv359vL298ff357HHHrPbH/Df5/DgwYMp+izd7m7O38T6yP7++++Ehobi6elJ4cKFGTNmDDNmzEhwz0DRokV59NFH2bhxIzVr1sTDw4PixYvz5ZdfJljPxYsX6devH8HBwbi7u1OyZEnGjRuX4D8n8dcTPz8//P396dq1KxcvXrzjdsB/1+z169fz7LPPkjt3bnx9fenSpQsXLlywq/vdd9/RokULChYsiLu7OyVKlGD06NHExsba1bv9+ya572WAyZMnU6FCBby8vAgICOD++++3u9ZkRzmcHYCkzHPPPcf8+fN58cUXKV++PJGRkWzcuJE///yTatWq2epduHCBhx9+mLZt29KhQwfmz5/PkCFDuO+++2jevDlg7dc4ffp0OnfuTK9evbh8+TKfffYZzZo147fffqNKlSp2654xYwbXr1+nd+/euLu7ExgYaPtgjx07FovFwpAhQzhz5gyTJk2iSZMm7Nq1C09PT8B6IW3evDnVq1dnxIgRuLi4MGPGDB588EE2bNhAzZo17dbXvn17SpUqxVtvvYUxJsl9MnLkSMLCwujZsyc1a9YkKiqKbdu2sWPHDpo2bZrs/tywYQPff/89ffr0ASAsLIxHH32UwYMH8/HHH/PCCy9w4cIFxo8fzzPPPMPq1atTdJzibd26lc2bN9OpUycKFy7MkSNH+OSTT2jYsCH79u3Dy8vLrv4LL7xA3rx5GT58eJKt0JMmTeKll17Cx8eH1157DYCgoKAkY0jpfk/puXW7zZs3AyRZZ/bs2Vy+fJlnn30Wi8XC+PHjadu2LYcPH062BaZbt258++23PP3009SuXZt169bRokWLJOt36NCBYsWKERYWxo4dO5g+fTr58uVj3Lhxtjpjx47ljTfeoEOHDvTs2ZOzZ88yefJkGjRowM6dO/H397fVjYyMpHnz5nTq1Imnnnoq2X38+OOPs3fvXl566SWKFi3KmTNnWLlyJceOHbN9+cycOZNnnnmGChUqMGzYMPz9/dm5cyfLly/niSeeAGDevHlcvXqV559/nty5c/Pbb78xefJk/vnnH+bNm5fk+h3dtvj/uG3atIk2bdoku9ykTJgwgSlTpjB48GCGDRtGy5Yt6dmzJw899BCurq6pWma8a9eu0bBhQw4ePMiLL75IsWLFmDdvHt26dePixYu8/PLLANSvX59vvvnGNt/58+fZu3cvLi4ubNiwgUqVKgHWz3nevHkpV64cAKdPn6Z27dq2/8zmzZuXZcuW0aNHD6KioujXr59dPKNHj8bNzY1XXnmFmJgY3NzcUrwte/fupX79+vj6+jJ48GBy5szJ1KlTadiwIevWraNWrVpAyq5jKTnPEvPzzz/TvHlzihcvzsiRI7l27RqTJ0/mgQceYMeOHQnmTclnKTF3c/7e7sSJEzRq1AiLxcKwYcPw9vZm+vTpSbZKHjx4kHbt2tGjRw+6du3K559/Trdu3ahevToVKlQArL+qhIaGcuLECZ599lmKFCnC5s2bGTZsGKdOnWLSpEkAGGN47LHH2LhxI8899xzlypVj0aJFdO3a1aFtePHFF/H392fkyJHs37+fTz75hKNHj9r+gwTW64KPjw8DBgzAx8eH1atXM3z4cKKionjnnXfuuI7EvpenTZtG3759adeuHS+//DLXr1/n999/59dff7Vda7IlI1mCn5+f6dOnT7J1QkNDDWC+/PJLW1lMTIzJnz+/efzxx21lN2/eNDExMXbzXrhwwQQFBZlnnnnGVhYeHm4A4+vra86cOWNXf82aNQYwhQoVMlFRUbbyb7/91gDm/fffN8YYExcXZ0qVKmWaNWtm4uLibPWuXr1qihUrZpo2bWorGzFihAFM586dU7JLTOXKlU2LFi2SrRO/zFsBxt3d3YSHh9vKpk6dagCTP39+u+0ZNmyYAezqdu3a1YSEhCRY5ogRI+y273ZbtmxJcHxmzJhhAFOvXj1z8+ZNu/rx025dd4UKFUxoaGiCZccfjzVr1hhjHNvvKTm3EvP6668bwFy+fNmuPP68yZ07tzl//ryt/LvvvjOA+eGHH2xltx+f7du3G8D069fPbpndunVLsI/j5731nDXGmDZt2pjcuXPb3h85csS4urqasWPH2tX7448/TI4cOezK4z9DU6ZMueP2X7hwwQDmnXfeSbLOxYsXTa5cuUytWrXMtWvX7KbdflxuFxYWZiwWizl69Kit7Pb95ci2xStdurRp3rz5HbfvTo4ePWpGjRplihUrZgBTuHBh8/rrr5vDhw+neBmhoaF25/OkSZMMYGbNmmUru3HjhqlTp47x8fGxfTbnzZtnALNv3z5jjDHff/+9cXd3N61atTIdO3a0zVupUiXTpk0b2/sePXqYAgUKmHPnztnF0alTJ+Pn52c7DvGfp+LFiyd6bBJz+/nZunVr4+bmZg4dOmQrO3nypMmVK5dp0KCBrexO17GUnGdJqVKlismXL5+JjIy0le3evdu4uLiYLl262MpS+llKSmrPX2OMCQkJMV27drW9f+mll4zFYjE7d+60lUVGRprAwMAE18OQkBADmPXr19vKzpw5Y9zd3c3AgQNtZaNHjzbe3t7mwIEDduseOnSocXV1NceOHTPGGLN48WIDmPHjx9vq3Lx509SvX98AZsaMGcnuh/hrdvXq1c2NGzds5ePHjzeA+e6772xlie2zZ5991nh5eZnr16/bym7/vknue/mxxx4zFSpUSDbG7EhdC7IIf39/fv31V06ePJlsPR8fH5566inbezc3N2rWrMnhw4dtZa6urraWhbi4OM6fP8/Nmze5//77E/05+fHHHydv3ryJrq9Lly7kypXL9r5du3YUKFCApUuXArBr1y7+/vtvnnjiCSIjIzl37hznzp0jOjqaxo0bs379+gQ/7Tz33HN32BtW/v7+7N27l7///jtF9W/VuHFju9aI+NaRxx9/3G574stv3X8pEd8aDdafDiMjIylZsiT+/v6J7uNevXrddWvWrRzZ7yk9t24XGRlJjhw58PHxSXR6x44dCQgIsL2vX78+kPy+XL58OWBtob7VSy+9lOQ8t58v9evXJzIy0vYz8MKFC4mLi6NDhw62/XDu3Dny589PqVKlEnSpcXd3p3v37kmuL56npydubm6sXbs2wc+G8VauXMnly5cZOnRogr7At3apuPV8iY6O5ty5c9StWxdjDDt37kwyBke3DSAgICBNhiwrUqQIw4cP59ChQ6xatYrQ0FAmTJhAiRIlaNKkCevXr3d4mUuXLiV//vx2Nw/mzJmTvn37cuXKFdatWwf8dy7Fr2PDhg3UqFGDpk2bsmHDBsD6E/GePXtsdY0xLFiwgJYtW2KMsdtfzZo149KlSwk+m127drU7NikVGxvLihUraN26NcWLF7eVFyhQgCeeeIKNGzfazs87XcdScp4l5tSpU+zatYtu3boRGBhoK69UqRJNmza1XaNvdafPUlJSe/4mZvny5dSpU8ful8HAwECefPLJROuXL1/edowB8ubNS5kyZeyuM/PmzaN+/fq2cz/+1aRJE2JjY23n0dKlS8mRIwfPP/+8bV5XV9dkrz+J6d27t92vTs8//zw5cuSw2+e37rPLly9z7tw56tevz9WrV/nrr7/uuI7Evpf9/f35559/EnThyu6UyGYR48ePZ8+ePQQHB1OzZk1GjhyZaEJQuHDhBH0OAwICElwAv/jiCypVqmTrk5U3b16WLFnCpUuXEiyzWLFiScZVqlQpu/cWi4WSJUva+jHFX5y7du1K3rx57V7Tp08nJiYmwTqTW9+t3nzzTS5evEjp0qW57777GDRokF3/uOQUKVLE7r2fnx8AwcHBiZY78gUC1p9Ihw8fbuuPlSdPHvLmzcvFixcd3sep4ch+T+m55ajb93F8Upvcvjx69CguLi4J9kfJkiVTvZ6///4bYwylSpVKsC/+/PPPBP08CxUqlKKfkN3d3Rk3bhzLli0jKCiIBg0aMH78eCIiImx1Dh06BHDH8Z2PHTtmSzh8fHzImzcvoaGhAImeL/Ec3TawJnR3uoHw/PnzRERE2F7JxWCxWHjwwQeZNWsWP/zwAwUKFGDVqlUsXLgw2XUk5ujRo5QqVSrBTYvxXQOOHj0KWLvUlCpVypa0btiwgfr169OgQQNOnjzJ4cOH2bRpE3FxcbYk5+zZs1y8eJFPP/00wb6K/4/L7fsrtZ/Ls2fPcvXqVcqUKZNgWrly5YiLi7P1Yb7TdSwl51li4vdVUjHE/8f2Vqn5zELqz9+k4k7s857UNeD2mOPjvjXmv//+m+XLlyc47vH9++OP+9GjRylQoECC/5wntg+Tc/v3oo+PDwUKFLDr37t3717atGmDn58fvr6+5M2b19YIlZJ9lti5OWTIEHx8fKhZsyalSpWiT58+Ce6JyI7URzaL6NChA/Xr12fRokWsWLGCd955h3HjxrFw4UJb31cgyVY9c0tf01mzZtGtWzdat27NoEGDyJcvH66uroSFhdm+eG+VmhaJePGtfu+8806Cvrfxbr9opHR9DRo04NChQ3z33XesWLGC6dOnM3HiRKZMmULPnj2TnTep/ZSS/ZcSL730EjNmzKBfv37UqVMHPz8/LBYLnTp1SvTO57vZx4lxZL+n9Ny6Xe7cubl58yaXL1+2a8WOl1b78k7utJ64uDgsFgvLli1LtG5qzz+Afv360bJlSxYvXsxPP/3EG2+8QVhYGKtXr6Zq1aopWkZsbCxNmzbl/PnzDBkyhLJly+Lt7c2JEyfo1q1bsnfKO7ptYE1Kbv+ivV3btm1trZ9g/Q9RUuMXnzlzhlmzZjFjxgz27NlDUFAQgwYNsmvVSg/16tVj1apVXLt2je3btzN8+HAqVqyIv78/GzZs4M8//8THx8d2HOL341NPPZVkn8f4vrXx0vpzmZiUXMfS4jxLidR8Zu/m/E0LKYk5Li6Opk2bMnjw4ETrli5dOl1iS8rFixcJDQ3F19eXN998kxIlSuDh4cGOHTsYMmRIivZZYudmuXLl2L9/Pz/++CPLly9nwYIFfPzxxwwfPpxRo0alx6ZkCkpks5ACBQrwwgsv8MILL3DmzBmqVavG2LFjk002EjN//nyKFy/OwoUL7VpmRowY4XBMt/8cZozh4MGDti+EEiVKAODr65vs3e2pFRgYSPfu3enevTtXrlyhQYMGjBw58o6JbHqbP38+Xbt2tRuz8/r16ym++zUpKX2qk6P7PTXnVtmyZQHr6AW3JwCpFRISQlxcHOHh4XbJ1sGDB1O9zBIlSmCMoVixYunyhVWiRAkGDhzIwIED+fvvv6lSpQoTJkxg1qxZtuOwZ8+eJFuU/vjjDw4cOMAXX3xBly5dbOUpGVnA0W27efMmx48fp1WrVsnWmzBhgl2LVsGCBRMsZ+nSpcyYMYMlS5YQFxdHs2bNePPNN3n00UdTPZxSSEgIv//+O3FxcXatsvE/td46pnL9+vWZMWMGc+bMITY2lrp16+Li4kK9evVsiWzdunVtiU7evHnJlSsXsbGx6XItulXevHnx8vJKdPzPv/76CxcXF7tff1JyHUvuPEtM/L5KKoY8efLg7e19t5t6V+dvYkJCQhL9vN/tNeDKlSt3PO4hISGsWrWKK1eu2P0n0NFxXP/++28aNWpke3/lyhVOnTrFI488AlhHxYiMjGThwoV2Y3+Hh4c7tJ7EeHt707FjRzp27MiNGzdo27YtY8eOZdiwYZl+qMPUUteCLCA2NjbBTw358uWjYMGCxMTEOLy8+Av7rf9j/fXXX9myZYvDy/ryyy+5fPmy7f38+fM5deqULQGqXr06JUqU4N1330107MqzZ886vM54kZGRdu99fHwoWbJkqvZJWnN1dU3QijF58uQEQ6s4ytvbO0XJcEr3+92cW3Xq1AFg27ZtKYz+zpo1awZYH/xwq8mTJ6d6mW3btsXV1ZVRo0YlOCbGmATnUUpdvXqV69ev25WVKFGCXLly2fbdQw89RK5cuQgLC0tQNz6WxD6Pxhjef//9O8bg6Lbt27eP69evU7du3WSXW716dZo0aWJ7lS9f3jZt5MiRFC5cmMcee4zdu3czfPhwjh49ypIlS2jTps1djQn6yCOPEBERwdy5c21lN2/eZPLkyfj4+Nh+rob/+smOGzeOSpUq2boB1a9fn1WrVrFt2za7vpOurq48/vjjLFiwgD179iRY991ci27n6urKQw89xHfffWf3c/Lp06eZPXs29erVw9fXF7jzdSwl51liChQoQJUqVfjiiy/srhl79uxhxYoVtqTqbt3N+ZuYZs2asWXLFnbt2mUrO3/+PF9//XWqY+zQoQNbtmzhp59+SjDt4sWL3Lx5E7Cefzdv3uSTTz6xTY+NjXX4+vPpp5/y77//2t5/8skn3Lx50/a9mNg+u3HjRoLrnqNuP5fc3NwoX748xhi7eLIbtchmAZcvX6Zw4cK0a9eOypUr4+Pjw88//8zWrVtT9ZSeRx99lIULF9KmTRtatGhBeHg4U6ZMoXz58g4PlB4YGEi9evXo3r07p0+fZtKkSZQsWZJevXoB1oH6p0+fTvPmzalQoQLdu3enUKFCnDhxgjVr1uDr68sPP/zg8DaAtZN/w4YNqV69OoGBgWzbts02jJSzPfroo3z11Vf4+flRvnx5tmzZws8//0zu3LnvarnVq1fnk08+YcyYMZQsWZJ8+fLx4IMPJqiX0v1+N+dW8eLFqVixIj///DPPPPPMXW3Xrdv3+OOPM2nSJCIjI23Dbx04cABIeYv0rUqUKMGYMWMYNmwYR44coXXr1uTKlYvw8HAWLVpE7969eeWVVxxe7oEDB2jcuDEdOnSgfPny5MiRg0WLFnH69Gk6deoEWFvEJ06cSM+ePalRo4ZtfOTdu3dz9epVvvjiC8qWLUuJEiV45ZVXOHHiBL6+vixYsCBF/bId3baVK1fi5eV1x+HpkjNnzhwaNWpEjx49aNy4caqOSVJ69+7N1KlT6datG9u3b6do0aLMnz+fTZs2MWnSJLsuLCVLliR//vzs37/f7macBg0aMGTIEAC7RBbg7bffZs2aNdSqVYtevXpRvnx5zp8/z44dO/j55585f/58mm3LmDFjWLlyJfXq1eOFF14gR44cTJ06lZiYGLuxve90HUvJeZaUd955h+bNm1OnTh169OhhG37Lz88v0TGZU+Nuzt/EDB48mFmzZtG0aVNeeukl2/BbRYoU4fz586k63wYNGsT333/Po48+ahuaKzo6mj/++IP58+dz5MgR8uTJQ8uWLXnggQcYOnQoR44coXz58ixcuNDhfr43btywHbP9+/fz8ccfU69ePdsvIXXr1iUgIICuXbvSt29fLBYLX3311V13u3rooYfInz8/DzzwAEFBQfz55598+OGHtGjRItHuX9lGRgyNIHcnJibGDBo0yFSuXNnkypXLeHt7m8qVK5uPP/7Yrl5oaGiiQ2/cPnxHXFyceeutt0xISIhxd3c3VatWNT/++GOSw3wkNuxL/PA033zzjRk2bJjJly+f8fT0NC1atLAbbiXezp07Tdu2bU3u3LmNu7u7CQkJMR06dDCrVq2y1YkfmuXs2bMp2i9jxowxNWvWNP7+/sbT09OULVvWjB071m7Yk6SG37p9uKmktjV+O+fNm2crS8nwWxcuXDDdu3c3efLkMT4+PqZZs2bmr7/+SjDUTPxwLVu3bk2wfYkNvxUREWFatGhhcuXKZQDb0EW3D78V7077PaXnVlLee+894+PjYzeUTHLnze37KbHjEx0dbfr06WMCAwONj4+Pad26tdm/f78BzNtvv51g3tvPl8T2mzHGLFiwwNSrV894e3sbb29vU7ZsWdOnTx+zf/9+W52kPkOJOXfunOnTp48pW7as8fb2Nn5+fqZWrVrm22+/TVD3+++/N3Xr1jWenp7G19fX1KxZ03zzzTe26fv27TNNmjQxPj4+Jk+ePKZXr15m9+7dCYb8SWx/pXTbjDGmVq1a5qmnnkrR9iXlypUrdzX/rW4ffssYY06fPm377Li5uZn77rsvyWGP2rdvbwAzd+5cW9mNGzeMl5eXcXNzSzDkWfzy+/TpY4KDg03OnDlN/vz5TePGjc2nn35qq5PY5/5Obj+3jTFmx44dplmzZsbHx8d4eXmZRo0amc2bN9vVudN1zJHzLDE///yzeeCBB2znXsuWLW3DlsVz9LN0u7s5f2+/JhpjvW7Vr1/fuLu7m8KFC5uwsDDzwQcfGMBERETYzZvY0GWJnVeXL182w4YNMyVLljRubm4mT548pm7duubdd9+1+86IjIw0Tz/9tPH19TV+fn7m6aefNjt37nRo+K1169aZ3r17m4CAAOPj42OefPJJuyHQjDFm06ZNpnbt2sbT09MULFjQDB482Pz0008JruOOfC9PnTrVNGjQwHa9L1GihBk0aJC5dOlSsnFndRZj0vjOC7knrF27lkaNGjFv3jzatWvn7HDESS5dukTx4sUZP348PXr0SLf17Nq1i6pVqzJr1qwkh+GR5O3atYtq1aqxY8eOJG8AFMms+vXrx9SpU7ly5UqaDlWYlmbOnEn37t3ZunUr999/v7PDuWeoj6yIpJqfnx+DBw/mnXfeSbO7kxN7rOWkSZNwcXGxuzFCHPP222/Trl07JbGS6d1+DYiMjOSrr76iXr16mTaJFedRH1kRuStDhgyx9UlMC+PHj2f79u00atSIHDlysGzZMpYtW0bv3r0TjPMrKTdnzhxnhyCSInXq1KFhw4aUK1eO06dP89lnnxEVFcUbb7zh7NAkE1IiKyKZSt26dVm5ciWjR4/mypUrFClShJEjR/Laa685OzQRyQCPPPII8+fP59NPP8VisVCtWjU+++wz/SIjiVIfWRERERHJktRHVkRERESyJCWyIiIiIpIl3XN9ZOPi4jh58iS5cuVK04G8RUREROTuGWO4fPkyBQsWtHtcdWLuuUT25MmTuvNZREREJJM7fvw4hQsXTrbOPZfIxj+m7fjx47ZnXYuIiIhI5hAVFUVwcHCKHq17zyWy8d0JfH19lciKiIiIZFIp6QKqm71EREREJEtSIisiIiIiWZISWRERERHJkpTIioiIiEiWpEQ2A+ghwCIiIiJpT4lsOvtlF7R9CY6ccHYkIiIiItmLEtl0ZAyMnAw79kLznjBniVpnRURERNKKEtl0ZLHA52FQuwpcvQ5D3oVnR8CFS86OTERERCTrUyKbzgrmg9nvwrDekDMH/LQBHuoB67c6OzIRERGRrE2JbAZwdYXnOsPij6FkCJyJhKcHw6gP4foNZ0cnIiIikjUpkc1AFUvBj1OgS2vr+88XQKvn4M9DTg1LREREJEtSIpvBPD1g9MswIwzyBMD+cGj1PEyfB3Fxzo5OREREJOtQIuskD9aGnz6DxnXgxr8w+mNrd4OIs86OTERERCRrUCLrRHkC4LOx8FZ/8HCHjduhWU9Yus7ZkYmIiIhkfkpkncxigSdbwdJP4b7ScDEKnh8Jr4yDK1edHZ2IiIhI5qVENpMoUQQWfgh9nrAmt/OWwyO9YMc+Z0cmIiIikjkpkc1E3HLC4F4wdyIUCoKjJ6HdSzBxJtyMdXZ0IiIiIpmLEtlMqFZlWDYdHmsMsXEw6Qto1xeOnnB2ZCIiIiKZhxLZTMrPBz54Hd5/DXy9Yec+aN4Lvl0Gxjg7OhERERHnUyKbybVuYm2drVUJoq/BoPHWm8EuXHJ2ZCIiIiLOpUQ2CyicH755D4b0ghyusGy9dZiujdudHZmIiIiI8yiRzSJcXeGFJ2DxR1AiGE6fgydfgTc/gus3nB2diIiISMZTIpvF3FcGlnwKT7Wyvv9sPrR9EU6cdm5cIiIiIhlNiWwW5OkBY/vD529BoB/s/RtaPge//e7syEREREQyjhLZLKxxHfhhCpQvCZEX4YmBMPsHZ0clIiIikjGUyGZxhfPDgg/g0Ybw700Y9h68Psn6t4iIiEh2pkQ2G/DyhA+Hw6Ae1sfbfvUdPPWKtZVWREREJLtSIptNWCzw4lMwfQz4eMEvu6HVc7DvoLMjExEREUkfSmSzmSZ1YdFHEFIQ/jkNbV+CpeucHZWIiIhI2lMimw2VLgrffwL174dr161PApvwOcTFOTsyERERkbSjRDab8veFmW9Dj3bW9x98Bc+NgCtXnRuXiIiISFpRIpuN5XCF4X1gwhBwywk/bYQ2feDoCWdHJiIiInL3lMjeA9o9DN9Ogny54cARaPUCbNzu7KhERERE7o4S2XtE1fLw4xSoUhYuRkGXwTBjARjj7MhEREREUkeJ7D0kKA/MfR/aNoXYOBj5IQx+B2JuODsyEREREccpkb3HeLjBe8Pg9efBxQW+XQadBsCZ886OTERERMQxSmTvQRYL9OoAM8PA1wd27IWWz8Lv+50dmYiIiEjKKZG9h4XWtI43WzIEIs5Bu76waKWzoxIRERFJGSWy97hihWHxR9YngsXcgH5vwVtTIDbW2ZGJiIiIJM+piez69etp2bIlBQsWxGKxsHjx4mTrr127FovFkuAVERGRMQFnU7m8Ydpo6POk9f3UufDMa3DpinPjEhEREUmOUxPZ6OhoKleuzEcffeTQfPv37+fUqVO2V758+dIpwnuHiwsM7gkfvgEe7rD2V2j9Ahw65uzIRERERBKXw5krb968Oc2bN3d4vnz58uHv75/2AQktH4RiwdDrdTh8HB57AaaNgTpVnB2ZiIiIiL0s2Ue2SpUqFChQgKZNm7Jp06Zk68bExBAVFWX3kuRVLAU/TIEa98HlaOg62Pp4WxEREZHMJEslsgUKFGDKlCksWLCABQsWEBwcTMOGDdmxY0eS84SFheHn52d7BQcHZ2DEWVeeAJj1LjSrDzH/wnMjrGPOioiIiGQWFmPu7iGlUVFRrF69mjJlylCuXLnUB2KxsGjRIlq3bu3QfKGhoRQpUoSvvvoq0ekxMTHExMTYxRscHMylS5fw9fVNdbz3ipux8Op7MHep9f2wZ+G5Ts6NSURERLKvqKgo/Pz8UpSrOdwi26FDBz788EMArl27xv3330+HDh2oVKkSCxYsSF3Ed6FmzZocPHgwyenu7u74+vravSTlcrjCuFf+S17Dplpfd/ffHxEREZG753Aiu379eurXrw/AokWLMMZw8eJFPvjgA8aMGZPmAd7Jrl27KFCgQIav915isVhbYoc9a30/ZQ4MedfaWisiIiLiLA6PWnDp0iUCAwMBWL58OY8//jheXl60aNGCQYMGObSsK1eu2LWmhoeHs2vXLgIDAylSpAjDhg3jxIkTfPnllwBMmjSJYsWKUaFCBa5fv8706dNZvXo1K1ascHQzJBWe6wQBvjB0grWrwcXL8MHr4OHm7MhERETkXuRwi2xwcDBbtmwhOjqa5cuX89BDDwFw4cIFPDw8HFrWtm3bqFq1KlWrVgVgwIABVK1aleHDhwNw6tQpjh37byDTGzduMHDgQO677z5CQ0PZvXs3P//8M40bN3Z0MySVOj4CU0aBe074aQN0G2od2UBEREQkozl8s9fHH3/Myy+/jI+PDyEhIezYsQMXFxcmT57MwoULWbNmTXrFmiYc6UAsSdu80zrW7JWr1uG6vhhnHelARERE5G44kqulatSCbdu2cfz4cZo2bYqPjw8AS5Yswd/fnwceeCB1UWcQJbJp548D0HUIRF6E4sHw1XgonN/ZUYmIiEhWlu6J7K1iY2P5448/CAkJISAg8zfJKZFNW4ePw1OD4MRpyJ8HvnoHShd1dlQiIiKSVaXr8Fv9+vXjs88+A6xJbGhoKNWqVSM4OJi1a9emKmDJuooHw4LJUCoEIs5B+5dh5z5nRyUiIiL3AocT2fnz51O5cmUAfvjhB8LDw/nrr7/o378/r732WpoHKJlfgbww732oWg4uRsETA2HDNmdHJSIiItmdw4nsuXPnyJ/f2hFy6dKltG/fntKlS/PMM8/wxx9/pHmAkjUE+MHXE6D+/XD1OnQfBkvWOjsqERERyc4cTmSDgoLYt28fsbGxLF++nKZNmwJw9epVXF1d0zxAyTq8PeGzsfBoQ/j3JvR5E2Z97+yoREREJLtyOJHt3r07HTp0oGLFilgsFpo0aQLAr7/+StmyZdM8QMla3N2sD0l4qpX1MbavTYTJs/RIWxEREUl7Dj/Za+TIkVSsWJHjx4/Tvn173N3dAXB1dWXo0KFpHqBkPa6uMKYfBPrBB1/Bu5/BhUvw+vPg4vB/nUREREQSd9fDb2U1Gn4rY30+H0Z9ZP27TVN4ZzDkdPi/TyIiInKvSNfhtwDWrVtHy5YtKVmyJCVLlqRVq1Zs2LAhVcFK9vZMO5j4Kri6wKKV0PsNuHbd2VGJiIhIduBwIjtr1iyaNGmCl5cXffv2pW/fvnh6etK4cWNmz56dHjFKFte2KUwbY+0/u/oXeHowXLri7KhEREQkq3O4a0G5cuXo3bs3/fv3tyt/7733mDZtGn/++WeaBpjW1LXAebb+Ac8Mg6hoKFccvnwH8gU6OyoRERHJTNK1a8Hhw4dp2bJlgvJWrVoRHh7u6OLkHlLjPvj2fcgbCH8ehsdfgmMnnR2ViIiIZFUOJ7LBwcGsWrUqQfnPP/9McHBwmgQl2Ve5ErDgAyhS0JrEdugHR084OyoRERHJihy+f3zgwIH07duXXbt2UbduXQA2bdrEzJkzef/999M8QMl+QgrB/A+sj7I9eBQ6DYC5E63JrYiIiEhKpWr4rUWLFjFhwgRbf9hy5coxaNAgHnvssTQPMK2pj2zmceY8dOoPh45BwXxKZkVERMSxXE3jyIpTnY6Ezv3h0HEoFARzJ0FwfmdHJSIiIs6S7uPIiqSVoNzwzUQoHgwnTkOnfvBPhLOjEhERkawgRX1kAwICsFgsKVrg+fPn7yogufcE5YY570HH/hD+j/XfbydZW2hFREREkpKiRHbSpEnpHIbc64Ly/JfMHjnxXzJbMJ+zIxMREZHMSn1kJVM5dRY69oOjJ603fn07CQrkdXZUIiIiklHUR1ayrAJ5Yc7E/8aZ7dgPIs46OyoRERHJjJTISqZTMJ+1m0FwAWvLbKcBSmZFREQkISWykikVCrIms4XzW28A6zwQTp9zdlQiIiKSmSiRlUyrcH5rN4PCQXD4OHQeYB13VkRERATuIpE9ePAgP/30E9euXQPgHrtnTDJIcH7rOLOFgqwPTeg8wPpEMBERERGHE9nIyEiaNGlC6dKleeSRRzh16hQAPXr0YODAgWkeoEiRAtZuBgXzWR9n26k/nFUyKyIics9zOJHt378/OXLk4NixY3h5ednKO3bsyPLly9M0OJF4RQpak9kCea3JbOcBSmZFRETudQ4nsitWrGDcuHEULlzYrrxUqVIcPXo0zQITuV1IIZg7EfLngb+PwhMD4dwFZ0clIiIizuJwIhsdHW3XEhvv/PnzuLu7p0lQIkkJKQRzJ1mfBHbgiDWZjbzo5KBERETEKRxOZOvXr8+XX35pe2+xWIiLi2P8+PE0atQoTYMTSUzR/7fM5ssN+8Otyez5S86OSkRERDKaw4+o3bNnD40bN6ZatWqsXr2aVq1asXfvXs6fP8+mTZsoUaJEesWaJvSI2uzj0DHrwxLOREK54vDNexDg5+yoRERE5G6k6yNqK1asyIEDB6hXrx6PPfYY0dHRtG3blp07d2b6JFaylxJFrMlr3kD48zA88QpcUMusiIjIPcPhFtmsTi2y2c/fR6Fzfzh7ASqUgtnvgr8OrYiISJaUri2yJUuWZOTIkfz999+pDlAkLZUKgdnvQZ4A2Ps3PPkKXLrs7KhEREQkvTmcyPbp04clS5ZQpkwZatSowfvvv09ERER6xCaSYqWLwjcTILc/7FEyKyIick9I1QMRtm7dyl9//cUjjzzCRx99RHBwMA899JDdaAYiGa10MWuf2UA/+OMAPDUIoq44OyoRERFJLw4nsvFKly7NqFGjOHDgABs2bODs2bN0797doWWsX7+eli1bUrBgQSwWC4sXL77jPGvXrqVatWq4u7tTsmRJZs6cmboNkGypzC3J7O/7ofswuHrN2VGJiIhIekh1Igvw22+/0a9fP9q0acOBAwdo3769Q/NHR0dTuXJlPvrooxTVDw8Pp0WLFjRq1Ihdu3bRr18/evbsyU8//ZSa8CWbKlscZr0Lvj6wbQ/0fB2u33B2VCIiIpLWHB614MCBA3z99dd88803hIeH8+CDD/Lkk0/Stm1bfHx8Uh+IxcKiRYto3bp1knWGDBnCkiVL2LNnj62sU6dOXLx4keXLl6doPRq14N6xYx889QpEX4PGdWDKKHDL6eyoREREJDnpOmpB2bJlWb58OX369OGff/7hp59+okuXLneVxKbUli1baNKkiV1Zs2bN2LJlS5LzxMTEEBUVZfeSe0O18vD5W+DuBqu2QL+xEBvr7KhEREQkreRwdIb9+/dTqlSp9IjljiIiIggKCrIrCwoKIioqimvXruHp6ZlgnrCwMEaNGpVRIUomU7sKfDoaer4GS9aBlyeMHwQud9WpRkRERDIDh7/OnZXEptawYcO4dOmS7XX8+HFnhyQZrGFN+HA4uLrAvOUw/AO4tx4DIiIikj2lqEU2MDCQAwcOkCdPHgICArBYLEnWPX/+fJoFd7v8+fNz+vRpu7LTp0/j6+ubaGssgLu7O+7u7ukWk2QND9eHCUOhfxh89R14ecCwZyGZU1lEREQyuRQlshMnTiRXrly2v5NLZNNTnTp1WLp0qV3ZypUrqVOnjlPikaylTVO4HgNDJ8DUueDtCS93dXZUIiIiklopSmS7dv3v275bt25ptvIrV65w8OBB2/vw8HB27dpFYGAgRYoUYdiwYZw4ccL2oIXnnnuODz/8kMGDB/PMM8+wevVqvv32W5YsWZJmMUn21vlRuHod3vwI3psJnp7Qu4OzoxIREZHUcLiPrKurK2fOnElQHhkZiaurq0PL2rZtG1WrVqVq1aoADBgwgKpVqzJ8+HAATp06xbFjx2z1ixUrxpIlS1i5ciWVK1dmwoQJTJ8+nWbNmjm6GXIP69EOXulh/XvsJzDre+fGIyIiIqnj8DiyLi4uREREkC9fPrvykydPUqJECa5dy9yPUdI4sgLWm73GT4ePZ1vfvzcUHtf/h0RERJzOkVwtxcNvffDBB4D1wQXTp0+3Gzc2NjaW9evXU7Zs2VSGLJKxLBYY3NP6+NqZi+CV8eDpAY+EOjsyERERSakUJ7ITJ04EwBjDlClT7LoRuLm5UbRoUaZMmZL2EYqkE4sFRrxo7TP77TLoOwY83OHB2s6OTERERFIixYlseHg4AI0aNWLhwoUEBASkW1AiGcXFBd4eaB3N4PvV8NxwmPE2PFDN2ZGJiIjInTh8s9eaNWvsktjY2Fh27drFhQsX0jQwkYzi6grvDYOmdSHmX+tTwLbtcXZUIiIicicOJ7L9+vXjs88+A6xJbIMGDahWrRrBwcGsXbs2reMTyRA5c8CHI6D+/dauBt2Hwh8HnB2ViIiIJMfhRHbevHlUrlwZgB9++IEjR47w119/0b9/f1577bU0D1Ako3i4wadvQo37ICoanh4EB8KdHZWIiIgkxeFENjIykvz58wOwdOlS2rdvT+nSpXnmmWf4448/0jxAkYzk5QkzwqBSGbgQBU+8AkdOODsqERERSYzDiWxQUBD79u0jNjaW5cuX07RpUwCuXr3q8AMRRDKjXN7w1XgoWxzOnocnBsKJ086OSkRERG7ncCLbvXt3OnToQMWKFbFYLDRp0gSAX3/9VePISrbh7wuz3oHiwdYk9omBcDrS2VGJiIjIrRxOZEeOHMn06dPp3bs3mzZtwt3dHbA+unbo0KFpHqCIs+QNhNkToHB+a/eCp16B85ecHZWIiIjEc/gRtVmdHlErjjp2Etq9DKfPQcVSMPs98PO583wiIiLiOEdytVQlstHR0axbt45jx45x48YNu2l9+/Z1dHEZSomspMbBY9DhZYi8CNUrwFfvgLens6MSERHJftI1kd25cyePPPIIV69eJTo6msDAQM6dO4eXlxf58uXj8OHDdxV8elMiK6m19yB06g9RV6BuVevoBh7uzo5KREQke3EkV3O4j2z//v1p2bIlFy5cwNPTk19++YWjR49SvXp13n333VQHLZLZVSgJX46ztsRu3gm9h8P1G3eeT0RERNKHw4nsrl27GDhwIC4uLri6uhITE0NwcDDjx4/n1VdfTY8YRTKNquXh87esLbHrfoPnhkOMklkRERGncDiRzZkzJy4u1tny5cvHsWPHAPDz8+P48eNpG51IJlS7yn/J7Jpf4bkRSmZFREScweFEtmrVqmzduhWA0NBQhg8fztdff02/fv2oWLFimgcokhk9UA0+GwvubrD6F3hhFNz419lRiYiI3FscTmTfeustChQoAMDYsWMJCAjg+eef5+zZs3z66adpHqBIZlWvurVl1t0Nft4MfZTMioiIZCiHRi0wxnD8+HHy5cuHh4dHesaVbjRqgaS19Vuh52sQ8y80qwcfjYCcOZwdlYiISNaUbqMWGGMoWbKk+sKK3KJBDZg2Btxzwk8boc+b8O9NZ0clIiKS/TmUyLq4uFCqVCkiI/XQeZFbhdaET8eAW074aQO8NFrJrIiISHpzuI/s22+/zaBBg9izZ096xCOSZTWsCVPftCazy9ZD3zFKZkVERNKTw0/2CggI4OrVq9y8eRM3Nzc8Pe2f03n+/Pk0DTCtqY+spLdVW+DZ4dYk9tGG8P7rkMPV2VGJiIhkDY7kag7fkjJx4kQsFkuqgxPJ7hrXgSmjrOPL/rgWXFxg4qtKZkVERNKawy2yWZ1aZCWjrNwEz4+0tsy2elDJrIiISEqk26gFAK6urpw5cyZBeWRkJK6u+pYWidf0Afh4hDV5/X41DHwbYmOdHZWIiEj24XAim1QDbkxMDG5ubncdkEh28tD/x5XN4QqLf4aB45TMioiIpJUU95H94IMPALBYLEyfPh0fHx/btNjYWNavX0/ZsmXTPkKRLO7h+vDhcOuTvxatBAvw7hDQDxgiIiJ3J8V9ZIsVKwbA0aNHKVy4sF03Ajc3N4oWLcqbb75JrVq10ifSNKI+suIsS9bBS29CbBy0awbjBymZFRERuV26jFoQHh4OQKNGjVi4cCEBAQF3F6XIPaZFKPCG9WEJ838Ci8WazLo43MFHREREIBXDb61ZsyY94hC5J7RoCHFx8PJYmLfcmsS+PVDJrIiISGo4nMjGxsYyc+ZMVq1axZkzZ4iLi7Obvnr16jQLTiQ7avkgGKzJ7Nyl1pbZsAFKZkVERBzlcCL78ssvM3PmTFq0aEHFihX1cASRVGj1oLVltn8YzFkCLhYY21/JrIiIiCMcTmTnzJnDt99+yyOPPJIe8YjcM1o3AWNgwNsw+0drEjumn7WFVkRERO7M4fYfNzc3SpYsmR6xiNxz2jSFCUOsyeus7+H1SdbkVkRERO7M4UR24MCBvP/++0k+GEFEHNP2IXh38H/J7PAPlMyKiIikhMNdCzZu3MiaNWtYtmwZFSpUIGfOnHbTFy5cmGbBidwr2j1svQFs0Hj4crE1qR35ovrMioiIJMfhr0l/f3/atGlDaGgoefLkwc/Pz+6VGh999BFFixbFw8ODWrVq8dtvvyVZd+bMmVgsFruXh4dHqtYrkpm0f9g6rqzFAl8sgj5vwvUYZ0clIiKSeTncIjtjxow0DWDu3LkMGDCAKVOmUKtWLSZNmkSzZs3Yv38/+fLlS3QeX19f9u/fb3uvkRMku+jQHNxyWltml66DU2fhs7GQ29/ZkYmIiGQ+qf7h8uzZs2zcuJGNGzdy9uzZVAfw3nvv0atXL7p370758uWZMmUKXl5efP7550nOY7FYyJ8/v+0VFBSU6vWLZDatm8BX48EvF+zcB637wKFjzo5KREQk83E4kY2OjuaZZ56hQIECNGjQgAYNGlCwYEF69OjB1atXHVrWjRs32L59O02aNPkvIBcXmjRpwpYtW5Kc78qVK4SEhBAcHMxjjz3G3r17Hd0MkUytdhVY9CEEF4BjJ6HNi/DrbmdHJSIikrk4nMgOGDCAdevW8cMPP3Dx4kUuXrzId999x7p16xg4cKBDyzp37hyxsbEJWlSDgoKIiIhIdJ4yZcrw+eef89133zFr1izi4uKoW7cu//zzT6L1Y2JiiIqKsnuJZAUlisDij6BqObh0GZ4aBIt/dnZUIiIimYfDieyCBQv47LPPaN68Ob6+vvj6+vLII48wbdo05s+fnx4x2qlTpw5dunShSpUqhIaGsnDhQvLmzcvUqVMTrR8WFmZ3M1pwcHC6xyiSVvIEwJyJ0LwB3PjX+ljbybM0PJeIiAikIpG9evVqon1S8+XL53DXgjx58uDq6srp06ftyk+fPk3+/PlTtIycOXNStWpVDh48mOj0YcOGcenSJdvr+PHjDsUo4mwe7vDxCOjd0fr+3c9g8Dvw703nxiUiIuJsDieyderUYcSIEVy/ft1Wdu3aNUaNGkWdOnUcWpabmxvVq1dn1apVtrK4uDhWrVqV4mXFxsbyxx9/UKBAgUSnu7u721qO418iWY2LC7z2HIx+2fr3t8ug21CIuuLsyERERJzH4eG33n//fZo1a0bhwoWpXLkyALt378bDw4OffvrJ4QAGDBhA165duf/++6lZsyaTJk0iOjqa7t27A9ClSxcKFSpEWFgYAG+++Sa1a9emZMmSXLx4kXfeeYejR4/Ss2dPh9ctktV0aQ2FguDFN2Hjdnj8JZgRBoVT9gOGiIhItuJwIluxYkX+/vtvvv76a/766y8AOnfuzJNPPomnp6fDAXTs2JGzZ88yfPhwIiIiqFKlCsuXL7d1Xzh27Bgutzze6MKFC/Tq1YuIiAgCAgKoXr06mzdvpnz58g6vWyQralwH5n0A3YfBgSPW4blmvAX3lXF2ZCIiIhnLYsy9ddtIVFQUfn5+XLp0Sd0MJEs7ecaazP51GDw94MM3oEldZ0clIiJydxzJ1RzuIxsWFpbowwo+//xzxo0b5+jiRCSVCuaD+R9Ag/vh2nXo9QbMXOTsqERERDKOw4ns1KlTKVu2bILyChUqMGXKlDQJSkRSJpc3fB4GnVpAXByM+ADe/AhiY50dmYiISPpzOJGNiIhIdISAvHnzcurUqTQJSkRSLmcOeHsgDP7//Y6fzYfnR1lbaUVERLIzhxPZ4OBgNm3alKB806ZNFCxYME2CEhHHWCzQ50mY/Aa45YSfNkCn/nD2vLMjExERST8Oj1rQq1cv+vXrx7///suDDz4IwKpVqxg8eLDDj6gVkbTV6kEokBd6vg67/rKOaDDzbSgV4uzIRERE0p7DoxYYYxg6dCgffPABN27cAMDDw4MhQ4YwfPjwdAkyLWnUArkXHD5ufWDC0ZPg6wNT34S6VZ0dlYiIyJ05kqulevitK1eu8Oeff+Lp6UmpUqVwd3dPVbAZTYms3CsiL0Kv12H7Xms/2vGDoO1Dzo5KREQkeRmSyGZVSmTlXnL9Bgx4C5ass77v3w1e7mLtUysiIpIZpes4siKSdXi4wYfD4fnO1vcTZ8LAtyHmhlPDEhERSRNKZEWyORcXGNobwgaAqwssWAEtesO2Pc6OTERE5O4okRW5RzzREmaEQZ4A+PsotOsLb7wPV646OzIREZHUUSIrcg8JrQmrZkL7h8EY+HIxNOkGq7Y4OTAREZFUUCIrco/x94V3h8DX70KRgnDqLDzzKrw0Gs5dcHZ0IiIiKadEVuQeVa86rPgMene09qP9fjU07gbzl1tba0VERDI7JbIi9zBPD3jtOfj+YyhfEi5GwcBx8PRgOHbK2dGJiIgkT4msiHBfGfj+E+voBu5usGEbPPQMTPsWbsY6OzoREZHEKZEVEcD69K/nO8NPn0HtKnDtOoz5BNr0gX0HnR2diIhIQkpkRcROscIw5z0Y9wr4esPv++HRZ2H8NLge4+zoRERE/qNEVkQSsFigUwv4+Qto3gBi4+Cj2fBwT/hll7OjExERsVIiKyJJCsoNU0bBp6MhKA+E/wMd+8PQd+HSFWdHJyIi9zolsiJyR83qwc8z4MmW1vffLLE+SGH5BqeGJSIi9zglsiKSIr4+8NYA+HYSFA+GM5Hw7HDr6/Q5Z0cnIiL3IiWyIuKQWpVh2XR48SnI4WptlW3SDb75EeLinB2diIjcS5TIiojDPNxgUA/4cSpULgtR0TB0AnQeAAfCnR2diIjcK5TIikiqlSsBiz6E4X2sTwn7ZTc0fcZ6Q9jin+H6DWdHKCIi2ZnFmHvrqepRUVH4+flx6dIlfH19nR2OSLZxPAJGfwwrN/3XxcDfFx5/CDo/CqVCnBufiIhkDY7kakpkRSRNnTwD3y6DuUutf8erWQk6t4BHQsHD3XnxiYhI5qZENhlKZEUyRmwsrNtqvQls1RbrQxXAOvpBfCttmWLOjVFERDIfJbLJUCIrkvEizsK3y2HuEvjn9H/l1StYE9pHG1r72IqIiCiRTYYSWRHniY2FDdth9o/w86ZbWmm9oXVTeOJR6w1kIiJy71IimwwlsiKZw+lImL/c+pSw46f+K69aztpK27IReHk6Lz4REXEOJbLJUCIrkrnExcGmHdZW2hUb4WastdzHC1o3sSa1FUs5N0YREck4SmSToURWJPM6ex4W/GRNao+e/K+8Uhlrt4OWD1oTXBERyb6UyCZDiaxI5hcXB1t2WUc8WL4B/r1pLXd3s7bOVioDlcpa/y1eGFz0aBcRkWxDiWwylMiKZC2RF2H+TzBnCRw+nnC6jxdULG1NaiuXsf4bXAAslgwPVURE0oAS2WQokRXJmoyBQ8fh9/3wx37Y/RfsPQjXYxLW9feFSqXhvjJQ+f8tt/nzKLkVEckKlMgmQ4msSPZxMxYOHrUmtb/vhz8OwJ+H4Ma/CevmDfx/l4RbXnkCMj5mERFJXpZLZD/66CPeeecdIiIiqFy5MpMnT6ZmzZpJ1p83bx5vvPEGR44coVSpUowbN45HHnkkRetSIiuSvcXcgAPhsHv/f623+8P/G7P2VgXz2Se2IQWtya2G/RIRcZ4slcjOnTuXLl26MGXKFGrVqsWkSZOYN28e+/fvJ1++fAnqb968mQYNGhAWFsajjz7K7NmzGTduHDt27KBixYp3XJ8SWZF7z7XrsO+QNbH9/S/4/QAcOmbtrpAYTw/I7Q95AyB3gPXvPAGQx9/6Ps//X7n9IcAXXF0zbltERLK7LJXI1qpVixo1avDhhx8CEBcXR3BwMC+99BJDhw5NUL9jx45ER0fz448/2spq165NlSpVmDJlyh3Xp0RWRACuXIU9B6xJ7R//75Zw8oy1RdcRLi4Q6GdNam0Jb0DCvwP8rKMu3PrKoQRYRCQBR3K1HBkUU6Ju3LjB9u3bGTZsmK3MxcWFJk2asGXLlkTn2bJlCwMGDLAra9asGYsXL07PUEUkm/HxgtpVrK94xsDV63DuPJy7CJEX4NyFpP++EGUdKuzc/987ytXFPrF1cwP3nAkT3kRft9VzcbG2DLtYwOJiXbaLBVz+X+by/zKL5b96Li7/f/2/zJJEPUv8Kz7w//8RX35LkX3ZrdNuL7PYz5Ocu5mu+/tE0obFBYoUcHYUCTk1kT137hyxsbEEBQXZlQcFBfHXX38lOk9ERESi9SMiIhKtHxMTQ0zMf7c1R0VF3WXUIpJdWSzg7QnehSCk0J3r34yF85esiW/kxf8S2tv/jrwIF6OsN6HFj4kL1n67V69bXyIimVmgH+xc7OwoEnJqIpsRwsLCGDVqlLPDEJFsKIcr5Au0vlIqNtaa0Mbc+O91/Yb9+5gbCevc6RUbB3EGTNz///7/+7j//x0bZ50WZ277O/a2+W77Oy7O2lId3wnt1n/j+6XZTYv/+5ZpCea9fZ7k3GF6cpPTu+Oc82+VFsk4Xh7OjiBxTk1k8+TJg6urK6dPn7YrP336NPnz5090nvz58ztUf9iwYXZdEaKioggODr7LyEVEUsfVFTxdrTeUiYjI3XHqgx3d3NyoXr06q1atspXFxcWxatUq6tSpk+g8derUsasPsHLlyiTru7u74+vra/cSERERkazP6V0LBgwYQNeuXbn//vupWbMmkyZNIjo6mu7duwPQpUsXChUqRFhYGAAvv/wyoaGhTJgwgRYtWjBnzhy2bdvGp59+6szNEBEREZEM5vREtmPHjpw9e5bhw4cTERFBlSpVWL58ue2GrmPHjuHi8l/Dcd26dZk9ezavv/46r776KqVKlWLx4sUpGkNWRERERLIPp48jm9E0jqyIiIhI5uVIrubUPrIiIiIiIqmlRFZEREREsiSn95HNaPE9KfRgBBEREZHMJz5HS0nv13sukb18+TKAxpIVERERycQuX76Mn59fsnXuuZu94uLiOHnyJA8++CDbtm1L1TJq1KjB1q1bU1Q3/gEMx48f181laciRY5CZZabtyMhY0mtdabncu13W3czv6Ly6zqS9zPTZvBuZaTt0jUn7ZaV2GZn9GmOM4fLlyxQsWNBu5KrE3HMtsi4uLhQuXJgcOXKk+mC4uro6PK8expC2UnMMMqPMtB0ZGUt6rSstl3u3y7qb+VM7r64zaSczfTbvRmbaDl1j0n5ZqV1GVrjG3KklNt49e7NXnz59nDKvpI3scgwy03ZkZCzpta60XO7dLkvXmKwtuxyDzLQdusak/bJSu4zMdF7crXuua0FG07i1IpLedJ0RkfSUma8x92yLbEZxd3dnxIgRuLu7OzsUEcmmdJ0RkfSUma8xapEVERERkSxJLbIiIiIikiUpkRURERGRLEmJrIiIiIhkSUpkRURERCRLUiLrZD/++CNlypShVKlSTJ8+3dnhiEg206ZNGwICAmjXrp2zQxGRbOb48eM0bNiQ8uXLU6lSJebNm5fhMWjUAie6efMm5cuXZ82aNfj5+VG9enU2b95M7ty5nR2aiGQTa9eu5fLly3zxxRfMnz/f2eGISDZy6tQpTp8+TZUqVYiIiKB69eocOHAAb2/vDItBLbJO9Ntvv1GhQgUKFSqEj48PzZs3Z8WKFc4OS0SykYYNG5IrVy5nhyEi2VCBAgWoUqUKAPnz5ydPnjycP38+Q2NQInsX1q9fT8uWLSlYsCAWi4XFixcnqPPRRx9RtGhRPDw8qFWrFr/99ptt2smTJylUqJDtfaFChThx4kRGhC4iWcDdXmNERJKTlteY7du3ExsbS3BwcDpHbU+J7F2Ijo6mcuXKfPTRR4lOnzt3LgMGDGDEiBHs2LGDypUr06xZM86cOZPBkYpIVqRrjIikp7S6xpw/f54uXbrw6aefZkTY9oykCcAsWrTIrqxmzZqmT58+tvexsbGmYMGCJiwszBhjzKZNm0zr1q1t019++WXz9ddfZ0i8IpK1pOYaE2/NmjXm8ccfz4gwRSSLSu015vr166Z+/frmyy+/zKhQ7ahFNp3cuHGD7du306RJE1uZi4sLTZo0YcuWLQDUrFmTPXv2cOLECa5cucKyZcto1qyZs0IWkSwkJdcYEZHUSsk1xhhDt27dePDBB3n66aedEqcS2XRy7tw5YmNjCQoKsisPCgoiIiICgBw5cjBhwgQaNWpElSpVGDhwoEYsEJEUSck1BqBJkya0b9+epUuXUrhwYSW5IpIiKbnGbNq0iblz57J48WKqVKlClSpV+OOPPzI0zhwZujZJoFWrVrRq1crZYYhINvXzzz87OwQRyabq1atHXFycU2NQi2w6yZMnD66urpw+fdqu/PTp0+TPn99JUYlIdqFrjIikp6xyjVEim07c3NyoXr06q1atspXFxcWxatUq6tSp48TIRCQ70DVGRNJTVrnGqGvBXbhy5QoHDx60vQ8PD2fXrl0EBgZSpEgRBgwYQNeuXbn//vupWbMmkyZNIjo6mu7duzsxahHJKnSNEZH0lC2uMU4ZKyGbWLNmjQESvLp27WqrM3nyZFOkSBHj5uZmatasaX755RfnBSwiWYquMSKSnrLDNcZijDFOyJ9FRERERO6K+siKiIiISJakRFZEREREsiQlsiIiIiKSJSmRFREREZEsSYmsiIiIiGRJSmRFREREJEtSIisiIiIiWZISWRERERHJkpTIioiIiEiWpERWRCSLGDlyJFWqVHF2GKm2atUqypUrR2xsLJA+27N27VosFgsXL15M0+WmFUe3+dy5c+TLl49//vkn/YISycKUyIpkcw0bNqRfv37ODiNDZPVEL7sbPHgwr7/+Oq6urum2jrp163Lq1Cn8/PwAmDlzJv7+/um2vvSWJ08eunTpwogRI5wdikimpERWRDK9GzduZOj6jDHcvHkzQ9eZ3W3cuJFDhw7x+OOPp+t63NzcyJ8/PxaLJV3Xk5G6d+/O119/zfnz550dikimo0RWJBvr1q0b69at4/3338disWCxWDhy5AgAe/bsoXnz5vj4+BAUFMTTTz/NuXPnbPM2bNiQl156iX79+hEQEEBQUBDTpk0jOjqa7t27kytXLkqWLMmyZcts88T/rLtkyRIqVaqEh4cHtWvXZs+ePXZxbdy4kfr16+Pp6UlwcDB9+/YlOjraNr1o0aKMHj2aLl264OvrS+/evQEYMmQIpUuXxsvLi+LFi/PGG2/w77//AtaWt1GjRrF7927bts6cOZMjR45gsVjYtWuXbfkXL17EYrGwdu1au7iXLVtG9erVcXd3Z+PGjcTFxREWFkaxYsXw9PSkcuXKzJ8/P9l9/vHHH1OqVCk8PDwICgqiXbt2tmlxcXGMHz+ekiVL4u7uTpEiRRg7dqxtenLbl5Tp06dTrlw5PDw8KFu2LB9//HGy9ROT3PH48ssv8fHx4e+//7bVf+GFFyhbtixXr14F/jtenTt3xtvbm0KFCvHRRx/ZrWPOnDk0bdoUDw+PJOOIi4vjzTffpHDhwri7u1OlShWWL19uV2fz5s1UqVIFDw8P7r//fhYvXmx3fG/tWrB27Vq6d+/OpUuXbOfEyJEj77g/MtsxrFChAgULFmTRokV3jF3knmNEJNu6ePGiqVOnjunVq5c5deqUOXXqlLl586a5cOGCyZs3rxk2bJj5888/zY4dO0zTpk1No0aNbPOGhoaaXLlymdGjR5sDBw6Y0aNHG1dXV9O8eXPz6aefmgMHDpjnn3/e5M6d20RHRxtjjFmzZo0BTLly5cyKFSvM77//bh599FFTtGhRc+PGDWOMMQcPHjTe3t5m4sSJ5sCBA2bTpk2matWqplu3brZ1h4SEGF9fX/Puu++agwcPmoMHDxpjjBk9erTZtGmTCQ8PN99//70JCgoy48aNM8YYc/XqVTNw4EBToUIF27ZevXrVhIeHG8Ds3LnTtvwLFy4YwKxZs8Yu7kqVKpkVK1aYgwcPmsjISDNmzBhTtmxZs3z5cnPo0CEzY8YM4+7ubtauXZvo/t66datxdXU1s2fPNkeOHDE7duww77//vm364MGDTUBAgJk5c6Y5ePCg2bBhg5k2bZptenLbZ4wxI0aMMJUrV7a9nzVrlilQoIBZsGCBOXz4sFmwYIEJDAw0M2fOTPE5kpLj0b59e1OjRg3z77//mh9//NHkzJnTbNu2ze545cqVy4SFhZn9+/ebDz74wLi6upoVK1bY6lSqVMm8/fbbduu+fXvee+894+vra7755hvz119/mcGDB5ucOXOaAwcOGGOMuXTpkgkMDDRPPfWU2bt3r1m6dKkpXbq03fGNP5YXLlwwMTExZtKkScbX19d2Tly+fDnZ/ZFZj2HHjh1N165dk41d5F6kRFYkmwsNDTUvv/yyXdno0aPNQw89ZFd2/PhxA5j9+/fb5qtXr55t+s2bN423t7d5+umnbWWnTp0ygNmyZYsx5r8kYs6cObY6kZGRxtPT08ydO9cYY0yPHj1M79697da9YcMG4+LiYq5du2aMsSZGrVu3vuO2vfPOO6Z69eq297cnCcYYhxLZxYsX2+pcv37deHl5mc2bN9str0ePHqZz586JxrNgwQLj6+troqKiEkyLiooy7u7udknP3W5fiRIlzOzZs+3mGT16tKlTp06K15GS43H+/HlTuHBh8/zzz5ugoCAzduxYu/ohISHm4Ycftivr2LGjad68ue29n5+f+fLLL+3q3L49BQsWTLDsGjVqmBdeeMEYY8wnn3xicufObYvLGGOmTZuWZCJrjDEzZswwfn5+KdsZJvMew/79+5uGDRumeL0i94oczmgFFhHn2r17N2vWrMHHxyfBtEOHDlG6dGkAKlWqZCt3dXUld+7c3HfffbayoKAgAM6cOWO3jDp16tj+DgwMpEyZMvz555+2df/+++98/fXXtjrGGOLi4ggPD6dcuXIA3H///Qlimzt3Lh988AGHDh3iypUr3Lx5E19fX4e3Pym3rvPgwYNcvXqVpk2b2tW5ceMGVatWTXT+pk2bEhISQvHixXn44Yd5+OGHadOmDV5eXvz555/ExMTQuHHjJNfvyPZFR0dz6NAhevToQa9evWzlN2/etN3olBIpOR4BAQF89tlnNGvWjLp16zJ06NAEy7n1mMe/nzRpku39tWvXku1WEBUVxcmTJ3nggQfsyh944AF2794NwP79+21dVuLVrFkzxduaEpn1GHp6etq6cojIf5TIityDrly5QsuWLRk3blyCaQUKFLD9nTNnTrtpFovFriz+hpq4uDiH1v3ss8/St2/fBNOKFCli+9vb29tu2pYtW3jyyScZNWoUzZo1w8/Pjzlz5jBhwoRk1+fiYr0VwBhjK0uqz+Kt67xy5QoAS5YsoVChQnb13N3dE50/V65c7Nixg7Vr17JixQqGDx/OyJEj2bp1K56ensnG6ej2xcc3bdo0atWqZTfNkVEBUno81q9fj6urK6dOnSI6OppcuXKleB1gvfv+woULDs3jDJn1GJ4/f568efPexZaJZE9KZEWyOTc3N9u4nfGqVavGggULKFq0KDlypP1l4JdffrElQRcuXODAgQO2ltZq1aqxb98+SpYs6dAyN2/eTEhICK+99pqt7OjRo3Z1EtvW+C//U6dO2VpSb73xKynly5fH3d2dY8eOERoamuI4c+TIQZMmTWjSpAkjRozA39+f1atX88gjj+Dp6cmqVavo2bNnqrbvVkFBQRQsWJDDhw/z5JNPpji+26XkeGzevJlx48bxww8/MGTIEF588UW++OILuzq//PJLgvfxxxygatWq7Nu3L8l1+Pr6UrBgQTZt2mS3vzdt2mRrdS1TpgyzZs0iJibG9p+JrVu3Jrt9iZ0Td5IZj+GePXto2LChQ9shci9QIiuSzRUtWpRff/2VI0eO4OPjQ2BgIH369GHatGl07tyZwYMHExgYyMGDB5kzZw7Tp0+/63E+33zzTXLnzk1QUBCvvfYaefLkoXXr1oD1ru7atWvz4osv0rNnT7y9vdm3bx8rV67kww8/THKZpUqV4tixY8yZM4caNWqwZMmSBHdxFy1alPDwcHbt2kXhwoXJlSsXnp6e1K5dm7fffptixYpx5swZXn/99TtuQ65cuXjllVfo378/cXFx1KtXj0uXLrFp0yZ8fX3p2rVrgnl+/PFHDh8+TIMGDQgICGDp0qXExcVRpkwZPDw8GDJkCIMHD8bNzY0HHniAs2fPsnfvXnr06JGi7bvdqFGj6Nu3L35+fjz88MPExMSwbds2Lly4wIABA+64jXDn43H58mWefvpp+vbtS/PmzSlcuDA1atSgZcuWdnfzb9q0ifHjx9O6dWtWrlzJvHnzWLJkiW16s2bNEiS/txs0aBAjRoygRIkSVKlShRkzZrBr1y5bt4cnnniC1157jd69ezN06FCOHTvGu+++C5DkcFtFixblypUrrFq1isqVK+Pl5YWXl1eSMWTGY3j16lW2b9/OW2+9leyyRO5JTu6jKyLpbP/+/aZ27drG09PTACY8PNwYY8yBAwdMmzZtjL+/v/H09DRly5Y1/fr1M3FxccaYxG8SCwkJMRMnTrQrA8yiRYuMMf/daPPDDz+YChUqGDc3N1OzZk2ze/duu3l+++0307RpU+Pj42O8vb1NpUqV7G7ySWw9xhgzaNAgkzt3buPj42M6duxoJk6caHcjz/Xr183jjz9u/P39DWBmzJhhjDFm3759pk6dOsbT09NUqVLFrFixItGbveJvEIoXFxdnJk2aZMqUKWNy5sxp8ubNa5o1a2bWrVuX6L7esGGDCQ0NNQEBAcbT09NUqlTJdpObMcbExsaaMWPGmJCQEJMzZ05TpEgR89Zbb6V4+xK7me3rr782VapUMW5ubiYgIMA0aNDALFy4MNH4kpLc8ejevbu57777zPXr1231J0yYYAIDA80///xjjLEer1GjRpn27dsbLy8vkz9/frs7/Y2x3vTn4eFh/vrrryS3JzY21owcOdIUKlTI5MyZ01SuXNksW7bMbjmbNm0ylSpVMm5ubqZ69epm9uzZBrAtN7Fj+dxzz5ncuXMbwIwYMSLZfZEZj+Hs2bNNmTJlko1b5F5lMeaWjmMiIndh7dq1NGrUiAsXLmTppymJY4oWLUq/fv3u+AS5QYMGERUVxdSpU9Ns3V9//bVtrNg79WHNqmrXrk3fvn154oknnB2KSKajByKIiEiGeO211wgJCXHo5sDbffnll2zcuJHw8HAWL17MkCFD6NChQ7ZNYs+dO0fbtm3p3Lmzs0MRyZTUR1ZERDKEv78/r7766l0tIyIiguHDhxMREUGBAgVo37693ZO17mTDhg00b948yenxIwlkFnny5GHw4MHODkMk01LXAhERuWdcu3aNEydOJDnd0dE0RMS5lMiKiIiISJakPrIiIiIikiUpkRURERGRLEmJrIiIiIhkSUpkRURERCRLUiIrIiIiIlmSElkRERERyZKUyIqIiIhIlqREVkRERESypP8BDWGBkjlzzgkAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"loss at scale 1: 1.882 . loss at scale 100: 0.000\n"
]
}
],
"source": [
"# viz: contrastive loss vs logit_scale on the aligned batch\n",
"I, T = aligned_batch(N=16, d=32, noise=0.05, rng_seed=SEED)\n",
"scales = np.linspace(0.0, math.log(100.0), 25)\n",
"losses = [clip_loss(I, T, torch.tensor(float(s))).item() for s in scales]\n",
"plt.figure(figsize=(7, 3))\n",
"plt.plot(np.exp(scales), losses, color=\"#1E40FF\")\n",
"plt.xscale(\"log\"); plt.xlabel(\"temperature scale exp(logit_scale)\"); plt.ylabel(\"contrastive loss\")\n",
"plt.title(\"sharper similarities (higher scale) -> lower loss on aligned pairs\")\n",
"plt.tight_layout(); plt.show()\n",
"print(f\"loss at scale 1: {losses[0]:.3f} . loss at scale 100: {losses[-1]:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "80b3de0c",
"metadata": {},
"source": [
"> **Interpretation.** On well-aligned pairs, raising the scale drives the loss toward zero because the softmax piles onto the correct diagonal entry. Real training learns this scale but clamps it at `exp(log(100)) = 100`: past that the softmax becomes one-hot, the gradient vanishes, and training stalls. This is the temperature-collapse failure mode.\n",
"\n",
"> **Key takeaways.** CLIP's loss is symmetric softmax cross-entropy on a similarity matrix, the diagonal is the correct match, and it is InfoNCE with in-batch negatives. L2-normalization is mandatory because cosine similarity is direction-only. A learned, clamped temperature sets how peaky the match must be.\n"
]
},
{
"cell_type": "markdown",
"id": "f7d83efe",
"metadata": {},
"source": [
"## Part 4 - Zero-shot and cross-attention\n",
"\n",
"> **Objectives**\n",
"> - Turn a frozen shared embedding space into an open-vocabulary classifier.\n",
"> - Implement the cross-attention primitive `Q <- modality A, K,V <- modality B` and verify its shapes.\n",
"> - See why one fusion primitive powers VLMs and Stable Diffusion alike; close with a GPU-fenced finetune appendix.\n",
"\n",
"Once images and captions live in one space, the text encoder *is* your classifier. Encode the prompt `\"a photo of a {class}\"` for each class, encode the image, pick the nearest class by cosine similarity. No labeled training data, no new model. We demonstrate the mechanism on a clean synthetic space (real CLIP zero-shot needs the pretrained weights), then build the cross-attention primitive that lets one modality read from another.\n"
]
},
{
"cell_type": "markdown",
"id": "7ed4d04a",
"metadata": {},
"source": [
"### 4.1 Zero-shot classification: prompts as a classifier\n",
"\n",
"The mechanism in one function: given image features `(1, d)` and a stack of class-prompt features `(C, d)`, the prediction is the argmax cosine similarity. We construct a synthetic shared space with `C` class prototypes so the mechanism is assertable: each \"image\" is a noisy copy of its class prototype, each \"prompt\" is the clean prototype, and the classifier should recover the class.\n"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "a51b5c6b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.425824Z",
"iopub.status.busy": "2026-06-10T20:02:35.425746Z",
"iopub.status.idle": "2026-06-10T20:02:35.429188Z",
"shell.execute_reply": "2026-06-10T20:02:35.428867Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"200 synthetic images . 10 class prompts . embed dim 128\n"
]
}
],
"source": [
"# a synthetic shared embedding space with known class prototypes\n",
"torch.manual_seed(SEED)\n",
"N_CLASS, EMBED = 10, 128\n",
"prototypes = F.normalize(torch.randn(N_CLASS, EMBED), dim=-1) # one direction per class\n",
"\n",
"def make_image_features(labels, noise=0.15):\n",
" \"Each image feature is its class prototype plus noise, normalized.\"\n",
" base = prototypes[labels]\n",
" return F.normalize(base + noise * torch.randn(len(labels), EMBED), dim=-1)\n",
"\n",
"# class-prompt features ARE the clean prototypes (the text encoder's output for each class)\n",
"prompt_features = prototypes\n",
"labels = torch.arange(N_CLASS).repeat(20) # 200 synthetic test images\n",
"img_features = make_image_features(labels)\n",
"print(f\"{len(labels)} synthetic images . {N_CLASS} class prompts . embed dim {EMBED}\")"
]
},
{
"cell_type": "markdown",
"id": "da225188",
"metadata": {},
"source": [
"### Exercise 16.6 - Zero-shot classify by cosine similarity\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"Write `zero_shot_classify(image_features, prompt_features)`. Inputs are L2-normalized `(M, d)` images and `(C, d)` class prompts. Return the predicted class index for each image (shape `(M,)`) as the argmax cosine similarity. The check asserts the synthetic accuracy is high (the prototypes are well separated) and that the output dtype and shape are right.\n"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "a7d09572",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.429935Z",
"iopub.status.busy": "2026-06-10T20:02:35.429860Z",
"iopub.status.idle": "2026-06-10T20:02:35.432865Z",
"shell.execute_reply": "2026-06-10T20:02:35.432610Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 16.6 zero-shot classify: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def zero_shot_classify(image_features, prompt_features):\n",
" \"\"\"image_features: (M, d) normalized . prompt_features: (C, d) normalized -> (M,) class ids.\"\"\"\n",
" # TODO 1: sims = image_features @ prompt_features.T -> (M, C) cosine similarities\n",
" sims = None\n",
" # TODO 2: predictions = argmax over the class axis (dim=-1)\n",
" preds = None\n",
" attempted(sims, preds)\n",
" return preds\n",
"\n",
"def _zero_shot_checks():\n",
" preds = zero_shot_classify(img_features, prompt_features)\n",
" check_shape(preds, (len(labels),))\n",
" assert preds.dtype == torch.long, f\"predictions should be integer class ids, got {preds.dtype}\"\n",
" acc = (preds == labels).float().mean().item()\n",
" assert acc > 0.9, f\"well-separated prototypes should give >90% zero-shot accuracy, got {acc:.1%}\"\n",
"\n",
"check(\"16.6 zero-shot classify\", _zero_shot_checks)"
]
},
{
"cell_type": "markdown",
"id": "0ea44cda",
"metadata": {},
"source": [
"Hint 1 (conceptual)One matrix multiply gives every image-vs-every-prompt similarity. The predicted class is the column index of the largest similarity in each row.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"sims = image_features @ prompt_features.T # (M, C)\n",
"preds = sims.argmax(dim=-1) # (M,)\n",
"```\n",
"\n",
"\n",
"Help: \"argmax returns floats / wrong shape\"`argmax(dim=-1)` returns `long` indices automatically; if your shape is `(M, C)` you forgot the `dim` argument and reduced the whole tensor to a scalar.\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "876a0010",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.433528Z",
"iopub.status.busy": "2026-06-10T20:02:35.433466Z",
"iopub.status.idle": "2026-06-10T20:02:35.435917Z",
"shell.execute_reply": "2026-06-10T20:02:35.435584Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 16.6 zero-shot classify\n",
"synthetic zero-shot accuracy: 100.0% (no training; the prompts are the classifier)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines zero_shot_classify; the checks below re-verify the reference.\n",
"def zero_shot_classify(image_features, prompt_features):\n",
" sims = image_features @ prompt_features.T # (M, C)\n",
" return sims.argmax(dim=-1) # (M,)\n",
"\n",
"check(\"16.6 zero-shot classify\", _zero_shot_checks, required=True)\n",
"acc = (zero_shot_classify(img_features, prompt_features) == labels).float().mean().item()\n",
"print(f\"synthetic zero-shot accuracy: {acc:.1%} (no training; the prompts are the classifier)\")"
]
},
{
"cell_type": "markdown",
"id": "8e12779c",
"metadata": {},
"source": [
"> **Caveat:** this works because the synthetic prototypes are well separated and the \"text encoder\" is perfect. Real zero-shot CLIP fails on classes whose linguistic name does not match what the text encoder learned (fine-grained bird species, culturally specific concepts). The mechanism is exact; the quality is entirely about whether the two encoders learned a shared space, which is a data-and-scale question.\n"
]
},
{
"cell_type": "markdown",
"id": "6e3ca799",
"metadata": {},
"source": [
"### 4.2 Cross-attention: the universal fusion primitive\n",
"\n",
"Every multimodal model that fuses two modalities does it with the same operation: cross-attention, where the *queries* come from one modality and the *keys and values* come from another.\n",
"\n",
"$$ \\text{CrossAttn}(A, B): \\quad Q = A W_Q,\\ \\ K = B W_K,\\ \\ V = B W_V, \\quad \\text{out} = \\text{softmax}\\!\\Big(\\tfrac{QK^{\\top}}{\\sqrt{d_k}}\\Big) V. $$\n",
"\n",
"The attention matrix is `(T_A, T_B)`: one row per query token from $A$, one column per key token from $B$. In Flamingo, $A$ is the language model's tokens and $B$ is the compressed image features. In Stable Diffusion's UNet, $A$ is the spatial image latent and $B$ is the text embedding. Same primitive, swapped operands. Once you see `Q <- one modality, K,V <- another`, you see it everywhere.\n"
]
},
{
"cell_type": "markdown",
"id": "9cd6547b",
"metadata": {},
"source": [
"### Exercise 16.7 - Cross-attention shapes\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"Implement `cross_attention(A, B, Wq, Wk, Wv, n_heads)`. Queries project from `A` of shape `(batch, T_a, d)`, keys and values from `B` of shape `(batch, T_b, d)`. Return the attention output `(batch, T_a, d)` and the attention weights `(batch, n_heads, T_a, T_b)`. This is the hard sub-skill behind every fusion family, so it stands alone (spec §3.4). Mind the `1/sqrt(d_k)` scaling.\n"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "9ed0f7b5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.436598Z",
"iopub.status.busy": "2026-06-10T20:02:35.436530Z",
"iopub.status.idle": "2026-06-10T20:02:35.441233Z",
"shell.execute_reply": "2026-06-10T20:02:35.440924Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 16.7 cross-attention: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def cross_attention(A, B, Wq, Wk, Wv, n_heads):\n",
" \"\"\"A: (batch, T_a, d) queries . B: (batch, T_b, d) keys+values.\n",
" Returns (out (batch, T_a, d), attn (batch, n_heads, T_a, T_b)).\"\"\"\n",
" batch, T_a, d = A.shape\n",
" T_b = B.shape[1]\n",
" d_k = d // n_heads\n",
" # TODO 1: project: q = A @ Wq.T, k = B @ Wk.T, v = B @ Wv.T (each Linear is bias-free here)\n",
" q = k = v = None\n",
" # TODO 2: reshape each to (batch, n_heads, T, d_k) via view + transpose(1, 2)\n",
" # q -> (batch, n_heads, T_a, d_k); k, v -> (batch, n_heads, T_b, d_k)\n",
" # TODO 3: scores = q @ k.transpose(-2, -1) / sqrt(d_k) -> (batch, n_heads, T_a, T_b)\n",
" scores = None\n",
" # TODO 4: attn = softmax over the LAST axis (the keys, T_b)\n",
" attn = None\n",
" # TODO 5: out = (attn @ v) -> (batch, n_heads, T_a, d_k); merge heads back to (batch, T_a, d)\n",
" out = None\n",
" attempted(q, scores, attn, out)\n",
" return out, attn\n",
"\n",
"def _cross_attn_checks():\n",
" torch.manual_seed(SEED)\n",
" batch, T_a, T_b, d, h = 2, 10, 20, 64, 8\n",
" A = torch.randn(batch, T_a, d); B = torch.randn(batch, T_b, d)\n",
" Wq, Wk, Wv = (nn.Linear(d, d, bias=False) for _ in range(3))\n",
" out, attn = cross_attention(A, B.clone(), Wq.weight, Wk.weight, Wv.weight, h)\n",
" check_shape(out, (batch, T_a, d))\n",
" check_shape(attn, (batch, h, T_a, T_b))\n",
" # attention rows are a distribution over the T_b keys\n",
" assert torch.allclose(attn.sum(-1), torch.ones(batch, h, T_a), atol=1e-5), \\\n",
" \"each query's attention must sum to 1 over the key positions\"\n",
"\n",
"check(\"16.7 cross-attention\", _cross_attn_checks)"
]
},
{
"cell_type": "markdown",
"id": "d83feb23",
"metadata": {},
"source": [
"Hint 1 (conceptual)It is ordinary multi-head attention, except `k` and `v` come from `B` while `q` comes from `A`. The attention matrix is `(T_a, T_b)`, not square, because the query and key sequences differ in length.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"q = (A @ Wq.T).view(batch, T_a, n_heads, d_k).transpose(1, 2) # (b, h, T_a, d_k)\n",
"k = (B @ Wk.T).view(batch, T_b, n_heads, d_k).transpose(1, 2) # (b, h, T_b, d_k)\n",
"v = (B @ Wv.T).view(batch, T_b, n_heads, d_k).transpose(1, 2)\n",
"scores = q @ k.transpose(-2, -1) / math.sqrt(d_k) # (b, h, T_a, T_b)\n",
"attn = F.softmax(scores, dim=-1)\n",
"out = (attn @ v).transpose(1, 2).reshape(batch, T_a, d)\n",
"```\n",
"\n",
"\n",
"Help: \"softmax over the wrong axis / rows don't sum to 1\"Softmax must be over the key axis (`dim=-1`, length `T_b`). Each query distributes its attention across the keys, so each row of the `(T_a, T_b)` matrix sums to 1.\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "0ed61e3c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.441965Z",
"iopub.status.busy": "2026-06-10T20:02:35.441901Z",
"iopub.status.idle": "2026-06-10T20:02:35.445325Z",
"shell.execute_reply": "2026-06-10T20:02:35.445015Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 16.7 cross-attention\n",
"[ ok ] Q from modality A, K/V from modality B: the (T_a, T_b) attention is the fusion point\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines cross_attention; the checks below re-verify the reference.\n",
"def cross_attention(A, B, Wq, Wk, Wv, n_heads):\n",
" batch, T_a, d = A.shape\n",
" T_b = B.shape[1]\n",
" d_k = d // n_heads\n",
" q = (A @ Wq.T).view(batch, T_a, n_heads, d_k).transpose(1, 2) # (b, h, T_a, d_k)\n",
" k = (B @ Wk.T).view(batch, T_b, n_heads, d_k).transpose(1, 2) # (b, h, T_b, d_k)\n",
" v = (B @ Wv.T).view(batch, T_b, n_heads, d_k).transpose(1, 2) # (b, h, T_b, d_k)\n",
" scores = q @ k.transpose(-2, -1) / math.sqrt(d_k) # (b, h, T_a, T_b)\n",
" attn = F.softmax(scores, dim=-1)\n",
" out = (attn @ v).transpose(1, 2).reshape(batch, T_a, d) # (b, T_a, d)\n",
" return out, attn\n",
"\n",
"check(\"16.7 cross-attention\", _cross_attn_checks, required=True)\n",
"print(\"[ ok ] Q from modality A, K/V from modality B: the (T_a, T_b) attention is the fusion point\")"
]
},
{
"cell_type": "markdown",
"id": "e38b2f7d",
"metadata": {},
"source": [
"> **Interpretation.** The attention matrix is `(T_a=10, T_b=20)`: rectangular, because text queries and image keys differ in count. Swap which modality supplies the queries and you have moved between the three fusion families in the draft. This single rectangular-attention op is the entire structural difference between a VLM and a unimodal transformer.\n"
]
},
{
"cell_type": "markdown",
"id": "69784e97",
"metadata": {},
"source": [
"## Safety lens, the typographic attack\n",
"\n",
"The same property that makes CLIP useful, it reads text rendered inside an image and treats it as evidence about content, is exploitable. Goh et al. 2021 found a \"Spider-Man neuron\" that fires on photographs of Spider-Man, drawings of it, *and* the word \"spider\" written on paper. So you can fool CLIP into calling an apple an iPod by writing \"iPod\" on it. We reproduce the mechanism in the synthetic shared space: a clean \"apple\" image gets a small push toward the \"iPod\" text direction, the way rendered text nudges the real image encoder, and watch the prediction flip.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "bc3b82e2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.446169Z",
"iopub.status.busy": "2026-06-10T20:02:35.446094Z",
"iopub.status.idle": "2026-06-10T20:02:35.449407Z",
"shell.execute_reply": "2026-06-10T20:02:35.449143Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"clean image classified as: class 0\n",
"after writing 'iPod' on it: class 8\n",
"[ ok ] a hostile label rendered in the image overrode the visual content\n"
]
}
],
"source": [
"# typographic attack, mechanism reproduced in the synthetic space\n",
"APPLE, IPOD = 0, 8 # two class prototype indices standing in for 'apple' and 'iPod'\n",
"torch.manual_seed(SEED)\n",
"clean = F.normalize(prototypes[APPLE] + 0.10 * torch.randn(EMBED), dim=-1) # a clean apple image\n",
"clean_pred = zero_shot_classify(clean.unsqueeze(0), prompt_features).item()\n",
"\n",
"# the attack: push the image embedding toward the 'iPod' text direction (rendered-text injection)\n",
"alpha = 1.5 # attack strength: rendered \"iPod\" text outweighs the visual evidence (sim_iPod > sim_apple)\n",
"attacked = F.normalize(clean + alpha * prototypes[IPOD], dim=-1)\n",
"attacked_pred = zero_shot_classify(attacked.unsqueeze(0), prompt_features).item()\n",
"print(f\"clean image classified as: class {clean_pred}\")\n",
"print(f\"after writing 'iPod' on it: class {attacked_pred}\")\n",
"assert clean_pred == APPLE and attacked_pred == IPOD, \\\n",
" \"the typographic push should flip the prediction from apple to iPod\"\n",
"print(\"[ ok ] a hostile label rendered in the image overrode the visual content\")"
]
},
{
"cell_type": "markdown",
"id": "5c73ec78",
"metadata": {},
"source": [
"> **Caveat:** this is the *mechanism*, not the real exploit. The point is structural: because CLIP fuses rendered text into the same embedding space as visual content, text in an image is a control channel. The deployed version, indirect prompt injection through an image (Greshake et al. 2023), jailbreaks production VLMs the same way: the vision encoder turns \"ignore previous instructions\" rendered as pixels into tokens the LM trusts. The habit to adopt: run OCR with a safety filter on uploaded images *before* they reach your model.\n"
]
},
{
"cell_type": "markdown",
"id": "609ef329",
"metadata": {},
"source": [
"### 4.3 Appendix: a real finetune (GPU-fenced)\n",
"\n",
"A real ViT finetune (more layers, full FashionMNIST, many epochs, or loading a pretrained backbone) is an hour-scale job that belongs on a GPU. Per the spec, GPU sections print why they skipped and what they would show; they never error on CPU. On CPU this cell does nothing but report. On CUDA it runs a short higher-capacity finetune.\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "dbc103e3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.450121Z",
"iopub.status.busy": "2026-06-10T20:02:35.450058Z",
"iopub.status.idle": "2026-06-10T20:02:35.452875Z",
"shell.execute_reply": "2026-06-10T20:02:35.452458Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"skipped: needs a CUDA GPU.\n",
"It would train a d_model=128, 6-layer ViT on all 60,000 FashionMNIST images for\n",
"5 epochs and reach ~90% test accuracy, vs the ~80% the tiny CPU run reaches on 6k images.\n"
]
}
],
"source": [
"# gpu-only: a heavier finetune; prints-and-skips on CPU, never errors\n",
"if device == \"cuda\":\n",
" torch.manual_seed(SEED)\n",
" big = ViT(d_model=128, n_heads=8, n_layers=6).to(device)\n",
" opt_g = torch.optim.AdamW(big.parameters(), lr=3e-4)\n",
" full_loader = DataLoader(train_full, batch_size=256, shuffle=True,\n",
" generator=torch.Generator().manual_seed(SEED))\n",
" big.train()\n",
" for epoch in range(5):\n",
" for xb, yb in full_loader:\n",
" xb, yb = xb.to(device), yb.to(device)\n",
" opt_g.zero_grad()\n",
" F.cross_entropy(big(xb), yb).backward()\n",
" opt_g.step()\n",
" g_loss, g_acc = evaluate(big, test_loader)\n",
" print(f\"GPU finetune: test loss {g_loss:.3f} . test acc {g_acc:.1%}\")\n",
"else:\n",
" print(\"skipped: needs a CUDA GPU.\")\n",
" print(\"It would train a d_model=128, 6-layer ViT on all 60,000 FashionMNIST images for\")\n",
" print(\"5 epochs and reach ~90% test accuracy, vs the ~80% the tiny CPU run reaches on 6k images.\")"
]
},
{
"cell_type": "markdown",
"id": "599249c8",
"metadata": {},
"source": [
"## Test yourself\n",
"\n",
"Three parts: concept self-checks, two auto-checked problems, and a capstone. Solutions are folded; try before you peek. Every answer is in this notebook; if unsure, re-run that section.\n"
]
},
{
"cell_type": "markdown",
"id": "e6a31cc5",
"metadata": {},
"source": [
"### Part A, Concepts\n",
"\n",
"1. How many tokens does ViT-B/16 feed its transformer for a `224x224` image, and why that number? Answer197: `(224/16)^2 = 196` patches plus one `[CLS]` token. You computed this in Exercise 16.1.\n",
"2. The patch embedding is a single `Conv2d`. What linear map is it equivalent to? AnswerFlatten each `C*P*P` patch and apply one `Linear(C*P*P, d_model)` whose weight is the conv kernel reshaped to `(d_model, C*P*P)`. You asserted this equivalence to `1e-5` in section 1.2.\n",
"3. Why does a ViT block drop the causal mask that a GPT block has? AnswerVision is not autoregressive. Every patch may attend to every patch in both directions, so the attention matrix is full (rows sum to 1 over all keys, upper triangle non-zero), as the falsification cell in 1.4 asserted.\n",
"4. In one sentence, what is CLIP's training loss? AnswerSymmetric softmax cross-entropy on the scaled image-text similarity matrix: the diagonal (matching pairs) should win its row and its column. It is InfoNCE with in-batch negatives.\n",
"5. Why must CLIP L2-normalize its encoder outputs before the similarity matrix? AnswerCosine similarity is direction-only; without normalization the dot product is dominated by magnitude, not direction, and the loss optimizes the wrong quantity. Section 3.2 showed two same-direction vectors have cosine 1 regardless of length.\n",
"6. The broken run in section 2.3 learns noticeably worse than the correct one without crashing. What single missing line causes it, and what is the one-cell diagnostic? AnswerA missing `opt.zero_grad()`: PyTorch accumulates gradients, so the effective gradient grows every iteration; at a moderate learning rate the optimizer rescales it enough that the run degrades quietly rather than diverging. The diagnostic: a second `backward()` without zeroing roughly doubles the gradient norm, which the probe cell asserted.\n",
"7. In cross-attention, what are the dimensions of the attention matrix when text (10 tokens) queries image features (20 tokens)? Answer`(10, 20)`: one row per query (text) token, one column per key (image) token. Rectangular because the two sequences differ in length, as Exercise 16.7 verified.\n",
"8. Zero-shot CLIP turns what object into your classifier? AnswerThe text encoder. Encode `\"a photo of a {class}\"` for each class to get class vectors, then classify by nearest cosine similarity. No training, open vocabulary.\n",
"9. Why does a 2-layer MLP suffice as the projector in LLaVA, but BLIP-2 needs a Q-Former? AnswerCLIP's vision encoder is already aligned to language by contrastive training, so the projector only does a coordinate change. A non-aligned encoder (DINO, EVA-CLIP) needs the Q-Former's heavier learned cross-attention to align as well as project.\n",
"10. What makes the typographic attack work, mechanistically? AnswerCLIP fuses text rendered in an image into the same embedding space as visual content, because in web data, text in images is a strong content signal. So a hostile label written on an object becomes a control channel that can override the visual evidence, as section's safety cell flipped apple to iPod.\n"
]
},
{
"cell_type": "markdown",
"id": "141be5fb",
"metadata": {},
"source": [
"### Part B, Auto-checked problems\n",
"\n",
"#### B1, Image-to-text retrieval accuracy\n",
"`Difficulty 2/5 · ~8 min`\n",
"\n",
"Write `retrieval_accuracy(image_features, text_features)`: given normalized `(N, d)` features for N matched pairs, return the fraction of images whose most-similar caption is their own (top-1 image-to-text retrieval). This is what CLIP's row-wise accuracy measures. The check uses the aligned batch from 16.4 (should be ~1.0) and a shuffled batch (should be near chance).\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "e07e23d6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.453574Z",
"iopub.status.busy": "2026-06-10T20:02:35.453509Z",
"iopub.status.idle": "2026-06-10T20:02:35.456893Z",
"shell.execute_reply": "2026-06-10T20:02:35.456444Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] B1 retrieval accuracy: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def retrieval_accuracy(image_features, text_features):\n",
" \"\"\"Fraction of images whose argmax-similarity caption is the matching (diagonal) one.\"\"\"\n",
" N = image_features.shape[0]\n",
" # TODO 1: sims = image_features @ text_features.T -> (N, N)\n",
" # TODO 2: preds = argmax over the caption axis (dim=-1)\n",
" # TODO 3: accuracy = mean(preds == arange(N))\n",
" acc = None\n",
" attempted(acc)\n",
" return acc\n",
"\n",
"def _retrieval_checks():\n",
" I, T = aligned_batch(N=16, d=32, noise=0.05, rng_seed=SEED)\n",
" acc_aligned = retrieval_accuracy(I, T)\n",
" assert acc_aligned > 0.9, f\"aligned pairs should retrieve near-perfectly, got {acc_aligned:.2f}\"\n",
" # break the alignment: shuffle text rows so the diagonal is no longer the match\n",
" perm = torch.randperm(16, generator=torch.Generator().manual_seed(SEED + 3))\n",
" acc_shuffled = retrieval_accuracy(I, T[perm])\n",
" assert acc_shuffled < 0.5, f\"shuffled pairs should retrieve poorly, got {acc_shuffled:.2f}\"\n",
"\n",
"check(\"B1 retrieval accuracy\", _retrieval_checks)"
]
},
{
"cell_type": "markdown",
"id": "9fa22f73",
"metadata": {},
"source": [
"Hint 1Reuse the `zero_shot_classify` pattern: the prediction for image `i` is `argmax(sims[i])`, and it is correct when it equals `i`.\n",
"Solution\n",
"\n",
"```python\n",
"def retrieval_accuracy(image_features, text_features):\n",
" N = image_features.shape[0]\n",
" preds = (image_features @ text_features.T).argmax(dim=-1)\n",
" return (preds == torch.arange(N)).float().mean().item()\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "4b6d0a88",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.457703Z",
"iopub.status.busy": "2026-06-10T20:02:35.457630Z",
"iopub.status.idle": "2026-06-10T20:02:35.460503Z",
"shell.execute_reply": "2026-06-10T20:02:35.460174Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] B1 retrieval accuracy\n",
"aligned retrieval accuracy: 100.0%\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines retrieval_accuracy; the checks below re-verify the reference.\n",
"def retrieval_accuracy(image_features, text_features):\n",
" N = image_features.shape[0]\n",
" preds = (image_features @ text_features.T).argmax(dim=-1)\n",
" return (preds == torch.arange(N)).float().mean().item()\n",
"\n",
"check(\"B1 retrieval accuracy\", _retrieval_checks, required=True)\n",
"I, T = aligned_batch(N=16, d=32, noise=0.05, rng_seed=SEED)\n",
"print(f\"aligned retrieval accuracy: {retrieval_accuracy(I, T):.1%}\")"
]
},
{
"cell_type": "markdown",
"id": "64a5f2ee",
"metadata": {},
"source": [
"#### B2, CLS-token pooling vs mean pooling\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"The image encoder reads the `[CLS]` token; the text encoder in the draft's lab mean-pools instead. Write `pool(x, mode, cls_index=0)` for a sequence `x` of shape `(B, T, d)`: `mode=\"cls\"` returns the token at `cls_index` (shape `(B, d)`), `mode=\"mean\"` returns the mean over the token axis (shape `(B, d)`). Raise `ValueError` for any other mode. The check verifies both shapes and that mean pooling equals the hand-computed average on a toy input.\n"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "bcc09ae1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.461322Z",
"iopub.status.busy": "2026-06-10T20:02:35.461250Z",
"iopub.status.idle": "2026-06-10T20:02:35.464768Z",
"shell.execute_reply": "2026-06-10T20:02:35.464531Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] B2 pooling: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def pool(x, mode, cls_index=0):\n",
" \"\"\"x: (B, T, d). mode 'cls' -> x[:, cls_index]; mode 'mean' -> mean over tokens. Else ValueError.\"\"\"\n",
" # TODO 1: if mode == \"cls\": return the token at cls_index, shape (B, d)\n",
" # TODO 2: if mode == \"mean\": return the mean over the token axis (dim=1), shape (B, d)\n",
" # TODO 3: otherwise raise ValueError with a helpful message\n",
" result = None\n",
" attempted(result)\n",
" return result\n",
"\n",
"def _pool_checks():\n",
" B, T, d = 2, 5, 4\n",
" x = torch.arange(B * T * d, dtype=torch.float).reshape(B, T, d)\n",
" check_shape(pool(x, \"cls\"), (B, d))\n",
" check_shape(pool(x, \"mean\"), (B, d))\n",
" # hand-computed mean of the first example, tokens 0..4: arithmetic mean of an arange step\n",
" expected_mean0 = x[0].mean(dim=0)\n",
" check_close(pool(x, \"mean\")[0], expected_mean0, msg=\"mean pooling should average over tokens\")\n",
" # cls picks token 0 exactly\n",
" assert torch.equal(pool(x, \"cls\"), x[:, 0]), \"cls pooling should return token 0\"\n",
" try:\n",
" pool(x, \"max\")\n",
" except ValueError:\n",
" return\n",
" raise AssertionError(\"pool should raise ValueError for an unknown mode\")\n",
"\n",
"check(\"B2 pooling\", _pool_checks)"
]
},
{
"cell_type": "markdown",
"id": "ba774f57",
"metadata": {},
"source": [
"Hint 1`x[:, cls_index]` indexes the token axis and drops it, giving `(B, d)`. `x.mean(dim=1)` averages over the token axis.\n",
"Solution\n",
"\n",
"```python\n",
"def pool(x, mode, cls_index=0):\n",
" if mode == \"cls\":\n",
" return x[:, cls_index]\n",
" if mode == \"mean\":\n",
" return x.mean(dim=1)\n",
" raise ValueError(f\"unknown pooling mode {mode!r}; use 'cls' or 'mean'\")\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "8bb7adee",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T20:02:35.465498Z",
"iopub.status.busy": "2026-06-10T20:02:35.465426Z",
"iopub.status.idle": "2026-06-10T20:02:35.468571Z",
"shell.execute_reply": "2026-06-10T20:02:35.468104Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] B2 pooling\n",
"[ ok ] cls pooling and mean pooling both return (B, d)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines pool; the checks below re-verify the reference.\n",
"def pool(x, mode, cls_index=0):\n",
" if mode == \"cls\":\n",
" return x[:, cls_index]\n",
" if mode == \"mean\":\n",
" return x.mean(dim=1)\n",
" raise ValueError(f\"unknown pooling mode {mode!r}; use 'cls' or 'mean'\")\n",
"\n",
"check(\"B2 pooling\", _pool_checks, required=True)\n",
"print(\"[ ok ] cls pooling and mean pooling both return (B, d)\")"
]
},
{
"cell_type": "markdown",
"id": "71e81662",
"metadata": {},
"source": [
"### Part C, Capstone: TinyCLIP on FashionMNIST captions\n",
"\n",
"Build a tiny CLIP that aligns FashionMNIST images with their class captions, then do zero-shot classification with held-out class prompts. You already have every piece: the `ViT` image encoder (drop the classification head, project to an embedding), a small text encoder over the caption `\"a photo of a {class}\"`, the `clip_loss`, and `zero_shot_classify`.\n",
"\n",
"Deliverables:\n",
"1. An image encoder that outputs L2-normalized `(B, embed_dim)` features (reuse `PatchEmbedding` + `Block`, replace the head with a projection + normalize).\n",
"2. A character-level text encoder over the 10 class captions, mean-pooled and normalized.\n",
"3. A training loop using `clip_loss` with the temperature clamped to `[0, log(100)]`.\n",
"4. Zero-shot accuracy on a held-out test split, using the 10 class-caption features as the classifier.\n",
"\n",
"Self-assessment (pass / partial / fail):\n",
"- (a) both encoders output unit-norm features (assert `feat.norm(dim=-1) == 1`);\n",
"- (b) the training loss drops below `log(10) = 2.30` (better than uniform over 10 classes);\n",
"- (c) zero-shot accuracy clears 50% (well above the 10% chance line; the tiny scale will not reach real-CLIP numbers);\n",
"- (d) you clamp `log_temperature` so it does not collapse;\n",
"- (e) the notebook runs top-to-bottom.\n",
"\n",
"My solution (reference, ~1-2 min on CPU at this scale)\n",
"\n",
"```python\n",
"CHAR_VOCAB = \" abcdefghijklmnopqrstuvwxyz0123456789-\"\n",
"CHAR_TO_ID = {c: i + 1 for i, c in enumerate(CHAR_VOCAB)} # 0 = pad\n",
"CAP_LEN = 24\n",
"\n",
"def encode_caption(name):\n",
" text = f\"a photo of a {name.lower()}\"\n",
" ids = [CHAR_TO_ID.get(c, 0) for c in text][:CAP_LEN]\n",
" ids = ids + [0] * (CAP_LEN - len(ids))\n",
" return torch.tensor(ids, dtype=torch.long)\n",
"\n",
"class ImgEncoder(nn.Module):\n",
" def __init__(self, d_model=64, embed=128, n_layers=3):\n",
" super().__init__()\n",
" self.pe = PatchEmbedding(28, 7, 1, d_model)\n",
" self.cls = nn.Parameter(torch.zeros(1, 1, d_model))\n",
" self.pos = nn.Parameter(torch.zeros(1, self.pe.n_patches + 1, d_model))\n",
" self.blocks = nn.ModuleList([Block(d_model, 4) for _ in range(n_layers)])\n",
" self.ln = nn.LayerNorm(d_model); self.proj = nn.Linear(d_model, embed)\n",
" nn.init.trunc_normal_(self.cls, std=0.02); nn.init.trunc_normal_(self.pos, std=0.02)\n",
" def forward(self, x):\n",
" x = self.pe(x); x = torch.cat([self.cls.expand(x.shape[0], -1, -1), x], 1) + self.pos\n",
" for b in self.blocks: x = b(x)\n",
" return F.normalize(self.proj(self.ln(x[:, 0])), dim=-1)\n",
"\n",
"class TxtEncoder(nn.Module):\n",
" def __init__(self, vocab=len(CHAR_VOCAB) + 1, d_model=64, embed=128, n_layers=3):\n",
" super().__init__()\n",
" self.tok = nn.Embedding(vocab, d_model); self.pos = nn.Embedding(CAP_LEN, d_model)\n",
" self.blocks = nn.ModuleList([Block(d_model, 4) for _ in range(n_layers)])\n",
" self.ln = nn.LayerNorm(d_model); self.proj = nn.Linear(d_model, embed)\n",
" def forward(self, t):\n",
" x = self.tok(t) + self.pos(torch.arange(t.shape[1], device=t.device))\n",
" for b in self.blocks: x = b(x)\n",
" return F.normalize(self.proj(self.ln(x.mean(1))), dim=-1)\n",
"\n",
"torch.manual_seed(SEED)\n",
"img_enc, txt_enc = ImgEncoder(), TxtEncoder()\n",
"log_temp = nn.Parameter(torch.tensor(2.65)) # log(1/0.07)\n",
"caps = torch.stack([encode_caption(n) for n in CLASS_NAMES]) # (10, CAP_LEN)\n",
"opt = torch.optim.AdamW(list(img_enc.parameters()) + list(txt_enc.parameters()) + [log_temp], lr=3e-4)\n",
"\n",
"img_enc.train(); txt_enc.train()\n",
"for epoch in range(1 if FAST else 4):\n",
" for xb, yb in train_loader:\n",
" opt.zero_grad()\n",
" i_feat = img_enc(xb) # (B, 128)\n",
" t_feat = txt_enc(caps[yb]) # caption for each image's true class\n",
" loss = clip_loss(i_feat, t_feat, log_temp)\n",
" loss.backward(); opt.step()\n",
" with torch.no_grad(): log_temp.clamp_(0.0, math.log(100.0))\n",
"print(f\"final clip loss {loss.item():.3f}\")\n",
"\n",
"# zero-shot: the 10 class captions ARE the classifier\n",
"img_enc.eval(); txt_enc.eval()\n",
"with torch.no_grad():\n",
" class_feats = txt_enc(caps) # (10, 128)\n",
" correct = total = 0\n",
" for xb, yb in test_loader:\n",
" preds = zero_shot_classify(img_enc(xb), class_feats)\n",
" correct += (preds == yb).sum().item(); total += yb.numel()\n",
"print(f\"zero-shot accuracy: {correct / total:.1%}\")\n",
"```\n",
"At this scale TinyCLIP reaches roughly 55-70% zero-shot, far above 10% chance and far below a real CLIP. The lesson is the mechanism, not the number: the same shared-space loss that needs 400M pairs to shine works in miniature here.\n"
]
},
{
"cell_type": "markdown",
"id": "8964eebf",
"metadata": {},
"source": [
"## Reflection\n",
"\n",
"Write ~150 words on the dumbest bug you hit in this notebook and how you found it. Maybe your `seq_len` returned 4 instead of 16 because you forgot to square the grid. Maybe your cross-attention rows did not sum to 1 because you softmaxed over the query axis. Maybe you stared at a flat loss curve before remembering `zero_grad`. Nobody grades this. Writing it is the point: the debugging move you name here is the one you will reach for next time, and naming the symptom-to-cause link is most of what separates someone who fixes ViT shape bugs in a minute from someone who burns an afternoon on them.\n"
]
},
{
"cell_type": "markdown",
"id": "9e4b148c",
"metadata": {},
"source": [
"## Going further\n",
"\n",
"- Dosovitskiy et al. 2020, *An Image Is Worth 16x16 Words* - the ViT paper. The whole architecture you built, at ImageNet scale.\n",
"- Radford et al. 2021, *Learning Transferable Visual Models From Natural Language Supervision* (CLIP) - the contrastive recipe and the zero-shot results.\n",
"- lucidrains `vit-pytorch` - Phil Wang's reference implementations of every ViT variant; read it to see how the patterns recombine.\n",
"- Goh et al. 2021, *Multimodal Neurons in Artificial Neural Networks* (Distill) - the Spider-Man neuron and the typographic attack you reproduced in miniature.\n",
"- Liu et al. 2023, *Visual Instruction Tuning* (LLaVA) - the frozen-CLIP + MLP-projector + LM recipe that is the cheap default for VLMs.\n",
"- Alayrac et al. 2022, *Flamingo* - the Perceiver resampler and gated cross-attention for interleaved image-text.\n",
"\n",
"## What this enables\n",
"\n",
"- **Ch 17 - Efficient Inference**: VLM context is dominated by image tokens (256 per image vs ~10-20 text tokens per turn), so KV cache for multimodal models is mostly visual. The architecture here is what determines that cost.\n",
"- **Ch 18 - Generative Models**: Stable Diffusion conditions its UNet on text through the exact `Q <- latent, K,V <- text` cross-attention you implemented in 16.6. You now know where the text reaches in.\n",
"- **Ch 24 - Safety + Red-Team**: typographic attacks, visual prompt injection, and adversarial-image jailbreaks are the deep-dive. The mechanism you flipped from apple to iPod is where it starts.\n",
"\n",
"We trained a 215k-param ViT to ~80% on a FashionMNIST subset. A real ViT-B/16 reaches ~84% top-1 on ImageNet's 1000 classes, and only after JFT-300M pretraining: the lesson that transformers trade inductive bias for data. The gap between our run and that one is entirely data and scale, not architecture.\n"
]
},
{
"cell_type": "markdown",
"id": "98d56947",
"metadata": {},
"source": [
"---\n",
"*Built top-to-bottom. If every check above printed `[ ok ]`, you've reproduced the chapter. Total running time and last-verified stamp written by CI.*\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "obvix-nb",
"language": "python",
"name": "obvix-nb"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"obvix": {
"title": "Ch 16: Multimodal Transformers"
}
},
"nbformat": 4,
"nbformat_minor": 5
}