{
"cells": [
{
"cell_type": "markdown",
"id": "b44e9a52",
"metadata": {},
"source": [
"# Ch 10 — PyTorch Foundations (notebook)\n",
"\n",
"`[← 09 intro-to-neural-networks]` · **this notebook** · `[11 training-deep-networks →]`\n",
"\n",
"Runs top-to-bottom in ~5 min on free Colab CPU. Last verified 2026-06-11.\n",
"\n",
"**What you'll build**\n",
"- A from-scratch `Linear` layer that you verify, element for element, against `torch.nn.Linear`.\n",
"- A from-scratch `cross_entropy` checked against `F.cross_entropy` to `1e-6`, and a from-scratch `SGD` checked against `torch.optim.SGD`.\n",
"- A `Dataset` + `DataLoader` over real FashionMNIST images, then the four-line training loop that turns them into a classifier above 80% accuracy.\n",
"- One training run you break on purpose (the forgotten `zero_grad`) and then fix, so the bug becomes muscle memory instead of a surprise.\n",
"\n",
"**How this notebook works.** Code cells with a `# TODO` are yours to fill in. Run the cell to grade yourself: `[ ok ]` passed, `[FAIL]` shows what went wrong, `[ -- ]` means not attempted yet. Every exercise has a hint ladder (open only as many as you need) and a folded solution below it. The notebook runs top-to-bottom even if you fill in nothing, because the solution cells redefine the pieces the later cells depend on. See Ch 00 for the full protocol.\n"
]
},
{
"cell_type": "markdown",
"id": "fccd6eb4",
"metadata": {},
"source": [
"## Before you start\n",
"\n",
"1. In micrograd (Ch 09), what did `.backward()` actually do to each `Value`? AnswerIt walked the computation graph in reverse topological order and accumulated, into each node's `.grad`, the partial derivative of the final scalar with respect to that node. PyTorch's `.backward()` is the same idea, scaled to n-dimensional tensors and a hundred operations.\n",
"2. Two arrays of shape `(3, 1, 5)` and `(4, 5)` are added. What shape comes out, or does it error? Answer`(3, 4, 5)`. Right-justify the shapes: `(3,1,5)` over `(_,4,5)`. Each axis is equal or one side is 1, so it broadcasts. The size-1 axes are stretched. Same rule as NumPy.\n",
"3. Predict before you run: a `nn.Linear(4, 8)` has how many learnable numbers? Answer40. The weight is `(8, 4)` = 32 numbers, the bias is `(8,)` = 8 more. Always `out*in + out`. We assert this in Part 4.\n"
]
},
{
"cell_type": "markdown",
"id": "a38eae21",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Three setup cells, no installs: Colab and a standard local environment already\n",
"have torch, torchvision, numpy, and matplotlib.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "61d391ef",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:55.579726Z",
"iopub.status.busy": "2026-06-10T19:01:55.579648Z",
"iopub.status.idle": "2026-06-10T19:01:57.170045Z",
"shell.execute_reply": "2026-06-10T19:01:57.169522Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"numpy 2.2.6 · torch 2.12.0+cpu · torchvision 0.27.0+cpu · device cpu\n"
]
}
],
"source": [
"import numpy as np\n",
"import torch\n",
"import torchvision\n",
"import matplotlib.pyplot as plt\n",
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
"print(f\"numpy {np.__version__} · torch {torch.__version__} · torchvision {torchvision.__version__} · device {device}\")\n",
"if np.__version__ < \"2.0\":\n",
" print(\"WARN: written for NumPy 2.x; older versions may differ slightly\")\n",
"if torch.__version__ < \"2.0\":\n",
" print(\"WARN: written for torch 2.x; a few APIs (weights_only, set_to_none default) differ on 1.x\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "6b88334a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.171055Z",
"iopub.status.busy": "2026-06-10T19:01:57.170914Z",
"iopub.status.idle": "2026-06-10T19:01:57.177434Z",
"shell.execute_reply": "2026-06-10T19:01:57.176903Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"FAST=False · EPOCHS=3 · TRAIN_N=12000 · TEST_N=4000 · BATCH=128\n"
]
}
],
"source": [
"import os, random\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"SEED = 0\n",
"FAST = bool(os.environ.get(\"NB_FAST\")) # CI smoke mode: ~10x fewer steps, same code paths\n",
"EPOCHS = 1 if FAST else 3 # training-loop passes over the data\n",
"TRAIN_N = 2000 if FAST else 12000 # FashionMNIST training images we use (full set is 60000)\n",
"TEST_N = 1000 if FAST else 4000 # held-out images for the accuracy report\n",
"BATCH = 128 # batch size; fixed because it does not need to shrink for FAST\n",
"rng = np.random.default_rng(SEED) # the one numpy generator we pass around\n",
"torch.manual_seed(SEED); random.seed(SEED)\n",
"# CPU is the canonical target. We keep torch single-threaded so timings and the\n",
"# committed outputs are stable across machines (BLAS threading is the usual culprit).\n",
"torch.set_num_threads(1)\n",
"print(f\"FAST={FAST} · EPOCHS={EPOCHS} · TRAIN_N={TRAIN_N} · TEST_N={TEST_N} · BATCH={BATCH}\")\n",
"\n",
"# ── house self-check harness (identical across all chapter notebooks) ──\n",
"import numpy as _np\n",
"\n",
"def check(label, test_fn, required=False):\n",
" \"\"\"Run one self-check. test_fn raises AssertionError (with a teaching\n",
" message) on failure, NotImplementedError if the stub is unfilled.\n",
" required=True is used only in solution cells; it is what CI grades.\"\"\"\n",
" try:\n",
" test_fn()\n",
" except NotImplementedError:\n",
" if required:\n",
" raise AssertionError(f\"{label}: reference solution incomplete\")\n",
" print(f\"[ -- ] {label}: not attempted yet — fill in the TODO above, then re-run.\")\n",
" return False\n",
" except AssertionError as e:\n",
" if required:\n",
" raise\n",
" print(f\"[FAIL] {label}: {e}\")\n",
" return False\n",
" print(f\"[ ok ] {label}\")\n",
" return True\n",
"\n",
"def attempted(*vals):\n",
" \"\"\"Treat None placeholders as 'not attempted'.\"\"\"\n",
" if any(v is None for v in vals):\n",
" raise NotImplementedError\n",
"\n",
"def check_shape(x, want):\n",
" assert tuple(x.shape) == tuple(want), \\\n",
" f\"shape {tuple(x.shape)}, expected {tuple(want)} — check your reshape/transpose order\"\n",
"\n",
"def check_close(got, want, atol=1e-5, rtol=1e-4, msg=\"\"):\n",
" g, w = _np.asarray(got, dtype=float), _np.asarray(want, dtype=float)\n",
" assert g.shape == w.shape, f\"shape {g.shape} vs expected {w.shape}. {msg}\"\n",
" bad = ~_np.isclose(g, w, atol=atol, rtol=rtol)\n",
" assert not bad.any(), \\\n",
" f\"{bad.mean():.2%} of values wrong (max diff {abs(g - w).max():.3g}). {msg}\""
]
},
{
"cell_type": "markdown",
"id": "ea508f25",
"metadata": {},
"source": [
"> **Note:** seeds make this notebook's printed numbers reproduce on CPU. Library versions and BLAS threading can shift the last digit or two; quoted numbers hold for the pinned environment. If your final accuracy is 0.842 and the page says 0.845, you did nothing wrong.\n"
]
},
{
"cell_type": "markdown",
"id": "87da834c",
"metadata": {},
"source": [
"## The map\n",
"\n",
"> **Part 1 — Tensors as the API surface.** Make tensors four ways; read `dtype`, `device`, `requires_grad`; watch `from_numpy` share memory.\n",
"> **Part 2 — Operations and broadcasting.** Broadcasting rules, the `view`/`reshape`/`contiguous` distinction, and `einsum` checked against `@`.\n",
"> **Part 3 — Autograd.** Build a graph, call `.backward()`, verify the gradient against calculus; see accumulation, `no_grad`, and `detach`.\n",
"> **Part 4 — nn.Module.** Parameters vs buffers; reimplement `nn.Linear` from scratch and prove it matches the real one elementwise.\n",
"> **Part 5 — Dataset and DataLoader.** Wrap FashionMNIST in a `Dataset`, batch it with a `DataLoader`, and look at the images.\n",
"> **Part 6 — The training loop.** The four lines that matter, a from-scratch `SGD` and `cross_entropy` checked against torch, a model trained to >80%, and a bug staged then fixed.\n"
]
},
{
"cell_type": "markdown",
"id": "6e2a786c",
"metadata": {},
"source": [
"## Part 1 — Tensors as the API surface\n",
"\n",
"> **Objectives.** Create tensors the four standard ways. Read the three fields that\n",
"> distinguish a tensor from a NumPy array: `dtype`, `device`, `requires_grad`.\n",
"> See that `torch.from_numpy` shares memory, and that `requires_grad` propagates\n",
"> through operations.\n",
"\n",
"A PyTorch tensor is a multi-dimensional array with a few extra fields: `dtype` (the\n",
"element type), `device` (CPU or which GPU), `requires_grad` (whether autograd\n",
"tracks operations on it), and `grad` (the accumulated gradient that `.backward()`\n",
"fills in). Everything else in the framework is bookkeeping around tensor operations.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "e44e6334",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.178711Z",
"iopub.status.busy": "2026-06-10T19:01:57.178579Z",
"iopub.status.idle": "2026-06-10T19:01:57.182154Z",
"shell.execute_reply": "2026-06-10T19:01:57.181803Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"a (2, 2) torch.float32\n",
"c (2, 3, 4) torch.float32\n",
"d [0, 2, 4, 6, 8] torch.int64\n",
"f [1.0, 2.0, 3.0] torch.float64\n"
]
}
],
"source": [
"a = torch.tensor([[1.0, 2.0], [3.0, 4.0]]) # 1. from Python data\n",
"b = torch.zeros(3, 4) # 2. zeros / ones / arange / linspace\n",
"c = torch.randn(2, 3, 4) # standard normal, shape (2,3,4)\n",
"d = torch.arange(0, 10, step=2) # like np.arange -> [0,2,4,6,8]\n",
"e = torch.zeros_like(a) # 3. match an existing tensor's shape+dtype\n",
"f = torch.from_numpy(np.array([1.0, 2.0, 3.0])) # 4. from numpy (shares memory!)\n",
"print(\"a\", tuple(a.shape), a.dtype)\n",
"print(\"c\", tuple(c.shape), c.dtype)\n",
"print(\"d\", d.tolist(), d.dtype)\n",
"print(\"f\", f.tolist(), f.dtype)"
]
},
{
"cell_type": "markdown",
"id": "d128aed8",
"metadata": {},
"source": [
"> **Notice that** floating-point literals give `torch.float32` and integer ranges give `torch.int64`. Those are the defaults you will see everywhere; mismatched dtypes are a common source of silent bugs (an int tensor where a model wants floats).\n"
]
},
{
"cell_type": "markdown",
"id": "36a87a3e",
"metadata": {},
"source": [
"`torch.from_numpy` does not copy. The tensor and the array point at the same\n",
"memory, so mutating one mutates the other. This is a feature when you want zero-copy\n",
"interop and a footgun when you forget.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "c01bf940",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.183104Z",
"iopub.status.busy": "2026-06-10T19:01:57.183024Z",
"iopub.status.idle": "2026-06-10T19:01:57.185409Z",
"shell.execute_reply": "2026-06-10T19:01:57.184866Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"after mutating the numpy array, the tensor sees: [999.0, 20.0, 30.0]\n",
"[ ok ] from_numpy and the array share one buffer\n"
]
}
],
"source": [
"arr = np.array([10.0, 20.0, 30.0])\n",
"t = torch.from_numpy(arr)\n",
"arr[0] = 999.0 # mutate the numpy side\n",
"print(\"after mutating the numpy array, the tensor sees:\", t.tolist())\n",
"assert t[0].item() == 999.0, \\\n",
" \"from_numpy shares memory; if this fails your torch build copied instead of viewing\"\n",
"print(\"[ ok ] from_numpy and the array share one buffer\")"
]
},
{
"cell_type": "markdown",
"id": "f8177bd7",
"metadata": {},
"source": [
"> **Common confusion:** `torch.tensor(arr)` *copies*; `torch.from_numpy(arr)` *shares*. If you want independence, copy explicitly with `torch.tensor(arr)` or `t.clone()`.\n"
]
},
{
"cell_type": "markdown",
"id": "0286a93d",
"metadata": {},
"source": [
"The `device` field says where the tensor lives. We are CPU-only here, but the\n",
"defensive idiom is to move things with `.to(device)`, which is a no-op when the\n",
"tensor is already there.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "b7f45caa",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.186943Z",
"iopub.status.busy": "2026-06-10T19:01:57.186775Z",
"iopub.status.idle": "2026-06-10T19:01:57.190086Z",
"shell.execute_reply": "2026-06-10T19:01:57.189736Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"x lives on: cpu\n",
"skipped: no GPU; .to(device) returned the same CPU tensor. On a GPU it would read 'cuda:0'.\n"
]
}
],
"source": [
"# gpu-only: on a CUDA box this would move x to the GPU; on CPU it is a no-op.\n",
"x = torch.randn(4, 4)\n",
"x = x.to(device) # safe to call unconditionally\n",
"print(f\"x lives on: {x.device}\")\n",
"if device == \"cuda\":\n",
" print(\"CUDA available; tensors above moved to GPU\")\n",
"else:\n",
" print(\"skipped: no GPU; .to(device) returned the same CPU tensor. On a GPU it would read 'cuda:0'.\")"
]
},
{
"cell_type": "markdown",
"id": "f860dbc9",
"metadata": {},
"source": [
"`requires_grad=True` is the flag that turns a tensor into an autograd *leaf*: a\n",
"node whose `.grad` will be filled in. Parameters (the things an optimizer updates)\n",
"have it set. Any tensor produced by an operation on a `requires_grad` input inherits\n",
"the flag automatically. We will lean on this hard in Part 3.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "ea6c852c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.191190Z",
"iopub.status.busy": "2026-06-10T19:01:57.191104Z",
"iopub.status.idle": "2026-06-10T19:01:57.193918Z",
"shell.execute_reply": "2026-06-10T19:01:57.193318Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"leaf.requires_grad: True\n",
"out.requires_grad : True (inferred from leaf)\n",
"out.grad_fn : \n",
"[ ok ] requires_grad propagated through the power op\n"
]
}
],
"source": [
"leaf = torch.randn(5, requires_grad=True) # a leaf: gradient will be populated\n",
"out = leaf ** 2 # not a leaf, but requires_grad inferred\n",
"print(\"leaf.requires_grad:\", leaf.requires_grad)\n",
"print(\"out.requires_grad :\", out.requires_grad, \"(inferred from leaf)\")\n",
"print(\"out.grad_fn :\", out.grad_fn) # the op that produced it\n",
"assert out.requires_grad and out.grad_fn is not None, \\\n",
" \"an op on a requires_grad tensor must itself require grad and carry a grad_fn\"\n",
"print(\"[ ok ] requires_grad propagated through the power op\")"
]
},
{
"cell_type": "markdown",
"id": "0750eccf",
"metadata": {},
"source": [
"### Exercise 10.1 — Construct a tensor to spec\n",
"`Difficulty 1/5 · ~5 min`\n",
"\n",
"Fill in `make_spec()` so it returns a tensor of shape `(3, 4)`, dtype `torch.float64`,\n",
"every entry equal to `7.0`. One line is enough. This is the \"can you read the\n",
"docs for `torch.full`/`torch.ones`\" warm-up; the check verifies shape, dtype, and value.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "3ebf5019",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.195280Z",
"iopub.status.busy": "2026-06-10T19:01:57.195103Z",
"iopub.status.idle": "2026-06-10T19:01:57.200094Z",
"shell.execute_reply": "2026-06-10T19:01:57.199707Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 10.1 make_spec: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def make_spec():\n",
" \"\"\"Return a (3,4) float64 tensor filled with 7.0.\"\"\"\n",
" # TODO 1: build the tensor. torch.full(size, fill_value, dtype=...) is the direct tool.\n",
" out = None\n",
" attempted(out)\n",
" return out\n",
"\n",
"def _check_spec():\n",
" t = make_spec()\n",
" check_shape(t, (3, 4))\n",
" assert t.dtype == torch.float64, f\"dtype {t.dtype}, expected torch.float64 — pass dtype=torch.float64\"\n",
" assert torch.all(t == 7.0), \"every entry should be 7.0\"\n",
"\n",
"check(\"10.1 make_spec\", _check_spec)"
]
},
{
"cell_type": "markdown",
"id": "15054241",
"metadata": {},
"source": [
"Hint 1 (conceptual)You want one constructor that takes a size, a fill value, and a dtype. `torch.full` is exactly that.\n",
"\n",
"Hint 2 (the line)`out = torch.full((3, 4), 7.0, dtype=torch.float64)`.\n",
"\n",
"Help — \"dtype torch.float32, expected torch.float64\"You built the tensor but did not request the dtype. Pass `dtype=torch.float64` to the constructor; do not rely on the default, which is float32 for floats.\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "0d78a01c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.201181Z",
"iopub.status.busy": "2026-06-10T19:01:57.201003Z",
"iopub.status.idle": "2026-06-10T19:01:57.205719Z",
"shell.execute_reply": "2026-06-10T19:01:57.205386Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 10.1 make_spec\n",
"tensor([[7., 7., 7., 7.],\n",
" [7., 7., 7., 7.],\n",
" [7., 7., 7., 7.]], dtype=torch.float64)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines make_spec; the check below re-verifies the reference.\n",
"def make_spec():\n",
" return torch.full((3, 4), 7.0, dtype=torch.float64)\n",
"\n",
"check(\"10.1 make_spec\", _check_spec, required=True)\n",
"print(make_spec())"
]
},
{
"cell_type": "markdown",
"id": "ff230901",
"metadata": {},
"source": [
"> **Key takeaways.** A tensor is an array plus `dtype`, `device`, `requires_grad`, `grad`. Float literals default to float32, int ranges to int64. `from_numpy` shares memory; `torch.tensor` copies. `requires_grad` flows through operations, and the producing op is recorded as `grad_fn`.\n"
]
},
{
"cell_type": "markdown",
"id": "dfa08eab",
"metadata": {},
"source": [
"## Part 2 — Operations and broadcasting\n",
"\n",
"> **Objectives.** Apply the broadcasting rule deliberately. Tell `view` from\n",
"> `reshape` from `contiguous` and know which one raises. Write `einsum` and prove\n",
"> it equals the `@` it replaces.\n",
"\n",
"Tensor operations broadcast exactly like NumPy. Right-justify the two shapes; each\n",
"axis must be equal or one of them must be 1; size-1 axes are stretched. The micro-demo\n",
"below uses tiny shapes so you can see the result in full.\n"
]
},
{
"cell_type": "markdown",
"id": "3a816526",
"metadata": {},
"source": [
"> **Predict:** what shape does `a + b` produce below? Work it out before running.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "392bb815",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.206758Z",
"iopub.status.busy": "2026-06-10T19:01:57.206676Z",
"iopub.status.idle": "2026-06-10T19:01:57.209345Z",
"shell.execute_reply": "2026-06-10T19:01:57.208799Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"a: (3, 1, 5) + b: (4, 5) -> (3, 4, 5)\n",
"[ ok ] broadcast to (3, 4, 5)\n"
]
}
],
"source": [
"a = torch.randn(3, 1, 5)\n",
"b = torch.randn( 4, 5) # right-justifies to (1, 4, 5)\n",
"out = a + b\n",
"print(\"a:\", tuple(a.shape), \"+ b:\", tuple(b.shape), \"->\", tuple(out.shape))\n",
"assert out.shape == (3, 4, 5), \"right-justify: (3,1,5) over (1,4,5) -> (3,4,5)\"\n",
"print(\"[ ok ] broadcast to (3, 4, 5)\")"
]
},
{
"cell_type": "markdown",
"id": "f1011e64",
"metadata": {},
"source": [
"> **Interpretation.** The size-1 axis of `a` (the middle) and the missing leading\n",
"axis of `b` (treated as 1) both stretch. No data is copied to do it; broadcasting is a\n",
"stride trick, which is why it is cheap.\n"
]
},
{
"cell_type": "markdown",
"id": "b215e13f",
"metadata": {},
"source": [
"The subtle trio is `view` vs `reshape` vs `contiguous`. `view` is a zero-copy\n",
"reinterpretation that *requires the tensor to be contiguous* in memory. `permute` and\n",
"`transpose` produce non-contiguous tensors, so `.view()` on their output raises. The\n",
"fixes are `.reshape()` (copies only if it must) or `.contiguous().view()` (an explicit\n",
"copy first). The deliberate failure below shows the exact error, then both fixes.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "e1e23ee4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.211102Z",
"iopub.status.busy": "2026-06-10T19:01:57.210925Z",
"iopub.status.idle": "2026-06-10T19:01:57.214316Z",
"shell.execute_reply": "2026-06-10T19:01:57.213875Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"y contiguous? False\n",
"view raised, as expected: view size is not compatible with input tensor's size and stride (at le\n",
"[ ok ] contiguous().view and reshape agree: (2, 12)\n"
]
}
],
"source": [
"x = torch.randn(2, 3, 4)\n",
"y = x.permute(0, 2, 1) # (2, 4, 3), now non-contiguous\n",
"print(\"y contiguous?\", y.is_contiguous())\n",
"try:\n",
" y.view(2, 12) # this is the staged failure\n",
"except RuntimeError as err:\n",
" print(\"view raised, as expected:\", str(err).splitlines()[0][:70])\n",
"z1 = y.contiguous().view(2, 12) # fix 1: explicit copy, then view\n",
"z2 = y.reshape(2, 12) # fix 2: reshape copies for you\n",
"assert z1.shape == z2.shape == (2, 12)\n",
"assert torch.equal(z1, z2), \"both fixes must produce the same numbers\"\n",
"print(\"[ ok ] contiguous().view and reshape agree:\", tuple(z1.shape))"
]
},
{
"cell_type": "markdown",
"id": "1c797353",
"metadata": {},
"source": [
"> **Common confusion:** most of the time `.reshape()` is the right call and you\n",
"need not think about it. Reach for `.view()` only when you want a guarantee that no\n",
"copy happened (performance-critical code); it will raise rather than silently copy.\n"
]
},
{
"cell_type": "markdown",
"id": "355ca8f4",
"metadata": {},
"source": [
"### Exercise 10.2 — Batched matmul with einsum\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"`einsum` is the most expressive operation in PyTorch: the string names the axes,\n",
"repeated letters are summed (contracted), and the right-hand side names what to keep.\n",
"Implement `batched_matmul(A, B)` for `A: (b, i, j)` and `B: (b, j, k)` returning\n",
"`(b, i, k)` using a single `torch.einsum`. The check compares it against `A @ B`, which\n",
"is the ground-truth oracle.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "37117978",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.215226Z",
"iopub.status.busy": "2026-06-10T19:01:57.215129Z",
"iopub.status.idle": "2026-06-10T19:01:57.219990Z",
"shell.execute_reply": "2026-06-10T19:01:57.219117Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 10.2 einsum batched matmul: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def batched_matmul(A, B):\n",
" \"\"\"A: (b, i, j), B: (b, j, k) -> (b, i, k), via one einsum.\n",
" The j axis is the contraction (it appears in both inputs, not in the output).\"\"\"\n",
" # TODO 1: write the einsum string. Same letter on both inputs = contract it.\n",
" # Keep b, i, k; sum over j.\n",
" out = None\n",
" attempted(out)\n",
" return out\n",
"\n",
"def _check_einsum():\n",
" A = torch.randn(5, 3, 4)\n",
" B = torch.randn(5, 4, 2)\n",
" got = batched_matmul(A, B)\n",
" check_shape(got, (5, 3, 2))\n",
" check_close(got, A @ B, atol=1e-5, msg=\"einsum must match the @ operator's batched matmul\")\n",
"\n",
"check(\"10.2 einsum batched matmul\", _check_einsum)"
]
},
{
"cell_type": "markdown",
"id": "64197686",
"metadata": {},
"source": [
"Hint 1 (conceptual)Three axes per input. `b` and `i` and `k` survive; `j` is the one shared between the two operands, so it is the axis you contract.\n",
"\n",
"Hint 2 (pseudocode)`out = torch.einsum(\"bij,bjk->bik\", A, B)`. Read it as: for each batch b, ordinary matrix multiply.\n",
"\n",
"Help — \"shape (5,3,2), expected ...\" or a contraction errorIf the letters on the output side include `j`, you forgot to contract it (it must NOT appear on the right of `->`). If the letters are in the wrong order, the output axes come out permuted.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "862b297f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.222757Z",
"iopub.status.busy": "2026-06-10T19:01:57.222630Z",
"iopub.status.idle": "2026-06-10T19:01:57.226195Z",
"shell.execute_reply": "2026-06-10T19:01:57.225893Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 10.2 einsum batched matmul\n",
"einsum result shape: (2, 2, 2)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines batched_matmul; the check below re-verifies against `@`.\n",
"def batched_matmul(A, B):\n",
" return torch.einsum(\"bij,bjk->bik\", A, B)\n",
"\n",
"check(\"10.2 einsum batched matmul\", _check_einsum, required=True)\n",
"A = torch.randn(2, 2, 3); B = torch.randn(2, 3, 2)\n",
"print(\"einsum result shape:\", tuple(batched_matmul(A, B).shape))"
]
},
{
"cell_type": "markdown",
"id": "94e343b4",
"metadata": {},
"source": [
"> **Key takeaways.** Broadcasting is right-justified, equal-or-1, and copy-free. `view` needs contiguity and raises otherwise; `reshape` copies when it must; `permute`/`transpose` break contiguity. `einsum` is the readable contraction: repeated index = summed, right-hand side = kept.\n"
]
},
{
"cell_type": "markdown",
"id": "addfec93",
"metadata": {},
"source": [
"## Part 3 — Autograd\n",
"\n",
"> **Objectives.** Build a computation graph, call `.backward()`, and verify the\n",
"> gradient against pencil-and-paper calculus. Then see the three behaviors that trip\n",
"> people up: gradients *accumulate*, `torch.no_grad()` stops graph-building, and\n",
"> `.detach()` cuts a tensor out of the graph.\n",
"\n",
"When you operate on `requires_grad` tensors, PyTorch records a directed acyclic graph.\n",
"Each output stores a `grad_fn` (its operation's backward rule) and remembers its\n",
"inputs. `.backward()` on a scalar walks that graph in reverse and accumulates the\n",
"partial derivatives into each leaf's `.grad`. The graph is dynamic, rebuilt every\n",
"forward pass, which is why ordinary Python `if`/`for` inside a model just works.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "e282c4d9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.227334Z",
"iopub.status.busy": "2026-06-10T19:01:57.227217Z",
"iopub.status.idle": "2026-06-10T19:01:57.231276Z",
"shell.execute_reply": "2026-06-10T19:01:57.230261Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"y.grad_fn: \n",
"x.grad: 16.0 (analytic 3*2^2 + 4 = 16)\n",
"[ ok ] autograd matches the hand derivative\n"
]
}
],
"source": [
"x = torch.tensor([2.0], requires_grad=True)\n",
"y = x ** 3 + 4 * x # scalar function of x\n",
"print(\"y.grad_fn:\", y.grad_fn) # the most recent op (an add)\n",
"y.backward()\n",
"# Analytically dy/dx = 3x^2 + 4; at x=2 that is 3*4 + 4 = 16.\n",
"print(\"x.grad:\", x.grad.item(), \"(analytic 3*2^2 + 4 = 16)\")\n",
"assert torch.allclose(x.grad, torch.tensor([16.0])), \\\n",
" \"gradient of x^3 + 4x at x=2 is 3x^2+4 = 16; if not, the graph did not reach x\"\n",
"print(\"[ ok ] autograd matches the hand derivative\")"
]
},
{
"cell_type": "markdown",
"id": "fe472fd2",
"metadata": {},
"source": [
"> **Interpretation.** We never wrote the derivative. PyTorch composed the local\n",
"derivatives of `**` and `*` and `+` by the chain rule, exactly as your micrograd did,\n",
"and landed on 16. That is the entire magic; the rest of autograd is performance.\n"
]
},
{
"cell_type": "markdown",
"id": "d6ea8391",
"metadata": {},
"source": [
"**Gradients accumulate.** A second `.backward()` adds to `.grad` rather than\n",
"replacing it. This is the single fact behind the whole `zero_grad` ritual in a training\n",
"loop, and behind the deliberate failure in Part 6. Watch it directly.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "cefbed18",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.232367Z",
"iopub.status.busy": "2026-06-10T19:01:57.232258Z",
"iopub.status.idle": "2026-06-10T19:01:57.236127Z",
"shell.execute_reply": "2026-06-10T19:01:57.235749Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"after one backward: 4.0\n",
"after a second backward: 8.0 (accumulated, not replaced)\n",
"after zero_grad + backward: 4.0 (back to one update)\n",
"[ ok ] gradients accumulate until you zero them\n"
]
}
],
"source": [
"x = torch.tensor([2.0], requires_grad=True)\n",
"(x ** 2).backward() # d/dx x^2 = 2x = 4\n",
"print(\"after one backward:\", x.grad.item())\n",
"(x ** 2).backward() # adds another 4 -> 8\n",
"print(\"after a second backward:\", x.grad.item(), \"(accumulated, not replaced)\")\n",
"assert torch.allclose(x.grad, torch.tensor([8.0])), \"two backwards accumulate to 8\"\n",
"x.grad.zero_() # clear it\n",
"(x ** 2).backward()\n",
"print(\"after zero_grad + backward:\", x.grad.item(), \"(back to one update)\")\n",
"assert torch.allclose(x.grad, torch.tensor([4.0]))\n",
"print(\"[ ok ] gradients accumulate until you zero them\")"
]
},
{
"cell_type": "markdown",
"id": "1aec3d59",
"metadata": {},
"source": [
"**`torch.no_grad()` skips graph construction**, saving memory and time; wrap\n",
"every evaluation loop in it. **`.detach()` cuts one tensor out of the graph**: it shares\n",
"data but is treated as a constant leaf with no `grad_fn`. Both are shown below, with\n",
"asserts that pin down the behavior.\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "6082d168",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.237123Z",
"iopub.status.busy": "2026-06-10T19:01:57.237051Z",
"iopub.status.idle": "2026-06-10T19:01:57.240306Z",
"shell.execute_reply": "2026-06-10T19:01:57.239938Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"inside no_grad, requires_grad: False\n",
"live.requires_grad: True · cut.requires_grad: False\n",
"[ ok ] no_grad and detach behave as documented\n"
]
}
],
"source": [
"w = torch.randn(3, requires_grad=True)\n",
"with torch.no_grad():\n",
" frozen = (w * 2).sum() # no graph built inside the block\n",
"print(\"inside no_grad, requires_grad:\", frozen.requires_grad)\n",
"assert frozen.requires_grad is False, \"no_grad should produce a tensor with no graph\"\n",
"\n",
"live = (w * 2).sum() # graph built here\n",
"cut = live.detach() # same value, but severed from the graph\n",
"print(\"live.requires_grad:\", live.requires_grad, \"· cut.requires_grad:\", cut.requires_grad)\n",
"assert live.requires_grad and not cut.requires_grad, \"detach severs the graph but keeps the value\"\n",
"assert torch.allclose(live, cut), \"detach shares the value; only the graph link is gone\"\n",
"print(\"[ ok ] no_grad and detach behave as documented\")"
]
},
{
"cell_type": "markdown",
"id": "7a9e01f3",
"metadata": {},
"source": [
"> **Caveat:** `.detach()` shares storage with the original. If you then mutate\n",
"the detached tensor in place, you corrupt the original too. When in doubt, `.detach().clone()`.\n"
]
},
{
"cell_type": "markdown",
"id": "095e1ef7",
"metadata": {},
"source": [
"### Exercise 10.3 — Differentiate a vector function\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"Implement `grad_of_sumsq(x)`: given a 1-D tensor `x`, compute the gradient of\n",
"`f(x) = sum(x**2)` with respect to `x`, using autograd, and return it. The closed form\n",
"is `2*x`, so the check verifies your autograd answer against `2*x` directly. The point\n",
"is the mechanic: make a leaf, build the scalar, call `.backward()`, read `.grad`.\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "04f3c44c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.241321Z",
"iopub.status.busy": "2026-06-10T19:01:57.241238Z",
"iopub.status.idle": "2026-06-10T19:01:57.246755Z",
"shell.execute_reply": "2026-06-10T19:01:57.245488Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 10.3 grad of sum of squares: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def grad_of_sumsq(x):\n",
" \"\"\"x: 1-D tensor of values. Return the gradient of sum(x**2) wrt x (shape == x.shape).\n",
" Do NOT hardcode 2*x; compute it with autograd so the check is meaningful.\"\"\"\n",
" x = x.detach().clone().requires_grad_(True) # a fresh leaf we own\n",
" # TODO 1: build the scalar f = sum(x**2)\n",
" f = None\n",
" # TODO 2: call f.backward() to populate x.grad\n",
" # TODO 3: return x.grad (the gradient tensor)\n",
" attempted(f)\n",
" raise NotImplementedError\n",
"\n",
"def _check_grad():\n",
" x = torch.tensor([1.0, -2.0, 3.0])\n",
" g = grad_of_sumsq(x)\n",
" check_shape(g, (3,))\n",
" check_close(g, 2 * x, atol=1e-6, msg=\"gradient of sum(x^2) is 2x\")\n",
"\n",
"check(\"10.3 grad of sum of squares\", _check_grad)"
]
},
{
"cell_type": "markdown",
"id": "cd660ed6",
"metadata": {},
"source": [
"Hint 1 (conceptual)Three lines: build the scalar, backward, return the leaf's `.grad`. The leaf is already made for you.\n",
"\n",
"Hint 2 (pseudocode)\n",
"```python\n",
"f = (x ** 2).sum()\n",
"f.backward()\n",
"return x.grad\n",
"```\n",
"\n",
"\n",
"Help — \"x.grad is None\" or returns NoneYou returned before calling `.backward()`, or you summed into a non-scalar. `.backward()` only works on a scalar; `(x**2).sum()` is scalar, `(x**2)` is not. Make sure you `return x.grad`, not `f`.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "5bd29e53",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.248130Z",
"iopub.status.busy": "2026-06-10T19:01:57.248021Z",
"iopub.status.idle": "2026-06-10T19:01:57.251476Z",
"shell.execute_reply": "2026-06-10T19:01:57.250738Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 10.3 grad of sum of squares\n",
"grad at [1,-2,3]: [2.0, -4.0, 6.0]\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines grad_of_sumsq; the check below re-verifies against 2*x.\n",
"def grad_of_sumsq(x):\n",
" x = x.detach().clone().requires_grad_(True)\n",
" f = (x ** 2).sum()\n",
" f.backward()\n",
" return x.grad\n",
"\n",
"check(\"10.3 grad of sum of squares\", _check_grad, required=True)\n",
"print(\"grad at [1,-2,3]:\", grad_of_sumsq(torch.tensor([1.0, -2.0, 3.0])).tolist())"
]
},
{
"cell_type": "markdown",
"id": "2e815f10",
"metadata": {},
"source": [
"> **Key takeaways.** `.backward()` fills `.grad` on the leaves by the chain rule, matching hand calculus. Gradients accumulate until you zero them. `no_grad` stops graph-building (use it for eval); `detach` severs one tensor while sharing its value.\n"
]
},
{
"cell_type": "markdown",
"id": "4ddecb46",
"metadata": {},
"source": [
"## Part 4 — nn.Module\n",
"\n",
"> **Objectives.** See how `nn.Module` discovers parameters; tell a `Parameter` from a\n",
"> registered buffer from a plain attribute. Reimplement `nn.Linear` from scratch and\n",
"> prove, elementwise, that it computes the same thing as the real one.\n",
"\n",
"`nn.Module` is how PyTorch organizes parameters. A module stores learnable tensors as\n",
"`nn.Parameter`, defines a `forward()`, and discovers everything through attribute\n",
"assignment. Submodules register automatically. The param count below is a thing you\n",
"should be able to predict: `out*in + out` per linear layer.\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "5c2cf146",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.252984Z",
"iopub.status.busy": "2026-06-10T19:01:57.252837Z",
"iopub.status.idle": "2026-06-10T19:01:57.257295Z",
"shell.execute_reply": "2026-06-10T19:01:57.256910Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total parameters: 101770\n",
"[ ok ] parameter count is 100480 + 1290 = 101770\n"
]
}
],
"source": [
"mlp = nn.Sequential(\n",
" nn.Linear(784, 128), # 128*784 + 128 = 100480\n",
" nn.ReLU(),\n",
" nn.Linear(128, 10), # 10*128 + 10 = 1290\n",
")\n",
"n_params = sum(p.numel() for p in mlp.parameters())\n",
"print(\"total parameters:\", n_params)\n",
"assert n_params == 100480 + 1290, \"100480 + 1290 = 101770; check the out*in + out arithmetic\"\n",
"print(\"[ ok ] parameter count is 100480 + 1290 = 101770\")"
]
},
{
"cell_type": "markdown",
"id": "2ade6cb3",
"metadata": {},
"source": [
"> **Common confusion:** a plain `self.t = torch.randn(...)` attribute is *not* a\n",
"parameter and is silently skipped by `model.parameters()`, so the optimizer never\n",
"updates it. Learnable tensors must be `nn.Parameter`. Non-learnable tensors that should\n",
"still move with `.to(device)` and serialize with the model (a mask, BatchNorm running\n",
"stats) are *buffers*, registered with `self.register_buffer(...)`.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "b32bbbb6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.258331Z",
"iopub.status.busy": "2026-06-10T19:01:57.258183Z",
"iopub.status.idle": "2026-06-10T19:01:57.262541Z",
"shell.execute_reply": "2026-06-10T19:01:57.261871Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"parameters: ['weight']\n",
"buffers : ['scale']\n",
"[ ok ] weight is a parameter, scale is a buffer, plain is neither\n"
]
}
],
"source": [
"class WithBufferAndParam(nn.Module):\n",
" def __init__(self):\n",
" super().__init__() # rule 1: always first\n",
" self.weight = nn.Parameter(torch.randn(2, 2)) # learnable -> in parameters()\n",
" self.register_buffer(\"scale\", torch.tensor([2.0])) # not learned, but tracked\n",
" self.plain = torch.zeros(2) # plain attribute -> ignored\n",
" def forward(self, x):\n",
" return (x @ self.weight) * self.scale\n",
"\n",
"m = WithBufferAndParam()\n",
"param_names = [n for n, _ in m.named_parameters()]\n",
"buffer_names = [n for n, _ in m.named_buffers()]\n",
"print(\"parameters:\", param_names)\n",
"print(\"buffers :\", buffer_names)\n",
"assert param_names == [\"weight\"], \"only nn.Parameter attributes are parameters\"\n",
"assert buffer_names == [\"scale\"], \"register_buffer makes a tracked non-parameter\"\n",
"print(\"[ ok ] weight is a parameter, scale is a buffer, plain is neither\")"
]
},
{
"cell_type": "markdown",
"id": "2333f606",
"metadata": {},
"source": [
"> **Note:** the difference matters at `.to(device)` time. Parameters and buffers\n",
"move; the plain attribute stays on CPU and silently breaks a GPU forward pass. This is\n",
"exactly why a causal mask in attention is a *buffer*, not a plain tensor.\n"
]
},
{
"cell_type": "markdown",
"id": "d490b9a1",
"metadata": {},
"source": [
"### Exercise 10.4 — Reimplement nn.Linear from scratch\n",
"`Difficulty 3/5 · ~20 min`\n",
"\n",
"This is the load-bearing exercise of the chapter. Implement `MyLinear`, a `nn.Module`\n",
"that does what `nn.Linear` does: a `weight` parameter of shape `(out, in)`, an optional\n",
"`bias` of shape `(out,)`, and a forward of `x @ weight.T + bias`. We will not check\n",
"init values (those are random), but we *will* prove your module matches a real\n",
"`nn.Linear` by copying the real layer's weights into yours and asserting the outputs\n",
"are identical. That is the strongest possible self-check: your code is right iff it\n",
"reproduces the reference exactly.\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "f8a1e9d2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.263912Z",
"iopub.status.busy": "2026-06-10T19:01:57.263800Z",
"iopub.status.idle": "2026-06-10T19:01:57.268867Z",
"shell.execute_reply": "2026-06-10T19:01:57.268419Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 10.4 MyLinear matches nn.Linear: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class MyLinear(nn.Module):\n",
" \"\"\"Reimplementation of nn.Linear. weight: (out, in); bias: (out,).\"\"\"\n",
" def __init__(self, in_features: int, out_features: int, bias: bool = True):\n",
" super().__init__()\n",
" self.in_features = in_features\n",
" self.out_features = out_features\n",
" # TODO 1: self.weight = nn.Parameter of shape (out_features, in_features)\n",
" # (any init is fine for the shape; we overwrite it before checking)\n",
" self.weight = None\n",
" # TODO 2: if bias, self.bias = nn.Parameter of shape (out_features,); else None\n",
" self.bias = None\n",
" attempted(self.weight)\n",
"\n",
" def forward(self, x): # x: (batch, in_features) -> (batch, out_features)\n",
" # TODO 3: return x @ self.weight.T, plus self.bias if it is not None\n",
" raise NotImplementedError\n",
"\n",
"def _check_mylinear():\n",
" ref = nn.Linear(6, 4) # the oracle\n",
" mine = MyLinear(6, 4)\n",
" # copy the reference's exact weights into ours, then outputs must match bit-for-bit-ish\n",
" with torch.no_grad():\n",
" mine.weight.copy_(ref.weight)\n",
" mine.bias.copy_(ref.bias)\n",
" x = torch.randn(5, 6)\n",
" check_shape(mine(x), (5, 4))\n",
" # outputs carry grad (params require it); detach before the numpy comparison\n",
" check_close(mine(x).detach(), ref(x).detach(), atol=1e-6,\n",
" msg=\"same weights must give the same output as nn.Linear\")\n",
" # parameter discovery: weight + bias = 2 params, totalling 4*6 + 4 = 28 numbers\n",
" n = sum(p.numel() for p in mine.parameters())\n",
" assert n == 28, f\"got {n} params; nn.Linear(6,4) has 4*6 + 4 = 28\"\n",
"\n",
"check(\"10.4 MyLinear matches nn.Linear\", _check_mylinear)"
]
},
{
"cell_type": "markdown",
"id": "4990fcc3",
"metadata": {},
"source": [
"Hint 1 (conceptual)The weight shape is `(out, in)` (PyTorch convention, the transpose of what you might expect). The forward is `x @ weight.T + bias`. Register both as `nn.Parameter` so `.parameters()` finds them.\n",
"\n",
"Hint 2 (pseudocode)\n",
"```python\n",
"self.weight = nn.Parameter(torch.randn(out_features, in_features) * 0.01)\n",
"self.bias = nn.Parameter(torch.zeros(out_features)) if bias else None\n",
"# forward:\n",
"out = x @ self.weight.T\n",
"return out + self.bias if self.bias is not None else out\n",
"```\n",
"\n",
"\n",
"Help — \"got 0 params\" or parameters() is emptyYou assigned a plain `torch.randn(...)` instead of wrapping it in `nn.Parameter(...)`. Only `nn.Parameter` attributes show up in `parameters()`. Wrap both `weight` and `bias`.\n",
"\n",
"Help — shape error in forward (mat1 and mat2 shapes cannot be multiplied)`x` is `(batch, in)` and `weight` is `(out, in)`. To contract the `in` axes you need `weight.T` (shape `(in, out)`), so `x @ self.weight.T` is `(batch, out)`. You probably wrote `x @ self.weight`.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "ec200f0e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.270578Z",
"iopub.status.busy": "2026-06-10T19:01:57.270425Z",
"iopub.status.idle": "2026-06-10T19:01:57.274634Z",
"shell.execute_reply": "2026-06-10T19:01:57.274227Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 10.4 MyLinear matches nn.Linear\n",
"MyLinear(6,4) params: 28\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines MyLinear; the check below re-verifies against nn.Linear.\n",
"class MyLinear(nn.Module):\n",
" def __init__(self, in_features: int, out_features: int, bias: bool = True):\n",
" super().__init__()\n",
" self.in_features = in_features\n",
" self.out_features = out_features\n",
" self.weight = nn.Parameter(torch.randn(out_features, in_features) * 0.01)\n",
" self.bias = nn.Parameter(torch.zeros(out_features)) if bias else None\n",
"\n",
" def forward(self, x):\n",
" out = x @ self.weight.T\n",
" return out + self.bias if self.bias is not None else out\n",
"\n",
"check(\"10.4 MyLinear matches nn.Linear\", _check_mylinear, required=True)\n",
"print(\"MyLinear(6,4) params:\", sum(p.numel() for p in MyLinear(6, 4).parameters()))"
]
},
{
"cell_type": "markdown",
"id": "506754e6",
"metadata": {},
"source": [
"> **Interpretation.** With identical weights, your forward and PyTorch's agree to\n",
"`1e-6`. The remaining difference is float rounding from different operation orders, not\n",
"a bug. You have now seen that `nn.Linear` is genuinely just `x @ W.T + b` with the\n",
"parameters registered for the optimizer to find.\n",
"\n",
"> **Key takeaways.** `nn.Parameter` attributes are learnable and discovered by\n",
"`parameters()`; `register_buffer` tracks non-learnable tensors that still move and\n",
"serialize; plain attributes are invisible to both. `nn.Linear` is `x @ weight.T + bias`\n",
"with `weight` shaped `(out, in)`. Copying reference weights into your module and\n",
"comparing outputs is the gold-standard correctness check.\n"
]
},
{
"cell_type": "markdown",
"id": "eaac4d78",
"metadata": {},
"source": [
"## Part 5 — Dataset and DataLoader\n",
"\n",
"> **Objectives.** Download FashionMNIST idempotently, wrap a subset in a custom\n",
"> `Dataset`, batch it with a `DataLoader`, and look at the images so the numbers stay\n",
"> attached to something real.\n",
"\n",
"A `Dataset` is anything with `__len__` and `__getitem__`. A `DataLoader` wraps it to\n",
"produce shuffled, batched tensors. FashionMNIST is 70,000 grayscale 28x28 images of\n",
"clothing in 10 classes; torchvision downloads and caches it under `root=\"data\"`, and a\n",
"second run is a no-op.\n",
"\n",
"> **Runtime:** the first run downloads ~26MB (about 10-30s). Every run after that reads\n",
"from the cache and is instant.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "fde0b1a2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.275808Z",
"iopub.status.busy": "2026-06-10T19:01:57.275632Z",
"iopub.status.idle": "2026-06-10T19:01:57.325306Z",
"shell.execute_reply": "2026-06-10T19:01:57.324798Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"train images: 60000 · test images: 10000 · classes: 10\n",
"one example: (1, 28, 28) label 9 = Ankle boot\n",
"[ ok ] images are (1,28,28) floats in [0,1]\n"
]
}
],
"source": [
"from torchvision import datasets, transforms\n",
"to_tensor = transforms.ToTensor() # uint8 [0,255] image -> float32 [0,1] tensor, shape (1,28,28)\n",
"# root=\"data\" + download=True is idempotent: it fetches once, then reads the cache.\n",
"train_full = datasets.FashionMNIST(root=\"data\", train=True, download=True, transform=to_tensor)\n",
"test_full = datasets.FashionMNIST(root=\"data\", train=False, download=True, transform=to_tensor)\n",
"CLASSES = [\"T-shirt\", \"Trouser\", \"Pullover\", \"Dress\", \"Coat\",\n",
" \"Sandal\", \"Shirt\", \"Sneaker\", \"Bag\", \"Ankle boot\"]\n",
"print(f\"train images: {len(train_full)} · test images: {len(test_full)} · classes: {len(CLASSES)}\")\n",
"img0, label0 = train_full[0]\n",
"print(\"one example:\", tuple(img0.shape), \"label\", label0, \"=\", CLASSES[label0])\n",
"assert img0.shape == (1, 28, 28) and 0.0 <= img0.min() and img0.max() <= 1.0, \\\n",
" \"ToTensor should give a (1,28,28) float tensor scaled to [0,1]\"\n",
"print(\"[ ok ] images are (1,28,28) floats in [0,1]\")"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "0a108e17",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.326415Z",
"iopub.status.busy": "2026-06-10T19:01:57.326327Z",
"iopub.status.idle": "2026-06-10T19:01:57.749907Z",
"shell.execute_reply": "2026-06-10T19:01:57.749483Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAj0AAAJOCAYAAABcJ7ZuAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAegJJREFUeJzt3WdgVNXaPvw7hCISQuhVE4oQSGEgBAEltIjYqAIKCioCtoNyLBzPsTcULKj4t8FDFAFBiYAK+oAnqEdAag4daaEXCSWEHtjvB1/m4b72ZtZMMqn7+n3yzszs2ZlZ2bOcdXGvEMuyLCEiIiIq4UoV9gkQERERFQROeoiIiMgVOOkhIiIiV+Ckh4iIiFyBkx4iIiJyBU56iIiIyBU46SEiIiJX4KSHiIiIXIGTHiIiInKFIj/pOX78uISFhcmQIUP8fszChQvF4/E43taxY0eZNWtWQOfg63i5lZKSIhs3bgzqMSnvPB6PeDweadasmYSGhnrr/v37+/X4F154QR577DHH2+bMmSMjR4687GNnzZolS5Ysyc1pUxHDcUTBFhUVJU2aNJHmzZtLo0aNpEePHrJo0aLCPq1ip3Rhn4DJ9OnTJSEhQVJTU+Xdd9+VsLCwwj6loEhJSZGIiAiJjo4u7FOhS6Snp4uISEZGhng8Hm8dDN27d5fu3bs73paTkyOzZs0Sj8cjbdq0CdpzUuHgOKL8MH36dO//gKempsrNN98sP/74o1x77bXe+1y4cEFEREqVKvLfaRSKIv+qTJw4UUaNGiVJSUkyffp0789TUlIkOTlZ7rzzTomLi5NWrVrJtm3bbI/PysqSrl27yksvvWS77fjx4zJ06FBp3bq1xMfHy7Bhw+Ts2bOO55GTkyODBg2S2NhYSUhIUBexsWPHSkxMjMTFxcnAgQPl2LFjIiKSnZ0t9913n8TGxkpsbKy8+OKLIiIyYcIEWb58uYwcOVI8Ho/MnTs3Ly8RFYLNmzfLddddJ82bN5e4uDh55plnvLft27dPbrvtNmnWrJl07txZDh8+LCJ/jdmePXuKyF/fHsbExMiQIUPE4/HIlClTZM6cOTJ27FjxeDwyYcKEwvi1qIBxHFFu9e7dWx544AF588035YUXXpA+ffrIjTfeKLGxsbJv3z758ccf5frrr5eEhARp3bq1pKWlicjlx9y3334r8fHx4vF4JDY2VmbPnl2Yv17+sYqwdevWWXXr1rVycnKs2bNnW23btvXeNmnSJCs8PNzatm2bZVmWNWrUKGvYsGGWZVlWWlqa1bx5c2vnzp1Wy5Ytrc8++8z7uA4dOljffPONZVmWNXToUO9tFy5csIYMGWKNGTPGdh5paWmWiFgLFiywLMuypk+fbjVp0sS6cOGCNXfuXCs6Oto6cuSI95gPPPCAZVmW9dRTT1kDBgywzp8/b2VnZ1sej8f68ssvbedBRc/27dutSpUqXfb2ESNGWK+99pq3zszMtCzLsp5//nkrMjLSOnTokGVZltW/f3/v/SZNmmT16NHDsqy/xlRISIi1cOFC7zEGDx5svfPOO8H9RahQcRxRsERGRlqrVq1SP0tNTbWaNm1qPf/881bt2rWt/fv3W5ZlWVu3brXatGljHTt2zLIsy9q8ebNVq1Yt6/Tp05cdc/Hx8daiRYssy7Ks8+fPez/TSpoivbw1ceJEGTRokISGhsrNN98sw4cPlw0bNkjTpk1FRKRt27ZSv35973+///773sceOHBAkpKSZMKECdKlSxfH48+aNUsWL14sb7/9toiInDp1SkJDQx3vGxUV5T1Ov379ZNiwYbJr1y5ZsGCB9O/fXyIiIkRE5MEHH5S+ffuKiMiCBQvkrbfeklKlSkmFChVk0KBBMn/+fL/X9anoSkpKkieffFKys7OlQ4cOkpyc7L2tW7duUrVqVRH5a1yuWbPG8RgNGjSQDh06FMj5UtHEcUR5YVmW979vvvlmqVmzpoiI/PDDD7JlyxZJSkry3l6qVCnZuXPnZcdcly5d5NFHH5Xbb79dunbtGvQca1FRZJe3zp07J5MnT5bPPvtMoqKipFGjRnLy5EmZOHGi9z5XXHGF979DQ0MlJyfHW0dEREhsbKx89913amBcyrIsmTlzpqSnp0t6erps2rRJPv74Y7/OLyQkREJCQhx/7usxVDyNGDHCG0Zds2aN9OnTR3777Tdp0qSJjB8/Xm699VbvfX2Ny0uVlHwa+Y/jiIJp2bJlEhsbKyJ6HFiWJTfccIP3sy09PV327Nkj11xzzWXH3Ntvvy2TJk2SK6+8UgYPHixjxowplN8pvxXZSc+cOXOkQYMGsmfPHsnIyJCMjAxZsmSJTJ48Wc6dO2d8fLly5SQ1NVX27t0rQ4cO9Ya7LtWzZ0954403vBeTI0eOyJYtWxyPl5GR4V0T/frrr6VmzZpSr149SU5OlhkzZkhWVpaIiHz88cfStWtXERFJTk6WiRMnimVZcuLECZk8ebL3tvDwcG/2h4q+9957z3vxiIuLk82bN0vNmjVl0KBBMmbMmKD8axmOiZKP44iCZfbs2fLhhx/K448/brvtxhtvlAULFsjq1au9P1u6dKmIyGXH3MaNGyUmJkYeeeQRefDBB0vsvwAsspOeiRMnysCBA9XPmjZtKnXr1pVvv/3Wr2OUKVNGpk6dKufPn5eBAwfa/k/pnXfekfLly4vH45H4+Hjp0qWLZGRkOB4rJiZGUlJSJC4uTkaPHi3Tpk2TkJAQuemmm+Tee++Vtm3bSlxcnGRlZcno0aNFROTZZ5+VMmXKSFxcnFx77bXSvXt36devn4iIDBs2TF577TUGmYupr7/+WuLi4qRFixbSv39/+eijj/J8zLvvvltmzJghLVq0YADVJTiOKBD9+/f3/pP1iRMnyty5c9W/3LqoUaNGMnXqVBk+fLg0b95cmjZtKuPGjRORy4+5f/7znxITEyMtWrSQyZMnywsvvFCAv1nBCbEut/ZDREREVIIU2W96iIiIiIKJkx4iIiJyBU56iIiIyBU46SEiIiJX4KSHiIiIXIGTHiIiInIFTnqIiIjIFfzee4tbKJQchdmaKT/GER4zr79fdHS0qsePH6/qr776StWrVq1S9dmzZ23HxC7iF1vHX9SrVy9Vb926VdVjx45V9dGjR23PUdBK2jjKqxo1aqj6nnvuUfXnn3+u6v379wf9HHC/JBzLM2fOVLU/3e3zm9vHUVRUlKo7duyo6h49eqg6MzNT1V988YWqV65cqWocAyIiffr0UTXuT3ny5Emfz/HJJ5/YjlnY/B1H/KaHiIiIXIGTHiIiInIFTnqIiIjIFfzee6sorH1ScBSnNfRg5HUw53DHHXeoGte3z58/r+oKFSqounz58qquWrVqwOeE/vjjD1VfuHBB1U2aNFH1gQMHVP3jjz+q+s0331T12rVr83qKNsVpHOWHsLAwVeO4evTRR1WNWa9Dhw75vN0pG1axYkVVlytXTtX16tVT9ezZs1W9ePFiVWM+rTCU9HF00003qXrkyJGqPnXqlKrLli2r6tOnT6saxwDmA2vWrKlqp020cfPtffv2qfrYsWOqxnFWt25dVf/000+qHjFihO058xszPURERESX4KSHiIiIXIGTHiIiInIFZnpcqKStoYeHh6sa+6HEx8erulQpPdc/fvy4qnENHXuZYOanTJkyqq5UqZLtHE+cOKFqzOwE+p5cccUVqsacEeYCfv31V9sx7r777oCeE5W0cZRXffv2VTVmNf71r3+puk6dOqrGLAbmKEREjhw5ours7GxVz58/X9XTpk1TNeaQZs2aZXuOglbSxlHDhg1V/cILL6ga83hXXnmlqvH6hNcKzONcddVVPs8HH+/0M8zw4HPgNfDw4cOqxowP9hF74oknfJ5jMDDTQ0RERHQJTnqIiIjIFTjpISIiIlfwe++tkiA3PV+wJ8L111+v6nnz5gX0nKGhoarGtdPcMK1LF+aaeUFITU1VdWRkpKoPHjyoalzPLl1a/xnge4KvL94fb8f+KyL29x3hOr4J5kUwh4TveVJSku0YuCfPxo0bAzoH0jBHhbkG3MMNe5mcOXNG1U6ZHjzmihUrVD1p0iRV169fX9V//vmn7ZgUXI8//riqTa85/u1jXg+vR1hv375d1ZjPweOJ2K+BTmPtUphjxGvgjh07VI29g2655RZVf//99z6fLz/xmx4iIiJyBU56iIiIyBU46SEiIiJXcFWmB9dOcZ2yUaNGtsfcf//9qsYsBfZfwWzF0qVLVW3K8GA+xCnrgfcxHdOUJylOEhISbD/DDA9manD9GV8PXPPGnhOmPhrYwwKfT8Q+1vA9xF4/+J5iL6Hdu3f7vL/p+UXsY7sgemmUZNgzp1q1aqrG3MPf//53VeO+WdWrV7c9B+Y3MjMzfT6nKX9GwZeSkqJq3GsLMz7YtwdzpHh9QbhHG44BJ1lZWarGzzUTfE7sTbZr1y5VF2aGB/GbHiIiInIFTnqIiIjIFTjpISIiIlfgpIeIiIhcwVVBZgywYrizc+fOtsckJyerGgOk2NQJQ6833HCDqidMmKBqDLFhUzmnACrCTQSx8dTJkyeNxyguOnXqZPsZvgdY4+uB4wCbwo0aNUrVe/fuVTWOAdw4ct++fbZzxPAzBgHxnPE9bdmypar/9re/qdoU3nbadPD2229XNYPMeWMKk5sCpvge7t+/33YfvL5g6B6vF3g9KemNSosC/McrixcvVnX37t1V/fvvv6sa/3bxPcfwOl5LcBzhP65xOiY+JwadnUL1vo73j3/8w+f9CxO/6SEiIiJX4KSHiIiIXIGTHiIiInKFEMvPRV43NLX69NNPbT/r1auXqrHpEtY//vijqlu0aKFqbEK3fPlyVa9Zs0bVGzZssJ1T69atVZ2YmKjqRYsWqRrXlHHTwoKU13G0ZMkS289q1Kihamzkh2vemJfBDfratGmj6q5du6oacxS4yePw4cNt57h27VpVly9fXtWYM8KsV3p6uqo3b96savydTZsWitg3HMVNAv/44w/bYy5VmPmQong9uu2221RdoUIFVWMjU3zPg9FEFLNb4eHhqsY8yHfffZfn58wrt42jrVu3qvrnn39WNTYvxPcUm2Di3z5yGlemhqr4OYWZHWxGmJaWpupvv/3W5znlB3/HEb/pISIiIlfgpIeIiIhcgZMeIiIicoUS3acH12txzQ976LRq1cp2DFwvxXX6xo0b+6yXLVum6i1btqga8yVt27ZVde/evW3nhOux+By4kST2oSnOmjdvbvsZ5qqwJw72wEGYe0A//PCDqjGb0axZM1U79bv55ptvVI35D1xTX7lypapxo1XM6OC4xH4tTn16du7cqWoce6ZMD2n4t4zjDvulYNbC1E9KxJxBwbGPNWa9KPjwbxn/Vq+//npVv/rqqz6Ph33W8HiYD8TNQ502QMaf4WeE00bXvm4vjAxPbvGbHiIiInIFTnqIiIjIFTjpISIiIlco1pmevPZYePnll1Vdu3Zt42OwXwGur2JPGFy/xdwQruNjlgMzQE7P+fDDD6u6QYMGqsY9looT7B2DPSxE7K8HZiFwnOAaOPYuMZ0Drn/juHFao8dzwFwW3o75GoT7gZn2YHLK9ODaf/v27VX92Wef+TwH0jAnge8p1piLMN3fn8fg3wLePxi9gMg30x5suDcf9u2pX7++qjELhjlT/NvG+zvlc7DXD+6tZRpHO3bssB2zuOA3PUREROQKnPQQERGRK3DSQ0RERK5QrDM9ed2z5ciRI6p2yvRg7gF7b+A6PvbqwPVVzJPgeizmKtq1a2c7J1xfxb2nsK9McTZq1ChV4+snYl+fxjwLPgbfE1y/xtxV1apVVV2lShVV4z41NWvWtJ0jZnjwHMqWLavqiIgIVffv31/VlStXVjWOU9wbB293ek6nPlXkP/y7xP4qmKcx5XNwHDsxXQNLUo+ukgrHQcWKFVWNnxH4GZSVlaVq/LvGa42IPXuKTLmkgwcP+ry9KOM3PUREROQKnPQQERGRK3DSQ0RERK5QrDM9eYU9d5z6GZjW6Y8dO6Zq7PkSFRWlalyDN/XuwHMUMfdgueqqq2yPKa4WLVqk6lq1atnu06hRI1XjXlq4L9XmzZtVja/nkiVLVI2vL9b4eKdeKKYeLngMHAfYmwP3xcJxYsqPiNh7/cyaNct2H/Kfab+iQPfaMh3PiWlPJcz/Uf7D9xHf9927d6s6Pj7e5+PxPcXPFMwYOmXDcA82zPxhDqhatWqq3rNnj+2YlzLtP1aY+E0PERERuQInPUREROQKnPQQERGRKxTrTI8pD4NrmdhDp06dOqp26mmBP8MeCdjvADM/2G8FMz+YxcAeC5jlELH3YFm9erWq8fcszv1XPvzwQ5+1iL1nzTXXXKPqBx98UNUdOnRQ9eHDh1W9du1aVR89elTVuGYejP2MTGMZ19hNY2DgwIF5PifyDcedac83zF7kJrODMB+CWQocN5hvw2yHU08Xyl8ZGRmqxnGBnwk47vDxmJ/BPmMi9h51+Bj83MNzKkoZnUDxmx4iIiJyBU56iIiIyBU46SEiIiJXKNaZHlwjxzV1zPTg/kXY8+XPP/+0PYdpryxcI8ceOZj5wUwQ7smEa/JOe03hGu0HH3ygao/H4/OYJQ2uTy9dulTVuD7duXNnVeM4wjV0fI9N/VacYL4Da9P+OjiOMIuB/Ywo/+G4MvVPMfHn/qbsF8Kxin3FmOEpfNgjx3Q9MfV3wmuD0/Hwmol9eHD/L4S5xuKE3/QQERGRK3DSQ0RERK7ASQ8RERG5QrEOe2BWBXMPCPuv4Bq80zqlKSeEe9ngGjn25cHnwPVXzI/g2quIfa+WAQMGqHrs2LGqxr2kijPMNIjYX1McB5iVyMrKUrXpPTZlLUz9WILB1AsIewn583hc68+P8y7JTJnCwoDnhNkwKnimjA72vMFsKV7PnD4TfN3u9LmIWdGDBw+qunr16qrOzs72+ZzFCb/pISIiIlfgpIeIiIhcgZMeIiIicoWgZXow14Dr2079JPAx2LMm0LVQk7lz56r6xIkTqsZ+CSL2ni24Zo7rr6aeCfg7In9eA3yO+Ph4VWMvjpLEKXdiek23bt2qasz0BJoNw3PITabHKZvk6xxMfTHwd0JOf3+YXaLAmDI8+Lcb6F5bTn/7gR7DtB8h3u5PzykKjOk1xp44uLcW7udYpUoVn8936NAhVeP+jiL2vftM1zy8XkVGRvq8f1Hem4vf9BAREZErcNJDRERErsBJDxEREbkCJz1ERETkCrkOMpsauhVEkCkpKUnVffr0UfV1112nagyEYeNADC2L2EOu+HviMfF1weZgGGzG0CsezwmeJzaO6t27t6q//fZb4zGLM1NYEwPqpk1gceziGDAFl51CyqaNIvEY2DgTw4h4vKIcHCypTH/LpnFhChXnptmhaWyaNtflBqTBZwqH4z+GwSa6u3btUjVeC/A9q1mzpqqdQsoZGRk+j4FB53379qm6Tp06tmMWF/ymh4iIiFyBkx4iIiJyBU56iIiIyBVynekJtLGZU0MlXBe85pprfN6OWZXGjRurGnMQuEaOeZmqVauqeu/evbZzxLVOXAPHDUdx/RTXXxctWqTqsLAwVWNOyWk9GJsPYnO+Nm3a2B5TkpmaAeJraNpQFGtTQzg8vj9ZjEAzPqZGd6bXgJuJBh++h6Y60I1rg8F0zECbHVLwtW/fXtXbtm1T9Y4dO1SNn0nYmDQ8PFzVmM8RMecca9eu7eOMRWrVqqVq/BzEDUyLUhNMjngiIiJyBU56iIiIyBU46SEiIiJXyHWmB3MjL7/8sqqrV6+u6oiICNsxMFuBWYijR4+qGnuRHD9+XNW4Lonr2biOifmafv362c5x+fLlqsbN4TBHFBUVZTvGpeLi4nweD3syOPXtKV++vKoxF2TaDM7t6tatq+ojR46oGsehKeOTH1kMfA7MbZk2+KX8F+zX3J9+T8iUG8JzxBp7UFHemfIrV111laqbNWumasz04GdntWrVVL1lyxZVV6hQQdX169e3nSN+tmIOyAR7ww0YMEDV48aNU3VR2siW3/QQERGRK3DSQ0RERK7ASQ8RERG5gt8LurgW/N5776ka/10/5nWc+vqY9pnCnjimPZUQ9ifArMvrr79uPN6DDz6oauzlgz0TfvrpJ1Xj+iz2IsJeQZhLKlOmjO2cTHkP3MulpAu0B41pnyrTuAu0P4uIOa+Ba974vmN2DI/nNE583Z/yzrSXluk9N/XI8ec9C7SnFJ4DXiOx5wsFzpRfufHGG1W9fv16VeOebvieYG50z549qo6Ojjaez+7du1UdHx+v6gMHDqgaP6cwB4k5yUaNGqkac0eFid/0EBERkStw0kNERESuwEkPERERuYLfmZ5BgwapGvMxW7duVTX2jsFaxHk/rkthTgHXn7GnDeZtcN8rXKf87LPPVN2zZ0/bOXz77beqxvVU/L0SEhJU3alTJ1XjmjtmeMqVK6dqzJc4wSwBvm7YF8LtMB+DeTXM/ODtuEZu6o0iYn+f8THYLwVvN+XfnPpgUf7Cv7NA+zcFujdXbphyRni9ofyH+ZnVq1erGq8f+Blges/86R+F1zCsMauKnyGYMzLljpjpISIiIipgnPQQERGRK3DSQ0RERK7gd6bn4MGDqsY8jWlPKry/iD0Pg2uXuB/I4cOHVb1jxw6fx8O+O7hOidmNb775xnaOa9asUTWuVWIuCbMbuMcJ9tTBczD1a3G6D2YD8HVs3Lix7RhuFug+MIFmL5x6pZjyHaaeLng7jhvcj810fMo7zGGZ9kMriPfA1IMKrz+mvj6Ud/iZsW/fPlVjXx7c1wrHWaB/+05jAq+BppwQZgpr1qypauwVhHtvFiUc8UREROQKnPQQERGRK3DSQ0RERK7gd6YH1+xwfRr38qhQoYKqq1WrZjsm5l0OHTqkatxDCtc2cR0S8y+4Voq5I1zPxucXEWnatKmqT5w4oWrMKuGeJHiO+BymjA/eLmJfw61Vq5aqjx07pmqPx2M7hpsFmmMINIsRjEwPHsOU6cGeVJT/TD208D3DHEVB5GnwHPB6wnGT/66++mpV4zjAzzUcV/g5hr2X8PGocuXKtp/h9QOPgfX27dtVjXtIYg887KmH2VfM5xYkftNDRERErsBJDxEREbkCJz1ERETkCn5netLT01Wdmpqq6vvuu0/VuA/Wtm3bbMfEvjnYZwczOphlwbVP7IuBvYJM+9A47W+EPRXwMab1VdPvaOrrg7WIOQdUv359VeN6a0mT1/4n/uxV4+v5THkdf54j0N4/OO4C/R0o7/D6Y8pd+TNO8so0TvDa0ahRI1XjdZ7yDv828T3Czx3MWeHnIH5mmPYCdNr3EscmflbWrVtX1cuXL1d1UlKSqvFzEj8HMVfETA8RERFRPuOkh4iIiFyBkx4iIiJyBb8zPWj06NGqxrXgJ554QtW4/4iIvWcN5lewJw6ujeKaOq4j4v1N+xk57XOFP8PnxNtN6/Z4O+ZtcP0V+xuI2NdwsU/P6tWrVf3FF1+oevLkyT7PsbgJdG8sXBMPtFcJvv44zpz2ugn0HE0CzfRw763gq1Onjs/bTb2WTOPIn/cMn8O0Lx+OTafeZBRc2KMOP0OwH11sbKyqsU9PVlaWz+Phe4z96Zweg9nT+Ph4VX///feqxs9qPB5meEy9hAoSv+khIiIiV+Ckh4iIiFyBkx4iIiJyBU56iIiIyBX8TheZAnPz5s3zWXfq1Ml2TAxDR0ZGqho3LcNzwOAfhqUw7IkOHjyoaqfgIG60ik2csrOzfZ4TMm0AiI2qnDYlnD9/vqo3bNig6kWLFvk8B9JMDd0wDIr3N9Ui5oApMm1AiticsOBh+BP/UQO+h6Z/WJGbhpN4/cDH4LjDfyixY8cO43NQ3mCQGf+WMzMzVY2fe/i5ho0AMUSMm17jPwhyOgcT/JzD58Bxhs9Zu3ZtVW/atCmg5w8mftNDRERErsBJDxEREbkCJz1ERETkCn5nenDNLlBpaWm2n7Vp08bnY6Kjo1WNa6PYIKlevXqqzsjIUDWuf2/dutXn81PxEGjjPdwMt3HjxqrG5l449rHGLIfT34ppU0DTxrXIlBcx3Z/ybunSparGcRQREaHqU6dO+TyeqZGgSODvI2YpcJz98ccfAR2PAoc5KsxtYiM/hM0JsbkqXiuqV6+uamx+KCJSoUIFn4/Bz9qGDRuqGq9npsyvU4PEwsJveoiIiMgVOOkhIiIiV+Ckh4iIiFyh6OwC5mDjxo0B3X/t2rX5dCZUkmDWAte3cY3c1GcDa6eNa01MPVp27dqlatwkFdfckT+9gygwmM34/PPPVY29yXAc4bjzZ+NaZOoxtX37dlVjthJ/Bwq+a665RtX4nmBmB+F7jH/72C8K+7QNGDDAdky8xv30008+nxNrvIZiXx7TuCtM/KaHiIiIXIGTHiIiInIFTnqIiIjIFUIsPxs/mPYKouKjMHu25Mc4wmOafr+xY8equly5cqrG/k+mjA6ud+M+NU7nZOrJgnkb7M2BvT2wZ8x3333n44yDo6SNo7yeQ6CvR5UqVVRdq1YtVYeHhxuPsX//fp815j1QXn+HYCjp4wjzM/i3bupxg3k93C/N1J/OLfwdR/ymh4iIiFyBkx4iIiJyBU56iIiIyBX8zvQQERERFWf8poeIiIhcgZMeIiIicgVOeoiIiMgVOOkhIiIiV+Ckh4iIiFyBkx4iIiJyBU56iIiIyBU46SEiIiJX4KSHiIiIXIGTHiIiInIFTnqIiIjIFTjpISIiIlfgpIeIiIhcgZMeIiIicoUSM+nxeDzi8XikWbNmEhoa6q379+/v1+NfeOEFeeyxxxxvmzNnjowcOfKyj501a5YsWbIkN6dNhSwqKkqaNGniHTsffPCBz/unpKRIz549RURk4cKF4vF48v8kqVhITU2VhIQE8Xg8Eh0dLZ07d5YLFy7ky3OFhITI0aNH8+XYVHh4Pcp/pQv7BIIlPT1dREQyMjLE4/F462Do3r27dO/e3fG2nJwcmTVrlng8HmnTpk3QnpMKzvTp08Xj8ciOHTskPj5e2rdvL/Hx8YV9WkpOTo6ULl1i/lxLnH379smwYcNkxYoVEhkZKSIiK1eulJCQkEI+M2ccT0UXr0f5q8R80+OvzZs3y3XXXSfNmzeXuLg4eeaZZ7y37du3T2677TZp1qyZdO7cWQ4fPiwi9tl0TEyMDBkyRDwej0yZMkXmzJkjY8eOFY/HIxMmTCiMX4uCIDIyUpo0aSIDBgxQ3/qNHz9e7rnnHuPjJ0+eLPHx8RIfHy+33HKL7NmzR0REGjduLMuXL/feLyUlRXr16iUiIvv375d+/fpJ69atbeMxKipKRo0aJa1bt5bBgwcH55ekfHHgwAEJDQ2VKlWqeH/WsmVLCQkJkaioKHnuueekbdu2Ur9+fXnllVe89/H1/j/xxBOSmJgoHo9HkpKSZNOmTbbntSxLRo0aJd27d5eTJ0/K5s2b5ZZbbpHExESJj4+X8ePHe+8bEhIizz//vCQmJsrTTz+dT68EBQuvR/nDdZOe8ePHy6233ir//e9/Zc2aNfL3v//de9vvv/8uKSkpsn79eqlRo4Z8/PHHjsfYsGGDDBo0SNLT02Xw4MHSvXt3efLJJyU9PV3uv//+gvpVKMjWrFkjGzdulObNmwf82LVr18qTTz4p8+bNk9WrV0u7du28Y+Gee+6RlJQU730nTZok9913n4iIDB48WB5++GFZunSprFq1SpYvXy5fffWV976ZmZny+++/y5QpU/L2y1G+io+Pl+uvv14iIyOlV69eMnbsWO+HjIjI0aNHZfHixbJs2TJ1m6/3f9SoUbJs2TJJT0+Xhx56SB599FH1nGfOnJE777xTsrOz5ZtvvpFy5crJnXfeKW+99ZYsW7ZMlixZIp988oksW7bM+5jQ0FDvOVDRxutR/iie30/lQVJSkjz55JOSnZ0tHTp0kOTkZO9t3bp1k6pVq4qISNu2bWXNmjWOx2jQoIF06NChQM6X8l///v2lfPnycuWVV8r//M//yNq1awPOS6SlpUm3bt2kbt26IiLy0EMPyUsvvSTnz5+XQYMGSYsWLeStt96SPXv2yB9//CE33XSTnDhxQn766Sc5cOCA9zjZ2dnq/+jvueeeIrtEQv+nVKlSMnPmTNm4caP8/PPPMm/ePHn11Ve9/0c9YMAAERGpVq2aNGjQQLZv3y4RERE+3//58+fL+++/L8ePH5cLFy54v3m+6JZbbpEePXrIs88+KyIi69evl3Xr1skdd9zhvc/x48dl/fr1kpiYKCLi/XCjoovXo/xV4ic9I0aMkF9++UVE/vq6r0+fPtKuXTuZP3++jB8/XsaNGydz584VEZErrrjC+7jQ0FDJyclxPGZYWFj+nzgVmItr6Bdt3LhRzp8/761Pnz4d8DEvvTDUq1dPWrVqJbNnz5Z169bJXXfdJaVLl/Yed8mSJWrsXYpjrXiJjo6W6OhoGT58uHTr1k3mzJkjIs7XFsuyRMT5/d+5c6c88sgjsmzZMmnYsKGsXr1akpKS1H06d+4s8+fPl0cffVTCw8PFsiypUqWKzzwjx1PRx+tR/irxy1vvvfeepKenS3p6usTFxcnmzZulZs2aMmjQIBkzZkxQ/tVVeHi4HDt2LAhnS0VBo0aNZPny5XL+/Hk5efKkzJw50/iYTp06yQ8//CB79+4VEZGPPvpIunTpIqGhoSIicu+998r//M//yOeff+79v+2wsDDp1KmTvP76697j7N27V3bv3p0PvxXlpz179shvv/3mrY8cOSLbt2+Xhg0bXvYxvt7/Y8eOSZkyZaR27dpiWZbK5lz0z3/+U3r37i3JycmSmZkpTZo0kfDwcJk0aZL3Plu2bLF9Q0TFC69HwVXiJz3o66+/lri4OGnRooX0799fPvroozwf8+6775YZM2ZIixYtGGQuAXr37i116tSRpk2byq233iotWrQwPiY2NlbGjh0r3bp1k/j4ePn111/l008/9d7eo0cPWbZsmdSsWVOaNm3q/fmUKVNky5YtEhsbK3FxcdK7d2/JzMzMl9+L8k9OTo689NJL0rhxY/F4PNK+fXsZPHiw9OjRw+fjLvf+x8XFyR133CExMTGSmJgoV199tePjH3vsMRk6dKh07txZDh06JN99952kpqZKfHy89x9cnDp1Kj9+ZSogvB4FV4h18TtWIiIiohLMdd/0EBERkTtx0kNERESuwEkPERERuQInPUREROQKnPQQERGRK3DSQ0RERK7ASQ8RERG5Aic9RERE5Ap+771V3DcZyy916tRR9cW230VZYfajzOs4cnp8Xn+fGjVqqLpz586qvrg78UW4+d+GDRtUffbsWdtzREREqLpdu3aqxu1Q/vnPf6o60K66+Drlx3tenMcRFR0lbRyZjpnX3xc3u966dauqc7NtRFRUlKovblB70aU7rRdV/r6u/KaHiIiIXIGTHiIiInIFv/feKgpfJ//000+qrly5sqpxY7ShQ4eqOiMjI+DnxOWrtLQ0VZcvX17VO3bsUHW3bt1UfeLEiYDPIdiK09fJuVmmqVatmqofffRRVScnJ6u6XLlyqsb3CG+Pjo5WdcWKFY3ndO7cOVXjV9D79u1TNY4r3Cn7l19+UfX777+v6iNHjhjPKa+K0ziioqukjaNSpfR3CRcuXPB5/3r16qn64q7nFz3++OOqDg8Pz8PZ+ef8+fOqzsnJUfWoUaNU/e677wZ0fHyNRMyvkwmXt4iIiIguwUkPERERuQInPUREROQKxSrTs3DhQlU3bNhQ1Zi9wFzE8ePHVT1z5kxV33XXXbbnDA0NVfXp06dVjf98Gf9pcfPmzW3HLGzFaQ3dn0wPjoNvv/1W1QcOHFA1voeYt8H17DNnzqga8zVhYWE+H+90jLJly6q6evXqqi5dWneTwPtjffLkSVV/9NFHqv7mm29s55RXxWkcUdFVnMdRbrIpK1euVPU111yj6iuuuELV+LeNmUO8P+b58DNKRKR27dqqvvLKK30+J36W4jUPr4kLFixQ9cCBA23ngALNQiFmeoiIiIguwUkPERERuQInPUREROQKxSrTgxmcVq1aqRrXIatUqaJqzE3gGiL2PhERiY+PVzXmQzB7gX16cEuDoqA4r6E7mTFjhqqxTw+uN5cpU0bV+HpgxgfXljGfgzVmhkTsebNKlSr5PCfT64RjFzM+eLyePXvajpGdne3zOUxK2jiiwlGcxlFu+oYtXrxY1fi5tX//flXjtQKfA3OmeDvmc5xyR/hZiTlEvH6YtsHB++M1ePbs2ap2uh6hQF9rZnqIiIiILsFJDxEREbkCJz1ERETkCqXNdyk6tm3bpuo2bdqoGvcHwayFaf3WaW+u9u3bq3rPnj2qxv4FuJ5KwYX9JUREatWqpepjx46pGvMuOE7wPatQoYKqTf0jcD3cqU8P9tLA5zDtdYO3Yx4Hc0R4/Ntuu812TtOmTbP9jIguz5/cSK9evVR97bXXqhr33cPPJczH4PUGzwFr7Efn9LmH1zS8D15v8HMOzwmvVzt37lR1165dVX3TTTfZzmnevHmqzq+sF7/pISIiIlfgpIeIiIhcgZMeIiIicgVOeoiIiMgVilWQef369arGJk0IN2Y7e/asqrHxoBNsyoSBL2xOmJWVZTwm5V7lypVtP8MgM4bwMMiMIV8M4WFzMAzt4Rjwp8EZjlV8jOk58HfCRpuHDh1SNf7ON9xwg+2cGGQm8g3/bp3+kQJKTU1VNf5tVqxYUdW4ISg2R8XPGFOzQgwp5yYQjI/B3xtvN4Wx8R+XzJ071/ac+I9UsGkjvg543fYXv+khIiIiV+Ckh4iIiFyBkx4iIiJyhWKV6cHGgLj2iWuZuK64b98+Va9cuVLV2NTJ6TlN2Qxcu6Tgcsph4XuCGR8cF1hjY7+9e/eqeuvWrarGJpaYHXPacBTvg2MXMzj4e956660+nyMiIkLVYWFhqsYcExGZmTI8uJGmiD2jg41EIyMjfd7f1PgPOW0oGmymhoj4OuE1Ga9/ThuYduzYUdVffvmlz+fILX7TQ0RERK7ASQ8RERG5Aic9RERE5ArFKtODWQvMRZh6n2AOAvv+YAZIxL5eipkd7OniT88Wyj1c5xUR+fXXX1U9cOBAVcfGxqr6tddeU/XGjRsDOgfcoBQ348NaxJ6pwQ1Icc0be+g8/fTTql62bJmqa9asqeqTJ0+qukGDBrZzIqK8adu2rfE+mNcz9eBCpjwNyo/PINM5mH4n/GzF65+ISKtWrVSN1/pgbUDKb3qIiIjIFTjpISIiIlfgpIeIiIhcoVhlenAPk6ioKFVjNgMzPKZ9s5zgfl2mtUvMGVFwjRkzxvYzzG6lpaWpetWqVaoODw9XNY4bfI9xP7XMzExVm/bOETGvgVeqVEnVMTExqsZeQZhbwl4geI5nzpyxnRMFlylLYdozCcexU4Yhr/sPYUYRnzM3MK+B5xSsLEZR5NRvBjM8pswOjhu8fuDrG+jeXE7Pb+pd5s8xfJ0jXm/wNcEMo4j9mvbEE0/4fM7c4jc9RERE5Aqc9BAREZErcNJDRERErlCsMj379+/3ebtp7y3THiVOa8+BrqceOXLE53NQ3vz444+2n3Xp0kXVffr0UXXXrl1V/dlnn6n6wQcfVDXuY9WoUSNV475WpqyGiH1NG7NimK344osvVI37wo0aNcrn8XAc9u7d23ZO7dq1U/Xhw4dt9yH/BZpdwSyHP48PNMODY/uZZ55Rdd26dQM6nhM35RibN2+u6mrVqtnugxlA7EmDf6t4O2ZRTTksU+00rkyPMcFzMvXMq1y5sqrxNRAJfGznFr/pISIiIlfgpIeIiIhcgZMeIiIicoVilelBpt4jpjVyvN1pXRP7E2Bt6ulCwfX666/bfobrybhH24YNG1R92223qfq5557z+Zx4fBx3OCacxh2uV2PuB7NjmBvCjM7SpUtVjXk37FW0efNm2zkxw5O/TJmd3GQY7rzzTlW3aNFC1X379lU19pHBXme4xxse3x+YV3vqqadU/corrwR8zKIKM5xO+T18n3HfPdNnCF4L8HbM05hud/pcMx3D1JcHH2+6vuHtTsevV6+ez+cMFn7TQ0RERK7ASQ8RERG5Aic9RERE5ArFOtMTaG8B0/5Hpr1znO6Dx3TaU4SCJzU11fYz7NPTqlUrVc+bN0/Vc+bMUXWNGjVUvXPnTlWb8jfYZ8OfPd1wjfvkyZOqxj4WuF9YZGSkqh977DGft3fs2NF2DrgnWXp6+uVOlxyYrgWmTCH2f8I8DvZRErH3nMI92Xbv3q1qzBjifoU333yzz3P0xx133KHqa6+9Ns/HLKpatmyparwWiNjfd8y/4N825q4wz+fU08bX8/nzuYj3ccom+brddH/8ncuXL69q7DsmYt8/EMfR77//7vM5/cVveoiIiMgVOOkhIiIiV+Ckh4iIiFyhWGd6THtpIVyDN61LOj0Hrp9ivwHMh1BwNWvWzPYzXBPHnjVLlixR9XXXXafq2NhYVfuzl9al/NnrxpQfw+fAY+LvNHXqVFVjHmfbtm2q3rVrl+2c/vjjD9vPSip/epeY9kdDpswO7uH26quvqrp///6qxlzXvn37bMfE/kyYKcHsxMaNG1WNvVBefvll23NcCq9neM4iIm+//baqo6OjVZ2QkKDqFStW+HzOoszUE0fEPrYC3ZsMnwPzf+XKlVM1fgZhptCfPj0mONbxHI4dO6Zq7E1k6tvjdEzMKeamh5QTftNDRERErsBJDxEREbkCJz1ERETkCsU602Pqq2PKUZj2D3F6DK6X4mOwDwYFV4MGDWw/w/cEcwuYh8HsBL6H2EMi0H1mTPvWOME1cMwBVK9eXdX4O1SsWFHV+BpgvkREpFatWqrGHFBxlpseXKYMD8L+UH369FH1gAEDVJ2Zmanq9evXqxrHFfZmEhGpWrWqqjHPhuMCe1bh3wKe45NPPunz+GvWrLGdE2YxsG+VU0+W4sqf38XUlwf/tv3JCeXl/sGA54zX3EAzP07njHsa4jgKFn7TQ0RERK7ASQ8RERG5Aic9RERE5AolOtOD64aBZoCcmPIbzPTkL6e14NOnT6sa3xNch7/yyitVbdqHBmvTmrrTOeJj8DnxMdgzBs/h0KFDtue4VJUqVVTttB9YnTp1VF2SMj2mflr+GDFihKofeOABVdesWVPVuO8V5l/wHPDxyKm/imlfJ3zMn3/+qWqnnNClFi1apOpevXr5vL+IyDPPPKPqhx56SNW4l91dd91lPGZR9c9//lPVTj14TH118G8T/5b9+RzKb3i9wVwSjjP8HbF/FF6DsZ+UiD0/1rNnT1Wb9rrzF7/pISIiIlfgpIeIiIhcgZMeIiIicgVOeoiIiMgVilWQuXHjxqrGsCeGq5zCm5fyJ+hsanKGobVq1ar5fE7Km9yEhA8fPqxqDNGZQsWmwBze7s+GoxiAxCAgjl08J2wyZwpzO22aig0Ni7OWLVuq+oYbblB1kyZNVO3U+AyD3WFhYao+evSoqvfs2aPqSpUq+XwOrHGcYGNBDIOK2MdRoJtNYlgUx03r1q1VvXfvXlXjayJiD3Bv3rxZ1fgPB4YOHWo7RnGBzVGxoZ6I/W8Z6x07dqgar0fBCuwGE54TBptxXJiaFzpdj/AxGRkZPo+RW/ymh4iIiFyBkx4iIiJyBU56iIiIyBWKVaanadOmqsa1ZMxJOK2JX8rUdM4JZitwTRcbjrVr107V2PyL8g7fR8wxHDhwQNVOjbF8MWWGTPkbp5+ZshlOa96XMm2Oic/n1JzP9BxF2SOPPKLq3r17q9qUk3B6/fB6gRkbPAbmGHBcnDhxQtWYCTLlb5xyR3gOmBfB9xRfBzwm/s5ZWVmqxszikSNHbOeE98HnLM7Zsbp166oa80lOTULxPqbGfqbrl6kBZW6uRwivD1ibPvcwz4afxZgdc2qSiePoqquu8nHGucdveoiIiMgVOOkhIiIiV+Ckh4iIiFyhWGV6unTpomrT5nuB9jvwpw8Arr/iY7Zu3arqBx98UNXM9OSNP+8Rvu+YQ8AcAx4T18hNvZn86etjOm/TMfEcMDeBeRGnPAjy5z5F1eTJk1W9bNkyVWOWLjY2VtWRkZG2Y2L2pHLlyqrGrIQp91C9enWftSnbgX3InM7BlNfIzs5WNeaMMG+C4xDPAbMZTvfB58D8x/fff6/qp556ynbMoqJ9+/Y+b3fKyuHrga8xvoa4ASnmYUzXp2B8rgUKfyfMv+E54t+WUw89fF3yK3PIb3qIiIjIFTjpISIiIlfgpIeIiIhcoVhletq0aaNqXPs09d3BtU3T3lxOcA0dcxG4Ltm2bduAn4PyF75npgyPP3u0XcqfNXTTfl24Zo7ngJmeLVu2qNrj8fg8noh/famKKjz3tWvXqvr333/3+XjsbyMiUr9+fVU3atRI1VFRUarGvbpwXJnGEY477PmCeRwRkczMTFVjlstU495bmMVAmE/xZ8zg74EZn6Kwl5S/8DMGOe29ZbpeRERE+Lw/Pqdp3ODtWDv97ZuyYIH2CcPPPbwdc0tOx8c8WX7hNz1ERETkCpz0EBERkStw0kNERESuUKwyPbimjv1X/OmXcilTzx1/4DFw35VatWqpGrMETmvCdHnHjx+3/axChQqqNq1XYx4G159NfTEQ3t8p94A/M409XNc35UN27typ6latWqnaaZwV5723MKuCY6B27dqq9ieLcvjwYVUvXLhQ1ZjZMeU9TO+xKR/o9P5gxgZziXgM3B8MewXhHkjYwwp/R6ccJF7z8G8Uj7Fjxw7bMYqqn3/+2eftTp8ZeL0w9XPCLAv+rZrGEb4neH+nXkKB7v1nulbg74TnhLVTfqegsl78poeIiIhcgZMeIiIicgVOeoiIiMgVinSmB/e+qVatmqoPHDigalzPNmUt8HantU9TzwVcY//f//1fVfft21fVCQkJquZeXL7h6+u07ovvUVZWls9jmnILCJ8TzwnHjT/5EdM+Tqb9dfDxGRkZqsbf0Wls432KM+wFg7U/MOtleg0xL4N5PdPrizkJU9bDn2MgzNfs3btX1ThWcVzh7+B0Tqa8BvYCwnMoym655Raftzv1wMGfYY4KP7dMPbnw9cVrA77e+J46XY9M1xNTPhbHhWnfLH8yPU7XqPzAb3qIiIjIFTjpISIiIlfgpIeIiIhcoUhnenD/IFNvAVNmx9QXA7MaIuZsBa5NNmnSRNW4ltm0aVNVM9Pjm2mPKhH7a7xnzx6fxzT1vTD15TGtmTutoZt6d5jOCe9fsWJFVf/xxx+qNuUALneebob7UmGNsE8YlTzdunXzebtTHhD77ODf6oMPPqjqL774QtX4OYS5LPxbxkyQ6doiYr6u4nNgXg0/OytVqqRq7G8UGRmpauyz5Y+aNWuqGrNR/uI3PUREROQKnPQQERGRK3DSQ0RERK5QpDM9t912m6oPHTqkalxPxXVIrLGvBmYanPpq4Nol9oDBc8C9tjDzExcXZ3sO8p8/fXpMmZ5Ae1Dg/XGN3JQBEgm8R5Qpb4Nr6OvWrVM1nrPTfmTM9BD5ZsrX4J5vIubrwTfffKPq999/X9UDBgxQNWaCqlatqmrse4T5GyemrCrmhLBHHl6vfv/9d1W/++67qu7QoYPP57/czy7VvXt3VX/66ac+7385/KaHiIiIXIGTHiIiInIFTnqIiIjIFYp0pqdhw4aqxrVNzM9gbuHw4cM+74+Zoe+++852Dtir48orr1Q1rvEiXPONiYnxeX/yzZ9Mz86dO30eA/to/Pnnn6rG99S0B5I/eRzTHm5Ym/pi4LjCHBMez2m9HHv5EJGG1xv8DMpNvxn0j3/8w2dtgtcGPEd/9t4yZXpM+xkGyumc8HqEn734ec1MDxEREZEPnPQQERGRK3DSQ0RERK7ASQ8RERG5QpFOMmKwuGPHjj7vj2HN8uXL+7x/dna28RwwxIoBL4Sh1tOnT6t6zZo1xuek/+PPZp7IFLrDkDDW2HCySpUqqsb3GMeIP+eI98GgM/4OGFyuU6eOqnGcYVM1p9Cy0wa7RPR/7r//flX36dNH1fgPW0Tsf8t4vQg2/NvHuijYvn27qqtXr267D4bCMaD922+/BeVc+E0PERERuQInPUREROQKnPQQERGRKxTpTA82H/rkk09UjbkI3JDUtIGZPxtF4jFxo0fMf2BjqPDwcFXjRmzkG27u6ZSpwkyN0+aal5o5c6aq8T06ePCgqjEPY2pW6JSfMWWTcCzicxw7dkzVy5cv93kO+Hincza9TkRuhzmTyMhIVTvlTPAzYtq0aUE9J9NmwqYNlZ2Y7mPazNu0ofKPP/6oasxKidg/O7///ntVv/HGGz7P0V+86hEREZErcNJDRERErsBJDxEREblCkc70oLi4OFWbet7gxpKoRo0axuesWbOmqrH3D+Y3cF3yxhtvVPWOHTuMz0n/B19vfzbzjIiI8HnM0aNH5/m8ijpcU3fK75heJyLScDNj7PElYv8MqFevns9jYg+uEydO+Ly/KV9TGDB7iRnC9PR0VWMWVkQkLCxM1R988EFwTg7wmx4iIiJyBU56iIiIyBU46SEiIiJXKFaZnrVr16oa8x3XX3+9qps1a6bqzp07q9qfvTxwXRFzQF9++aWq582bZzwm+e/w4cOq/uOPP2z32b17t6p///13n8c07Y3lT1+Lom7KlCmqbtCgge0+K1euLKjTISoR8Nrx5JNP2u6D16x9+/b5PKYpe1ocmK6Z2Pvs1KlTtvtgD7b8yirxmx4iIiJyBU56iIiIyBU46SEiIiJXCLFKQoCBiIiIyIDf9BAREZErcNJDRERErsBJDxEREbkCJz1ERETkCpz0EBERkStw0kNERESuwEkPERERuQInPUREROQKnPQQERGRK3DSQ0RERK7ASQ8RERG5Aic9RERE5Aqc9BAREZErcNJDRERErlBiJj2pqamSkJAgHo9HoqOjpXPnznLhwoWgHb9Vq1aycOFC4/2ioqIkPT09aM9LBYvjiILB4/GIx+ORZs2aSWhoqLfu37+/X49/4YUX5LHHHnO8bc6cOTJy5MjLPnbWrFmyZMmS3Jw2FZDjx49LWFiYDBkyxO/HLFy4UDwej+NtHTt2lFmzZgV0Dr6Ol1spKSmycePGoB4z2EoX9gkEw759+2TYsGGyYsUKiYyMFBGRlStXSkhISCGfGRUnHEcULBcnrBkZGeLxeII6ge3evbt0797d8bacnByZNWuWeDweadOmTdCek4Jr+vTpkpCQIKmpqfLuu+9KWFhYYZ9SUKSkpEhERIRER0cX9qlcVon4pufAgQMSGhoqVapU8f6sZcuWEhISIk888YQkJiaKx+ORpKQk2bRpk/c+ISEh8tprr0nr1q2lfv36MmnSJO9tixYtEo/HI7GxsXLvvfdKTk6O97a3337be8zExERZvHhxwfyilK84jqigbd68Wa677jpp3ry5xMXFyTPPPOO9bd++fXLbbbdJs2bNpHPnznL48GER+euDpWfPniLy1/+tx8TEyJAhQ8Tj8ciUKVNkzpw5MnbsWPF4PDJhwoTC+LXIYOLEiTJq1ChJSkqS6dOne3+ekpIiycnJcuedd0pcXJy0atVKtm3bZnt8VlaWdO3aVV566SXbbcePH5ehQ4dK69atJT4+XoYNGyZnz551PI+cnBwZNGiQxMbGSkJCgpqcjx07VmJiYiQuLk4GDhwox44dExGR7Oxsue+++yQ2NlZiY2PlxRdfFBGRCRMmyPLly2XkyJHi8Xhk7ty5eXmJ8o9VApw/f97q3bu3VblyZatnz57WmDFjrN27d1uWZVkHDx703m/atGnWjTfe6K1FxHrzzTcty7KsDRs2WGFhYda5c+esM2fOWPXq1bPmz59vWZZl/fjjj5aIWGlpabZjLl682GrSpIm3joyMtFatWpVfvyrlI44jCrbt27dblSpVuuztI0aMsF577TVvnZmZaVmWZT3//PNWZGSkdejQIcuyLKt///7e+02aNMnq0aOHZVmWlZaWZoWEhFgLFy70HmPw4MHWO++8E9xfhIJm3bp1Vt26da2cnBxr9uzZVtu2bb23TZo0yQoPD7e2bdtmWZZljRo1yho2bJhlWX+9182bN7d27txptWzZ0vrss8+8j+vQoYP1zTffWJZlWUOHDvXeduHCBWvIkCHWmDFjbOeRlpZmiYi1YMECy7Isa/r06VaTJk2sCxcuWHPnzrWio6OtI0eOeI/5wAMPWJZlWU899ZQ1YMAA6/z581Z2drbl8XisL7/80nYeRVWJWN4qVaqUzJw5UzZu3Cg///yzzJs3T1599VVZvny5LF26VN5//305fvy4XLhwwft/SxcNHDhQRESio6OldOnSsn//fjl8+LCULl1akpOTRUSka9eu0qBBA+9jVq1aJa+++qpkZmZK6dKlZdOmTXLq1CkpX758wf3SFHQcR1TQkpKS5Mknn5Ts7Gzp0KGDd6yIiHTr1k2qVq0qIiJt27aVNWvWOB6jQYMG0qFDhwI5X8q7iRMnyqBBgyQ0NFRuvvlmGT58uGzYsEGaNm0qIn+91/Xr1/f+9/vvv+997IEDByQpKUkmTJggXbp0cTz+rFmzZPHixfL222+LiMipU6ckNDTU8b5RUVHe4/Tr10+GDRsmu3btkgULFkj//v0lIiJCREQefPBB6du3r4iILFiwQN566y0pVaqUVKhQQQYNGiTz58/3O69W2ErEpOei6OhoiY6OluHDh0u3bt1k6tSpMm7cOFm2bJk0bNhQVq9eLUlJSeoxV1xxhfe/Q0ND1fLDpS7mOs6ePSu9e/eWtLQ0SUxMlKysLKlUqZKcOXOGH1YlBMcR5ZcRI0bIL7/8IiIikydPlj59+ki7du1k/vz5Mn78eBk3bpx3WcDfMVVS8iBucO7cOZk8ebKUKVNGpk6dKiIiJ0+elIkTJ8qbb74pIr7f94iICGnUqJF899130rlzZ8e8oWVZMnPmTGncuHHA5xcSEuJ4TF+5xuKWeSwRmZ49e/bIb7/95q2PHDki27dvl0qVKkmZMmWkdu3aYlmWjB8/3q/jRUdHS05OjqSlpYnIXzPbrVu3iojI6dOn5ezZs3L11VeLiKhZOBVvHEeU39577z1JT0+X9PR0iYuLk82bN0vNmjVl0KBBMmbMmKD8q6vw8HBv/oKKljlz5kiDBg1kz549kpGRIRkZGbJkyRKZPHmynDt3zvj4cuXKSWpqquzdu1eGDh3q+C9Le/bsKW+88YZ3snTkyBHZsmWL4/EyMjK816evv/5aatasKfXq1ZPk5GSZMWOGZGVliYjIxx9/LF27dhURkeTkZJk4caJYliUnTpyQyZMne28rDmOvREx6cnJy5KWXXpLGjRuLx+OR9u3by+DBg+XRRx+VO+64Q2JiYiQxMdH7AWNStmxZmT59uowcOVLi4uJk6tSp0rx5cxH560195ZVXpHXr1pKQkCBly5bNz1+NChDHERW0r7/+WuLi4qRFixbSv39/+eijj/J8zLvvvltmzJghLVq0YJC5iJk4caJ3Kfyipk2bSt26deXbb7/16xgXvyU6f/68DBw40PYN4DvvvCPly5cXj8cj8fHx0qVLF8nIyHA8VkxMjKSkpEhcXJyMHj1apk2bJiEhIXLTTTfJvffeK23btpW4uDjJysqS0aNHi4jIs88+K2XKlJG4uDi59tprpXv37tKvXz8RERk2bJi89tprRTrIHGJZllXYJ0FERESU30rENz1EREREJpz0EBERkStw0kNERESuwEkPERERuQInPUREROQKnPQQERGRK3DSQ0RERK7ASQ8RERG5gt97bxW3/TXo8gqzHyXHUcnhtnFUoUIFVb/88suqbteunao/++wzVX/44Yf5c2I+XNwk8qL7779f1fPmzVP1uHHj8vuUbNw2jgpakyZNbD/r1q2bqnED5dOnT6t60aJFqt6zZ0+Qzu7/4HsR6Ljw9/78poeIiIhcgZMeIiIicgVOeoiIiMgV/N5w1A1rn27BNXQKhpI+jnDH86SkJFWHhoaq+sCBA6pu1qyZqg8dOqTqXbt2qfqPP/5QdVZWlu2cqlSpomrMEZUtW1bV4eHhqt67d6+qw8LCfJ7TsGHDVL1t2zbbOeVVSR9HwRZo9uXf//637WeJiYmqLlOmjKrLlSvn85gTJkxQdfPmzVVdvnx5Vf/666+qfvzxx23HPHXqlKrx7+v8+fM+z4mZHiIiIqJLcNJDRERErsBJDxEREbmCqzI9+DuUKqXnfBcuXLA9xvTymF6XvK5X45q9iL1nAvZhwGwAngPX0IPLn9+poF/zyZMnq/qdd96x3WflypWqxnX8M2fO+HyOkjaOOnXqpOp//OMfqs7MzFR1xYoVVY3XE8w1VK9eXdVXXnmlqvfv36/qFStW2M6xVatWqr7iiitUfezYMVVjzqhGjRqqxv4sERERqj5+/Liqe/XqZTunvCpp4yi/+fO5dal169bZfoZZL8yCnT17VtU4LjBvg2P93LlzqsbM0Pvvv287pxEjRvg8JmZ+EDM9RERERJfgpIeIiIhcgZMeIiIicgW/995yg9ysLed1Pbpjx46qjouLU/U111xje8xrr72malyX7tq1q6pN2Qy3CbTPhen+WDvlBAJ9TlwDxzXy2NhYVc+cOVPVjRs3VjXmT0REevbsGdA5lXT4d5ORkaFqzDzl5OSounRpfTnFvjx4fxwTmJPAPj8i9j2RTpw4oWrM4NStW1fVJ0+eVDXmQ3BPJcx+XHfddbZz+u2332w/o/xjyvRgPufqq6+2HQPHAY49zPDgOMMsWIMGDVSNmSAc62+//bbtnJApq5Rb/KaHiIiIXIGTHiIiInIFTnqIiIjIFYp1pifQnATebtrLw8mgQYNUvWTJElW3b99e1dh7APe+iY+PV/XmzZtVjb1UREQee+wxVaenp1/2fMnOnwyOr/vj+jfCNXcRe94De07gYzDDg/s+paam+rz/xo0bVf3www/7OGPnY7hNnTp1VI17X2GmB18vHBemvkeYk8Acl9O4xGsWZm6w9w9mNzDzg2MbxyHejtc3EWZ68pupvxzq3LmzqnF/NRGR7OxsVWMOCOHYxGPiWMfr3Zo1a4znVKtWLVVj36pA+xNdDr/pISIiIlfgpIeIiIhcgZMeIiIicgVOeoiIiMgVinWQOdiio6NtP8NAFjYTxA0AK1eurOqUlBRV//LLL6rGoHJCQoKqExMTbeeEjZ8aNWqk6i1bttgeQ5cXaFM+UwDe6XZTSBhDeVdddZWqv//+e1VjEBFDtH//+99VjU3nRAL/hwAliVMYFEPBuHkn1rjZJ8JrB9YIw6L4d+50H/w98Dnwdny8aRNHHJfY9JLyH/5dOo2LS+FnBgaCRUSOHj2qanxf8TlxnFSrVs3n/TEwP3v2bFXfcMMNtnPCz0I872BtDstveoiIiMgVOOkhIiIiV+Ckh4iIiFyhWGd6As0gYOOudu3aqdpp7RMblE2cOFHVI0eOVDU2H3znnXdUXaNGDVXj77Bp0yZVY8ZHxL4eipsQMtMTmLw2vapZs6aqMdclIlK1alVVYxYMj4HZjCNHjqgax2qlSpVUvWLFCh9nTPXr17f9DMdB+fLlVY2ZHnxP8D3D9xw3HMWGbphZcMoM4X0wK4bngPfHsY01NjNEuIEp5b9As3eYO3W6P47dBQsWqBo3EMVjVK9eXdWrVq1SdYsWLVSNWTJsrioismPHDtvPLpWbZsJO+E0PERERuQInPUREROQKnPQQERGRKxTrTA/2JsH1aVyHxE3OMAsTGxtrew5cHx0+fLiqu3Xrpuoff/zx8icsIgcPHvR5O2Z+Dh8+bLsPrqvfd999qsYNANeuXevzOd3ONI4aNmyo6nHjxqk6IiJC1dijQkQkJiZG1dg3B29fuHChz/vjBoG4maWpJ0xumDZaLU5wc0MR+2uI4wCzFZhBwNcHeynh4ytUqKBqzPw4ZctMGR7M5OAx8HfEbBjmHitWrKjqzMxM2zlhvuPPP/+03YdyD8cVjhOE1yscZyIibdu2VTW+rzhWMduK16d69eqpetq0aar+17/+dfkTvsxz5lffMH7TQ0RERK7ASQ8RERG5Aic9RERE5ArFOtNjyvAg3D8E+3J07tzZ9pgvvvhC1Q888EAgpxgw7O2B+wGJiCxfvlzVuE6P/T/wmKSZ9sXaunWrqu+55x5VO+Uc8gpzEdizZc2aNaqeMWOGqrFflFMeB/9+8D64xm7KEhQnuHeQiMi+fftUjb2P2rdvr+opU6aoGl/z2rVrqxr/LvF6hOPQ6XqGvUpwHybsh4LHxExhmzZtVI1jYsOGDap2uh41adJE1cz0BJepP83111+vasxYrVu3zvaYKlWqqBp7i2EfH8yaYhYM93/EcVOU8JseIiIicgVOeoiIiMgVOOkhIiIiVyjWmZ5A/x0/9k/55ZdffNZOcD8e7PVjOidTLwLMATj16cHfY968eaquU6eOqiMjI32eEwUGMzyYDXPKz5hyQygtLU3VvXv3VjWuuXfo0EHVb7zxhqr92bfGdJ+SlA3D3IOIvY9Xp06dVI05INw/Da8f8fHxqj569KiqMT+D48hpzGB/JhxrmP3C7MbOnTtVjX19rr32Wp/H27Vrl+2cPB6Pqv/zn//Y7kO5Z/pMufvuu1Xtz/UIP1cwX4b5PcyK4f3RV199peq33npL1Y8//rjtMfh75lffHn7TQ0RERK7ASQ8RERG5Aic9RERE5ArFOtOTV6Y9l0Ts66Om2/3JTviCWQPcv0fEvtaJvwdmE0pSf5WiwLTW7E9+B/dMwvfo888/V3Xfvn1VjeMO+2Rg9sy0Bi8i0qxZM1V/8MEHqt69e7fxGMXFhAkTbD+bP3++qrF3yYgRI1SNe95FR0erGvN+2FMH8zl4/cEchYh97OExMaODe2clJiaqul+/fqoeOXKkqnFPJac+ZdgnjPIGr+emz5QbbrhB1YcOHVI1voci9j3WcKzhuEKmfN/kyZNVjde32bNn2x7To0cPVXPvLSIiIqI84KSHiIiIXIGTHiIiInKFEMvPhTNcS6a/mNZfA+018Oijj6r69ttvt93nu+++U/XUqVNVfdVVV6k6KytL1bhvU0HiOPoLZnKc8mSXwt5AmzZtUjXmQ55//nlVO+VxUlNTfT4nZlpiYmJUjXtNFaSiMI569eql6oceekjV+JpjTgJzXfg74e1O90GY3apfv76q8XrltN9gQcuv7IY/isI4QqZrA/Z/+u9//6vqbdu2qdppvzSEe7I1bNhQ1TiW8TOlRYsWqsZ+UNddd52qnfo95fW98Hcc8ZseIiIicgVOeoiIiMgVOOkhIiIiVyjSfXrya++NYMIMj9M+J77uj7DHwqpVq2z3wT1/Pv74Y1XjeuyiRYt8Pif5Fug4dFqbzutYxjV17L+Ceyxh7svp+XAdH/sLLVy4UNX79u3z61yLA6f3CLMUpr2wMBuHPbXwNcfjYW8U7GXiT98wvN7gc2LfHqeeLb6Yrmciee9NRpop39e1a1dV47jE7Bj2ixKx58XwelKuXDlV498+9pPDc7j66qtV/fLLL9vOAaWkpKj6nnvuMT4mN/hNDxEREbkCJz1ERETkCpz0EBERkSsU6UxPUczwmAS6vu3xeFSNPRe+/PJL22NuvfVWVd94442qxp4tTj0RyH/BGIemdXqT5s2bq3r16tWqrlOnjqrvuOMOVTv16njxxRdVXaFCBVXjXlQlidN7in+7pvfsxIkTPm/HbMUVV1yhaszwmPI5IvYsEp4jPgeeoz/7wvk6fnG8Jhd1pl5vmK/BPeDS09NV3aRJE1Xj54GIfWxi3x2EWdMGDRqoGrNmeDzM5+zYscP2HB07dlQ1fs5hTjG3+E0PERERuQInPUREROQKnPQQERGRKxTpTE9xYFqPRaNGjVI19lf58MMPVX333XfbjoH7MM2dO1fVkZGRqsb1Wwouf3rwYF8MHCf4GDzmmTNnVI1r5rnZt+Zf//qXqnEsf/XVVwEfsyTB1wMzOKY+O5ifwcwU3o7ZDTyeiD07geOofPnyqsZx88cff9iO6Utx6JVW3Jk+M5555hlV496KR48eVTXuexUdHW07Jo5d7DFlglkv035h2C/KKWeEY/Wmm25SNfYSmjZtmn8nC/hNDxEREbkCJz1ERETkCpz0EBERkStw0kNERESuwCBzHmEILSoqStUvvPCCqjEc+eeff6r69ttvV/XmzZttz4mhWGxMF2gDsuIOw5YYqjOFfDF0l9dGgv48hykQumzZMlWnpaWpGhtSmjgFB3EsYsMwbEhGmunvDhsFIlOw2QmOIwyk4jFMwWfcgBQ3ts1NQJ58M4V+ETb2O3LkiKrxPTx+/LiqsZGpiMg111yj6sqVK6s6IyND1WFhYao2/eMYvL5deeWVqnYKTv/000+qfvjhh30+R27xmx4iIiJyBU56iIiIyBU46SEiIiJXKLBMD+YHRALfnDPY8Jyc1q8xC4FNlrDx09ixY1WNmRxsLPX444+r2p/mX7hJKW7+tnjxYuMxigpT8zN/mqPhzwp7XDkxrdvPnDlT1WvWrFH1vffe6/PxppwA5sBE7Ovsq1at8vkcbmP6W2zbtq2qMU+D1w683mAzNlNjQRFzpgevT/ic+Bw1atRQNWZ6TJkgsjNds0zXgttuu03VmNnBZoT4nuLmwpjHEbFvbI1jFRvc4jjCc8DfyTROtm3bZvvZ/fff7/MxwcJveoiIiMgVOOkhIiIiV+Ckh4iIiFyhwDI9/qwFm3pCBHuzOzwnp9wRrpHXrVtX1ZjJ+fe//63qNm3aqLpv374BnyfC1wHPG8+5KMtNhidQmLu67777VI05LOydhEz5Gaf+LKdPn1b1yy+/rGrMVvTp08fnOSBTTsDpdhw3W7du9XkMt/VsMb2mjRo1UjVuEIqZKczfYGYHc1dOfXtM54Rjz7TRY5MmTVS9cuVKVXOD0cDl9TV76aWXVI09c0w5URwD2DtOROT6669XNW5Ei5+NHTt29Hn7qVOnVI1jHZl6WDkJ1mcDv+khIiIiV+Ckh4iIiFyBkx4iIiJyhSK191Z+rx+b1gT9yR3hXlp79+5VdfPmzVXdv3//AM7QP3ie1apVU7VpX5TChGu9+J5gZgFzEpiFEREZOnSoqvfv3+/zHOrXr6/qHj16qBpzDgjPEX8HzO+I2Nfd+/Xrp+qbb77Z53NiLw5cQzfljHBvHaf7/Oc///F5DiU90xNoryPMYeH7bsqroXLlyqna6e8Y//bxnE17b+HtgY51CpxpXLVq1UrV+BmCe+AlJiaqGvfi2r59u6q3bNliO6eKFSuqumXLlqrG/bt+++03VWNWFccuPh5/52PHjtnOySRY8wN+00NERESuwEkPERERuQInPUREROQKBZbpcVrPxjW6iIgIVdesWVPVtWvXVvXChQsDOofcrAm++OKLqsaMSXx8vKp79eoV0PGd9kTy9XxOj8FMT1Hm1HskELj2LGIfJ/g+45r6wYMHVV29enVV49433377rc9z8mdcTZ06VdU//PCDqk09cjDDEyh8jURETpw4oepFixbl6TmKO1PmBvc0yszMVDWOI8w1YI7ClL9xYtovEG/HsYnXjoYNG/p8PlN+zek5SjJ8j5zeM3zNTLmoMWPGqBr7N+Hri9kx3JsL+/I47eG2adMmVa9fv17VeL3AvbjWrl2rasyG4e+MYx1zSAWJ3/QQERGRK3DSQ0RERK7ASQ8RERG5QoFlevxZ923WrJmqsbdJVlaWqnFvm7zuOYX7aomItGvXTtW4Z0j79u3z9Jz4uvjTFwMfc/XVV+fpHApSUlKSqvHcv/76a1Xj+nWdOnWMz4E9IA4fPqxqzMdgtmXcuHGqNmV60OzZs20/i42NVXXPnj0DOmZeYV5OJPCcUEnv02P6/fB6hBkd/LvE3iW47xXeH2/Hx4uYewFhPyfMFWFGELMW2EfLn9yRP/3NigscA/h6YO+k3PQxevLJJ1V97bXXqvrnn39WNX4G4XuI1zvMdTntg4X5WOw5he6//35VY58ej8ejahx3mCUz7W+Yn/hNDxEREbkCJz1ERETkCpz0EBERkSv4nekx7VsV6OOdjlHYfUI++eQT288aN26s6ltuuSWoz4nr4f7kJvAx0dHRQT2n/NSgQQNVf/zxx6rGvbWys7NV7ZTpwftgDgGzGNjXAl9PXBPHPhoTJkxQ9RtvvKHqTp062c5x/vz5qsYeL/kN1/BF7Bk5Ezf1Y3GCf2fYtwd7j+B+Z5gHwZwD1pjPEbFnevCYmN3C2/HxmFGsVKmSqnHfp5Ke68IxbtrHEK8VIvbrzd/+9jdV//3vf1c1fu7VqlXL5+3YqywsLEzV/uy9aMoide/eXdWYa7zpppsCOj6OG3/23srrnONy+E0PERERuQInPUREROQKnPQQERGRK/id6cnrepo/j8c1vLlz56oa++iMHj1a1dOmTQvonJ577jlVd+vWzXafd999V9W450hhwLV/zA4UZSkpKaoeOnSoqmNiYlSNv5tTT5D9+/erukKFCqrGnAPmFDDXgLCvBtbYc8Kp/83zzz/v8zmw/0lu+n/44tSn5+jRowEdI9jnVNxUqVJF1ThuMEuG+RjMceHfsWnPOBF7zxXMs+FzYr8UzKBgjXkS/Ftxm9tvv13VkyZNUrVTpgezWPi+Yp4Fe3itWLFC1XFxcaresmWLz9vxnJz2O8T3GfeMNPUmM+0ZifA12Lt3r/ExOP6D1Q+K3/QQERGRK3DSQ0RERK7ASQ8RERG5gt8Lcx07dlQ19gLAnh/YswL3NxIROXPmjKqxhwTWDRs2VPXjjz+u6p9++knVBw8eVHXXrl1VPWLECFXjniciIv/4xz9sP8tP/mSfcK0TX6fiJCMjQ9W4p8uuXbtU7bSPTM2aNVWN2TAce7inEb7m+HjcuwvHLTpw4IDtZ6YsWF4zc/g7Ya4Isx4izud5KcysFOdx5g9TD5r69eurGq+B+HjMlm3btk3VTntrXQr7AInYr6t4DrgfGOZLcOziOWPPF1TS+/RgP6uxY8eqGve9wsyUiLkHDWZucBy0bdtW1UuWLFE19jrDc8B9tJze09TUVFXPmjXr8ifswNQLCF8nvL75kyfMr7HGb3qIiIjIFTjpISIiIlfgpIeIiIhcgZMeIiIicgW/g8xRUVE+6+rVq6saQ3hODZIwIIrNzzDEOmXKFFWvXr1a1V26dFF1u3btVB0fH6/q3377TdUYjBaxB7YwdGYKteaHkydPqvp///d/C/wcggUbTA4YMEDVuDmoU7gNG7RhsA/fQxxnGI7GGoPjGETEoODAgQNt54iC3YzQFPpzasCIQX/k1BzPzbA5Gv7tY2gYxx1eA8uWLatqDD5jM0QRke3bt/s8BjI1eHP6hwGBHK+kwY028T3ARqhXXnml7Rh4fcC/PbzdtOl0q1atVL1nzx5VL1u2TNUJCQmqxs9qEZE+ffrYfnYp0+ec0z9M8vV4hK9jQSrZI5iIiIjo/8dJDxEREbkCJz1ERETkCn5nenCjyEBVrVrV9jPMa+D6qSnPERkZqWrM8GCjLtzAdOrUqarGDJGTwsjwIGwSN3LkSFW//PLLBXk6eYJN+/A9xk1gX3rpJdsxEhMTVe3U1C0//frrr6pOS0sr0OcXMWeC8G9DxLzpX14bJpY0mNEx5WMwM4XvEV5L8PFOGyxiDhIzJZhvCzQ7ZmpAWdI3nf38889V3bdvX1U3bdpU1fgZI2LeOBbfV3xNMbOJGSBs0ot5WtxcuFOnTrZzNMHmgsgpo+vrdtyg1JQJErH/3qZz8he/6SEiIiJX4KSHiIiIXIGTHiIiInIFvzM9eZWZmenXz8gMN+n84IMPCudECsAPP/zgs3bSuHFjVWPfCuzXVLduXVVXrlzZ5/GxT8YDDzxgPCfMKgU7G2HKmo0ZM8b2s02bNvl8jGlTQbfBcYXZCcwx4O04rrDHTrVq1VTtlE275pprVI2bS7Zo0ULVixYtUjVmUHBcuv09x416k5OTVY0508GDB9uOceutt6q6ZcuWqjb1VgoU9gG65ZZbVL1w4cKgPp+IyObNm33ejmN/69atql63bp3xOZwybcHAb3qIiIjIFTjpISIiIlfgpIeIiIhcIcTysxmHaW8fKj4Ks/8Kx1HJUdLGkWlPJNybDzM42JcHe94cOnRI1dh3BLNltWvXtp3jypUrVY35ENxnCd8j7AHj8XhU/cQTT6ga+wLhaySS9+xFSRtHJpgNa9Cggaox+4XvAeZjtmzZEsSz+4vpbwF17NhR1fi3gL9Dfuy95e844jc9RERE5Aqc9BAREZErcNJDREREruB3poeIiIioOOM3PUREROQKnPQQERGRK3DSQ0RERK7ASQ8RERG5Aic9RERE5Aqc9BAREZErcNJDRERErsBJDxEREbkCJz1ERETkCpz0EBERkStw0kNERESuwEkPERERuQInPUREROQKnPQQERGRK5SYSU9qaqokJCSIx+OR6Oho6dy5s1y4cCFox2/VqpUsXLjQeL+oqChJT08P2vNSweI4otzK77FzqZCQEDl69Gi+HJvy3/HjxyUsLEyGDBni92MWLlwoHo/H8baOHTvKrFmzAjoHX8fLrZSUFNm4cWNQjxlspQv7BIJh3759MmzYMFmxYoVERkaKiMjKlSslJCSkkM+MihOOI8qt4jZ2cnJypHTpEnH5L5amT58uCQkJkpqaKu+++66EhYUV9ikFRUpKikREREh0dHRhn8pllYhveg4cOCChoaFSpUoV789atmwpISEh8sQTT0hiYqJ4PB5JSkqSTZs2ee8TEhIir732mrRu3Vrq168vkyZN8t62aNEi8Xg8EhsbK/fee6/k5OR4b3v77be9x0xMTJTFixcXzC9K+YrjiHLL19iJioqS5557Ttq2bSv169eXV155xXuf/fv3S79+/aR169YSFxcnzzzzjPc2X2PuIsuyZNSoUdK9e3c5efKkbN68WW655RZJTEyU+Ph4GT9+vPe+ISEh8vzzz0tiYqI8/fTT+fRKkD8mTpwoo0aNkqSkJJk+fbr35ykpKZKcnCx33nmnxMXFSatWrWTbtm22x2dlZUnXrl3lpZdest12/PhxGTp0qLRu3Vri4+Nl2LBhcvbsWcfzyMnJkUGDBklsbKwkJCSob5fHjh0rMTExEhcXJwMHDpRjx46JiEh2drbcd999EhsbK7GxsfLiiy+KiMiECRNk+fLlMnLkSPF4PDJ37ty8vET5xyoBzp8/b/Xu3duqXLmy1bNnT2vMmDHW7t27LcuyrIMHD3rvN23aNOvGG2/01iJivfnmm5ZlWdaGDRussLAw69y5c9aZM2esevXqWfPnz7csy7J+/PFHS0SstLQ02zEXL15sNWnSxFtHRkZaq1atyq9flfIRxxHllq+xExkZaf3tb3+zLMuy/vzzTys8PNx7W9euXa2FCxdalmVZ586ds2688UZrxowZlmWZx9z+/fut/v37Ww899JCVk5Nj5eTkWAkJCdaGDRssy7KsEydOWHFxcdbSpUu9j3nxxRfz+ZUgk3Xr1ll169a1cnJyrNmzZ1tt27b13jZp0iQrPDzc2rZtm2VZljVq1Chr2LBhlmVZVlpamtW8eXNr586dVsuWLa3PPvvM+7gOHTpY33zzjWVZljV06FDvbRcuXLCGDBlijRkzxnYeaWlplohYCxYssCzLsqZPn241adLEunDhgjV37lwrOjraOnLkiPeYDzzwgGVZlvXUU09ZAwYMsM6fP29lZ2dbHo/H+vLLL23nUVSViO83S5UqJTNnzpSNGzfKzz//LPPmzZNXX31Vli9fLkuXLpX3339fjh8/LhcuXJDDhw+rxw4cOFBERKKjo6V06dKyf/9+OXz4sJQuXVqSk5NFRKRr167SoEED72NWrVolr776qmRmZkrp0qVl06ZNcurUKSlfvnzB/dIUdBxHlFu+xo6IyIABA0REpFq1atKgQQPZvn27REREyE8//SQHDhzwHic7O9v7jc78+fN9jrlbbrlFevToIc8++6yIiKxfv17WrVsnd9xxh/c+x48fl/Xr10tiYqKIiNx333359yKQXyZOnCiDBg2S0NBQufnmm2X48OGyYcMGadq0qYiI9xvBi//9/vvvex974MABSUpKkgkTJkiXLl0cjz9r1ixZvHixvP322yIicurUKQkNDXW8b1RUlPc4/fr1k2HDhsmuXbtkwYIF0r9/f4mIiBARkQcffFD69u0rIiILFiyQt956S0qVKiUVKlSQQYMGyfz586V///55f3EKQImY9FwUHR0t0dHRMnz4cOnWrZtMnTpVxo0bJ8uWLZOGDRvK6tWrJSkpST3miiuu8P53aGioWn641MW1+bNnz0rv3r0lLS1NEhMTJSsrSypVqiRnzpzhh1UJwXFEuYVjZ86cOSLiPD4syxIRkSVLlqjbRUR27twpjzzyiM8x17lzZ5k/f748+uijEh4eLpZlSZUqVXwG4EtKdqS4OnfunEyePFnKlCkjU6dOFRGRkydPysSJE+XNN98UEd/XkoiICGnUqJF899130rlzZ8fMmGVZMnPmTGncuHHA5xcSEuJ4TF/ZtKKaW7ucEpHp2bNnj/z222/e+siRI7J9+3apVKmSlClTRmrXri2WZan1bV+io6MlJydH0tLSROSvme3WrVtFROT06dNy9uxZufrqq0VE1CycijeOI8qty42dhg0bXvYxYWFh0qlTJ3n99de9P9u7d6/s3r1bjh07Zhxz//znP6V3796SnJwsmZmZ0qRJEwkPD1eZsi1btti+IaLCM2fOHGnQoIHs2bNHMjIyJCMjQ5YsWSKTJ0+Wc+fOGR9frlw5SU1Nlb1798rQoUMd/3Vgz5495Y033vBOlo4cOSJbtmxxPF5GRob3+vT1119LzZo1pV69epKcnCwzZsyQrKwsERH5+OOPpWvXriIikpycLBMnThTLsuTEiRMyefJk723h4eHe7E9RVSImPTk5OfLSSy9J48aNxePxSPv27WXw4MHy6KOPyh133CExMTGSmJjo/YAxKVu2rEyfPl1GjhwpcXFxMnXqVGnevLmI/PWmvvLKK9K6dWtJSEiQsmXL5uevRgWI44hy63Jjp0ePHj4fN2XKFNmyZYvExsZKXFyc9O7dWzIzMyUuLs6vMffYY4/J0KFDpXPnznLo0CH57rvvJDU1VeLj4yUmJkaGDBkip06dyo9fmXJh4sSJ3qXwi5o2bSp169aVb7/91q9jXPyW6Pz58zJw4EDbt8rvvPOOlC9fXjwej8THx0uXLl0kIyPD8VgxMTGSkpIicXFxMnr0aJk2bZqEhITITTfdJPfee6+0bdtW4uLiJCsrS0aPHi0iIs8++6yUKVNG4uLi5Nprr5Xu3btLv379RERk2LBh8tprrxXpIHOIdfE7ViIiIqISrER800NERERkwkkPERERuQInPUREROQKnPQQERGRK3DSQ0RERK7ASQ8RERG5Aic9RERE5Ap+b0NR3FpN0+UVZmsmjqOSg+OIgoHjiILB33HEb3qIiIjIFTjpISIiIlfgpIeIiIhcgZMeIiIicgVOeoiIiMgVOOkhIiIiV+Ckh4iIiFyBkx4iIiJyBU56iIiIyBU46SEiIiJX4KSHiIiIXIGTHiIiInIFTnqIiIjIFTjpISIiIlfgpIeIiIhcgZMeIiIicgVOeoiIiMgVOOkhIiIiV+Ckh4iIiFyBkx4iIiJyhdKFfQJERESUP/r27Wv72fDhw1W9fv16Vf/000+qnj17dvBPrJDwmx4iIiJyBU56iIiIyBU46SEiIiJXYKbHICoqStX16tVT9X/+858CPBsiIiL/XXvttbafhYeHqzoxMVHVf/vb31T97rvvqvqxxx7L0zlVqFDB9rNnnnlG1TVq1FD1Aw88oOpz587l6rn5TQ8RERG5Aic9RERE5Aqc9BAREZErhFiWZfl1x5CQ/D6XQufUz+Dll19W9Q8//KDqTz/9VNXr1q0L/okZDBw4UNWbN29W9dKlS1Xt51ueL9wwjtyC44iCgeMoMKGhoao+f/68z/svWbLE9rPjx4+rumLFiqo+deqUqjt27KjqVq1aqXrFihU+zyEiIkLVCxcutN2natWqqi5fvryq+/Tpo+qff/5Z1f6OI37TQ0RERK7ASQ8RERG5Aic9RERE5Aolqk9PqVJ6DnfhwgVV161bV9XYewB78IiIbNu2TdVxcXGq/uSTT1R93XXX+Xey/7+wsDBV33fffaquVq2a7TG41pmdna3qvXv3BnQObofr+oFmDEaMGKHqlStX2u5z8OBBVbdp00bVf/75p6pXr16t6j179gR0Trnx9NNPqxrzaXPmzMn3cyAi30zXpypVqqi6fv36tvts3LhR1WXLllV1VlaWqrds2aLq5cuXq/rrr79W9Y4dO1T9+OOPqxo/V0VE9u/fr2rsJXTo0CHbY3KD3/QQERGRK3DSQ0RERK7ASQ8RERG5QonK9Jh6LlSuXFnVTZo0UXVGRobtMZi1wH1McH+Qu+66S9VpaWmqvvXWW1Xdq1cvVWNe55dffrGdU0pKiqoLozdQSYJ9L3JycnzePzk5WdVffvmlqnHMiIj07NlT1c2bN1f1yZMnVf3QQw+pGtfAly1bpmpcY8c1e9xDTkSkS5cuqo6MjFQ1jkVmegofXuNwLOI42bp1q8/HF2aPHModzKqiO++8U9VHjx613Qfzr9jrB3NBeH3atGmTqrt166ZqzKquX79e1WfPnrWdU6VKlVSN15+rrrpK1bn93OM3PUREROQKnPQQERGRK3DSQ0RERK5QpPbewnVGPLVgrz9jXsapT89vv/3m8xi4X0jbtm1VvXv3blWnp6er+vPPP1f12rVrVb1v3z6fzy9if29Kl9ZRrXPnzqnabXvdmPo3oejoaFX369dP1ThOcB8bpzV0fA9Onz6taswR4Xr2sWPHVI3jrlatWqrGvkBO+/PMmDFD1QMGDFB148aNVT1o0CBVu20cmc4hr69HgwYNVP3cc8/Z7oO5ww4dOqgac1fvvPNOns4pNx5++GFV4zUPr6luH0fBhp8ZeO0QETlz5oyq8fqErwtmdPB2vH7he4q95PD6JiJSpkwZVWOu6MMPP1T1Cy+84PM5L4ff9BAREZErcNJDRERErsBJDxEREblCgfXpwVyFiD1bYcpaBNuTTz6p6gULFtju06NHD1VjfmPXrl2qPnDggKofeeQRVf/8888Bn6cJrmXi+mxJ4rQGjz/D2inPcinsMTFy5EhVjx8/XtXY+wT7PTmpWbOmqvE9u/LKK1WNa+D493Pq1Cmft584cULVX331le2c8O8N+2BgXyunzFtJYsp3BJoxxIxCTEyMqrt3767q2rVrm05RYmNjVY0ZHnzPrr/+elX/5z//MT6HLwkJCbaf/b//9/9Ujec4e/ZsVZtykqSZsmPYgwvzfU650CuuuELV+JmBt+M54P0xR4pj//Dhw7ZzQHjMiIgIVWNeNrf4TQ8RERG5Aic9RERE5Aqc9BAREZEr+J3pCbQnBd7fn7wOrkXefffdqr7ppptU3blzZ+Mxffn9999VjX1LnJ4T8yGYpcA9Sm6//XZVmzI9uA8U7kciYu+ZgD0P6tSpo+ojR474fM7ixCl3gWML3xPM3OC+MdgP5b777lM1vt64v9GUKVN8nLF/cP36xhtvVLXH41E19nTBDBDmjqpXr257TswZYa4I98cpSpkeU44rNz2+8tov5uqrr1b1q6++qmr82965c6eqsQePUw4CM4W33XabqrFHFO7t16ZNG1VnZmaqGnMV+LeD+7OJ2HNCmDHBjA/5htdz7KmD4/T5559XNe79h2NGxD4W8ZrplMG9FGZ4sMbrUdmyZX3eX8T+e+MxOnbs6POc/MVveoiIiMgVOOkhIiIiV+Ckh4iIiFzB7723cI0v0IyP0+3jxo1TdWJioqpxLbJGjRqqXrJkiaofeughn+dggr0FRETuvPNOVeNeN7h+HR4ermrcx+nf//63qufPn69qzE1UrFjReJ647wmuy2/evFnV77//vu2YBcXUCyU3+xm1bt1a1dirBHsl4Xvw2WefqRrf06lTp6oac1q4X5rTejW+R3mFOQnc7wjHgNPYxuwXrrtjFuy9995T9eLFi/072XyQ1z2TnDILmN2qVq2aqjGzg+MM9yrDnNZ///tfVWNOC/dI6tu3r+0cMbOD1w+E4wCvL/g64hjAPInTOK5QoYKqsccLZsfw98LsRkEqCntvBXrNwxwX9mrasmWLqp1eX8ym4jnge4hM+xlithWPh+PMCf494PwA+6v98MMPxmOK8JseIiIicglOeoiIiMgVOOkhIiIiV/C7T09u+l6YrFu3TtUDBw5U9R9//KFq7D3Ss2dPVb/++uuqxj4YJk57VuE6POaGcG0S95VZtWqVqtesWaPq7du3q3rp0qU+j+8E19mrVq2qauzbUJhwLTjQcfXggw/afoYZHBxXCxcuVPUNN9yg6rS0NFXjfkVz585VNWakkNPvYFq3D3Rdf8SIEarGfZswV4R5FRF7DyjMi2AWYO/evT7PqTDhuMIsnSmPI2LP4GDfIswYYq8TfA/Xr1+v6vbt26sa+/AcPHhQ1U5/t5jx27Nnj+0+l8K8DV7P8HfGPdvw8fiaiNivYVlZWarGzB1mpYoyUz8ofzJBmHcJ9Jr39NNPq/qZZ55R9YYNG1SN+T0cpyL2rBY+JtA+PfjZadr/0CkbZroG4n6DzZs393mOl8NveoiIiMgVOOkhIiIiV+Ckh4iIiFyBkx4iIiJyBb+DzBioxUAbNhLyJ+j86aefqhobAWIA9aWXXlI1NifETRrx+FdddZWqcfM93MRRxN5UafXq1apetmyZqjH8iY/HTQVNDRkxaChiD2hjUK1u3bqqnjdvnu0YhcWfjWd92b17t+1nGBbHkB0G4rGZII6LlStXqhobumEIEGFozx+mvxcc20OHDlU1Nua65pprVI0bS4rYG4jh3zD+HkUpyHzXXXepevTo0aqeNGmSqjEUjGFbEftrdOjQIVXj3zZu4or3r1KliqoxRIyBeLxWjB8/3naOGPY0NQbEQKrT9eRS+Dth7fSPPTBcjY/Bf2hQnOTHP+BB3bt3V/WYMWNUjZu+4jgyXW+cNhzF4DJu9olBY/y98TqONV6Dcf6A1x5/HoNBZvz78he/6SEiIiJX4KSHiIiIXIGTHiIiInIFvzM9mHfBzb5wrRnXkp3W8LARVrly5VSNzQdxbfLs2bOq/uSTT1SNa36mhkobN260nSOeE2YnMJPjlDm5FK6p//LLL6rGhks//fST7RiY2cH1Wcyw5DVHU5RgozMRc06hVq1aqj59+rSqMWeFr2/Dhg39P0Fx3nAUmwfixrSYG8LMHJ5T7969Vb1r1y5V42ah+LcmYh//+PeE5xDsTVPzAnNqeG74dxkTExPwc2CGB3MP9evXVzW+nnjtwNcTj4fZPKemcthQEo9pagKHYxOvqfg747jx51qC4wg3HG3ZsqXxGMUFNoIVEUlOTlY1bix76623qho3D8brN+ZG8T3EcWRqFOgPHHum3JDpsxwf73Q8HDf4uYYZH7y/v/hNDxEREbkCJz1ERETkCpz0EBERkSuEWH42HsAN+3CNHNchsbeAU18MzFrgJoDYxwLX0HE9Gs8R+7fgmiD2W8HeJiIiO3bsUHVkZKSqsXcAZpfwOU0bu+E5YdZDxN6nAY9Rp04dVWMOad++fbZjFhTczBOzKfv371c1boLptFkh/gzXuHGzTawxp4CvJ65XT5kyRdW4xu6UfcF+Kvic+BhcrzatiWOuCXun4Lhz+hlmAWrUqKHqiRMnqvrAgQO2YxYUzLY4XV/yG75++J5irsGUtcAeO07wfcdjYp0ffWXyCsdmYbx3F+F70LFjR1U/99xzqsaeXvg3ImLfBBZ/X7ye4Eaz+J7hZ4hpM1C8vuFnlIj9+oIZHrym4e2mPA3eHz8XnfJq+Jz4OmCe8+abb1Y1bgx9Ofymh4iIiFyBkx4iIiJyBU56iIiIyBX8zvTgfio9evRQNa7Z4Zr74cOHbcfEDA+ubWIPinXr1qkaMzy4NhwXF6dq7F1i6vsjYl+zxfVZPCauueO6Pq7H4u+M65r4uovY+8zgY3CdeubMmarGPZYKEr7nuKcU5pHw9XPKkeAQxmwE9sTBDBA+B+Zv8D3D7Bf2+cHHizj37rmUqZdGfHy8qvE9NO1955TpQfj3hXmyIUOGqNrUkyo/4euF7zHW/vQdwVxVoL1K8DXHcYV/t3g8U1ZDxP57m8aNqX8a1njOOG792VcOj4mZFLzmFWY2DHOlaWlpqsaMlFO/K4SvEf7t4ecMXhOdetpdCl9ffI/xPXTaK9BpD7VLmfbews9OfI/xdcXXxOmc8PMer+P42YCfzdhj6nL4TQ8RERG5Aic9RERE5Aqc9BAREZEr+L33FuZlsCcF3m7qP+P0GMz9YH6madOmPp8D1/iwzw+u0eOaodOaOvaJwf25cH8v7IETHR2tanwd8Bwwm5GZmWk7J+xDs3PnTp/P6e9aZ0HA13P69OkBPd6pvwO+hpipwb4V+D5j/gPfI1MeBnvkOOUscFyYclm4vo3Pga8DrtHj345TFgP//jBLgJkdp1xeUYG/S2H2fqHi46GHHlI1ZlNwHOG11ykSi5+NmH8x9ezCv32EeRv828a+PE59w/CaidcTzC7h64J5P8zwYE5r7969qna6lpjmGHgN9Cdf5YTf9BAREZErcNJDRERErsBJDxEREbmC35ke7C2A2QzMkeD9MfsiYs9S4BodZiNw3dHUiwT7H2DPG+xngDkIp3PAtU1TjxzTOmXt2rVVjWulTvumYOYEXwfccwyfszhzyqbguDHtbUNEJGLPu5j2rcLPNaf90vB6ixke/IzA++P1HWu8Bpr6SflzjvhZiJ9D+Fm9cOFCVT/77LOq7tatm+05L+WUM8LfA1+3qlWr+jymv/hNDxEREbkCJz1ERETkCpz0EBERkSv4vfeWaY8X0/5E+O/4Rex9derWratqzGbg2iSuC2LOCDM+uE8W9gpw6meDz7F//36fxww0P4PrmPiaOO2RguuvuIaLvX5wXdrPtzxfmMYRFR8cRxQMRWkcvffee6ru1KmTqjGb6tQPCj8DsP8VXtPxcw2v56bXB4+Hz++UVcV8DD7mnXfeUfW4ceN8ngP68ccfVY19ypz+fjEvhfkqnFO0atVK1f6OI37TQ0RERK7ASQ8RERG5Aic9RERE5ApBy/RQ8VGU1tCp+OI4omAoyuMIc6GPPfaYqgcNGmR7TJ06dVSNOSDMjmKN2RbMZGJvOFOPHcwUiYi88sorqh49erTtPnmBGR7Mvjr1W8M9xw4dOqRqfF0bNmyoaqf9PZ3wmx4iIiJyBU56iIiIyBU46SEiIiJXYKbHhYryGjoVHxxHFAxFaRxhv7lg7FuIvX4SEhJUHRsbq+rIyEhVR0RE+Dw+ZllmzZql6tdff92Ps/Qt0Ndl8ODBqsZcEeaWROzZJuy7t2LFCp/PyT49RERERJfgpIeIiIhcgZMeIiIicgVOeoiIiMgVGGR2oaIUHKTii+OIgoHjiIKBQWYiIiKiS3DSQ0RERK7ASQ8RERG5Aic9RERE5Aqc9BAREZErcNJDRERErsBJDxEREbkCJz1ERETkCpz0EBERkStw0kNERESuwEkPERERuQInPUREROQKnPQQERGRK3DSQ0RERK7ASQ8RERG5QohlWVZhnwQRERFRfuM3PUREROQKnPQQERGRK3DSQ0RERK7ASQ8RERG5Aic9RERE5Aqc9BAREZErcNJDRERErsBJDxEREbkCJz1ERETkCv8fKIhr+SLmwcUAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: a 4x4 grid of FashionMNIST images with their class names\n",
"fig, axes = plt.subplots(4, 4, figsize=(6, 6))\n",
"for i, ax in enumerate(axes.flat):\n",
" img, label = train_full[i]\n",
" ax.imshow(img.squeeze(0).numpy(), cmap=\"gray\")\n",
" ax.set_title(CLASSES[label], fontsize=8)\n",
" ax.axis(\"off\")\n",
"plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "f9e6c61e",
"metadata": {},
"source": [
"> **Interpretation.** Low-resolution but recognizable clothing. The model's job is\n",
"to map each 28x28 = 784-pixel vector to one of 10 class labels. Harder than digits\n",
"(shirts and coats and pullovers blur together), which is why ~85% is a respectable\n",
"score here where MNIST digits would give ~98%.\n"
]
},
{
"cell_type": "markdown",
"id": "7cb82bf8",
"metadata": {},
"source": [
"We take a fixed subset (sizes from CONFIG) so the notebook stays inside the\n",
"CPU budget, and flatten each image to a 784-vector for the MLP. The custom `Dataset`\n",
"below is the minimal pattern: store the tensors, return one `(x, y)` pair per index.\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "7818cd95",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:57.750994Z",
"iopub.status.busy": "2026-06-10T19:01:57.750909Z",
"iopub.status.idle": "2026-06-10T19:01:58.698517Z",
"shell.execute_reply": "2026-06-10T19:01:58.698176Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"train features (12000, 784) · test features (4000, 784)\n",
"label range: 0 to 9\n"
]
}
],
"source": [
"def take_subset(ds, n):\n",
" # Stack the first n examples into flat (n, 784) features and (n,) integer labels.\n",
" xs = torch.stack([ds[i][0].view(-1) for i in range(n)]) # (n, 784)\n",
" ys = torch.tensor([ds[i][1] for i in range(n)]) # (n,)\n",
" return xs, ys\n",
"\n",
"Xtr, ytr = take_subset(train_full, TRAIN_N)\n",
"Xte, yte = take_subset(test_full, TEST_N)\n",
"print(\"train features\", tuple(Xtr.shape), \"· test features\", tuple(Xte.shape))\n",
"print(\"label range:\", int(ytr.min()), \"to\", int(ytr.max()))"
]
},
{
"cell_type": "markdown",
"id": "0c5d4ca2",
"metadata": {},
"source": [
"### Exercise 10.5 — Write a Dataset\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"Implement `FlatImageDataset`, a map-style `Dataset` over the `(X, y)` tensors above.\n",
"It needs `__len__` returning the number of examples and `__getitem__(i)` returning the\n",
"pair `(X[i], y[i])`. The check builds one, asserts the length, and asserts that a\n",
"`DataLoader` over it yields the right batch shapes.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "838acbe1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:58.699670Z",
"iopub.status.busy": "2026-06-10T19:01:58.699596Z",
"iopub.status.idle": "2026-06-10T19:01:58.703429Z",
"shell.execute_reply": "2026-06-10T19:01:58.703089Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 10.5 FlatImageDataset + DataLoader: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from torch.utils.data import Dataset, DataLoader\n",
"\n",
"class FlatImageDataset(Dataset):\n",
" \"\"\"Map-style dataset over feature matrix X (n, 784) and label vector y (n,).\"\"\"\n",
" def __init__(self, X, y):\n",
" self.X = X\n",
" self.y = y\n",
"\n",
" def __len__(self):\n",
" # TODO 1: return the number of examples\n",
" raise NotImplementedError\n",
"\n",
" def __getitem__(self, i):\n",
" # TODO 2: return the pair (features, label) for index i\n",
" raise NotImplementedError\n",
"\n",
"def _check_dataset():\n",
" ds = FlatImageDataset(Xtr, ytr)\n",
" assert len(ds) == TRAIN_N, f\"len should be {TRAIN_N}, got {len(ds)}\"\n",
" xb, yb = ds[0]\n",
" check_shape(xb, (784,))\n",
" assert yb.shape == (), \"a single label should be a scalar tensor\"\n",
" dl = DataLoader(ds, batch_size=BATCH, shuffle=False)\n",
" bx, by = next(iter(dl))\n",
" check_shape(bx, (BATCH, 784))\n",
" assert by.shape == (BATCH,), f\"batch labels should be ({BATCH},), got {tuple(by.shape)}\"\n",
"\n",
"check(\"10.5 FlatImageDataset + DataLoader\", _check_dataset)"
]
},
{
"cell_type": "markdown",
"id": "3d41bb20",
"metadata": {},
"source": [
"Hint 1 (conceptual)`__len__` is just the row count of `self.X`; `__getitem__(i)` returns the i-th feature row and the i-th label as a tuple.\n",
"\n",
"Hint 2 (pseudocode)\n",
"```python\n",
"def __len__(self):\n",
" return len(self.X)\n",
"def __getitem__(self, i):\n",
" return self.X[i], self.y[i]\n",
"```\n",
"\n",
"\n",
"Help — DataLoader gives the wrong batch shapeIf batches come out as `(BATCH, 1, 784)` you stored an extra axis; `Xtr` is already `(n, 784)`, so `self.X[i]` is `(784,)`. If `__len__` returns a tensor, wrap it in `int(...)` or return `len(self.X)`.\n"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "69426222",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:58.704296Z",
"iopub.status.busy": "2026-06-10T19:01:58.704228Z",
"iopub.status.idle": "2026-06-10T19:01:58.707726Z",
"shell.execute_reply": "2026-06-10T19:01:58.707366Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 10.5 FlatImageDataset + DataLoader\n",
"train batches: 93 · test batches: 32 (batch size 128)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines FlatImageDataset; the check below re-verifies batch shapes.\n",
"class FlatImageDataset(Dataset):\n",
" def __init__(self, X, y):\n",
" self.X = X\n",
" self.y = y\n",
" def __len__(self):\n",
" return len(self.X)\n",
" def __getitem__(self, i):\n",
" return self.X[i], self.y[i]\n",
"\n",
"check(\"10.5 FlatImageDataset + DataLoader\", _check_dataset, required=True)\n",
"train_ds = FlatImageDataset(Xtr, ytr)\n",
"test_ds = FlatImageDataset(Xte, yte)\n",
"# A seeded generator makes the shuffle reproducible (spec determinism policy).\n",
"g = torch.Generator().manual_seed(SEED)\n",
"train_dl = DataLoader(train_ds, batch_size=BATCH, shuffle=True, generator=g, drop_last=True)\n",
"test_dl = DataLoader(test_ds, batch_size=BATCH, shuffle=False)\n",
"print(f\"train batches: {len(train_dl)} · test batches: {len(test_dl)} (batch size {BATCH})\")"
]
},
{
"cell_type": "markdown",
"id": "3479b7bb",
"metadata": {},
"source": [
"> **Note:** `shuffle=True` matters during training (it decorrelates consecutive\n",
"batches); we pass a seeded `generator` so the shuffle is reproducible. `drop_last=True`\n",
"drops the final partial batch so every step sees a full `BATCH`. We leave `num_workers`\n",
"at its default 0 here: workers help on a GPU box with heavy I/O, but on this CPU subset\n",
"they would only add process-startup overhead.\n",
"\n",
"> **Key takeaways.** A `Dataset` is `__len__` + `__getitem__`. A `DataLoader` batches\n",
"and shuffles it; seed its generator for reproducibility. Set `shuffle=True` for\n",
"training, `drop_last=True` to keep batch sizes uniform, and reach for `num_workers>0`\n",
"only when data loading is the bottleneck.\n"
]
},
{
"cell_type": "markdown",
"id": "c4543f39",
"metadata": {},
"source": [
"## Part 6 — The training loop\n",
"\n",
"> **Objectives.** Build the from-scratch pieces (`my_cross_entropy`, a tiny `SGD`),\n",
"> verify each against torch, then assemble the four-line training step into a real run\n",
"> that classifies FashionMNIST above 80%. Then break it on purpose and fix it.\n",
"\n",
"The canonical PyTorch training step is four lines, and they have a fixed order:\n",
"\n",
"```\n",
"opt.zero_grad() # 1. clear last step's gradients (they accumulate)\n",
"logits = model(x) # 2. forward\n",
"loss.backward() # 3. backward, populates .grad on every parameter\n",
"opt.step() # 4. update parameters in place\n",
"```\n",
"\n",
"Before we run it, we build two of the pieces from scratch and check them, so nothing in\n",
"the loop is a black box.\n"
]
},
{
"cell_type": "markdown",
"id": "36a8a352",
"metadata": {},
"source": [
"### Exercise 10.6 — Numerically stable cross-entropy\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"Implement `my_cross_entropy(logits, targets)`: `logits` is `(batch, n_classes)`,\n",
"`targets` is `(batch,)` of integer class indices. Return the mean negative\n",
"log-likelihood. Do it the stable way (subtract the log-sum-exp, do not exponentiate\n",
"raw logits). The check compares against `F.cross_entropy`, the oracle, to `1e-6`.\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "02f08b8b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:58.708731Z",
"iopub.status.busy": "2026-06-10T19:01:58.708623Z",
"iopub.status.idle": "2026-06-10T19:01:58.713116Z",
"shell.execute_reply": "2026-06-10T19:01:58.712819Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 10.6 my_cross_entropy: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def my_cross_entropy(logits, targets):\n",
" \"\"\"logits: (batch, n_classes); targets: (batch,) int class indices.\n",
" Return mean negative log-likelihood, computed stably.\"\"\"\n",
" # TODO 1: log_probs = logits - logits.logsumexp(dim=-1, keepdim=True)\n",
" # (this is log_softmax; logsumexp is the stable softmax denominator)\n",
" log_probs = None\n",
" # TODO 2: pick out the log-prob of the true class for each row.\n",
" # log_probs.gather(1, targets.long().unsqueeze(1)).squeeze(1) does it.\n",
" chosen = None\n",
" attempted(log_probs, chosen)\n",
" # TODO 3: return the mean of -chosen\n",
" raise NotImplementedError\n",
"\n",
"def _check_ce():\n",
" torch.manual_seed(SEED)\n",
" logits = torch.randn(8, 5)\n",
" targets = torch.randint(0, 5, (8,))\n",
" check_close(my_cross_entropy(logits, targets), F.cross_entropy(logits, targets),\n",
" atol=1e-6, msg=\"must match torch.nn.functional.cross_entropy\")\n",
" # a uniform-logit row should give loss log(n_classes); here log(5) ~ 1.609\n",
" flat = torch.zeros(1, 5)\n",
" check_close(my_cross_entropy(flat, torch.tensor([2])), np.log(5), atol=1e-5,\n",
" msg=\"uniform logits over 5 classes -> loss log(5)\")\n",
"\n",
"check(\"10.6 my_cross_entropy\", _check_ce)"
]
},
{
"cell_type": "markdown",
"id": "a57af207",
"metadata": {},
"source": [
"Hint 1 (conceptual)Cross-entropy is the negative log-probability of the true class. Compute `log_softmax` stably with `logits - logits.logsumexp(...)`, gather the true-class entries, negate, average.\n",
"\n",
"Hint 2 (pseudocode)\n",
"```python\n",
"log_probs = logits - logits.logsumexp(dim=-1, keepdim=True)\n",
"chosen = log_probs.gather(1, targets.long().unsqueeze(1)).squeeze(1)\n",
"return (-chosen).mean()\n",
"```\n",
"\n",
"\n",
"Help — \"must match ... cross_entropy\" with ~50% wrongYou probably forgot the `.squeeze(1)` after `gather`, leaving a `(batch, 1)` shape that then broadcasts oddly, or you exponentiated the logits directly (overflow). Use the logsumexp form; do not call `.exp()` on raw logits.\n",
"\n",
"Help — \"index out of range\" in gather`gather` needs an int64 index of shape `(batch, 1)`. Use `targets.long().unsqueeze(1)`. If targets are floats, the `.long()` is what fixes it.\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "e20f77bf",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:58.713904Z",
"iopub.status.busy": "2026-06-10T19:01:58.713832Z",
"iopub.status.idle": "2026-06-10T19:01:58.717695Z",
"shell.execute_reply": "2026-06-10T19:01:58.717353Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 10.6 my_cross_entropy\n",
"loss on uniform logits over 10 classes: 2.3026 (log 10 = 2.3026; this is the expected loss at init)\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines my_cross_entropy; the check below re-verifies against F.cross_entropy.\n",
"def my_cross_entropy(logits, targets):\n",
" log_probs = logits - logits.logsumexp(dim=-1, keepdim=True)\n",
" chosen = log_probs.gather(1, targets.long().unsqueeze(1)).squeeze(1)\n",
" return (-chosen).mean()\n",
"\n",
"check(\"10.6 my_cross_entropy\", _check_ce, required=True)\n",
"print(\"loss on uniform logits over 10 classes:\",\n",
" f\"{my_cross_entropy(torch.zeros(1, 10), torch.tensor([0])).item():.4f}\",\n",
" f\"(log 10 = {np.log(10):.4f}; this is the expected loss at init)\")"
]
},
{
"cell_type": "markdown",
"id": "68056ed1",
"metadata": {},
"source": [
"> **Interpretation.** A freshly initialized 10-class classifier should print a loss\n",
"near `log(10) = 2.303`, because it has learned nothing and spreads its probability\n",
"evenly. That single number is the cheapest sanity check on a training run: if step 0 is\n",
"nowhere near `log(n_classes)`, something is wrong before you have trained at all\n",
"(Karpathy's \"verify the loss at init\").\n"
]
},
{
"cell_type": "markdown",
"id": "9831f856",
"metadata": {},
"source": [
"### Exercise 10.7 — A tiny SGD optimizer\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"Implement `step()` for `TinySGD`, plain SGD with momentum. For each parameter `p` with\n",
"gradient `p.grad`, maintain a velocity `v <- momentum*v + p.grad` and update\n",
"`p <- p - lr*v`. Do the update under `torch.no_grad()` and write to `p` in place so you\n",
"do not build a graph through the optimizer. The check trains a toy problem with both\n",
"`TinySGD` and `torch.optim.SGD` from the same seed and asserts the parameters end up\n",
"within `1e-5` of each other.\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "19d2840c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:58.718554Z",
"iopub.status.busy": "2026-06-10T19:01:58.718487Z",
"iopub.status.idle": "2026-06-10T19:01:58.724837Z",
"shell.execute_reply": "2026-06-10T19:01:58.724569Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 10.7 TinySGD matches torch.optim.SGD: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class TinySGD:\n",
" \"\"\"SGD with momentum. params: an iterable of leaf tensors with .grad populated.\"\"\"\n",
" def __init__(self, params, lr=0.1, momentum=0.9):\n",
" self.params = list(params)\n",
" self.lr = lr\n",
" self.momentum = momentum\n",
" self.v = [torch.zeros_like(p) for p in self.params] # velocity buffer per param\n",
"\n",
" def zero_grad(self):\n",
" for p in self.params:\n",
" if p.grad is not None:\n",
" p.grad.zero_()\n",
"\n",
" def step(self):\n",
" with torch.no_grad():\n",
" for i, p in enumerate(self.params):\n",
" if p.grad is None:\n",
" continue\n",
" # TODO 1: v[i] = momentum * v[i] + p.grad\n",
" # TODO 2: p -= lr * v[i] (in place, on the no_grad tensor)\n",
" raise NotImplementedError\n",
"\n",
"def _check_sgd():\n",
" # Two identical tiny linear models from the same seed, same data, same steps.\n",
" def make_model():\n",
" torch.manual_seed(SEED)\n",
" return nn.Linear(4, 2)\n",
" x = torch.randn(16, 4); y = torch.randint(0, 2, (16,))\n",
" m_ref = make_model(); m_mine = make_model()\n",
" opt_ref = torch.optim.SGD(m_ref.parameters(), lr=0.1, momentum=0.9)\n",
" opt_mine = TinySGD(m_mine.parameters(), lr=0.1, momentum=0.9)\n",
" for _ in range(20):\n",
" opt_ref.zero_grad(); F.cross_entropy(m_ref(x), y).backward(); opt_ref.step()\n",
" opt_mine.zero_grad(); F.cross_entropy(m_mine(x), y).backward(); opt_mine.step()\n",
" for pr, pm in zip(m_ref.parameters(), m_mine.parameters()):\n",
" # parameters require grad; detach before the numpy comparison\n",
" check_close(pm.detach(), pr.detach(), atol=1e-5,\n",
" msg=\"TinySGD must track torch.optim.SGD step for step\")\n",
"\n",
"check(\"10.7 TinySGD matches torch.optim.SGD\", _check_sgd)"
]
},
{
"cell_type": "markdown",
"id": "a1ebf6ca",
"metadata": {},
"source": [
"Hint 1 (conceptual)Two assignments per parameter. Update the velocity, then subtract `lr*velocity` from the parameter. The velocity is stored in `self.v[i]`.\n",
"\n",
"Hint 2 (pseudocode)\n",
"```python\n",
"self.v[i] = self.momentum * self.v[i] + p.grad\n",
"p -= self.lr * self.v[i]\n",
"```\n",
"The `-=` is in place; because we are inside `torch.no_grad()`, it does not record a graph.\n",
"\n",
"Help — parameters diverge from torch.optim.SGDPyTorch's SGD applies momentum as `v = momentum*v + grad` and `p -= lr*v` (the \"no dampening\" form). If you wrote `p -= lr*grad` you dropped momentum entirely; if you scaled grad by lr before adding to v, the formula differs. Match the two-line form above.\n",
"\n",
"Help — \"a leaf Variable that requires grad is being used in an in-place operation\"The in-place `p -= ...` must be inside `with torch.no_grad():`. It already is in the stub; make sure you did not move it out.\n"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "192ffe8f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:58.725758Z",
"iopub.status.busy": "2026-06-10T19:01:58.725692Z",
"iopub.status.idle": "2026-06-10T19:01:58.733671Z",
"shell.execute_reply": "2026-06-10T19:01:58.733352Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 10.7 TinySGD matches torch.optim.SGD\n",
"[ ok ] from-scratch momentum SGD reproduces torch.optim.SGD\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines TinySGD.step; the check below re-verifies against torch.optim.SGD.\n",
"class TinySGD:\n",
" def __init__(self, params, lr=0.1, momentum=0.9):\n",
" self.params = list(params)\n",
" self.lr = lr\n",
" self.momentum = momentum\n",
" self.v = [torch.zeros_like(p) for p in self.params]\n",
" def zero_grad(self):\n",
" for p in self.params:\n",
" if p.grad is not None:\n",
" p.grad.zero_()\n",
" def step(self):\n",
" with torch.no_grad():\n",
" for i, p in enumerate(self.params):\n",
" if p.grad is None:\n",
" continue\n",
" self.v[i] = self.momentum * self.v[i] + p.grad\n",
" p -= self.lr * self.v[i]\n",
"\n",
"check(\"10.7 TinySGD matches torch.optim.SGD\", _check_sgd, required=True)\n",
"print(\"[ ok ] from-scratch momentum SGD reproduces torch.optim.SGD\")"
]
},
{
"cell_type": "markdown",
"id": "a7fdfef7",
"metadata": {},
"source": [
"Now assemble the model. A two-layer MLP, `784 -> 128 -> 10`, the standard\n",
"FashionMNIST baseline. We run a `randn` smoke test on a dummy batch before training:\n",
"the output must be `(batch, 10)` logits, and the loss at init must sit near\n",
"`log(10) = 2.303`. This catches shape and init bugs in one second, before we spend\n",
"minutes training.\n"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "64c7cc50",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:58.734766Z",
"iopub.status.busy": "2026-06-10T19:01:58.734682Z",
"iopub.status.idle": "2026-06-10T19:01:58.738881Z",
"shell.execute_reply": "2026-06-10T19:01:58.738562Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"smoke test: logits (128, 10) · loss-at-init 2.332 (expect ~2.303)\n",
"[ ok ] model shape and init loss look right\n"
]
}
],
"source": [
"def make_mlp():\n",
" torch.manual_seed(SEED) # re-seed so the model inits identically on a re-run\n",
" return nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10))\n",
"\n",
"model = make_mlp()\n",
"dummy = torch.randn(BATCH, 784) # smoke test on noise\n",
"logits = model(dummy)\n",
"check_shape(logits, (BATCH, 10))\n",
"init_loss = F.cross_entropy(logits, torch.randint(0, 10, (BATCH,))).item()\n",
"print(f\"smoke test: logits {tuple(logits.shape)} · loss-at-init {init_loss:.3f} (expect ~{np.log(10):.3f})\")\n",
"assert 2.0 < init_loss < 2.7, \"loss at init should be near log(10)=2.303; if not, init is off\"\n",
"print(\"[ ok ] model shape and init loss look right\")"
]
},
{
"cell_type": "markdown",
"id": "348a594f",
"metadata": {},
"source": [
"Here is the training loop, with the four lines flagged by the exact comments\n",
"you will use in every chapter from here on. The eval pass is wrapped in\n",
"`torch.no_grad()` and toggles `model.eval()`/`model.train()`, the discipline that keeps\n",
"dropout and batchnorm honest (this MLP has neither, but the habit is the point).\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "62a24cb0",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:58.740055Z",
"iopub.status.busy": "2026-06-10T19:01:58.739987Z",
"iopub.status.idle": "2026-06-10T19:01:59.046599Z",
"shell.execute_reply": "2026-06-10T19:01:59.046136Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"training 3 epoch(s) on 12000 images...\n",
"epoch 0: train_loss 0.5966 val_loss 0.5912 val_acc 0.7903\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch 1: train_loss 0.4692 val_loss 0.5816 val_acc 0.7860\n",
"epoch 2: train_loss 0.4831 val_loss 0.4943 val_acc 0.8157\n"
]
}
],
"source": [
"@torch.no_grad()\n",
"def evaluate(model, dl):\n",
" model.eval()\n",
" total, correct, loss_sum = 0, 0, 0.0\n",
" for xb, yb in dl:\n",
" logits = model(xb)\n",
" loss_sum += F.cross_entropy(logits, yb, reduction=\"sum\").item()\n",
" correct += (logits.argmax(1) == yb).sum().item()\n",
" total += yb.size(0)\n",
" model.train()\n",
" return loss_sum / total, correct / total\n",
"\n",
"def train(model, train_dl, test_dl, lr=0.1, epochs=EPOCHS):\n",
" opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9)\n",
" log = []\n",
" for epoch in range(epochs):\n",
" for xb, yb in train_dl:\n",
" opt.zero_grad() # 1. clear last step's gradients\n",
" logits = model(xb) # 2. forward\n",
" loss = F.cross_entropy(logits, yb)\n",
" loss.backward() # 3. backward\n",
" opt.step() # 4. update\n",
" val_loss, val_acc = evaluate(model, test_dl)\n",
" log.append((epoch, loss.item(), val_loss, val_acc))\n",
" print(f\"epoch {epoch}: train_loss {loss.item():.4f} val_loss {val_loss:.4f} val_acc {val_acc:.4f}\")\n",
" return log\n",
"\n",
"print(f\"training {EPOCHS} epoch(s) on {TRAIN_N} images...\")\n",
"model = make_mlp()\n",
"history = train(model, train_dl, test_dl)"
]
},
{
"cell_type": "markdown",
"id": "6cf1e233",
"metadata": {},
"source": [
"> **Runtime:** this cell is the heaviest in the notebook. On CPU it is a few\n",
"seconds per epoch on the subset (well under the budget); the full 60k-image, more-epochs\n",
"version is the capstone and still finishes in a couple of minutes.\n",
"\n",
"> **Interpretation.** Loss falls and accuracy climbs each epoch. With the subset and a\n",
"plain MLP you should land somewhere in the low-to-mid 80s; the full-data capstone pushes\n",
"it higher. The four flagged lines are the entire engine, everything else (eval,\n",
"logging) is scaffolding around them.\n"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "0ba1167e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:59.047878Z",
"iopub.status.busy": "2026-06-10T19:01:59.047799Z",
"iopub.status.idle": "2026-06-10T19:01:59.119849Z",
"shell.execute_reply": "2026-06-10T19:01:59.119363Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAGGCAYAAACNCg6xAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAcZlJREFUeJzt3XlcVFXjBvBnZmBm2PddBFQ0d3MBUdwSBSvSFncT3Pq9paaZltaraPaGpZmWltmrYmWuWdbrLmklophLuaIoiBuLIPs+c39/jAwM6wzbsDzfz2c+Mveee+ZcZpCHc849VyQIggAiIiIiqpZY3w0gIiIiaioYnIiIiIi0xOBEREREpCUGJyIiIiItMTgRERERaYnBiYiIiEhLDE5EREREWmJwIiIiItISgxMRERGRlhiciOpBcHAwTE1NtSorEomwdOnS+m0Q6V1TeZ/d3d0RHBys72YQNVoMTtRihIWFQSQSVfhYuHChvptXrwYPHgyRSARPT88K9x89elT9vdizZ496e/H3TC6X4/79+xXW26VLF41t7u7ueP755zW2ZWVlISQkBF26dIGJiQlsbGzQo0cPzJkzBw8ePEBcXFyl703ZR1xcXKXnWVxm+vTpFe5///331WUePXqk3q5N0C37+ZHL5Wjfvj1mzZqFxMTEKo8loubDQN8NIGpoH3zwATw8PDS2lf3l35Byc3NhYFD/P4pyuRwxMTGIioqCl5eXxr5t27ZBLpcjLy+vwmPz8/OxYsUKfPHFFzq/bmFhIQYOHIjr168jKCgIs2fPRlZWFq5cuYIffvgBL774Ivr06YPvvvtO47hPP/0U9+7dw2effaax3c7Ortrz/PHHH/Hll19CKpVq7Nu+fXuV56mN4s9PXl4eTp48ia+++goHDhzA5cuXYWxsXON6G4vo6GiIxfybmqgyDE7U4owYMQK9e/fWdzPU5HJ5g7xO27ZtUVRUhO3bt2sEp7y8PPz000947rnn8OOPP1Z4bI8ePfDNN99g0aJFcHZ21ul1f/75Z1y4cAHbtm3DhAkTNPbl5eWhoKAAJiYmmDRpksa+HTt24PHjx+W2VycgIAC//PILDh48iJEjR6q3nzp1CrGxsXj55ZcrPU9tlP78TJ8+HTY2Nli9ejX27duH8ePH17je+lBUVASlUlkuQFZFJpPVY4v0Kzs7GyYmJvpuBjVx/LOC6Ik7d+7gjTfeQIcOHWBkZAQbGxuMHj263NBQYWEhli1bBk9PT8jlctjY2MDX1xdHjx4tV+f9+/cxatQomJqaws7ODvPnz4dCodAoU9HclwsXLmDEiBEwNzeHqakphg4ditOnT2uUKR46ioiIwLx582BnZwcTExO8+OKLSE5OrvAcx48fj507d0KpVKq3/frrr8jJycGYMWMq/d689957UCgUWLFiRaVlKnPr1i0AQP/+/cvtk8vlMDc317nOqri4uGDgwIH44YcfNLZv27YNXbt2rfPexWeeeQYAEBsbq/Ox9+/fx9SpU+Hg4ACZTIbOnTtj8+bNGmUKCgqwZMkS9OrVCxYWFjAxMcGAAQNw/PhxjXLFw52rVq3CmjVr0LZtW8hkMly9ehVLly6FSCRCTEwMgoODYWlpCQsLC0yZMgU5OTka9ZSd46TL50ypVGLp0qVwdnaGsbExhgwZgqtXr2o9b0qpVGLt2rXo2rUr5HI57OzsEBAQgL/++kvjHMPCwsodW/bnqPicr169igkTJsDKygq+vr5YtWoVRCIR7ty5U66ORYsWQSqV4vHjx+ptZ86cQUBAACwsLGBsbIxBgwYhIiKi2nOh5ovBiVqc9PR0PHr0SOMBAGfPnsWpU6cwbtw4fP755/jXv/6F8PBwDB48WOOXy9KlS7Fs2TIMGTIE69atw/vvv4/WrVvj/PnzGq+jUCjg7+8PGxsbrFq1CoMGDcKnn36KjRs3Vtm+K1euYMCAAfj777/xzjvvYPHixYiNjcXgwYNx5syZcuVnz56Nv//+GyEhIXj99dfx66+/YtasWRXWPWHCBDx8+BAnTpxQb/vhhx8wdOhQ2NvbV9omDw8PTJ48Gd988w0ePHhQZfvLcnNzAwB8++23EARBp2NrasKECfj111+RlZUFQNXzsnv37nI9XnWhOBja2NjodFxiYiL69u2LY8eOYdasWVi7di3atWuHadOmYc2aNepyGRkZ+O9//4vBgwfj448/xtKlS5GcnAx/f39cvHixXL1btmzBF198gddeew2ffvoprK2t1fvGjBmDzMxMhIaGYsyYMQgLC8OyZcu0aq82n7NFixZh2bJl6N27N1auXAlPT0/4+/sjOztbq9eYNm0a5s6dC1dXV3z88cdYuHAh5HJ5uT8adDF69Gjk5OTgo48+wowZMzBmzBiIRCLs2rWrXNldu3Zh+PDhsLKyAgD89ttvGDhwIDIyMhASEoKPPvoIaWlpeOaZZxAVFVXjNlETJxC1EFu2bBEAVPgQBEHIyckpd0xkZKQAQPj222/V27p37y4899xzVb5WUFCQAED44IMPNLY//fTTQq9evTS2ARBCQkLUz0eNGiVIpVLh1q1b6m0PHjwQzMzMhIEDB5Y7Hz8/P0GpVKq3v/XWW4JEIhHS0tLU2wYNGiR07txZEARB6N27tzBt2jRBEATh8ePHglQqFbZu3SocP35cACDs3r273GucPXtWuHXrlmBgYCC8+eabFdZbzM3NTeP7k5OTI3To0EEAILi5uQnBwcHCpk2bhMTExCq/h88995zg5uZWZZmyAAgzZ84UUlNTBalUKnz33XeCIAjC/v37BZFIJMTFxQkhISECACE5OVl9XFBQkGBiYlJl3cXfi2PHjgnJycnC3bt3hR07dgg2NjaCkZGRcO/evWrbVvp9njZtmuDk5CQ8evRIo9y4ceMECwsL9eexqKhIyM/P1yjz+PFjwcHBQZg6dap6W2xsrABAMDc3F5KSkjTKF59z6fKCIAgvvviiYGNjo7HNzc1NCAoKKnfe1X3OEhISBAMDA2HUqFEa9S1dulQAoFFnRX777TcBgMbnq1jx6xaf45YtW8qVKfv9LT7n8ePHlyvr4+NT7ucwKipK42ddqVQKnp6egr+/v8Z55+TkCB4eHsKwYcOqPB9qvtjjRC3O+vXrcfToUY0HABgZGanLFBYWIiUlBe3atYOlpaVGb5KlpSWuXLmCmzdvVvta//rXvzSeDxgwALdv3660vEKhwJEjRzBq1Ci0adNGvd3JyQkTJkzAyZMnkZGRoXHMa6+9BpFIpPEaCoWiwqEIQNUbs3fvXhQUFGDPnj2QSCR48cUXqz2XNm3a4NVXX8XGjRvx8OHDassXMzIywpkzZ7BgwQIAqqGfadOmwcnJCbNnz0Z+fr7WdWnLysoKAQEB2L59OwBVr1q/fv3UvV+14efnBzs7O7i6umLcuHEwNTXFTz/9BBcXF63rEAQBP/74IwIDAyEIgkbvp7+/P9LT09WfOYlEop6jpFQqkZqaiqKiIvTu3btcLycAvPzyy5VOoK/o85iSklLuM1WR6j5n4eHhKCoqwhtvvKFx3OzZs6utGwB+/PFHiEQihISElNtX+nV1VfacAWDs2LE4d+6curcQAHbu3AmZTKaeF3fx4kXcvHkTEyZMQEpKivr9yc7OxtChQ/HHH39oDHlTy8HgRC2Ol5cX/Pz8NB6A6uq2JUuWwNXVFTKZDLa2trCzs0NaWhrS09PVx3/wwQdIS0tD+/bt0bVrVyxYsAD//PNPudcpnqNRmpWVlcb8ibKSk5ORk5ODDh06lNvXsWNHKJVK3L17V2N769aty70GgEpfZ9y4cUhPT8fBgwexbds2PP/88zAzM6u0TaX9+9//RlFRkc5znSwsLPDJJ58gLi4OcXFx2LRpEzp06IB169Zh+fLlOtWlrQkTJuDo0aOIj4/Hzz//XGfDdMXB+/jx47h69Spu374Nf39/nepITk5GWloaNm7cCDs7O43HlClTAABJSUnq8lu3bkW3bt3Uc+rs7Oywf/9+jc9lsbJXjJam62dFl2OLA1S7du00yllbW6vLVuXWrVtwdnbWGFqsCxV9P0aPHg2xWIydO3cCUAXZ3bt3q+cVAlD/YRQUFFTuPfrvf/+L/Pz8Cr//1PzxqjqiJ2bPno0tW7Zg7ty58PHxgYWFBUQiEcaNG6fxl+XAgQNx69Yt7Nu3D0eOHMF///tffPbZZ9iwYYPG+kESiaRB2l3Z6wiVzCdycnLC4MGD8emnnyIiIkKnK8zatGmDSZMmYePGjTVe+8rNzQ1Tp07Fiy++iDZt2mDbtm348MMPa1RXVV544QXIZDIEBQUhPz+/ysnvuvDy8qr1VZnFn6dJkyYhKCiowjLdunUDAHz//fcIDg7GqFGjsGDBAtjb20MikSA0NFSjx6RY6Z7TsnT9rNTVsXWlsp6nshdclFbR98PZ2RkDBgzArl278N577+H06dOIj4/Hxx9/rC5T/B6tXLkSPXr0qLBubRe5peaFwYnoiT179iAoKAiffvqpelteXh7S0tLKlbW2tsaUKVMwZcoUZGVlYeDAgVi6dGmlCy9qy87ODsbGxoiOji637/r16xCLxXB1da3VawCq3pjp06fD0tISzz77rE7H/vvf/8b333+v8UumJqysrNC2bVtcvny5VvVUxsjICKNGjcL333+PESNGwNbWtl5epybs7OxgZmYGhUKh7vGszJ49e9CmTRvs3btXIzhUNKSlT8XDoDExMRq9PCkpKVr1aLVt2xaHDx9Gampqpb1OxT1XZX8mKxuWrsrYsWPxxhtvIDo6Gjt37oSxsTECAwM12gMA5ubm1b5H1LJwqI7oCYlEUu6v5y+++KLcX7MpKSkaz01NTdGuXbs6masjkUgwfPhw7Nu3T2MZhMTERPzwww/w9fWtk8v3X3nlFYSEhFS4SGR12rZti0mTJuHrr79GQkJCteX//vtvjVW6i925cwdXr16tcFiyrsyfPx8hISFYvHhxvb1GTUgkEvV6UhUFx9KX+Rf39JT+bJ45cwaRkZH131AdDB06FAYGBvjqq680tq9bt06r419++WUIglDhVX7F525ubg5bW1v88ccfGvu//PJLndv78ssvQyKRYPv27di9ezeef/55jTWeevXqhbZt22LVqlXqqzNLq2zJD2r+2ONE9MTzzz+P7777DhYWFujUqRMiIyNx7NixcpeZd+rUCYMHD0avXr1gbW2Nv/76C3v27Kl0CQBdffjhhzh69Ch8fX3xxhtvwMDAAF9//TXy8/PxySef1MlrWFhY1Oq+ae+//z6+++47REdHo3PnzlWWPXr0KEJCQvDCCy+gb9++MDU1xe3bt7F582bk5+fX6/3bunfvju7du2tVtrCwsMIhQ2tr63ITnuvCihUrcPz4cXh7e2PGjBno1KkTUlNTcf78eRw7dgypqakAVJ/LvXv34sUXX8Rzzz2H2NhYbNiwAZ06darwF7q+ODg4YM6cOfj000/xwgsvICAgAH///TcOHjwIW1vbaid4DxkyBK+++io+//xz3Lx5EwEBAVAqlfjzzz8xZMgQ9c/X9OnTsWLFCkyfPh29e/fGH3/8gRs3bujcXnt7ewwZMgSrV69GZmYmxo4dq7FfLBbjv//9L0aMGIHOnTtjypQpcHFxwf3793H8+HGYm5vj119/1fl1qeljcCJ6Yu3atZBIJNi2bRvy8vLQv39/HDt2rNzE3zfffBO//PILjhw5gvz8fLi5ueHDDz9UXzVWW507d8aff/6JRYsWITQ0FEqlEt7e3vj+++/h7e1dJ69RW+3atcOkSZOwdevWasu+/PLLyMzMxJEjR/Dbb78hNTUVVlZW8PLywttvv40hQ4Y0QIurV1BQUGHPVNu2beslODk4OCAqKgoffPAB9u7diy+//BI2Njbo3LmzxjBocHAwEhIS8PXXX+Pw4cPo1KkTvv/+e+zevVtjPa7G4OOPP4axsTG++eYbHDt2DD4+Pjhy5Ah8fX21WiF/y5Yt6NatGzZt2oQFCxbAwsICvXv3Rr9+/dRllixZguTkZOzZswe7du3CiBEjcPDgwSrXIavM2LFjcezYMZiZmVU4ZD148GBERkZi+fLlWLduHbKysuDo6Ahvb2/83//9n86vR82DSGjImX1ERNSipKWlwcrKCh9++CHef/99fTeHqNY4x4mIiOpEbm5uuW3Fq6APHjy4YRtDVE84VEdERHVi586dCAsLw7PPPgtTU1OcPHkS27dvx/Dhwyu8VyFRU8TgREREdaJbt24wMDDAJ598goyMDPWE8fpYp4tIXzjHiYiIiEhLnONEREREpCUGJyIiIiItcY5TBZRKJR48eAAzM7Na3ZWbiIiIGj9BEJCZmQlnZ2eIxVX3KTE4VeDBgwd1cj8wIiIiajru3r2LVq1aVVmGwakCZmZmAFTfwLq4LxgRERE1XhkZGXB1dVX//q8Kg1MFiofnzM3NGZyIiIhaCG2m53ByOBEREZGWGJyIiIiItMTgRERERKQlBiciIiIiLTE4EREREWmJwYmIiIhISwxORERERFpicCIiIqJGL/JBJEb+PBKRDyL12g4GJyIiImrUBEHA2vNrcTv9NtaeXwtBEPTWFgYnIiIiatROPTiFKylXAABXUq7g1INTemsLb7lCREREepevyEdyTjKScpKQlJuE5JxkJOckIzEnESfunlCXE4vE+OLCF+jn3E+rW6TUNQYnIiIiqjdFyiKk5KYgOfdJKHrySM4tCUbJuclIz0/Xqj6loFT3OvV36V/PrS+PwYmIiIh0JggCHuc/VvcSJec+CUFPeoqKe41S8lKgFJRa1SmTyGBvbA87IzvYG9vD1sgWR+8cRVJOEgSUzGvSZ68TgxMRERGpCYKA7MJsjSGz0r1ESTlJ6mBUpCzSqk6JSAJbI1s4GDvAzthOHYzsje1hZ2wHeyPVv+ZSc40gFHE/At9f+75cffrsdWJwIiIiaiHyivLUQ2Rlw1Dpr3OLcrWu01puXRKCngSi4jBU/LW13BpikW7XowmCgC8ufAERRBq9TcVEEOml14nBiYiIqIkrVBaq5hE96QlS9wqVCUYZBRla12kmNVP3BFUUjByMHWAjt4GhxLDezikhO6HC0AQAAgQkZCegUFkIqURaL22oCIMTERFRI6UUlHic91hziKyCIbTUvNRKA0ZZcom83BBZRcHIyMCons+ualKJFDue34HUvNRKy1jLrRs0NAEMTkRERA1OEARkFWZVPHeo9OX4uclazyMyEBnA1thWFYJKBaKy84nMDM30chl/TTiaOMLRxFHfzdDA4ERERFSHcoty8SjnkXrIrPRk6uKvk3OTtZ5HJIJIPY9I3TtUQU+RldxK53lEpDu9B6f169dj5cqVSEhIQPfu3fHFF1/Ay8ur0vJr1qzBV199hfj4eNja2uKVV15BaGgo5HJ5jeskIiKqTvE8ouLwU7z+UNmeosyCTK3rNJeaq4NP8byhskNoNkY2MBTXzzwi0p1eg9POnTsxb948bNiwAd7e3lizZg38/f0RHR0Ne3v7cuV/+OEHLFy4EJs3b0a/fv1w48YNBAcHQyQSYfXq1TWqk4iIWjaloERqXqq6J6iyRRof5z3Weh6RkYGRRiAqvsKs9NwiW2Nbvc8jIt2JBD3eKc/b2xt9+vTBunXrAABKpRKurq6YPXs2Fi5cWK78rFmzcO3aNYSHh6u3vf322zhz5gxOnjxZozorkpGRAQsLC6Snp8Pc3Ly2p0lERHogCAIyCzORlK05ZygxO7HkkvzcJDzKeYQiQct5RGIDzd6h4mD0JBQV9xSZGpo2mXlEpNvvfb31OBUUFODcuXNYtGiReptYLIafnx8iIyMrPKZfv374/vvvERUVBS8vL9y+fRsHDhzAq6++WuM6iYio6cktyi0ZLivVU1R6CC05Jxl5ijyt6hNBBBsjm0oXZiweQrOUWXIeUQunt+D06NEjKBQKODg4aGx3cHDA9evXKzxmwoQJePToEXx9fSEIAoqKivCvf/0L7733Xo3rBID8/Hzk5+ern2dkaL/OBRER1Z1CRSEe5T5S9xCVDUbF4SizUPt5RBYyi5LL7CsJRrZGtjAQ633aLzUBTepTcuLECXz00Uf48ssv4e3tjZiYGMyZMwfLly/H4sWLa1xvaGgoli1bVoctJSKi0ornEVV0hVnpy/GrWrOnLCMDo3K38LAzsoO9ScmQmZ2RHeQG8uorI9KS3oKTra0tJBIJEhMTNbYnJibC0bHiNRsWL16MV199FdOnTwcAdO3aFdnZ2Xjttdfw/vvv16hOAFi0aBHmzZunfp6RkQFXV9eanhoRUYshCAIyCjI0AlFFt/N4lPsICkGhVZ0GYoNKF2Ys3VNkYmjCeUTU4PQWnKRSKXr16oXw8HCMGjUKgGoid3h4OGbNmlXhMTk5ORCLNceWJRIJANUPb03qBACZTAaZTFb7kyIiakZyCnPKDZFVFIzyFfnVVwbVHe1t5Dbl7mNWNhhZyCw4j4gaLb0O1c2bNw9BQUHo3bs3vLy8sGbNGmRnZ2PKlCkAgMmTJ8PFxQWhoaEAgMDAQKxevRpPP/20eqhu8eLFCAwMVAeo6uokImrpChWFJZOpKwhGxV9nFWZpXaelzFJjzpCdUak1iZ4EIxsjG84joiZPr5/gsWPHIjk5GUuWLEFCQgJ69OiBQ4cOqSd3x8fHa/Qw/fvf/4ZIJMK///1v3L9/H3Z2dggMDMR//vMfrevUt8gHkVgRtQILvRbCx9lH380homZEoVSo5hFVMlxWPMm6JvOIqrq3mZ2xHWQS9tpTy6DXdZwaq/pax0kQBIzfPx5XUq6gs01nbH9uO8fniahaxfOIiq8wq6ynKCU3Ret5RIZiwwrnDpV+bm+smkdE1Nw1iXWcWqJTD07hSsoVAMCVlCv4d8S/0caiDSQiCSRiCSQiCQzEBhCLxOqvJSIJxGIxDEQGGuXU/z752kBkAIlYArFIrP669P5K6xaJGd6I9CinMKdkperc8leZFQejAmWBVvWJRWLYym1VIaiCFauLh9AsZBb82SeqAQanBiIIAr648IXGtl9u/aKn1mgqG7DUQUtkALG4fNAq/rqi8CYWiWsV5CoqV2d1VxQ8K6iPv0xarrocSi9QFJSsTl0mGJVepDG7MFvrOi1llhUvzFh8Kb6xHWzkNpCIJbVqOxFVjsGpgZTubSrNx8kHtka2UAgK1UOpQJFQBKWgVH+tUCqgFJTqr0uXVQgKFCnLlBcUUCrLl6+Mer+yPr8DTUdx4NKmt65WIbLUMer6tQyGFbWt2hBZXHfpXkwt6mspVzcJgoC159fidvptrD2/Fn2d+lYYohVKBVLyUjR6htRDaKV6jNLy07R+bRNDE43J1JX1FEkl0jo8YyKqCQanBlDc2yQWiaEUStKJWCRGRkEGvh72db33cgiCoApXpYPWk68VwpNg9uTrsuGsOMiV218c3IQiKJW611c6GJYNgxUFQ42yZf+trnyZY6q6L5VSUEIpKFGIQkC76SLNmgiiaoeJqwqRlfUeVlm+tj2NpUKitsPbF5Mvagylrzy7EhYyi3I9RY/yHmn8HFdFKpZWeLl92cvxOY+IqOlgcGoAlfU2KQUlrqRcwakHp9DfpX+9tkEkevLLDxL+1fpE6fBWbdArDoalewB1DZqVlK2wR7FUuXI9itr2QOoYZCsLAwIEFAlFKFJodxPU5uK7a99Vuk8iksDGyKbSRRqLh9DMpeYc+iVqZhic6llxb5MIIggofwGjCCJ8ceEL9HPux/9gG5hYJIZYIoYhDPXdlEahOGxVGeS0DZq1DHq17VHUNcjmKfKQU5hT7nvS37k/uth2KddTZC235jwiohaKwameFSoLkZCdUGFoAlR/zSdkJ6BQWcieINIrsUgMsUgMQ3HLCpLFy4RcS71Wbig9LT8NM3vM5B81RKTG4FTPpBIpdjy/o8oF56zl1gxNRHrSGIbSiajpYHBqAI4mjnA0qfwmw0SkHxxKJyJdtYzrjImIKqDLUDoREcAeJyJqwTiUTkS6YnAiohaNQ+lEpAsO1RERERFpicGJiIiISEsMTkRERERaYnAiIiIi0hKDExEREZGWGJyIiIiItMTgRERERKQlBiciIiIiLTE4EREREWmJwYmIiIhISwxORERERFpicCIiIiLSEoMTERERkZYYnIiIiIi0xOBEREREpCUGJyIiIiItMTgRERERaalRBKf169fD3d0dcrkc3t7eiIqKqrTs4MGDIRKJyj2ee+45dZng4OBy+wMCAhriVIiIiKgZM9B3A3bu3Il58+Zhw4YN8Pb2xpo1a+Dv74/o6GjY29uXK793714UFBSon6ekpKB79+4YPXq0RrmAgABs2bJF/Vwmk9XfSRAREVGLoPcep9WrV2PGjBmYMmUKOnXqhA0bNsDY2BibN2+usLy1tTUcHR3Vj6NHj8LY2LhccJLJZBrlrKysGuJ0iIiIqBnTa3AqKCjAuXPn4Ofnp94mFovh5+eHyMhIrerYtGkTxo0bBxMTE43tJ06cgL29PTp06IDXX38dKSkpddp2IiIiann0OlT36NEjKBQKODg4aGx3cHDA9evXqz0+KioKly9fxqZNmzS2BwQE4KWXXoKHhwdu3bqF9957DyNGjEBkZCQkEkm5evLz85Gfn69+npGRUcMzIiIiouZM73OcamPTpk3o2rUrvLy8NLaPGzdO/XXXrl3RrVs3tG3bFidOnMDQoUPL1RMaGoply5bVe3uJiIioadPrUJ2trS0kEgkSExM1ticmJsLR0bHKY7Ozs7Fjxw5Mmzat2tdp06YNbG1tERMTU+H+RYsWIT09Xf24e/eu9idBRERELYZeg5NUKkWvXr0QHh6u3qZUKhEeHg4fH58qj929ezfy8/MxadKkal/n3r17SElJgZOTU4X7ZTIZzM3NNR5EREREZen9qrp58+bhm2++wdatW3Ht2jW8/vrryM7OxpQpUwAAkydPxqJFi8odt2nTJowaNQo2NjYa27OysrBgwQKcPn0acXFxCA8Px8iRI9GuXTv4+/s3yDkRERFR86T3OU5jx45FcnIylixZgoSEBPTo0QOHDh1STxiPj4+HWKyZ76Kjo3Hy5EkcOXKkXH0SiQT//PMPtm7dirS0NDg7O2P48OFYvnw513IiIiKiWhEJgiDouxGNTUZGBiwsLJCens5hOyIiomZOl9/7eh+qIyIiImoqGJyIiIiItMTgRERERKQlBiciIiIiLTE4EREREWmJwYmIiIhISwxORERERFpicCIiIiLSEoMTERERkZYYnIiIiIi0xOBEREREpCUGJyIiIiItMTgRERERaYnBiYiIiEhLDE5EREREWmJwIiIiItISgxMRERGRlhiciIiIiLTE4ERERESkJQYnIiIiIi0xOBERERFpicGJiIiISEsMTkRERERaYnAiIiIi0hKDExEREZGWGJyIiIiItMTgRERERKQlBiciIiIiLTE4EREREWmpUQSn9evXw93dHXK5HN7e3oiKiqq07ODBgyESico9nnvuOXUZQRCwZMkSODk5wcjICH5+frh582ZDnAoRERE1Y3oPTjt37sS8efMQEhKC8+fPo3v37vD390dSUlKF5ffu3YuHDx+qH5cvX4ZEIsHo0aPVZT755BN8/vnn2LBhA86cOQMTExP4+/sjLy+voU6LiIiImiGRIAiCPhvg7e2NPn36YN26dQAApVIJV1dXzJ49GwsXLqz2+DVr1mDJkiV4+PAhTExMIAgCnJ2d8fbbb2P+/PkAgPT0dDg4OCAsLAzjxo2rts6MjAxYWFggPT0d5ubmtTtBIiIiatR0+b2v1x6ngoICnDt3Dn5+fuptYrEYfn5+iIyM1KqOTZs2Ydy4cTAxMQEAxMbGIiEhQaNOCwsLeHt7a10nERERUUUM9Pnijx49gkKhgIODg8Z2BwcHXL9+vdrjo6KicPnyZWzatEm9LSEhQV1H2TqL95WVn5+P/Px89fOMjAytz4GIiIhaDr3PcaqNTZs2oWvXrvDy8qpVPaGhobCwsFA/XF1d66iFRERE1JzoNTjZ2tpCIpEgMTFRY3tiYiIcHR2rPDY7Oxs7duzAtGnTNLYXH6dLnYsWLUJ6err6cffuXV1PhYiIiFoAvQYnqVSKXr16ITw8XL1NqVQiPDwcPj4+VR67e/du5OfnY9KkSRrbPTw84OjoqFFnRkYGzpw5U2mdMpkM5ubmGg8iIiKisvQ6xwkA5s2bh6CgIPTu3RteXl5Ys2YNsrOzMWXKFADA5MmT4eLigtDQUI3jNm3ahFGjRsHGxkZju0gkwty5c/Hhhx/C09MTHh4eWLx4MZydnTFq1KiGOi0iIiJqhvQenMaOHYvk5GQsWbIECQkJ6NGjBw4dOqSe3B0fHw+xWLNjLDo6GidPnsSRI0cqrPOdd95BdnY2XnvtNaSlpcHX1xeHDh2CXC6v9/MhIiKi5kvv6zg1RlzHiYiIqOVoMus4ERERETUlDE5EREREWmJwIiIiItISgxMRERGRlhiciIiIiLSk9+UIiIiImgqlUomCggJ9N4N0ZGhoCIlEUid1MTgRERFpoaCgALGxsVAqlfpuCtWApaUlHB0dIRKJalUPgxMREVE1BEHAw4cPIZFI4OrqWm5hZmq8BEFATk4OkpKSAABOTk61qo/BiYiIqBpFRUXIycmBs7MzjI2N9d0c0pGRkREAICkpCfb29rUatmNkJiIiqoZCoQCgujk9NU3FgbewsLBW9TA4ERERaam282NIf+rqvWNwIiIiItISgxMRERFpxd3dHWvWrNF7HfrEyeFEREQNRKEUEBWbiqTMPNibyeHlYQ2JuP6G/wYPHowePXrUWVA5e/YsTExM6qSuporBiYiIqAEcuvwQy369iofpeeptThZyhAR2QkCX2l0iXxuCIEChUMDAoPpIYGdn1wAtatw4VEdERFTPDl1+iNe/P68RmgAgIT0Pr39/HocuP6zz1wwODsbvv/+OtWvXQiQSQSQSIS4uDidOnIBIJMLBgwfRq1cvyGQynDx5Erdu3cLIkSPh4OAAU1NT9OnTB8eOHdOos+wwm0gkwn//+1+8+OKLMDY2hqenJ3755Red2hkfH4+RI0fC1NQU5ubmGDNmDBITE9X7//77bwwZMgRmZmYwNzdHr1698NdffwEA7ty5g8DAQFhZWcHExASdO3fGgQMHav5N0wKDExERkY4EQUBOQZFWj8y8QoT8cgVCRfU8+XfpL1eRmVeoVX2CUFFN5a1duxY+Pj6YMWMGHj58iIcPH8LV1VW9f+HChVixYgWuXbuGbt26ISsrC88++yzCw8Nx4cIFBAQEIDAwEPHx8VW+zrJlyzBmzBj8888/ePbZZzFx4kSkpqZq1UalUomRI0ciNTUVv//+O44ePYrbt29j7Nix6jITJ05Eq1atcPbsWZw7dw4LFy6EoaEhAGDmzJnIz8/HH3/8gUuXLuHjjz+GqampVq9dUxyqIyIi0lFuoQKdlhyuk7oEAAkZeei69IhW5a9+4A9jafW/vi0sLCCVSmFsbAxHR8dy+z/44AMMGzZM/dza2hrdu3dXP1++fDl++ukn/PLLL5g1a1alrxMcHIzx48cDAD766CN8/vnniIqKQkBAQLVtDA8Px6VLlxAbG6sOdd9++y06d+6Ms2fPok+fPoiPj8eCBQvw1FNPAQA8PT3Vx8fHx+Pll19G165dAQBt2rSp9jVriz1ORERELVDv3r01nmdlZWH+/Pno2LEjLC0tYWpqimvXrlXb49StWzf11yYmJjA3N1ff3qQ6165dg6urq0ZPWKdOnWBpaYlr164BAObNm4fp06fDz88PK1aswK1bt9Rl33zzTXz44Yfo378/QkJC8M8//2j1urXBHiciIiIdGRlKcPUDf63KRsWmInjL2WrLhU3pAy8Pa61euy6UvTpu/vz5OHr0KFatWoV27drByMgIr7zyCgoKCqqsp3jYrJhIJKrTGyEvXboUEyZMwP79+3Hw4EGEhIRgx44dePHFFzF9+nT4+/tj//79OHLkCEJDQ/Hpp59i9uzZdfb6ZbHHiYiISEcikQjGUgOtHgM87eBkIUdliw6IoLq6boCnnVb16bICtlQqVd8upjoREREIDg7Giy++iK5du8LR0RFxcXFav1ZNdOzYEXfv3sXdu3fV265evYq0tDR06tRJva19+/Z46623cOTIEbz00kvYsmWLep+rqyv+9a9/Ye/evXj77bfxzTff1GubGZyIiIjqkUQsQkigKgSUjTzFz0MCO9XLek7u7u44c+YM4uLi8OjRoyp7gjw9PbF3715cvHgRf//9NyZMmFCnPUcV8fPzQ9euXTFx4kScP38eUVFRmDx5MgYNGoTevXsjNzcXs2bNwokTJ3Dnzh1ERETg7Nmz6NixIwBg7ty5OHz4MGJjY3H+/HkcP35cva++MDgRERHVs4AuTvhqUk84Wsg1tjtayPHVpJ71to7T/PnzIZFI0KlTJ9jZ2VU5X2n16tWwsrJCv379EBgYCH9/f/Ts2bNe2lVMJBJh3759sLKywsCBA+Hn54c2bdpg586dAACJRIKUlBRMnjwZ7du3x5gxYzBixAgsW7YMgOrmyzNnzkTHjh0REBCA9u3b48svv6zfNgvaXtdYRkxMDG7duoWBAwfCyMgIgiA0m5sfZmRkwMLCAunp6TA3N9d3c4iISM/y8vIQGxsLDw8PyOXy6g+oREOvHE4lqnoPdfm9r/Pk8JSUFIwdOxa//fYbRCIRbt68iTZt2mDatGmwsrLCp59+qmuVRERELYJELIJPWxt9N4NqQeehurfeegsGBgaIj4+HsbGxevvYsWNx6NChOm0cERERUWOic4/TkSNHcPjwYbRq1Upju6enJ+7cuVNnDSMiIiJqbHTuccrOztboaSqWmpoKmUxWJ40iIiIiaox0Dk4DBgzAt99+q35evNDVJ598giFDhtRp44iIiIgaE52D0yeffIKNGzdixIgRKCgowDvvvIMuXbrgjz/+wMcff6xzA9avXw93d3fI5XJ4e3sjKiqqyvJpaWmYOXMmnJycIJPJ0L59e407IS9dulR9F+jiR/H9bYiIiIhqQ+c5Tl26dMGNGzewbt06mJmZISsrCy+99JI6zOhi586dmDdvHjZs2ABvb2+sWbMG/v7+iI6Ohr29fbnyBQUFGDZsGOzt7bFnzx64uLjgzp07sLS01CjXuXNnHDt2rOQkDXhnGSIiIqq9GiUKCwsLvP/++7V+8dWrV2PGjBmYMmUKAGDDhg3Yv38/Nm/ejIULF5Yrv3nzZqSmpuLUqVPqe+O4u7uXK2dgYFDhnaCJiIiIakPn4PTHH39UuX/gwIFa1VNQUIBz585h0aJF6m1isRh+fn6IjIys8JhffvkFPj4+mDlzJvbt2wc7OztMmDAB7777LiSSkpse3rx5E87OzpDL5fDx8UFoaChat26tVbuIiIiIKqNzcBo8eHC5baVXDNf2ZoKPHj2CQqGAg4ODxnYHBwdcv369wmNu376N3377DRMnTsSBAwcQExODN954A4WFhQgJCQEAeHt7IywsDB06dMDDhw+xbNkyDBgwAJcvX4aZmVmF9ebn5yM/P1/9PCMjQ6tzICIiau7c3d0xd+5czJ07t8L9wcHBSEtLw88//9yg7dIXnYPT48ePNZ4XFhbiwoULWLx4Mf7zn//UWcMqolQqYW9vj40bN0IikaBXr164f/8+Vq5cqQ5OI0aMUJfv1q0bvL294ebmhl27dmHatGkV1hsaGqq+7w0REVGdS7sL5KRUvt/YBrB0bbj2UI3pHJwsLCzKbRs2bBikUinmzZuHc+fOaVWPra0tJBIJEhMTNbYnJiZWOj/JyckJhoaGGsNyHTt2REJCAgoKCiCVSssdY2lpifbt2yMmJqbStixatAjz5s1TP8/IyICrKz/ARERUB9LuAut6AUX5lZcxkAGzzjE8NQE6L0dQGQcHB0RHR2tdXiqVolevXggPD1dvUyqVCA8Ph4+PT4XH9O/fHzExMVAqleptN27cgJOTU4WhCQCysrJw69atKq/4k8lkMDc313gQERHViZyUqkMToNpfVY9UDWzcuBHOzs4avzMBYOTIkZg6dSoA4NatWxg5ciQcHBxgamqKPn36aFyVXhP5+fl48803YW9vD7lcDl9fX5w9e1a9//Hjx5g4cSLs7OxgZGQET09PbNmyBYBq/vOsWbPg5OQEuVwONzc3hIaG1qo9dU3nHqd//vlH47kgCHj48CFWrFiBHj166FTXvHnzEBQUhN69e8PLywtr1qxBdna2+iq7yZMnw8XFRf1Ne/3117Fu3TrMmTMHs2fPxs2bN/HRRx/hzTffVNc5f/58BAYGws3NDQ8ePEBISAgkEgnGjx+v66kSERFVTBCAwhztyhblal+uILv6cobGQKm5xZUZPXo0Zs+ejePHj2Po0KEAVHf5OHTokHr9w6ysLDz77LP4z3/+A5lMhm+//RaBgYGIjo6u8UVV77zzDn788Uds3boVbm5u+OSTT+Dv74+YmBhYW1tj8eLFuHr1Kg4ePAhbW1vExMQgN1f1Pfr888/xyy+/YNeuXWjdujXu3r2Lu3fv1qgd9UXn4NSjRw+IRCIIgqCxvW/fvti8ebNOdY0dOxbJyclYsmQJEhIS0KNHDxw6dEg9YTw+Ph5icUmnmKurKw4fPoy33noL3bp1g4uLC+bMmYN3331XXebevXsYP348UlJSYGdnB19fX5w+fRp2dna6nioREVHFCnOAj5zrts7NAdqVe+8BIDWptpiVlRVGjBiBH374QR2c9uzZA1tbW/WdPrp3747u3burj1m+fDl++ukn/PLLL5g1a5bOp5CdnY2vvvoKYWFh6jnH33zzDY4ePYpNmzZhwYIFiI+Px9NPP43evXsD0FxWKD4+Hp6envD19YVIJIKbm5vObahvOgen2NhYjedisRh2dnaQy+U1asCsWbMqfXNOnDhRbpuPjw9Onz5daX07duyoUTuIiIiam4kTJ2LGjBn48ssvIZPJsG3bNowbN07dKZGVlYWlS5di//79ePjwIYqKipCbm4v4+Pgavd6tW7dQWFiI/v37q7cZGhrCy8sL165dA6AaPXr55Zdx/vx5DB8+HKNGjUK/fv0AqK7QGzZsGDp06ICAgAA8//zzGD58eC2/C3VL5+DUGNMfERFRgzI0VvX8aCPhH+16k6YeAhy7affaWgoMDIQgCNi/fz/69OmDP//8E5999pl6//z583H06FGsWrUK7dq1g5GREV555RUUFBRo/Rq6GjFiBO7cuYMDBw7g6NGjGDp0KGbOnIlVq1ahZ8+eiI2NxcGDB3Hs2DGMGTMGfn5+2LNnT721R1daBafPP/9c6wpLzzciIiJqlkQirYbLAAAGRtqX07ZOLcnlcrz00kvYtm0bYmJi0KFDB/Ts2VO9PyIiAsHBwXjxxRcBqHqg4uLiavx6bdu2hVQqRUREhLqjpbCwEGfPntVYB8rOzg5BQUEICgrCgAEDsGDBAqxatQoAYG5ujrFjx2Ls2LF45ZVXEBAQgNTUVFhbW9e4XXVJq+BUOp1WRSQSMTgRERE1IhMnTsTzzz+PK1euYNKkSRr7PD09sXfvXgQGBkIkEmHx4sXlrsLThYmJCV5//XUsWLAA1tbWaN26NT755BPk5OSo11JcsmQJevXqhc6dOyM/Px//+9//0LFjRwCqW7E5OTnh6aefhlgsxu7du+Ho6FjunrT6pFVwKjuviYiIiLRkbKNap6m6dZyMberl5Z955hlYW1sjOjoaEyZM0Ni3evVqTJ06Ff369YOtrS3efffdWt89Y8WKFVAqlXj11VeRmZmJ3r174/Dhw7CysgKgWo5o0aJFiIuLg5GREQYMGKCen2xmZoZPPvkEN2/ehEQiQZ8+fXDgwAGNC8X0TSSUvTyOkJGRAQsLC6Snp3NNJyIiQl5eHmJjY+Hh4VGzi6G4crjeVfUe6vJ7X+fJ4YDqkv9ffvkF8fHx5SaQrV69uiZVEhERNV+WrgxGzYTOwSk8PBwvvPAC2rRpg+vXr6NLly6Ii4uDIAgaE86IiIiImhudBw0XLVqE+fPn49KlS5DL5fjxxx9x9+5dDBo0CKNHj66PNhIRERE1CjoHp2vXrmHy5MkAAAMDA+Tm5sLU1BQffPABPv744zpvIBEREVFjoXNwMjExUc9rcnJywq1bt9T7Hj16VHctIyIiImpkdJ7j1LdvX5w8eRIdO3bEs88+i7fffhuXLl3C3r170bdv3/poIxEREVGjoHNwWr16NbKysgAAy5YtQ1ZWFnbu3AlPT09eUUdERETNms7B6aOPPlKvPGpiYoINGzbUeaOIiIiIGiOd5zglJycjICAArq6uWLBgAf7+++/6aBcRERE1Au7u7lizZo2+m9Fo6Byc9u3bh4cPH2Lx4sU4e/Ysevbsic6dO+Ojjz6q1Y0BiYiIiBq7Gt38xcrKCq+99hpOnDiBO3fuIDg4GN999x3atWtX1+0jIiIiajRqdde8wsJC/PXXXzhz5gzi4uLg4OBQV+0iIiJqliIfRGLkzyMR+SCyXl9n48aNcHZ2hlKp1Ng+cuRITJ06FQBw69YtjBw5Eg4ODjA1NUWfPn1w7NgxnV7n7NmzGDZsGGxtbWFhYYFBgwbh/PnzGmXS0tLwf//3f3BwcIBcLkeXLl3wv//9T70/IiICgwcPhrGxMaysrODv74/Hjx/X8MzrV42C0/HjxzFjxgw4ODggODgY5ubm+N///od79+7VdfuIiIiaDUEQsPb8WtxOv42159dCEIR6e63Ro0cjJSUFx48fV29LTU3FoUOHMHHiRABAVlYWnn32WYSHh+PChQsICAhAYGAg4uPjtX6dzMxMBAUF4eTJkzh9+jQ8PT3x7LPPIjMzEwCgVCoxYsQIRERE4Pvvv8fVq1exYsUKSCQSAMDFixcxdOhQdOrUCZGRkTh58iQCAwOhUCjq8LtRd3S+qs7FxQWpqakICAjAxo0bERgYCJlMVh9tIyIiapQEQUBuUa7Ox51+cBpXUq4AAK6kXMHx+OPo66zbGohGBkYQiUTVlrOyssKIESPwww8/YOjQoQCAPXv2wNbWFkOGDAEAdO/eHd27d1cfs3z5cvz000/45ZdfMGvWLK3a88wzz2g837hxIywtLfH777/j+eefx7FjxxAVFYVr166hffv2AIA2bdqoy3/yySfo3bs3vvzyS/W2zp07a/Xa+qBzcFq6dClGjx4NS0vLemgOERFR45dblAvvH7xrXc+cE3N0PubMhDMwNjTWquzEiRMxY8YMfPnll5DJZNi2bRvGjRsHsVg14JSVlYWlS5di//79ePjwIYqKipCbm6tTj1NiYiL+/e9/48SJE0hKSoJCoUBOTo66josXL6JVq1bq0FTWxYsXm9S9bnUOTjNmzKiPdhAREVEdCwwMhCAI2L9/P/r06YM///wTn332mXr//PnzcfToUaxatQrt2rWDkZERXnnlFfWt1bQRFBSElJQUrF27Fm5ubpDJZPDx8VHXYWRkVOXx1e1vbHQOTkRERC2dkYERzkw4o3V5QRAw5fAURD+OhlIomawtFonRwaoDtvhv0Wr4rfi1tSWXy/HSSy9h27ZtiImJQYcOHdCzZ0/1/oiICAQHB+PFF18EoOqB0nVpoYiICHz55Zd49tlnAQB3797VuHdtt27dcO/ePdy4caPCXqdu3bohPDwcy5Yt0+l19YXBiYiISEcikUjr4TIAiLgfgWup18ptVwpKXEu9hovJF9HfpX9dNlFt4sSJeP7553HlyhX1nT+KeXp6Yu/evQgMDIRIJMLixYvLXYVXHU9PT3z33Xfo3bs3MjIysGDBAo1epEGDBmHgwIF4+eWXsXr1arRr1w7Xr1+HSCRCQEAAFi1ahK5du+KNN97Av/71L0ilUhw/fhyjR4+Gra1tnXwP6lKtliMgIiKiqgmCgC8ufAERKu5REkGELy58UW9X2D3zzDOwtrZGdHQ0JkyYoLFv9erVsLKyQr9+/RAYGAh/f3+NHiltbNq0CY8fP0bPnj3x6quv4s0334S9vb1GmR9//BF9+vTB+PHj0alTJ7zzzjvqq+bat2+PI0eO4O+//4aXlxd8fHywb98+GBg0zr4dkVCf10I2URkZGbCwsEB6ejrMzc313RwiItKzvLw8xMbGwsPDA3K5XKdjCxQFGL5nOFLyUiotYyO3wZFXjkAqkda2qVSJqt5DXX7vN844R0RE1ExIJVLseH4HUvNSKy1jLbdmaGoiGJyIiIjqmaOJIxxNHPXdDKoDnONEREREpCUGJyIiIiIt6T04rV+/Hu7u7pDL5fD29kZUVFSV5dPS0jBz5kw4OTlBJpOhffv2OHDgQK3qJCIiItKGXoPTzp07MW/ePISEhOD8+fPo3r07/P39kZSUVGH5goICDBs2DHFxcdizZw+io6PxzTffwMXFpcZ1EhERaYsXojdddfXe6XU5Am9vb/Tp0wfr1q0DoLqDsqurK2bPno2FCxeWK79hwwasXLkS169fh6GhYZ3UWZH6Wo5AoRQQFZuKpMw82JvJ4eVhDYlYu5ViiYhIfwoLCxETEwNnZ2dYWFjouzlUAykpKUhKSkL79u0hkUg09jWJ5QgKCgpw7tw5LFq0SL1NLBbDz88PkZGRFR7zyy+/wMfHBzNnzsS+fftgZ2eHCRMm4N1334VEIqlRnQCQn5+P/Px89fOMjIw6OENNhy4/xLJfr+Jhep56m5OFHCGBnRDQxanOX4+IiOqOgYEBjI2NkZycDENDQ/VNcqnxEwQBOTk5SEpKgqWlZbnQpCu9BadHjx5BoVDAwcFBY7uDgwOuX79e4TG3b9/Gb7/9hokTJ+LAgQOIiYnBG2+8gcLCQoSEhNSoTgAIDQ2t13vkHLr8EK9/fx5lu/YS0vPw+vfn8dWkngxPRESNmEgkgpOTE2JjY3Hnzh19N4dqwNLSEo6OtV8Sokmt46RUKmFvb4+NGzdCIpGgV69euH//PlauXImQkJAa17to0SLMmzdP/TwjIwOurq510WQolAKW/Xq1XGgCAAGACMCyX69iWCdHDtsRETViUqkUnp6eKCgo0HdTSEeGhoa17mkqprfgZGtrC4lEgsTERI3tiYmJlSZCJyenciffsWNHJCQkoKCgoEZ1AoBMJoNMJqvF2VQuKjYVovR76CzKrLTM43QzRMWmwqetTb20gYiI6oZYLNb5livUvOgtOEmlUvTq1Qvh4eEYNWoUAFWPUnh4OGbNmlXhMf3798cPP/wApVKpHl++ceMGnJycIJWqlqrXtc76lpl4G7/J3oZcVFhpmTzBEH8kdgIYnIiIiBo1vc5umzdvHr755hts3boV165dw+uvv47s7GxMmTIFADB58mSNid6vv/46UlNTMWfOHNy4cQP79+/HRx99hJkzZ2pdZ0NzNMipMjQBgFxUiN/OX8P5+McN1CoiIiKqCb3OcRo7diySk5OxZMkSJCQkoEePHjh06JB6cnd8fLzGlQuurq44fPgw3nrrLXTr1g0uLi6YM2cO3n33Xa3rbGidXbRbzuDS/Qzs+PIUerhaYqqvB0Z0cYShhFdtEBERNSZ6XcepsarTdZweXAQ2Dqq22HGbcfgt0RRZSkPkwxDGxqYY2MkVQ7q4wszUFDAwAgxkgOGTf4ufizihnIiIqDaaxDpOpGlIyg4MKf1uFAH458mjKgZyzSBVNliVfW4gBwzlT46T1+y5gRzgGiZERNQCMTg1Fm2GAIbGQFEulIV5SMvIQHpmFlCYC5moEHIUwFhcBBkKIBKUJccV5akeSG/Y9kqkpcJZLYNYtc9LhT5x3VxOSkREVBMMTo2F31LAuQcA1Yx9awBWgoDTt1OxOSIWx64lQjWoKqCjvRGmeTvh+c7WkKMAKMoHCnNV/xbl1vHzvJKHsqikvYoC1SO/3JnUL7FhBb1qZXvEtOh5MzTSLbxJ+KNCREQMTo2aSCSCT1sb+LS1wZ2UbGyJiMPuv+7iWlIe5v8ai/+E38NEbze86uMGB9sGWFdEUaQZpArzqnmuYzirrD5lqasSlYVAfqEeAptBPfeqVfJcUvE9GakOpd0FclIq329sA1jWzYK4RNT0cXJ4Bep0cnjaXWBdL1VAqIyBDJh1Tqv/nDPyCrHr7F2EnYrDvce5AABDiQjPd3PG1P4e6NqqGd58UqmoJqjVYa9aYV5JGUUjWB1YJNGxF62OnksMW8aFB3X880lETZMuv/cZnCpQp8EJqJe/aIsUShy7lohNJ2NxNq5k/Scvd2tM9XXnLVzqglKpGaq06mWrg+eKhu5Oq4ioHoNZFT1tDX2lqJZXveK139VD6UTU/PCqusbG0rXO/1o1kIgR0MUJAV2c8M+9NGyJiMOvfz9AVFwqouJS0crKCMH93DGmjyvM5RzuqRGxGJAaqx4NSalUhac6DWZa9sSpCUBhjuqBBl6YtSGHRPMrvxUSEVFF2ONUgTrvcWogiRl5+C7yDraduYPHOap5QSZSCUb3dsWU/u5wszHRcwupURME1fBkvV1oUMXzCm+D3Yiwx4moWeNQXS011eBULK9QgZ8u3Mfmk7G4mZQFQDX64dfRAdN8PeDtYQ1RS5i/Qk2DIACKwnqYq1bd81yg9NIeVRm2HOg9FZCZ1u/3goj0gsGplpp6cComCAJOxjzCppOxOBGdrN7eyckcU309ENjdCTIDrotELdi9v4D/DtWurEQGeAwA2gcAHUYAFq3qt21E1GAYnGqpuQSn0mKSshB2KhZ7zt1DXqHqr2xbUxle7euGiX1bw9ZUpucWEumBtpPDzZyAzIea2xy6Ah0CgPYjAOenuZo+URPG4FRLzTE4FUvLKcD2qLvYeioOCRl5AACpgRijejhjSn8PdHRqXudLVCWtr6o7obo68MZBIPoQcC9Kc5jP1AHwHK7qiWozGJByPiFRU8LgVEvNOTgVK1QocfByAjadjMXfd9PU2/u1tcE0Xw8M6WAPMZczoOaupus4ZacAN4+oglTMb0BBqavzJDKgzSDVkF77AMDCpf7aT0R1gsGpllpCcCrtfPxjbDoZi0OXE6BQqj4O7jbGmNLfA6/0agUTGVetoGastuusFRUAd06qeqJuHATS4jX3O3Z7Mi8qAHDikB5RY8TgVEstLTgVu5+Wi29PxWF7VDwy8lT3pTOTG2C8V2tM9nFDK6sGXs+IqKkRBCDpWqkhvbPQWGrB1BFoP1w1L6rN4IZfI4yIKsTgVEstNTgVy84vwt7z97AlIg63H2UDACRiEQI6O2Kqrzt6trbicgZE2shKfjKkdwi49RtQkFWyz0AOeAx6MsE8ADB31l87iVo4BqdaaunBqZhSKeDEjSRsPhmHkzGP1Nu7t7LAVF8PPNvVCYYSDjsQaaUoH4g7qQpR0YeA9DJDek7dVT1RHQIApx4t416BRI0Eg1MtMTiVdz0hA1tOxuGni/dRUKS6msjRXI5Xfdwwwas1rEykem4hURMiCEDSVSD6oCpI3fsLGkN6Zk5Ae/8nQ3qDVPf5I6J6w+BUSwxOlUvJyse2M/H47vQdJGeqrkSSG4rxUs9WmNrfHe3szfTcQqImKCtJNaQXfRC4dRwozC7ZZ2Ckmg9VPKRn5qi3ZhI1VwxOtcTgVL38IgX2//MQm07G4sqDDPX2Qe3tMNXXAwM9bTkPiqgmCvOeDOk9mWCecU9zv/PTJUsdOHXnkB5RHWBwqiUGJ+0JgoCo2FRsjojFkauJKP40tbM3xdT+HnjxaRcYSXlbF6IaEQQg8XLJUgf3z2nuN3NWDel1GAF4DOSQHlENMTjVEoNTzcSn5CDsVBx2/XUXWfmq5QysjA0xwbs1Xu3rDkcLuZ5bSNTEZSYCNw8DNw6rrtIrzCnZZ2isGtIr7o0yc9BbM4maGganWmJwqp3MvELs+usewk7F4m5qLgDAQCzCc92cMLW/B7q7Wuq3gUTNQWEeEPdnyQTzjPua+517qnqi2gcAjl05pEdUBQanWmJwqhsKpYCjVxOxOSIWUbGp6u293aww1dcDwzs5wIDLGRDVniAACZeeLHVwEHhwXnO/eauSIT33AYAhe3+JSmNwqiUGp7p3+X46NkfE4te/H6BQofrIuVgaIbifO8b0cYWFkaGeW0jUjGQmqIbzbhxSXaVXlFuyz9AEaDvkyZCeP2Bqr792EjUSDE61xOBUf5Iy8vD96Tv4/kw8UrMLAADGUgnG9HZFcD93uNvyrvJEdaowF4j948mQ3mEg80GpnSLApVfJvfQcunBIj1okBqdaYnCqf3mFCuy7eB+bT8YhOlF1Z3mRCBj6lD2m+nrAp40NlzMgqmuCADz8+0lv1EHgwQXN/RauJQtvegwADGT6aSdRA2NwqiUGp4YjCAIiYlKwOSIWv11PUm9/ytEMU3098EJ3Z8gNuZwBUb3IeKi6Si/6EHD7RMVDeh1GAJ7+gKmd3ppJVN8YnGqJwUk/bidnYUtEHPacu4fcQgUAwNZUionebpjU1w12Zvzrl6jeFOSohvRuFA/pPSy1UwS06v1kSG8EYN+JQ3rUrDS54LR+/XqsXLkSCQkJ6N69O7744gt4eXlVWDYsLAxTpkzR2CaTyZCXl6d+HhwcjK1bt2qU8ff3x6FDh7RqD4OTfqXnFGLH2XhsPRWHB+mq91UqEeOFHs6Y2t8DnZz5nhDVK0EAHl4sWXjz4d+a+y1al9wCxt2XQ3rU5DWp4LRz505MnjwZGzZsgLe3N9asWYPdu3cjOjoa9vblr/YICwvDnDlzEB0drd4mEong4FCy2FtwcDASExOxZcsW9TaZTAYrKyut2sTg1DgUKpQ4fCUBm0/G4nx8mnq7TxsbTPX1wDNP2UMi5l+9RPUu48GTpQ4OAbG/A0Ulf6hCagq0febJkN5wwMRWf+0kqqEmFZy8vb3Rp08frFu3DgCgVCrh6uqK2bNnY+HCheXKh4WFYe7cuUhLS6u0zuDgYKSlpeHnn3+uUZsYnBqfC/GPsTkiDgcuPYRCqfrIutkYI7ifO0b3doWpzEDPLSRqIQpyVPOhiof0shJL7RQBrl4lE8ztO3JIj5qEJhOcCgoKYGxsjD179mDUqFHq7UFBQUhLS8O+ffvKHRMWFobp06fDxcUFSqUSPXv2xEcffYTOnTurywQHB+Pnn3+GVCqFlZUVnnnmGXz44YewsbGpsB35+fnIz89XP8/IyICrqyuDUyP0IC0X30bewfaoeKTnFgIAzGQGGNvHFUH93OFqbaznFhK1IEol8PCCKkBFHwQS/tHcb+lWstSBmy9gINVPO4mq0WSC04MHD+Di4oJTp07Bx8dHvf2dd97B77//jjNnzpQ7JjIyEjdv3kS3bt2Qnp6OVatW4Y8//sCVK1fQqlUrAMCOHTtgbGwMDw8P3Lp1C++99x5MTU0RGRkJiaT8FVpLly7FsmXLym1ncGq8cgqKsPf8fWyOiMXt5GwAgFgE+Hd2xFRfD/R2s+JyBkQNLf2+akjvxiHg9u+AouQPUkjNgHbPqHqiPIcDJhX/IUukD806OJVVWFiIjh07Yvz48Vi+fHmFZW7fvo22bdvi2LFjGDp0aLn97HFqupRKAb/fTMbmk7H48+Yj9fZurSwwtb8Hnu3qBKkBb+tC1OAKslVDesULb2aXLDcCkRho5fVkgvkIwK4Dh/RIr3QJTnqdGGJrawuJRILExESN7YmJiXB0dNSqDkNDQzz99NOIiYmptEybNm1ga2uLmJiYCoOTTCaDTMarQpoisViEIR3sMaSDPW4kZmJLRCz2nr+Pf+6lY+7Oiwg9eA2Tfdwx3qs1rE04TEDUYKQmwFPPqR5KpWqxzRsHVRPMEy8Bd0+rHseWAlbuqgDVIQBo3Y9DetSo6fVPcalUil69eiE8PFy9TalUIjw8XKMHqioKhQKXLl2Ck5NTpWXu3buHlJSUKstQ09fewQyhL3XDqYXPYP7w9rA3kyExIx8rD0fDJzQci/b+g5tPViknogYkFgOtegHP/Bt4/SQw9zLw7CqgnR8gkQKP44AzXwHfjgRWtgV2BwN/7wRyUqurmajB6f2qup07dyIoKAhff/01vLy8sGbNGuzatQvXr1+Hg4MDJk+eDBcXF4SGhgIAPvjgA/Tt2xft2rVDWloaVq5ciZ9//hnnzp1Dp06dkJWVhWXLluHll1+Go6Mjbt26hXfeeQeZmZm4dOmSVj1LvKqueSgoUuLApYfYdDIWl+6nq7cP8LTFVF8PDPK0g5jLGRDpV36W5lV62ckl+0RiwLWv6iq9DiMA2/Yc0qN60WSG6gBg7NixSE5OxpIlS5CQkIAePXrg0KFD6nWZ4uPjIRaXdIw9fvwYM2bMQEJCAqysrNCrVy+cOnUKnTp1AgBIJBL8888/2Lp1K9LS0uDs7Izhw4dj+fLlHI5rYaQGYox62gUjezjjrzuPsenPWBy5moA/bz7Cnzcfoa2dCab098DLPVvBSMrbuhDphcwU6Pi86qFUAg/OP5kXdQhIvAzEn1I9joUAVh6qANU+AHDrB0gM9d16aoH03uPUGLHHqfm6m5qDrafisPPsXWTmFwEALIwMMcG7NSb7uMHJwkjPLSQitbT4kqUO4v4EFAUl+2QWQLuhqiDVzg8wttZfO6nJazJX1TVWDE7NX1Z+EXb/dRdbIuIQn5oDAJCIRXi2qxOm+Xqgh6ulfhtIRJryM4Fbx58sd3AYyCm5ihYiCdC6b8m99Gw99ddOapIYnGqJwanlUCgFhF9LxOaIWJy+XTIRtWdrS0zzbQP/zg4wkHA5A6JGRakA7p8rGdJLuqq537ptyZBe674c0qNqMTjVEoNTy3TlQTo2n4zDr38/QIFCCQBwtpAjqJ87xvVpDQtj/udL1Cg9vvPkXnoHgbiTgLKwZJ/cAmg37MmQ3lDASLt7llLLwuBUSwxOLVtSZh62nY7H96fvICVbNafCWCrBK71aIbifO9rYmeq5hURUqbwM4PZx1XpRNw8DOSkl+0QS1aTy4nvp2bbTXzupUWFwqiUGJwKAvEIFfvn7ATafjMX1hJL1n4Y+ZY+pvh7o19aGt3UhasyUCuDeXyULbyZf09xv065kXpRrX0Ci9wvNSU8YnGqJwYlKEwQBkbdSsDkiFuHXk1D8E/OUoxmm9vfACz2cITfkcgZEjV5qrGpi+Y2DQFxEmSE9S8BzmCpItfMDjCz11UrSAwanWmJwosrEPspGWEQsdp+7h5wCBQDAxkSKid6tMcnHDfZmcj23kIi0kpcB3Ap/MqR3BMgttUq52ABo7VMywdymrf7aSQ2CwamWGJyoOum5hdh5Nh5bT93B/bRcAIChRITA7s6Y2t8DXVws9NxCItKaUgHcjSoZ0nsUrbnftn3JkF4rLw7pNUMMTrXE4ETaKlIoceRqIjafjMVfdx6rt3t7WGOqrwf8OjpAwtu6EDUtqbdLFt68EwEoi0r2GVkBnsOfDOkNVV21R00eg1MtMThRTVy8m4YtEbHY/89DFClVP1atrY0R3M8do3u3gpmcyxkQNTl56UBMuGq5g5tHgNySP5AgNgDc+j/pjQoArNvor51UKwxOtcTgRLXxMD0X30XewQ9R8UjLUU0+NZUZYExvV0zp7w5Xa2M9t5CIakRRBNyLKll489ENzf22HVQBqv0IwNULEPOikaaCwamWGJyoLuQWKLD3wj1sPhmLW8nZAACxCBjWyQHTfNugj7sVlzMgaspSbpUsvBkfWWZIz1o1pNchAGg7FJDzd0ljxuBUSwxOVJeUSgF/xjzCppOx+ONGsnp7FxdzTO3vgee7OUNqwNu6EDVpuWlAzLEnQ3pHgby0kn1iQ8C9v6onqkMAYOWup0ZSZRicaonBierLzcRMbI6Iw97z95BfpLqti52ZDJP7umGCd2vYmMr03EIiqjVFEXD3dMmQXkqM5n67jiVDeq16c0ivEWBwqiUGJ6pvj7ML8ENUPL6NjENiRj4AQGYgxotPu2BKfw90cDTTcwuJqM48ilEFqBuHgDunAEFRss/YRvMqPRl/9vWBwamWGJyooRQqlDhw6SE2nYzFP/fS1dsHeNpian8PDGpvBzGXMyBqPnIfq67Siz4IxBxVXbVXTGwIuPuWLLxp5aa/drYwDE61xOBEDU0QBJy78xibI2Jx6HICnqxmgDa2JpjS3x0v92oFYykX3SNqVhSFQPzpkgnmqbc099t3Kll406UXh/TqEYNTLTE4kT7de5yDrafisCPqLjLzVVfpmMsNMN67NYJ83OFsaaTnFhJRvXh0s2ReVPzpMkN6tkB7f1WQavsMIDPVXzubIQanWmJwosYgK78IP567hy0RsYhLyQEASMQijOjiiKm+HujZ2krPLSSiepOTqrpKL/qgamgvv9SQnkQKuA8oGdKzdNVfO5sJBqdaYnCixkSpFPDb9SRsjojFqVsp6u09XC0xzdcDAV0cYSjhcgZEzZaiULVOVPQh1f30Um9r7nfoUjKk59wTEPP/A10xONUSgxM1VlcfZGBLRCz2XXyAAoVqOQMnCzmC+rljfJ/WsDDmbV2ImjVBUA3pFd+Q+O5pQFCW7DexAzz9VcsdtBnCIT0tMTjVEoMTNXbJmfnYduYOvj99B4+yCgAARoYSvNxLtZxBWzv+Z0nUIuSkqhbcvFE8pJdRsk8iAzwGlPRGWbTSXzsbOQanWmJwoqYiv0iBX/9WLWdw7WHJf5hDOthhqq8HfNvZ8rYuRC1FUQEQf6pkSO9xnOZ+h64lC286P80hvVIYnGqJwYmaGkEQcPp2KjadjEX49UQU/1S3dzDF1P4eGPW0C+SGvJSZqMUQBCA5umRI716U5pCeqcOTe+mNANoMBqQmemtqY8DgVEsMTtSUxT3KRtipOOz+6y6yC1SXM1ubSDHRuzVe7esGe3O5nltIRA0uOwW4eeTJkN5vQEFmyT6JDGgzSDWk1z4AsHDRXzv1hMGplhicqDnIyCvErrN3sSUiDvfTcgEAhhIRnu/mjGm+HujiYqHnFhKRXhQVAHciShbeTLujud+xW8lSB049WsSQHoNTLTE4UXNSpFDi6NVEbI6Ixdm4x+rtXu7WmOrrjmGdHCHhbV2IWiZBAJKvlyy8eTcKQKlYYOpYsvBmm8GA1FhfLa1XDE61xOBEzdU/99Kw+WQs/vfPQxQ9ua9LKysjBPdzx5g+rjCXczkDohYt+5FqSC/6IHDrN6Agq2SfgRzwGPRkgnkAYO6sv3bWMQanWmJwouYuMSMP30XewbYzd/A4pxAAYCozwOjerRDczx1uNi17oigRASjKB+JOPhnSOwSkx2vud+quukKvw5MhvSZ8BW+TC07r16/HypUrkZCQgO7du+OLL76Al5dXhWXDwsIwZcoUjW0ymQx5eXnq54IgICQkBN988w3S0tLQv39/fPXVV/D09NSqPQxO1FLkFijw88X72HwyFjeTVH9ZikTAsI4OmOrrAW8Pay5nQESqIb2kqyVDevf+gsaQnpnTkyG9EaqJ5oZN656aTSo47dy5E5MnT8aGDRvg7e2NNWvWYPfu3YiOjoa9vX258mFhYZgzZw6io6PV20QiERwcHNTPP/74Y4SGhmLr1q3w8PDA4sWLcenSJVy9ehVyefVXFDE4UUsjCAL+vPkImyNicSI6Wb29s7M5pvb3wPPdnSAz4HIGRPREVrLmVXqF2SX7DIxU86GKh/TMHPXWTG01qeDk7e2NPn36YN26dQAApVIJV1dXzJ49GwsXLixXPiwsDHPnzkVaWlqF9QmCAGdnZ7z99tuYP38+ACA9PR0ODg4ICwvDuHHjqm0TgxO1ZDFJWdgSEYsfz99DXqFq3RdbUxle7euGiX1bw9ZUpucWElGjUpQPxP35ZOHNQ0D6Xc39zk+XDOk5dmuUQ3pNJjgVFBTA2NgYe/bswahRo9Tbg4KCkJaWhn379pU7JiwsDNOnT4eLiwuUSiV69uyJjz76CJ07dwYA3L59G23btsWFCxfQo0cP9XGDBg1Cjx49sHbt2nJ15ufnIz8/X/08IyMDrq6uDE7UoqXlFOCHqHh8e+oOEjJUQ+FSAzFG9XDGVF8PPOXInw0iKkMQgMQrJQtv3j8HjSE9c5eSq/Q8BjaaIT1dgpNBA7WpQo8ePYJCodAYZgMABwcHXL9+vcJjOnTogM2bN6Nbt25IT0/HqlWr0K9fP1y5cgWtWrVCQkKCuo6ydRbvKys0NBTLli2rgzMiaj4sjaV4Y3A7zBjQBgcvJ2DTyVj8fTcNu/66h11/3UP/djaY2t8DQzrYQ8zlDIgIUPUmOXZRPQYuALKSgBuHVT1Rt34DMu4Df21WPQyNVUN6xQtvmjmUry/tLpCTUvnrGdsAlq71djoV0WtwqgkfHx/4+Pion/fr1w8dO3bE119/jeXLl9eozkWLFmHevHnq58U9TkQEGErEeKG7M17o7oxzdx5jc0QsDl1OQERMCiJiUuBha4Ip/d3xcs9WMJE1uf9SiKg+mdoDPV9VPQrzngzpPZlgnnEfiD6gegCAc8+ShTcduwLp94B1vVRDgZUxkAGzzjVoeNLr/3K2traQSCRITEzU2J6YmAhHR+0mkxkaGuLpp59GTEwMAKiPS0xMhJOTk0adpYfuSpPJZJDJOG+DqDq93KzQy80K99Ny8e2pOPwQFY/YR9lYsu8KVh6Oxniv1gjq5w4Xy8bR/U5EjYihHPAcpnoInwIJl0pWL39wvuRx/D+AeSvApZrQBKj256Q0aHDS6zrqUqkUvXr1Qnh4uHqbUqlEeHi4Rq9SVRQKBS5duqQOSR4eHnB0dNSoMyMjA2fOnNG6TiKqmoulERY92xGnFw3FByM7w8PWBJl5Rdj4x20M/OQ4Zm47j3N3UtEIVjshosZIJAKcugGD3gFeOw68fQN44Qugw3OqIbyMe8C18vOcGwO996vPmzcPQUFB6N27N7y8vLBmzRpkZ2er12qaPHkyXFxcEBoaCgD44IMP0LdvX7Rr1w5paWlYuXIl7ty5g+nTpwNQLU0wd+5cfPjhh/D09FQvR+Ds7KwxAZ2Ias9EZoDJPu6Y5O2GEzeSsOlkLCJiUrD/0kPsv/QQ3V0tMbW/O57t6gRDSfO/3xUR1ZCZA9BzsupRmAvE/glc+A649ou+W1aO3oPT2LFjkZycjCVLliAhIQE9evTAoUOH1JO74+PjIS51g8HHjx9jxowZSEhIgJWVFXr16oVTp06hU6dO6jLvvPMOsrOz8dprryEtLQ2+vr44dOiQVms4EZHuxGIRnnnKAc885YDrCRnYcjIOP128j7/vpmHOjosIPXAdk/u5YYJXa1gaS/XdXCJqzAyNgPbDVfOjGmFw0vs6To0R13Eiqr1HWfn44Uw8vo28g0dZqnkKckMxXu7ZClP6u6OdvZmeW0hEjdqDi8DGQdWXe+13wLlHrV5Kl9/77DsnonphayrDm0M9EbFwCD4d3R2dnMyRV6jEtjPx8Fv9B4I2R+H3G8mcB0VETYreh+qIqHmTGUjwcq9WeKmnC6JiU7HpZCyOXkvE7zeS8fuNZHjam2JKfw+81NMFckPe1oWIGjcO1VWAQ3VE9Ss+JQdhp+Kw66+7yMovAgBYGRtigndrvNrXHY4WnI9I1OKl3W2wdZyazC1XGisGJ6KGkZlXiF1/3UPYqVjcTc0FABiIRXi+mxOm+nqgWytL/TaQiPQr7S5OXYrG13/cxqOsAvVmW1Mp/m9gG/Tr2qFO1nBicKolBieihqVQCjh6NRGbI2IRFZuq3t7bzQrTfD0wrJMDDLicAVGLc+jyQ7z+/XmUDSrFN3n6alJPBHRxKnuYzhicaonBiUh/Lt1Lx5aIWPz6zwMUKlT/PblYGiG4nzvGernCXG6o5xYSUX0SBAF5hUpk5BXi+c9PIjmr4qE6EQBHCzlOvvsMJLW8XyaDUy0xOBHpX1JGHr47fQfbzsQjNVvVRW8ilWB0b1cE93OHu62JnltIRIIgIL9IiZwCBbLzi5BdUITsfAVyCopUz4u/Lt7/5HlWfpHGMTn5Co1/lTokk+0z+sKnrU2tzoPBqZYYnIgaj7xCBfZdvI9NJ2NxIzELgOpuDUOfcsBUX3f4tLGBSFS7vzaJWopChfJJWFEgJ798gCkJPYqScmVCUOljcgoUKNIl5dSDteN6YGQPl1rVocvvfS5HQESNmtxQgrF9WmNMb1dExKRgc0QsfruehGPXEnHsWiI6Opljan93BHZ35nIG1KwUKZTIKaygp+ZJr0zpkFNRL45GCHrSm1OgUNZbe40MJTCRSWAsNYCJzAAmUgmMZQYwLd4mlai2ywxgLJXA5Ek5Y1nx16p/jWUSmMoMcDE+DRP+e6ba17U3a9ircNnjVAH2OBE1breSsxAWEYc95+4ht1ABQHWVzURvN0zq6wY7M5meW0gtjVIpIKew6l6crHzV/uJeHM0Q9OSYUr09+UX1F3JkBuIyAUZS8lxmUBJgpAYwrjAEaR5jLDWo9TyjshRKAb4f/4aE9Lxyk8MBznFqVBiciJqG9JxCbD8bj62n4vAwPQ8AIJWI8UIPZ0zt74FOzvz5pfIEQUBuoaLinhp170zJvJxKQ1CpkFMc4OuDoURUEmaq6cUxlRk86fGRqP/VDDmqck3lptvFV9UB0AhPvKqukWFwImpaChVKHL6SgE0nY3EhPk293aeNDab6euCZp+zr/K9hahjFk481e2QqDzClJxhrztFRaNRRX7/5xCLARGbwJMBIKgw0JqV6cVShp+JenOJjpAZNI+TUl0OXH2LZr1fVfxwBgJOFHCGBneokNAEMTrXG4ETUdJ2Pf4wtEXE4cOkhFE8mrbrZGGNKP3e80tsVpjJO7axPBUXKkgBTthenzHCVKshUMkdHfYxC/T7WNZEI6l4c0ydzbTR6cUrNtynbe2NcKuQU12EiM4DMQMyLFeqBQikgKjYVSZl5sDeTw8vDuk7/GGJwqiUGJ6Km70FaLr6NvIPtUfFIzy0EAJjJDTCujysm+7jD1dpYo3x9/8fcGBUplOUCTMVXVqkCjGpOTuk5OqoQpN5XUKRee6s+FM+lqTDASA3KzNEpHtKqvOfHyFDCkEMAGJxqjcGJqPnIKSjCj+fvY0tELG4nZwNQDaf4d3bENF8P9HKzwuErCfU+FFBbCqWAHI0AU3kvjnq4qtKeH1UdBfU8+bi4F0cj1JTtxSkzXFVh0JEZwMhQ0uyDLOkPg1MtMTgRNT9KpYDfbyZj88lY/HnzkXq7m40x7qTklCtfm8mngiCo59Lk5CtKhqAqCDSle3FKr69TtieoPicfSyVijUvCKw0wZa++Kr7svMzQlbGhhLfIoSaFwamWGJyImrfohExsiYjFj+fvVTu0ZGlkiDl+nsgtVJQKQaWCTZlenJz8IuQUKupt8rFELNK4kspEWmrdnHJXWpUPQaWHsYqPbemTj4kYnGqJwYmoZTh8+SH+78mlzvVBXDz5uKI5OBqhR1JpCCrb88PJx0R1jyuHExFpIU/LOT49XC3R3sFUY32cCntxirc/CUpyQ4YcouaGwYmIWixtb9XwbsBTtb6JKBE1DxzYJqIWy8vDGk4WclTWJySC6uo6Lw/rhmwWETViDE5E1GJJxCKEBHYCgHLhqfh5SGAnXgZPRGoMTkTUogV0ccJXk3rC0UJz2M7RQl5n98EiouaDc5yIqMUL6OKEYZ0cW9zK4USkOwYnIiKohu04AZyIqsOhOiIiIiItMTgRERERaYnBiYiIiEhLDE5EREREWmJwIiIiItISgxMRERGRlrgcQQUEQQCgulsyERERNW/Fv++Lf/9XhcGpApmZmQAAV1dXPbeEiIiIGkpmZiYsLCyqLCMStIlXLYxSqcSDBw9gZmYGkahuVw7OyMiAq6sr7t69C3Nz8zqtuzHheTYvPM/mhefZvPA8a08QBGRmZsLZ2RlicdWzmNjjVAGxWIxWrVrV62uYm5s36w94MZ5n88LzbF54ns0Lz7N2qutpKsbJ4URERERaYnAiIiIi0hKDUwOTyWQICQmBTCbTd1PqFc+zeeF5Ni88z+aF59mwODmciIiISEvscSIiIiLSEoMTERERkZYYnIiIiIi0xOBUB9avXw93d3fI5XJ4e3sjKiqqyvK7d+/GU089Bblcjq5du+LAgQMa+wVBwJIlS+Dk5AQjIyP4+fnh5s2b9XkKWtHlPL/55hsMGDAAVlZWsLKygp+fX7nywcHBEIlEGo+AgID6Po1q6XKeYWFh5c5BLpdrlGkO7+fgwYPLnadIJMJzzz2nLtPY3s8//vgDgYGBcHZ2hkgkws8//1ztMSdOnEDPnj0hk8nQrl07hIWFlSuj6897fdP1PPfu3Ythw4bBzs4O5ubm8PHxweHDhzXKLF26tNx7+dRTT9XjWVRP1/M8ceJEhZ/ZhIQEjXJN/f2s6OdOJBKhc+fO6jKN8f0MDQ1Fnz59YGZmBnt7e4waNQrR0dHVHtcYfn8yONXSzp07MW/ePISEhOD8+fPo3r07/P39kZSUVGH5U6dOYfz48Zg2bRouXLiAUaNGYdSoUbh8+bK6zCeffILPP/8cGzZswJkzZ2BiYgJ/f3/k5eU11GmVo+t5njhxAuPHj8fx48cRGRkJV1dXDB8+HPfv39coFxAQgIcPH6of27dvb4jTqZSu5wmoFmMrfQ537tzR2N8c3s+9e/dqnOPly5chkUgwevRojXKN6f3Mzs5G9+7dsX79eq3Kx8bG4rnnnsOQIUNw8eJFzJ07F9OnT9cIFTX5fNQ3Xc/zjz/+wLBhw3DgwAGcO3cOQ4YMQWBgIC5cuKBRrnPnzhrv5cmTJ+uj+VrT9TyLRUdHa5yHvb29el9zeD/Xrl2rcX53796FtbV1uZ/NxvZ+/v7775g5cyZOnz6No0ePorCwEMOHD0d2dnalxzSa358C1YqXl5cwc+ZM9XOFQiE4OzsLoaGhFZYfM2aM8Nxzz2ls8/b2Fv7v//5PEARBUCqVgqOjo7By5Ur1/rS0NEEmkwnbt2+vhzPQjq7nWVZRUZFgZmYmbN26Vb0tKChIGDlyZF03tVZ0Pc8tW7YIFhYWldbXXN/Pzz77TDAzMxOysrLU2xrj+1kMgPDTTz9VWeadd94ROnfurLFt7Nixgr+/v/p5bb9v9U2b86xIp06dhGXLlqmfh4SECN27d6+7htUxbc7z+PHjAgDh8ePHlZZpju/nTz/9JIhEIiEuLk69rbG/n4IgCElJSQIA4ffff6+0TGP5/ckep1ooKCjAuXPn4Ofnp94mFovh5+eHyMjICo+JjIzUKA8A/v7+6vKxsbFISEjQKGNhYQFvb+9K66xvNTnPsnJyclBYWAhra2uN7SdOnIC9vT06dOiA119/HSkpKXXadl3U9DyzsrLg5uYGV1dXjBw5EleuXFHva67v56ZNmzBu3DiYmJhobG9M76euqvvZrIvvW2OkVCqRmZlZ7mfz5s2bcHZ2Rps2bTBx4kTEx8frqYW106NHDzg5OWHYsGGIiIhQb2+u7+emTZvg5+cHNzc3je2N/f1MT08HgHKfw9Iay+9PBqdaePToERQKBRwcHDS2Ozg4lBtHL5aQkFBl+eJ/damzvtXkPMt699134ezsrPGBDggIwLfffovw8HB8/PHH+P333zFixAgoFIo6bb+2anKeHTp0wObNm7Fv3z58//33UCqV6NevH+7duwegeb6fUVFRuHz5MqZPn66xvbG9n7qq7GczIyMDubm5dfJz0BitWrUKWVlZGDNmjHqbt7c3wsLCcOjQIXz11VeIjY3FgAEDkJmZqceW6sbJyQkbNmzAjz/+iB9//BGurq4YPHgwzp8/D6Bu/l9rbB48eICDBw+W+9ls7O+nUqnE3Llz0b9/f3Tp0qXSco3l9ydv8kv1bsWKFdixYwdOnDihMXF63Lhx6q+7du2Kbt26oW3btjhx4gSGDh2qj6bqzMfHBz4+Purn/fr1Q8eOHfH1119j+fLlemxZ/dm0aRO6du0KLy8vje3N4f1saX744QcsW7YM+/bt05j7M2LECPXX3bp1g7e3N9zc3LBr1y5MmzZNH03VWYcOHdChQwf18379+uHWrVv47LPP8N133+mxZfVn69atsLS0xKhRozS2N/b3c+bMmbh8+bLe511piz1OtWBrawuJRILExESN7YmJiXB0dKzwGEdHxyrLF/+rS531rSbnWWzVqlVYsWIFjhw5gm7dulVZtk2bNrC1tUVMTEyt21wTtTnPYoaGhnj66afV59Dc3s/s7Gzs2LFDq/9s9f1+6qqyn01zc3MYGRnVyeejMdmxYwemT5+OXbt2lRv+KMvS0hLt27dvMu9lZby8vNTn0NzeT0EQsHnzZrz66quQSqVVlm1M7+esWbPwv//9D8ePH0erVq2qLNtYfn8yONWCVCpFr169EB4ert6mVCoRHh6u0QtRmo+Pj0Z5ADh69Ki6vIeHBxwdHTXKZGRk4MyZM5XWWd9qcp6A6uqG5cuX49ChQ+jdu3e1r3Pv3j2kpKTAycmpTtqtq5qeZ2kKhQKXLl1Sn0Nzej8B1aXA+fn5mDRpUrWvo+/3U1fV/WzWxeejsdi+fTumTJmC7du3aywpUZmsrCzcunWrybyXlbl48aL6HJrT+wmorlKLiYnR6o+axvB+CoKAWbNm4aeffsJvv/0GDw+Pao9pNL8/62yaeQu1Y8cOQSaTCWFhYcLVq1eF1157TbC0tBQSEhIEQRCEV199VVi4cKG6fEREhGBgYCCsWrVKuHbtmhASEiIYGhoKly5dUpdZsWKFYGlpKezbt0/4559/hJEjRwoeHh5Cbm5ug59fMV3Pc8WKFYJUKhX27NkjPHz4UP3IzMwUBEEQMjMzhfnz5wuRkZFCbGyscOzYMaFnz56Cp6enkJeXp5dzFATdz3PZsmXC4cOHhVu3bgnnzp0Txo0bJ8jlcuHKlSvqMs3h/Szm6+srjB07ttz2xvh+ZmZmChcuXBAuXLggABBWr14tXLhwQbhz544gCIKwcOFC4dVXX1WXv337tmBsbCwsWLBAuHbtmrB+/XpBIpEIhw4dUpep7vumD7qe57Zt2wQDAwNh/fr1Gj+baWlp6jJvv/22cOLECSE2NlaIiIgQ/Pz8BFtbWyEpKanBz6+Yruf52WefCT///LNw8+ZN4dKlS8KcOXMEsVgsHDt2TF2mObyfxSZNmiR4e3tXWGdjfD9ff/11wcLCQjhx4oTG5zAnJ0ddprH+/mRwqgNffPGF0Lp1a0EqlQpeXl7C6dOn1fsGDRokBAUFaZTftWuX0L59e0EqlQqdO3cW9u/fr7FfqVQKixcvFhwcHASZTCYMHTpUiI6ObohTqZIu5+nm5iYAKPcICQkRBEEQcnJyhOHDhwt2dnaCoaGh4ObmJsyYMUOv/2EV0+U8586dqy7r4OAgPPvss8L58+c16msO76cgCML169cFAMKRI0fK1dUY38/iy9HLPorPKygoSBg0aFC5Y3r06CFIpVKhTZs2wpYtW8rVW9X3TR90Pc9BgwZVWV4QVMswODk5CVKpVHBxcRHGjh0rxMTENOyJlaHreX788cdC27ZtBblcLlhbWwuDBw8Wfvvtt3L1NvX3UxBUl9wbGRkJGzdurLDOxvh+VnSOADR+5hrr70/RkxMgIiIiompwjhMRERGRlhiciIiIiLTE4ERERESkJQYnIiIiIi0xOBERERFpicGJiIiISEsMTkRERERaYnAiIiIi0hKDExFRLZ04cQIikQhpaWn6bgoR1TMGJyIiIiItMTgRERERaYnBiYiaPKVSidDQUHh4eMDIyAjdu3fHnj17AJQMo+3fvx/dunWDXC5H3759cfnyZY06fvzxR3Tu3BkymQzu7u749NNPNfbn5+fj3XffhaurK2QyGdq1a4dNmzZplDl37hx69+4NY2Nj9OvXD9HR0fV74kTU4BiciKjJCw0NxbfffosNGzbgypUreOuttzBp0iT8/vvv6jILFizAp59+irNnz8LOzg6BgYEoLCwEoAo8Y8aMwbhx43Dp0iUsXboUixcvRlhYmPr4yZMnY/v27fj8889x7do1fP311zA1NdVox/vvv49PP/0Uf/31FwwMDDB16tQGOX8iajgiQRAEfTeCiKim8vPzYW1tjWPHjsHHx0e9ffr06cjJycFrr72GIUOGYMeOHRg7diwAIDU1Fa1atUJYWBjGjBmDiRMnIjk5GUeOHFEf/84772D//v24cuUKbty4gQ4dOuDo0aPw8/Mr14YTJ05gyJAhOHbsGIYOHQoAOHDgAJ577jnk5uZCLpfX83eBiBoKe5yIqEmLiYlBTk4Ohg0bBlNTU/Xj22+/xa1bt9TlSocqa2trdOjQAdeuXQMAXLt2Df3799eot3///rh58yYUCgUuXrwIiUSCQYMGVdmWbt26qb92cnICACQlJdX6HImo8TDQdwOIiGojKysLALB//364uLho7JPJZBrhqaaMjIy0KmdoaKj+WiQSAVDNvyKi5oM9TkTUpHXq1AkymQzx8fFo166dxsPV1VVd7vTp0+qvHz9+jBs3bqBjx44AgI4dOyIiIkKj3oiICLRv3x4SiQRdu3aFUqnUmDNFRC0Te5yIqEkzMzPD/Pnz8dZbb0GpVMLX1xfp6emIiIiAubk53NzcAAAffPABbGxs4ODggPfffx+2trYYNWoUAODtt99Gnz59sHz5cowdOxaRkZFYt24dvvzySwCAu7s7goKCMHXqVHz++efo3r077ty5g6SkJIwZM0Zfp05EesDgRERN3vLly2FnZ4fQ0FDcvn0blpaW6NmzJ9577z31UNmKFSswZ84c3Lx5Ez169MCvv/4KqVQKAOjZsyd27dqFJUuWYPny5XBycsIHH3yA4OBg9Wt89dVXeO+99/DGG28gJSUFrVu3xnvvvaeP0yUiPeJVdUTUrBVf8fb48WNYWlrquzlE1MRxjhMRERGRlhiciIiIiLTEoToiIiIiLbHHiYiIiEhLDE5EREREWmJwIiIiItISgxMRERGRlhiciIiIiLTE4ERERESkJQYnIiIiIi0xOBERERFpicGJiIiISEv/D+9ZKoCWDjvGAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"final validation accuracy: 0.816\n",
"[ ok ] the loop learned something: 81.6% accuracy (vs 10% chance)\n"
]
}
],
"source": [
"# viz: the learning curve from the experiment log\n",
"ep = [r[0] for r in history]\n",
"plt.figure(figsize=(6, 4))\n",
"plt.plot(ep, [r[1] for r in history], \"o-\", label=\"train loss\")\n",
"plt.plot(ep, [r[2] for r in history], \"s-\", label=\"val loss\")\n",
"plt.plot(ep, [r[3] for r in history], \"^-\", label=\"val acc\")\n",
"plt.xlabel(\"epoch\"); plt.ylabel(\"value\"); plt.title(\"FashionMNIST MLP learning curve\")\n",
"plt.legend(); plt.tight_layout(); plt.show()\n",
"final_acc = history[-1][3]\n",
"print(f\"final validation accuracy: {final_acc:.3f}\")\n",
"# A working loop must clearly beat the 10% random-chance baseline. The full run lands\n",
"# in the 0.80s; the FAST smoke run (one epoch on 2000 images, ~15 updates) lands lower\n",
"# but still far above chance, so the floor is FAST-aware to avoid a seed-fragile gate.\n",
"floor = 0.45 if FAST else 0.78\n",
"assert final_acc > floor, \\\n",
" f\"final acc {final_acc:.3f} is below {floor:.2f}; far above 10% chance is the sign of a working loop\"\n",
"print(\"[ ok ] the loop learned something:\", f\"{final_acc:.1%} accuracy (vs 10% chance)\")"
]
},
{
"cell_type": "markdown",
"id": "41432520",
"metadata": {},
"source": [
"### Experiment log\n",
"\n",
"The change-and-measure table. Every named change records its measured numbers so you\n",
"can tell at a glance whether a re-run is on track (\"if you do not see ~0.83, something\n",
"is wrong\"). Numbers are the committed full-fidelity run; the FAST smoke run trains on\n",
"less data and lands lower, which is expected.\n",
"\n",
"| Config | Data | Epochs | Final val acc | Note |\n",
"|---|---|---|---|---|\n",
"| MLP 784-128-10, SGD lr=0.1 mom=0.9 | 12000 train / 4000 test | 3 | ~0.80-0.83 | the committed run |\n",
"| same, FAST smoke | 2000 train / 1000 test | 1 | ~0.60-0.65 | CI path, far fewer updates |\n",
"| broken: no zero_grad (next cell) | 2000 train | 1 | collapses | gradients accumulate; staged failure |\n",
"\n",
"These are ranges, not promises: BLAS threading and library versions move the last\n",
"digit. The shape of the curve, not the third decimal, is what you are checking.\n"
]
},
{
"cell_type": "markdown",
"id": "95d9e8be",
"metadata": {},
"source": [
"### A deliberate failure: forgetting zero_grad\n",
"\n",
"This is the most common training bug in PyTorch, and it does not raise an error. We saw\n",
"in Part 3 that `.backward()` *accumulates* into `.grad`. If you drop `opt.zero_grad()`\n",
"from the loop, every step adds the new gradient on top of every previous step's\n",
"gradient. The effective step size balloons and training falls apart, silently. We run\n",
"the broken loop first, watch it fail, then fix it.\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "ff911b5f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:59.120821Z",
"iopub.status.busy": "2026-06-10T19:01:59.120745Z",
"iopub.status.idle": "2026-06-10T19:01:59.179684Z",
"shell.execute_reply": "2026-06-10T19:01:59.179158Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"broken loop: loss started at 2.321, ended at 33.795\n",
"watch it: with grads accumulating, the loss does not settle; it spikes or diverges.\n"
]
}
],
"source": [
"# This loop is WRONG on purpose: the zero_grad line is missing.\n",
"def train_broken(model, train_dl, lr=0.1, steps=60):\n",
" opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9)\n",
" losses = []\n",
" seen = 0\n",
" for xb, yb in train_dl:\n",
" # opt.zero_grad() <-- the bug: this line is gone, so grads accumulate forever\n",
" logits = model(xb)\n",
" loss = F.cross_entropy(logits, yb)\n",
" loss.backward()\n",
" opt.step()\n",
" losses.append(loss.item())\n",
" seen += 1\n",
" if seen >= steps:\n",
" break\n",
" return losses\n",
"\n",
"broken_model = make_mlp()\n",
"broken_losses = train_broken(broken_model, train_dl)\n",
"print(f\"broken loop: loss started at {broken_losses[0]:.3f}, ended at {broken_losses[-1]:.3f}\")\n",
"print(\"watch it: with grads accumulating, the loss does not settle; it spikes or diverges.\")"
]
},
{
"cell_type": "markdown",
"id": "9dfcbdd1",
"metadata": {},
"source": [
"> **Predict:** before the fix, what is the diagnostic that proves the bug? Hint:\n",
"look at the gradient norm, not just the loss. AnswerThe L2\n",
"norm of the accumulated gradients grows step over step instead of staying roughly\n",
"stable. A healthy loop has a roughly constant grad norm; the broken loop's norm climbs\n",
"because each step's gradient is piled on the last.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "47269de5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:59.180635Z",
"iopub.status.busy": "2026-06-10T19:01:59.180550Z",
"iopub.status.idle": "2026-06-10T19:01:59.244353Z",
"shell.execute_reply": "2026-06-10T19:01:59.243933Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"broken grad-norm: first 0.92 -> last 26.76 (grows)\n",
"fixed grad-norm: first 0.84 -> last 2.11 (bounded)\n",
"[ ok ] the diagnostic confirms accumulation is the culprit\n"
]
}
],
"source": [
"# Diagnostic: grad norm under the broken loop grows; under the fixed loop it stays bounded.\n",
"def grad_norm_trace(model, zero_grad, steps=30):\n",
" opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)\n",
" norms, seen = [], 0\n",
" for xb, yb in train_dl:\n",
" if zero_grad:\n",
" opt.zero_grad()\n",
" F.cross_entropy(model(xb), yb).backward()\n",
" total = sum(p.grad.detach().norm() ** 2 for p in model.parameters()) ** 0.5\n",
" norms.append(total.item())\n",
" opt.step()\n",
" seen += 1\n",
" if seen >= steps:\n",
" break\n",
" return norms\n",
"\n",
"broken_norms = grad_norm_trace(make_mlp(), zero_grad=False)\n",
"fixed_norms = grad_norm_trace(make_mlp(), zero_grad=True)\n",
"print(f\"broken grad-norm: first {broken_norms[0]:.2f} -> last {broken_norms[-1]:.2f} (grows)\")\n",
"print(f\"fixed grad-norm: first {fixed_norms[0]:.2f} -> last {fixed_norms[-1]:.2f} (bounded)\")\n",
"assert broken_norms[-1] > fixed_norms[-1] * 2, \\\n",
" \"without zero_grad the accumulated grad norm should blow past the fixed one\"\n",
"print(\"[ ok ] the diagnostic confirms accumulation is the culprit\")"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "1f65fdc9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:59.245456Z",
"iopub.status.busy": "2026-06-10T19:01:59.245365Z",
"iopub.status.idle": "2026-06-10T19:01:59.309429Z",
"shell.execute_reply": "2026-06-10T19:01:59.309109Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAGGCAYAAABmPbWyAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAeehJREFUeJzt3XdYU2cbBvA7BAg7yAZBRMSBuBVr3YqrLRW1X6utdbd1V1utdqK21m7tUmtt1VbtdFVbt+JelaLFLaKggijIlpW83x+RlMg6gUAC3L/ryqWc+eQkJA/vec5zZEIIASIiIiIqk5mxAyAiIiKqCZg0EREREUnApImIiIhIAiZNRERERBIwaSIiIiKSgEkTERERkQRMmoiIiIgkYNJEREREJAGTJiIiIiIJmDQRUYXIZDLMnTu3SrY7ZcoUg2/XWPsxRaNHj0bDhg0rvP6qVasgk8nw999/Gy6oUly+fBn9+vWDUqmETCbDpk2btPu/du1ale+fqCgmTURU7Y4cOYK5c+ciNTXV2KFQGZYsWYJVq1YZNYZRo0bh33//xYIFC/Djjz+iQ4cORo2H6jYmTURU7Y4cOYJ58+YxaTJxxk6a7t+/j6NHj2LcuHGYMmUKRowYAW9vbzz//PO4f/8+fH19jRYb1U1MmoiqWFZWlrFD0FKr1cjJyTF2GESS3LlzBwDg6OioM10ul8PKygoymcwIUVFdxqSJ6rRr165BJpOV+ijq+PHjGDBgAJRKJWxsbNCjRw8cPnxYZ5m5c+dCJpPh3LlzePbZZ1GvXj107doVAFBQUIB3330X/v7+UCgUaNiwId544w3k5ubqHXdERAQ6dOgAKysr+Pv745tvvtHuu6jCup21a9eiRYsWUCgU2L59OwDgk08+waOPPgpnZ2dYW1ujffv2+P3334vtKzc3FzNmzICrqyvs7e3x5JNP4saNGyXGdeHCBcTFxZUZ+9y5czFr1iwAgJ+fn/ZYP1yfsmnTJgQFBUGhUKBFixbauIu6efMmxo4dC3d3d+1y33//fZn7f9jatWvRtGlTWFlZoX379jhw4IDO/NLqf0o63vfv38e0adPg4uKiPVY3b96UVP+Vl5eHd955B+3bt4dSqYStrS26deuGffv26SxX+J795JNPsHz5cu37qWPHjjh58mSx7RYeRysrKwQFBWHjxo2SjkvDhg1x9uxZ7N+/X/sa9ezZU2eZ3NxcvPLKK3B1dYWtrS0GDx6sTXSK2rZtG7p16wZbW1vY29vj8ccfx9mzZ8vc/9y5c7UjSbNmzYJMJtO+Dg/XNO3duxdmZmZ45513dLaxbt06yGQyLF26VNJzJiqPubEDIDImV1dX/PjjjzrT8vPzMWPGDFhaWmqn7d27FwMHDkT79u0RHh4OMzMzrFy5Er1798bBgwcRHByss43//e9/CAgIwPvvvw8hBABg/PjxWL16NZ566im8+uqrOH78OBYuXIjz589L/iIDgH/++QcDBgyAp6cn5s2bB5VKhfnz58PV1bXE5ffu3Ytff/0VU6ZMgYuLi/aL5/PPP8eTTz6J5557Dnl5efj555/xv//9D1u3bsXjjz+uXX/8+PFYs2YNnn32WTz66KPYu3evzvyimjdvjh49eiAiIqLU+IcMGYJLly7hp59+wqJFi+Di4gIAOvEfOnQIGzZswKRJk2Bvb48vvvgCQ4cORVxcHJydnQEAt2/fxiOPPKJNDF1dXbFt2zaMGzcO6enpmD59ernHcv/+/fjll18wbdo0KBQKLFmyBAMGDMCJEycQFBRU7voPGz16NH799Vc8//zzeOSRR7B///5Sj9XD0tPTsWLFCgwfPhwvvPACMjIy8N1336F///44ceIE2rRpo7P8unXrkJGRgZdeegkymQwfffQRhgwZgqtXr8LCwgIAsHPnTgwdOhSBgYFYuHAhkpOTMWbMGHh7e5cbz+LFizF16lTY2dnhzTffBAC4u7vrLDN16lTUq1cP4eHhuHbtGhYvXowpU6bgl19+0S7z448/YtSoUejfvz8+/PBDZGdnY+nSpejatSv++eefUgvShwwZAkdHR8yYMQPDhw/HY489Bjs7uxKX7d27NyZNmoSFCxciLCwM7dq1Q0JCAqZOnYqQkBBMmDCh3OdLJIkgIh2TJk0Scrlc7N27VwghhFqtFgEBAaJ///5CrVZrl8vOzhZ+fn6ib9++2mnh4eECgBg+fLjONqOiogQAMX78eJ3pM2fOFAC0+5IiNDRU2NjYiJs3b2qnXb58WZibm4uHf6UBCDMzM3H27Nli28nOztb5OS8vTwQFBYnevXsXi3vSpEk6yz777LMCgAgPDy+2vx49epT7HD7++GMBQMTGxhabB0BYWlqKK1euaKedPn1aABBffvmldtq4ceOEp6enuHv3rs76w4YNE0qlstjzK2k/AMTff/+tnXb9+nVhZWUlBg8erJ02atQo4evrW2z9wte60KlTpwQAMX36dJ3lRo8eXeKxelhBQYHIzc3VmXbv3j3h7u4uxo4dq50WGxsrAAhnZ2eRkpKinb5582YBQGzZskU7rU2bNsLT01OkpqZqp+3cuVMAKPE5PaxFixYlvp4rV64UAERISIjO78SMGTOEXC7X7i8jI0M4OjqKF154QWf9xMREoVQqi01/WOFz/fjjj0vcf9H3T1ZWlmjcuLFo0aKFyMnJEY8//rhwcHAQ169fL/d5EknF03NERfzwww9YsmQJPvroI/Tq1QsAEBUVhcuXL+PZZ59FcnIy7t69i7t37yIrKwt9+vTBgQMHoFardbbz8F+2f/31FwDglVde0Zn+6quvAgD+/PNPSfGpVCrs3r0bYWFh8PLy0k5v3LgxBg4cWOI6PXr0QGBgYLHp1tbW2v/fu3cPaWlp6NatGyIjI4vFPW3aNJ11SxvFEUKUOcokVUhICPz9/bU/t2rVCg4ODrh69ap2P+vXr0doaCiEENrX5O7du+jfvz/S0tJ0nkdpOnfujPbt22t/btCgAQYNGoQdO3ZApVLpFXPh6cNJkybpTJ86daqk9eVyuXZ0U61WIyUlBQUFBejQoUOJz+WZZ55BvXr1tD9369YNALTHKCEhAVFRURg1ahSUSqV2ub59+5b4fqiIF198UecUZbdu3aBSqXD9+nUAwK5du5Camorhw4frvEZyuRydOnUqduqxMmxsbLBq1SqcP38e3bt3x59//olFixahQYMGBtsHEU/PET0QFRWFCRMmYPjw4TrJzeXLlwFoLn0uTVpams4XmJ+fn87869evw8zMDI0bN9aZ7uHhAUdHR+2XTHmSkpJw//79YtsBUOK0kmIptHXrVrz33nuIiorSqasq+iVYGHfRBAYAmjZtKineiirpi65evXq4d+8eAE2BcGpqKpYvX47ly5eXuI2kpKRy9xMQEFBsWpMmTZCdnY07d+7Aw8NDcsyFx+rh413a61KS1atX49NPP8WFCxeQn5+vnV7Sa/jwMSp8/xUeo8L3VEnPsWnTppKSyvKUF0Ph707v3r1LXN/BwaHSMRTVpUsXTJw4EV9//TX69++PsWPHGnT7REyaiKD5kB86dCiaNGmCFStW6MwrHEX6+OOPi9WVFHq41qLoKE5Rxrjap6RYDh48iCeffBLdu3fHkiVL4OnpCQsLC6xcuRLr1q2r9hgfJpfLS5wuHtSHFb4mI0aMKDWZbdWqlUFiKe0103ckqjxr1qzB6NGjERYWhlmzZsHNzQ1yuRwLFy5ETExMseXLO0bVQerr9OOPP5aYgJqbG/YrKDc3VzvSGRMTg+zsbNjY2Bh0H1S3MWmiOk+tVuO5555Damoqdu/eXexDtnCUxcHBASEhIRXah6+vL9RqNS5fvozmzZtrp9++fRupqamS+824ubnBysoKV65cKTavpGmlWb9+PaysrLBjxw4oFArt9JUrV5YYd0xMjM7o0sWLFyXvqySVTR4Lr+RTqVQVfk2A/0ZCirp06RJsbGy0hen16tUrsZ/Uw6ODhccqNjZWZ3RH6uvy+++/o1GjRtiwYYPO8QkPD5e0/sMK31MlPUepr19lX6fC3x03N7dKvU5ShYeH4/z58/jkk08we/ZszJkzB1988UWV75fqDtY0UZ03b9487NixAz/99FOJp0Hat28Pf39/fPLJJ8jMzCw2v6RLrB/22GOPAdBckVTUZ599BgCSr7CSy+UICQnBpk2bcOvWLe30K1euYNu2bZK2UbgdmUymM1py7do1bNq0SWe5wjqph794Hn4ehaS0HAAAW1tbAKhwc0u5XI6hQ4di/fr1iI6OLjZfymsCAEePHtU5TRUfH4/NmzejX79+2lEUf39/pKWl4cyZM9rlEhISil3x2L9/fwCahpBFffnll5KfE6A7UnT8+HEcPXpU0voP8/T0RJs2bbB69WqkpaVpp+/atQvnzp2TtA1bW9tKNSDt378/HBwc8P777+ucbiwk9XWS4vjx4/jkk08wffp0vPrqq5g1axa++uor7N+/32D7IOJIE9Vp//77L9599110794dSUlJWLNmjc78ESNGwMzMDCtWrMDAgQPRokULjBkzBvXr18fNmzexb98+ODg4YMuWLWXup3Xr1hg1ahSWL1+O1NRU9OjRAydOnMDq1asRFhamLTqXYu7cudi5c6e2fkOlUuGrr75CUFAQoqKiJG3j8ccfx2effYYBAwbg2WefRVJSEr7++ms0btxYJzlo06YNhg8fjiVLliAtLQ2PPvoo9uzZU+roiZSWAwC0xddvvvkmhg0bBgsLC4SGhmqTKSk++OAD7Nu3D506dcILL7yAwMBApKSkIDIyErt370ZKSkq52wgKCkL//v11Wg4AmkS60LBhwzB79mwMHjwY06ZN014y36RJE52Eq3379hg6dCgWL16M5ORkbcuBS5cuASh/1OaJJ57Ahg0bMHjwYDz++OOIjY3FsmXLEBgYWGKyLsXChQvx+OOPo2vXrhg7dixSUlLw5ZdfokWLFpK22b59eyxduhTvvfceGjduDDc3t1Lrk0ri4OCApUuX4vnnn0e7du0wbNgwuLq6Ii4uDn/++Se6dOmCr776qkLPraicnByMGjUKAQEBWLBgAQDNa7hlyxaMGTMG//77r17vLaJSGfHKPSKj27dvn/bS85IeRf3zzz9iyJAhwtnZWSgUCuHr6yuefvppsWfPHu0yhZeh37lzp9i+8vPzxbx584Sfn5+wsLAQPj4+4vXXXxc5OTl6x71nzx7Rtm1bYWlpKfz9/cWKFSvEq6++KqysrHSWAyAmT55c4ja+++47ERAQIBQKhWjWrJlYuXJlscvohRDi/v37Ytq0acLZ2VnY2tqK0NBQER8fX6mWA0II8e6774r69esLMzMzncvHS4vZ19dXjBo1Smfa7du3xeTJk4WPj4+wsLAQHh4eok+fPmL58uXl7r9wP2vWrNEeh7Zt24p9+/YVW3bnzp0iKChIWFpaiqZNm4o1a9aUeKyysrLE5MmThZOTk7CzsxNhYWHi4sWLAoD44IMPyoxHrVaL999/X/j6+mpj2bp1a7GWB6Vdhl/4nB5+TdavXy+aN28uFAqFCAwMFBs2bCi1jcLDEhMTxeOPPy7s7e11XtvCS/5Pnjyps3zh79PDx3Dfvn2if//+QqlUCisrK+Hv7y9Gjx6t0+6hJFJbDhS2Ojh+/LjOcn///bcwNzcXEydOLPe5EkkhE6IaqwaJqMqEhYXh7NmzJdawkPFERUWhbdu2WLNmDZ577jljh0NElcCaJqIa6P79+zo/X758GX/99Vex21xQ9Xr4dQE09V9mZmbo3r27ESIiIkNiTRORicjMzCy3zsTV1RVyuRyNGjXC6NGj0ahRI1y/fh1Lly6FpaUlXnvttWqKlkry0Ucf4dSpU+jVqxfMzc2xbds2bNu2DS+++CJ8fHyMHR4RVRJPzxGZiLlz5+oUIJckNjYWDRs2xJgxY7Bv3z4kJiZCoVCgc+fOeP/999GuXbtqipZKsmvXLsybNw/nzp1DZmYmGjRogOeffx5vvvmmwXsSEVH1Y9JEZCKuXr2qvQVGabp27QorK6tqioiIiIpi0kREREQkAQvBiYiIiCSo9SfZ1Wo1bt26BXt7e6Pc94uIiIhMkxACGRkZ8PLygplZ+eNItT5punXrFq9aISIiolLFx8fD29u73OVqfdJkb28PQHNAHBwcjBwNERERmYr09HT4+Phoc4Xy1PqkqfCUnIODA5MmIiIiKkZq+Q4LwYmIiIgkYNJEREREJAGTJiIiIiIJmDQRERERScCkiYiIiEgCJk1EREREEtT6lgNERESkH5Va4ERsCpIycuBmb4VgPyfIzXhXDSZNREREpLU9OgHztpxDQlqOdpqn0grhoYEYEORpxMiMj6fniIiICIAmYZq4JlInYQKAxLQcTFwTie3RCZK3pVILHI1JxuaomzgakwyVWugVS2XXrwocaSIiIiKo1ALztpxDSamJACADMG/LOfQN9Cj3VF1lR6tMdbSLI01ERESEE7EpxUaYihIAEtJy8MWeyzh1/R5upd5HgUpdbLnKjlYZcrTL0DjSREREREhKLz1hKurzPZfx+Z7LAAAzGeBqr4CH0hqeDlZwc1Bg4z83Sx2tAoB3Np9FI1c7yACohIBKLaBWa/6fV6DGmxujDTLaVRWYNBEREdVhWbkF2PDPTSyNuCJp+SbudsjKVeF2eg4K1AK303NxOz0XpyXuLykjF/0WHahQrIWjXSdiU9DZ37lC26gMJk1ERES1jJSWAdeTs/DD0ev49e94ZOQUANCM5JRWbi0D4KG0wraXu0NuJoNaLXA3KxeJaTlISMtBYloODly+gz3nk8qNz9rCDFYWcsjNZDCTaR5yMxnu5xcgJSu/3PWTMqSNihkakyYiIqJapKwi6v4tPHDoyl2sOnwNey8mQTzIkBo622DUow3haG2BV37VjBkVTZ4K063w0EBt8mVmJoObvRXc7K3Qylszv4m7vaSk6fvRwSWOFB2NScbwb4+Vu76bvVW5y1QFJk1ERES1RGER9cOjRYlpOZiwJhIeDlZILFK71KOJK0Y/2hA9mrjC7EEyZG0pL5Z0eUi8ci3YzwmeSiskpuWUOGJVOFoV7OdUJetXNSZNREREtUB5LQMAIDE9BzYWZvhfBx+MfLQh/F3tii07IMgTfQM9KtQRXG4mQ3hoICauiSx2qq+k0SpDr1/VZEII43eLqkLp6elQKpVIS0uDg4ODscMhIiKqElJPbX0/qgN6N3ev0lhqSp8mfXMEjjQREVGtUtn7phl7fX23kVugwvGrKVhx8KqkbWfkFugVS0VUZrTKEOtXFSZNRERkMiqbcBh7hMMQIyRStnE7PQf7LiRh74UkHLpyF9l5KknbBqqviFpuJqtUW4DKrl8VeHqOiIgMpjJJjyESlpKKoAv3vnREuzK3Y+z1y9uGADAwyANxKdk4eytdZ76bvQI9mrhi1/nbSMvOL7OI+tDs3kYfsTEVPD1HREQVZqykp6yrviauiSw34ZB637Q+zdyRp1IjK7cAmbkFyM5TITO3AJk5+Ziz/t8yi6jnrP8XeQVq2Fiaw8pCDqsHvYasLMxgbmaGdzafldTJ2kwGCAGohYBKCO3/81UC4X+Uvg0A2BadCACQyYDW3o7o3cwNvZu5IdDTAWZmMu1xNMUi6tqAI01ERASgapIeKaMsKrVA1w/3lnrfs8IRkl0zeiA7rwDpOflIzylA+v18ZORofo6+mYafTsRLfKY124Qe/hjfzQ8udooS55vqzW5Nkb45ApMmIiKq8qTHzUGBH8d1Qk6+Cpk5BcjILdCO9py7lY6fT1ZvwmMmA2wV5rC1NIetQo58lRpxKffLXc/f1RZ2VhbIyVMhp0CFnHwVcvI1I1cF6ur5Ov18WBsMalO/zGUMUYxeF/D0HBER6UVKf58Zv5zGn2cSoHpwGilfpUbBg3/vZeeVmjAVbuN2esXvN1aUmQywt7KAg7U57BUP/rWyQG6+Cgcu3y13/WUj2qFnUzcozM0gk/2XREi9XP+9sJaV6mS9dEQ7dGzopLltiEwGmRke3EYEOHktBaO+P1nuNqQUcptiEXVtwKSJiKiOOxGbUmbSAwD381XYciahUvuxsjCDk40lbBXmsLMyh51C88jOK8D+S+UnPCtHd0TPpq46yU6hwtGu8jpJ9w30KHHEpbo6WfcrZf8A0LWxq0l3wyYmTUREdVpk3D18sO28pGXD2nihnW89mJuZwVwug4VcBnMzM1y9k4VFuy+Vu/7KUu43JjXh6d6k5IQJMH4nakN0sjb1btgEmBlz5wsXLkTHjh1hb28PNzc3hIWF4eLFizrL9OzZEzKZTOcxYcIEI0VMRFTzqdQC26MTMXTpEQxZcgSnb6RJWu+Zjg0wsnNDPNupAZ7u4IPBbb0R2toLU3o3hqfSCqV9lcugKUQubYSkMFkoXPbhdQFpycKAIE8sHdEOHkrd01ceSitJl/sbe31DbYOqjlELwQcMGIBhw4ahY8eOKCgowBtvvIHo6GicO3cOtra2ADRJU5MmTTB//nztejY2NpKLulkITkR1SVkFwPfzVPj9VDy+OxSLa8nZAABLuRmebOOJiIt3kJyZV+H+PoWF5EDJIyRSexQZ4qovY3f0ru6O4FRxNfrquTt37sDNzQ379+9H9+7dAWiSpjZt2mDx4sUV2iaTJiKqK0pLOqaHNMGNe9lYc+w67mXnAwCU1hZ4/hFfjHzUF272ViaT9DBZoOpUo5OmK1euICAgAP/++y+CgoIAaJKms2fPQggBDw8PhIaG4u2334aNjU2J28jNzUVubq725/T0dPj4+DBpIqJarbSWAQ9r4GSDcV398L8O3rCx1C1rZdJDdU2NTZrUajWefPJJpKam4tChQ9rpy5cvh6+vL7y8vHDmzBnMnj0bwcHB2LBhQ4nbmTt3LubNm1dsOpMmIqqtyuuTBAAWchkWP90GA1p6lpnEMOmhuqTGJk0TJ07Etm3bcOjQIXh7e5e63N69e9GnTx9cuXIF/v7+xeZzpImI6hqpPYJ+euER9u4hKqJGNrecMmUKtm7digMHDpSZMAFAp06dAKDUpEmhUEChKLm1PBFRbSKEwPHYFHyy84Kk5ZMyyu7FRERlM2rSJITA1KlTsXHjRkRERMDPz6/cdaKiogAAnp687JKIDMvYp6ak7j85MxcbIm/ipxNxuHo3S/L2pXSSJqLSGTVpmjx5MtatW4fNmzfD3t4eiYmauzcrlUpYW1sjJiYG69atw2OPPQZnZ2ecOXMGM2bMQPfu3dGqVStjhk5EtYyxi6DL279aLXD0ajJ+OhGHHWcTka/SVFbYWsrxRGsv7D5/GynltAxgJ2miyjFqTVNpnV1XrlyJ0aNHIz4+HiNGjEB0dDSysrLg4+ODwYMH46233mKfJiIymMrcrLboNiqadJW1fwEgrE19/BN/D9cf9FYCgFbeSgwPboDQ1l6wU5gbpGUAUV1TYwvBqwqTJiIqS3lXnunT2LEiSZeUK98K2SnMEdbWC8M6NkBQfWWJcRiiOSRRXVEjC8GJiIylvJvVCgAJaTkYu+oE/FzsYG0ph62lHDaW5rBVyKEwl2PelrMlnhYrnPbGhn+hUgnkqwVy8lXILVAjt0CFnHw1rt7JlJQwvdS9EV4OCSjWW6moAUGe6BvowZYBRFWESRMR1Qr61BMJIXD1bhb2nk/CL3/HSdr+/kt3sf/S3QrFlpKdj8k//VOhdQsFejmUmTAVkpvJ2FaAqIowaSKiGk/Kaam8AjVOxKZgz4Xb2HshSac+SIphHX3gaq9AVq4K2XkFyMpTITu3AHEp2biclFnu+o1cbFG/njUU5mZQmMuhsND8m5KVix1nb5e7Pq98IzI+Jk1EVKOVVk+UmJaDCWsiMbKzL+5k5OLg5bvIzC3QzreUm6FTIyf0auqKpRFXcTczt8wrzxYMblniyJXUxpILBrcscQSosKYpMS2HV74RmTgmTURUY6nUAvO2nCuznuiHo9e101zsFOjdzBW9m7mja4AL7BSaj0AvR2tMXBOpvVqtUGGKFB4aWOqpvmA/J3gqrSqc9MjNZAgPDazw/omo+pgZOwAioooqr4i70JB29bF5checeKMPPnqqNQYEeWgTJkBTQL10RDt4KHVPgXkorcq9VL8w6QH+S3IKSU16KrN/Iqo+HGkiohpL6m1BejRxRWsfxzKXqcyVZ4VJz8N1VR56XO7PK9+ITB+TJiKqsaQWR0tdrjJXnhki6eGVb0SmjUkTEdVYwX5OcLazRHJmXonzq7uImkkPUe3GmiYiqrHuZedBrS75pgYsoiYiQ2PSREQ1Ur5KjUlrI3EvOx/u9gq4Oyh05rOImogMjafniKhGWvDneZyITYGdwhxrX+gEPxc7FlETUZVi0kRENc5vf8dj1ZFrAIDPnm6Nxm72AMB6IiKqUjw9R0Q1SlR8Kt7cFA0AeLlPAPq18DByRERUVzBpIqIaIykjBxN+PIW8AjVCmrvj5T4Bxg6JiOoQJk1EVCPkFagxeW0kEtNz4O9qi0XPtIYZa5aIqBoxaSKiGmH+1rM4ee0e7BXmWD6yA+ytLIwdEhHVMUyaiMjk/XwiDmuOxUEmAz4f3gb+rnbGDomI6iAmTURk0k5dv4d3Np8FALwS0gS9m7kbOSIiqquYNBGRybqdnoOJa04hT6XGgBYemNyrsbFDIqI6jEkTEZmk3AIVJq45haSMXAS42eGTp1n4TUTGxeaWRGQSVGqh09F7U9QNRMalwsHKHN+O7AA7BT+uiMi4+ClEREa3PToB87acQ0JaTrF5Xwxvi4YutkaIiohIF5MmIjKq7dEJmLgmEqKU+Tn5qmqNh4ioNKxpIiKjUakF5m05V2rCJAMwb8s5qNSlLUFEVH2YNBGR0ZyITSnxlFwhASAhLQcnYlOqLygiolIwaSIio0nKKD1hqshyRERViUkTERmNm72VQZcjIqpKTJqIyGiC/ZxQz6b0e8jJAHgqrRDs51R9QRERlYJJExEZTcydTNzPK/nquMI2luGhgZCzqSURmQAmTURkFPey8jB+9d/IKVCjsZsdPBx0T8F5KK2wdEQ7DAjyNFKERES62KeJiKpdvkqNSWsjEZeSDe961vj1pc5QWlvodAQP9nPiCBMRmRQmTURU7eZvOYejV5NhaynHilEd4GRrCQDo7O9s5MiIiErH03NEVK3WHLuOH49dh0wGLHqmDZp5OBg7JCIiSZg0EVG1ORqTjLl/nAUAzOzXFP1aeBg5IiIi6Zg0EVG1iEvOxsS1p1CgFniytRcm9fQ3dkhERHph0kREVS4jJx/jfziJ1Ox8tPZW4qOnWkEmY5E3EdUsTJqIqEqp1ALTf47CpduZcLNX4JvnO8DKQm7ssIiI9MakiYiq1Mc7LmLPhSRYmpth+cgO8FDylihEVDMxaSKiKrPxnxtYtj8GAPDxU63QxsfRuAEREVUCkyYiqhL/xN3D7PX/AgAm9vTHoDb1jRwREVHlsLklERmESi20Hb3Nzcww949o5BWoEdLcDbP6NTV2eERElWbUkaaFCxeiY8eOsLe3h5ubG8LCwnDx4kWdZXJycjB58mQ4OzvDzs4OQ4cOxe3bt40UMRGVZHt0Arp+uBfDvz2Gl3+OwuR1kbiTmQdPpRUWD2sLM94OhYhqAaMmTfv378fkyZNx7Ngx7Nq1C/n5+ejXrx+ysrK0y8yYMQNbtmzBb7/9hv379+PWrVsYMmSIEaMmoqK2Rydg4ppIJKTlFJuXkJaDQ5fvGCEqIiLDkwkhhLGDKHTnzh24ublh//796N69O9LS0uDq6op169bhqaeeAgBcuHABzZs3x9GjR/HII4+Uu8309HQolUqkpaXBwYG3ayAyJJVaoOuHe0tMmABABsBDaYVDs3vz5rtEZHL0zRFMqhA8LS0NAODk5AQAOHXqFPLz8xESEqJdplmzZmjQoAGOHj1a4jZyc3ORnp6u8yCiqnEiNqXUhAkABDSjTSdiU6ovKCKiKmIySZNarcb06dPRpUsXBAUFAQASExNhaWkJR0dHnWXd3d2RmJhY4nYWLlwIpVKpffj4+FR16ER11slr0pKhpIzSEysioprCZK6emzx5MqKjo3Ho0KFKbef111/HK6+8ov05PT2diRORAanUArvOJWLFwVj8ff2epHXc7NnQkohqPpNImqZMmYKtW7fiwIED8Pb21k738PBAXl4eUlNTdUabbt++DQ+Pku+OrlAooFAoqjpkolqlaLsAN3srBPs5FatBysotwG9/x+P7w9cQl5INADA3AyzkctzPV5W43cKapmA/p6p+CkREVc6oSZMQAlOnTsXGjRsREREBPz8/nfnt27eHhYUF9uzZg6FDhwIALl68iLi4OHTu3NkYIRPVOtujEzBvyzmd2iRPpRXCQwMxIMgTCWn3serINfx0PA7pOQUAAKW1BUY80gCjOjdEZNw9TFwTCUBTw1SoMOUKDw1kETgR1QpGvXpu0qRJWLduHTZv3oymTf9rfqdUKmFtbQ0AmDhxIv766y+sWrUKDg4OmDp1KgDgyJEjkvbBq+eISlfYLuDhDwEZNAlQcMN6iIxLRYFas4Sfiy3GdvXD0Hb1YWNprrOdshIvIiJTpG+OYNSkSSYr+a/PlStXYvTo0QA0zS1fffVV/PTTT8jNzUX//v2xZMmSUk/PPYxJE1HJymsXUFQnPyeM79YIfZq5ldqoUsopPiIiU1KjkqbqwKSJqGRHY5Ix/Ntj5S73flgQnn3EtxoiIiKqXvrmCJWqacrMzIRardaZxsSEqGaQ2gbA1sokrhchIjI6vfs0xcbG4vHHH4etrS2USiXq1auHevXqwdHREfXq1auKGImoCkhtA8B2AUREGnr/CTlixAgIIfD999/D3d291LokIjJtwX5O8FRaITEtp1ghOMB2AURED9M7aTp9+jROnTqlc7UbEdU8cjMZwkMDMeFBu4Ci2C6AiKg4vU/PdezYEfHx8VURCxFVs24BrrBTFP/byUNphaUj2rFdABFREXqPNK1YsQITJkzAzZs3ERQUBAsLC535rVq1MlhwRFS11h6/jszcAvg6WWPB4JZIzspjuwAiolLonTTduXMHMTExGDNmjHaaTCaDEAIymQwqVcm3UyAi03I/T4XlB64CAKb0DkDXAFcjR0REZNr0TprGjh2Ltm3b4qeffmIhOFENtu5EHO5m5sHHyRphbesbOxwiIpOnd9J0/fp1/PHHH2jcuHFVxENE1SAnX4Vv9scAACb1bAwLud7ljUREdY7en5S9e/fG6dOnqyIWIqomv/4dj6SMXHgprTC0nbexwyEiqhH0HmkKDQ3FjBkz8O+//6Jly5bFCsGffPJJgwVHRIaXW6DC0gjNKNPEXo1hac5RJiIiKfS+95yZWekfsKZYCM57zxHpWnv8Ot7cGA0PByvsf60nFOZyY4dERGQUVX7vuYfvNUdENUdegRpL9mlGmSb0aMSEiYhID3qNy+fn58Pc3BzR0dFVFQ8RVaGN/9zAzdT7cLFTYFhwA2OHQ0RUo+iVNFlYWKBBgwYmdwqOiMpXoFLj6yKjTFYWHGUiItKH3hWgb775Jt544w2kpKRURTxEVEU2R91CXEo2nG0t8WwnjjIREelL75qmr776CleuXIGXlxd8fX1ha2urMz8ysvjNP4nIuFRqga/2XQEAvNC9EWws9f7VJyKq8/T+5AwLC6uCMIioKm09cwuxd7NQz8YCzz/ia+xwiIhqJL2TpvDw8KqIg4iqiFot8OVezSjTuK5+sFVwlImIqCIq/Ol56tQpnD9/HgDQokULtG3b1mBBEZHhbItOxJWkTDhYmWPkow2NHQ4RUY2ld9KUlJSEYcOGISIiAo6OjgCA1NRU9OrVCz///DNcXXmndCJToRllugwAGNvVDw5WFuWsQUREpdH76rmpU6ciIyMDZ8+eRUpKClJSUhAdHY309HRMmzatKmIkograee42LiRmwF5hjjGP+hk7HCKiGk3vkabt27dj9+7daN68uXZaYGAgvv76a/Tr18+gwRFRxQkh8MUezSjT6C4NobThKBMRUWXoPdKkVquL3aQX0DS+5C1WiEzHnvNJOJeQDltLOcZ24SgTEVFl6Z009e7dGy+//DJu3bqlnXbz5k3MmDEDffr0MWhwRFQxQgh88aCW6fnODVHP1tLIERER1Xx6J01fffUV0tPT0bBhQ/j7+8Pf3x9+fn5IT0/Hl19+WRUxEpGe9l+6gzM30mBtIcf4bhxlIiIyBL1rmnx8fBAZGYndu3fjwoULAIDmzZsjJCTE4MERkf6EEPj8QS3TiEcawMVOYeSIiIhqhwr1aZLJZOjbty/69u1r6HiIqJIOX0nGP3GpUJib4YXujYwdDhFRrVGhpGnPnj3Ys2cPkpKSihV/f//99wYJjIj0pxllugQAGB7cAG72VkaOiIio9tA7aZo3bx7mz5+PDh06wNPTEzKZrCriIiI9qNQCJ2JTcPjKXZy8dg8WZjJM6OFv7LCIiGoVvZOmZcuWYdWqVXj++eerIh4i0tP26ATM23IOCWk52mkW5maIir+HAUpPI0ZGRFS76H31XF5eHh599NGqiIWI9LQ9OgET10TqJEwAkJ2nwsQ1kdgenWCkyIiIah+9k6bx48dj3bp1VRELEelBpRaYt+UcRBnLzNtyDip1WUsQEZFUep+ey8nJwfLly7F79260atWqWHfwzz77zGDBEVHpTsSmFBthKkoASEjLwYnYFHT2d66+wIiIaim9k6YzZ86gTZs2AIDo6GideSwKJ6o+SRmlJ0wVWY6KU6lUyM/PN3YYRFRBFhYWkMvlBtue3knTvn37DLZzIqo4qe0E2HZAf0IIJCYmIjU11dihEFElOTo6wsPDwyADOxXq00RExhfs5wQPpRUSSzlFJwPgobRCsJ9T9QZWCxQmTG5ubrCxseEoOlENJIRAdnY2kpKSAACenpW/mphJE1ENJTeTIayNF5btv1psXuFXfHhoIORm/MLXh0ql0iZMzs6sBSOqyaytrQEASUlJcHNzq/SpOr2vniMi06BWC0RcvAMAsFXofhB4KK2wdEQ7DAhinyZ9FdYw2djYGDkSIjKEwt9lQ9QncqSJqIbacuYWLiRmwN7KHPtn9sLF2xlIysiBm73mlBxHmCqHp+SIagdD/i4zaSKqgfJVaizapbnH3IvdGsHJzhKd7XgqiYioKul9eu7GjRvIzMwsNj0/Px8HDhwwSFBEVLb1p27gWnI2nG0tMaarn7HDIar1ZDIZNm3aVOYyycnJcHNzw7Vr16olJmNatWoVHB0djR0Ghg0bhk8//bTa9ic5aUpISEBwcDB8fX3h6OiIkSNH6iRPKSkp6NWrl147P3DgAEJDQ+Hl5VXiG3L06NGQyWQ6jwEDBui1D6LaJrdAhS/2XAYATOzpDzsFB4xNlUotcDQmGZujbuJoTDK7s9dyCxYswKBBg9CwYUNjh2KSGjZsiMWLFxt0m2+99RYWLFiAtLQ0g263NJI/befMmQMzMzMcP34cqampmDNnDnr16oWdO3eiXr16ADSX9+kjKysLrVu3xtixYzFkyJASlxkwYABWrlyp/VmhUOi1D6LaZt3xONxKy4GHgxVGPOJr7HCoFCXdSNlTaYXw0MA6XaAvhIBKpYK5efUm+yqVCjKZDGZmVXP9U3Z2Nr777jvs2LGjSrZPJQsKCoK/vz/WrFmDyZMnV/n+JL97du/ejS+++AIdOnRASEgIDh8+DE9PT/Tu3RspKSkA9C+2GjhwIN577z0MHjy41GUUCgU8PDy0j8IEjaguys4rwNf7rgAApvUJgJWF4TrdkuGUdiPlxLScKr2Rcs+ePTFt2jS89tprcHJygoeHB+bOnauzTFxcHAYNGgQ7Ozs4ODjg6aefxu3bt0vd5ty5c4uN+MtkMqxatQoAoFarsXDhQvj5+cHa2hqtW7fG77//rl0/IiICMpkM27ZtQ/v27aFQKHDo0CHk5uZi2rRpcHNzg5WVFbp27YqTJ09Kfq5//PEHAgICYGVlhV69emH16tWQyWTahqSFp4/++OMPBAYGQqFQIC4uDidPnkTfvn3h4uICpVKJHj16IDIyUmfbly9fRvfu3WFlZYXAwEDs2rWr3Hj++usvKBQKPPLII9ppKpUK48aN0x6bpk2b4vPPPy+27vfff48WLVpAoVDA09MTU6ZM0c5LTU3FSy+9BHd3d1hZWSEoKAhbt27VvjaFd+gotHjxYp2RrtGjRyMsLAzvv/8+3N3d4ejoiPnz56OgoACzZs2Ck5MTvL29dQYnCl+zos1do6KiIJPJSj31GBMTg0GDBsHd3R12dnbo2LEjdu/erZ3fs2dPXL9+HTNmzNC+hwodOnQI3bp1g7W1NXx8fDBt2jRkZWVp5y9ZskT7Wru7u+Opp57S2XdoaCh+/vnnEuMyNMlJU1pamk7ColAosGHDBjRs2BC9evXSNo8ytIiICLi5uaFp06aYOHEikpOTy1w+NzcX6enpOg+i2mLl4Wu4m5kHX2cb/K+Dt7HDqTOEEMjOK5D0yMjJR/gfZ0u8kXLhtLl/nENGTr6k7ek7gr969WrY2tri+PHj+OijjzB//nztl75arcagQYOQkpKC/fv3Y9euXbh69SqeeeaZUrc3c+ZMJCQkaB+ffPIJbGxs0KFDBwDAwoUL8cMPP2DZsmU4e/YsZsyYgREjRmD//v0625kzZw4++OADnD9/Hq1atcJrr72G9evXY/Xq1YiMjETjxo3Rv39/7R/hZYmNjcVTTz2FsLAwnD59Gi+99BLefPPNYstlZ2fjww8/xIoVK3D27Fm4ubkhIyMDo0aNwqFDh3Ds2DEEBATgscceQ0ZGhvYYDRkyBJaWljh+/DiWLVuG2bNnlxvTwYMH0b59e51parUa3t7e+O2333Du3Dm88847eOONN/Drr79ql1m6dCkmT56MF198Ef/++y/++OMPNG7cWLv+wIEDcfjwYaxZswbnzp3DBx98oHevob179+LWrVs4cOAAPvvsM4SHh+OJJ55AvXr1cPz4cUyYMAEvvfQSbty4odd2i8rMzMRjjz2GPXv24J9//sGAAQMQGhqKuLg4AMCGDRvg7e2N+fPna99LgCbZGjBgAIYOHYozZ87gl19+waFDh7SJ499//41p06Zh/vz5uHjxIrZv347u3bvr7Ds4OBgnTpxAbm5uheOXTEjUsmVL8fvvvxebnp+fL8LCwkSDBg2EmZmZ1M0VA0Bs3LhRZ9pPP/0kNm/eLM6cOSM2btwomjdvLjp27CgKCgpK3U54eLiA5rNJ55GWllbh2IhMQWp2nmgZvl34zt4qNkTGGzucWuv+/fvi3Llz4v79+9ppWbn5wnf2VqM8snLzJcfeo0cP0bVrV51pHTt2FLNnzxZCCLFz504hl8tFXFycdv7Zs2cFAHHixIlyt3/06FFhZWUlfvnlFyGEEDk5OcLGxkYcOXJEZ7lx48aJ4cOHCyGE2LdvnwAgNm3apJ2fmZkpLCwsxNq1a7XT8vLyhJeXl/joo4/KjWP27NkiKChIZ9qbb74pAIh79+4JIYRYuXKlACCioqLK3JZKpRL29vZiy5YtQgghduzYIczNzcXNmze1y2zbtq3E76iiBg0aJMaOHVtu7JMnTxZDhw7V/uzl5SXefPPNEpfdsWOHMDMzExcvXixxfnh4uGjdurXOtEWLFglfX1/tz6NGjRK+vr5CpVJppzVt2lR069ZN+3NBQYGwtbUVP/30kxDiv9es8FgKIcQ///wjAIjY2FghhOb4KpXKMp9rixYtxJdffqn92dfXVyxatEhnmXHjxokXX3xRZ9rBgweFmZmZuH//vli/fr1wcHAQ6enppe7n9OnTAoC4du1aifNL+p0ulJaWpleOIHmkaeDAgVi+fHmx6ebm5vjtt9/Qpk0bvf8iKs+wYcPw5JNPomXLlggLC8PWrVtx8uRJRERElLrO66+/jrS0NO0jPj7eoDERGcu3B64iPacATdzt8GTr+sYOh0xUq1atdH729PTUngk4f/48fHx84OPjo50fGBgIR0dHnD9/vsztxsXFISwsDDNnzsTTTz8NALhy5Qqys7PRt29f2NnZaR8//PADYmJidNYvHJkCNKML+fn56NKli3aahYUFgoODy40DAC5evIiOHTvqTAsODi62nKWlZbHjcfv2bbzwwgsICAiAUqmEg4MDMjMztSMihcfIy8tLu07nzp3Ljen+/fuwsip+n8evv/4a7du3h6urK+zs7LB8+XLtvpKSknDr1i306dOnxG1GRUXB29sbTZo0KXf/ZWnRooVOLZe7uztatmyp/Vkul8PZ2blSZ4wyMzMxc+ZMNG/eHI6OjrCzs8P58+e1z7U0p0+fxqpVq3TeP/3794darUZsbCz69u0LX19fNGrUCM8//zzWrl2L7OxsnW0Udv1+eHpVkFyJt2DBglIDMjc3x/r163Hz5k2DBVaSRo0awcXFBVeuXCn1TaZQKFgsTrXO3cxcfH84FgDwSt+mbFxZzawt5Dg3v7+kZU/EpmD0yvJrc1aN6SjpvoDWetatWVhY6Pwsk8mgVqv12sbDsrKy8OSTT6Jz586YP3++dnrhFdR//vkn6tfXTeQf/hy2tbWtVAwVYW1tXazWdtSoUUhOTsbnn38OX19fKBQKdO7cGXl5eZXal4uLC+7du6cz7eeff8bMmTPx6aefonPnzrC3t8fHH3+M48ePa+MrL/6ymJmZFRusKKnrdUnvibLeJ4UJVtFtl9dNe+bMmdi1axc++eQTNG7cGNbW1njqqafKPa6ZmZl46aWXMG3atGLzGjRoAEtLS0RGRiIiIgI7d+7EO++8g7lz5+LkyZPalgeFp3RdXV3L3JchSE6azM3N4eDgUOr8uLg4TJgwATt37jRIYCW5ceMGkpOTDXLTPaKaZGlEDLLzVGjlrUT/Fu7GDqfOkclksLGU9nHZLcAVng9upFzS2HvhjZS7BbhWe/LbvHlzxMfHIz4+XjvadO7cOaSmpiIwMLDEdYQQGDFiBNRqNX788UedJKRogXWPHj0kx+Hv7w9LS0scPnwYvr6aK0Dz8/Nx8uRJTJ8+vdz1mzZtir/++ktnmtQi8sOHD2PJkiV47LHHAADx8fG4e/eudn7hMUpISNB+1xw7dqzc7bZt2xZr1qwptq9HH30UkyZN0k4rOgJnb2+Phg0bYs+ePSW27GnVqhVu3LiBS5culTja5OrqisTERAghtK9LVFRUubGWpzD5SEhI0NYyl7fdw4cPY/To0doLuzIzM4sVjVtaWkKlUulMa9euHc6dO6et4yqJubk5QkJCEBISgvDwcDg6OmLv3r3aq+6jo6Ph7e0NFxcXfZ5mhRjs2suMjAzs2bNHr3UyMzMRFRWlfTFiY2MRFRWFuLg4ZGZmYtasWTh27BiuXbuGPXv2YNCgQdpiQaK6IiHtPn48dh0A8Gq/pry9h4mTm8kQHqpJQB5+pYx9I+WQkBC0bNkSzz33HCIjI3HixAmMHDkSPXr00Dl9VtTcuXOxe/dufPPNN8jMzERiYiISExNx//592NvbY+bMmZgxYwZWr16NmJgYREZG4ssvv8Tq1atLjcPW1hYTJ07ErFmzsH37dpw7dw4vvPACsrOzMW7cuHKfx0svvYQLFy5g9uzZuHTpEn799Vft1Xzl/X4EBATgxx9/xPnz53H8+HE899xzOiM6ISEhaNKkCUaNGoXTp0/j4MGDJRaZP6x///44e/aszmhTQEAA/v77b+zYsQOXLl3C22+/XSy5mzt3Lj799FN88cUXuHz5svb4AUCPHj3QvXt3DB06FLt27UJsbCy2bduG7du3A9BckXbnzh189NFHiImJwddff41t27aVG2t5GjduDB8fH8ydOxeXL1/Gn3/+WW4DyYCAAGzYsAFRUVE4ffo0nn322WIjnA0bNsSBAwdw8+ZNbaI6e/ZsHDlyBFOmTEFUVBQuX76MzZs3awvBt27dii+++AJRUVG4fv06fvjhB6jVajRt2lS73YMHD6Jfv36Vft6SSKp8kiAqKkrvQvDCYrOHH6NGjRLZ2dmiX79+wtXVVVhYWAhfX1/xwgsviMTERL32oW+RF5GpmbP+jPCdvVX8b9kRoVarjR1OrVdW0ag+tv17Szzy/m6dou5H3t8ttv17y0CRFtejRw/x8ssv60wbNGiQGDVqlPbn69eviyeffFLY2toKe3t78b///a/Mz9UePXqU+Dm9cuVKIYQQarVaLF68WDRt2lRYWFgIV1dX0b9/f7F//34hRMlFxUJojvPUqVOFi4uLUCgUokuXLpKK0Qtt3rxZNG7cWCgUCtGzZ0+xdOlSAUD7upVWqBwZGSk6dOggrKysREBAgPjtt9+KFShfvHhRdO3aVVhaWoomTZqI7du3l1sILoQQwcHBYtmyZdqfc3JyxOjRo4VSqRSOjo5i4sSJYs6cOcWKt5ctW6Y9fp6enmLq1KnaecnJyWLMmDHC2dlZWFlZiaCgILF161bt/KVLlwofHx9ha2srRo4cKRYsWFCsEHzQoEE6+yvpffLwMTh06JBo2bKlsLKyEt26dRO//fZbmYXgsbGxolevXsLa2lr4+PiIr776qth+jh49Klq1aiUUCoUomn6cOHFC9O3bV9jZ2QlbW1vRqlUrsWDBAiGEpii8R48eol69esLa2lq0atVKeyGCEJr3kVKpFEePHi3pJdEuY6hCcJkQhqnePn36NNq1a1ds6M3Y0tPToVQqkZaWVubpRSJTdD05C30+3Y8CtcBvEzqjY8Pya2CocnJychAbGws/P78SC3v1oVILnIhN4Y2Uq8GCBQuwbNkyo1788+eff2LWrFmIjo6usiaapGvp0qXYuHFjmaVBZf1O65sj8P4LRCZs8e7LKFAL9GjiyoSpBpKbydDZnzdSrgpLlixBx44d4ezsjMOHD+Pjjz/WaQppDI8//jguX76Mmzdv6lyhSFXHwsJCezqzOkhOmtq2bVvmueLquNSPqC65mJiBTVGaK1Jn9mtaztJEtceECROKFVUXGjFiBJYtW4bLly/jvffeQ0pKCho0aIBXX30Vr7/+ejVHWpyUQnYynPHjx1fr/iQnTWFhYVUYBhE97LNdFyEEMDDIAy29lcYOh6jazJ8/HzNnzixxXuEplEWLFmHRokXVGRaR9KQpPDy8KuMgoiLO3EjFjrO3IZMBr/StXGM7oprGzc0Nbm5uxg6DqBhWqhGZoE92XgIADG5THwHu9kaOhoiIACZNRCbn+NVkHLh0B+ZmMkwP4SgTEZGpYNJEZEKEEPhk50UAwDMdfdDA2cbIERERUSG2HCAyAYX9fPZfSsLJa/dgKZdhau8AY4dFRERF6D3S9MMPPyA3N7fY9Ly8PPzwww8GCYqoLtkenYCuH+7F8G+PYdn+qwAAC7kZouLvlbMmERFVJ72TpjFjxiAtLa3Y9IyMDIwZM8YgQRHVFdujEzBxTSQS0nJ0pmfnqTBxTSS2RycYKTKq7VatWqW9S3xZZDIZNm3aVOXx1DY9e/aU1LOpe/fuWLdunfbnxMRE9O3bF7a2ttrXpzpeg4fjfeSRR7B+/foq3WdNpHfSJIrcTbmoGzduQKlkLxkiqVRqgXlbzqGk+xgVTpu35RxUaoPc6YiqU2o8cCuq9Eeq8W71UeiZZ57BpUuXtD/PnTsXbdq0MV5AddAff/yB27dvY9iwYdppixYtQkJCAqKiorSvT0JCAgYOHFitsb311luYM2dOsZvu1nV6dwSXyWTo06cPzM3/W1WlUiE2NhYDBgyokiCJaqMTsSnFRpiKEgAS0nJwIjaFt+KoSVLjga/aAwXFyxi0zBXAlFOAo/FutWFtbQ1ra2uj7V8f+fn5sLCwqHX7/eKLLzBmzBid+9TFxMSgffv2CAj4r6bRw8OjymIozcCBAzF+/Hhs27YNjz/+eLXv31RJHmkKCwvDoEGDIIRA//79MWjQIO1j2LBh+Oabb0pte09ExSVllJ4wVWQ5MhHZyWUnTIBmfnayQXe7detWODo6am+aHhUVBZlMhjlz5miXGT9+PEaMGAFA9/TcqlWrMG/ePJw+fVr7x/GqVau06929exeDBw+GjY0NAgIC8Mcff5QaR0REhHYbRR+jR4/WLrN582a0a9cOVlZWaNSoEebNm4eCggLtfJlMhqVLl+LJJ5+Era0tFixYAEBzc1Z/f39YWlqiadOm+PHHHyUfnwsXLqBr166wsrJCYGAgdu/erXPa69q1a5DJZPjll1/Qo0cPWFlZYe3atUhOTsbw4cNRv3592NjYoGXLlvjpp590tp2VlYWRI0fCzs4Onp6e+PTTT8uN586dO9i7dy9CQ0O10xo2bIj169fjhx9+0DlmReP84YcfYGdnh8uXL2vXmzRpEpo1a6a9nVl0dDQGDhwIOzs7uLu74/nnn8fdu3f1ilcul+Oxxx7Dzz//LOn41hV6dwRv2LAhnnnmmUrf/ZuornOzl/Y7JHU5qkJCAPkS769ZcF/6cnlZ5S9nYQOUcd/PQt26dUNGRgb++ecfdOjQAfv374eLiwsiIiK0y+zfvx+zZ88utu4zzzyD6OhobN++Hbt37wYAnXKLefPm4aOPPsLHH3+ML7/8Es899xyuX78OJ6fiN5F+9NFHkZDwXy3e+fPn8dhjj6F79+4AgIMHD2LkyJH44osv0K1bN8TExODFF18EoHvniblz5+KDDz7A4sWLYW5ujo0bN+Lll1/G4sWLERISgq1bt2LMmDHw9vZGr169yjw2KpUKYWFhaNCgAY4fP46MjAy8+uqrJS47Z84cfPrpp2jbti2srKyQk5OD9u3bY/bs2XBwcMCff/6J559/Hv7+/ggODgYAzJo1C/v378fmzZvh5uaGN954A5GRkWWe7jx06BBsbGzQvHlz7bSTJ09i5MiRcHBwwOeff17iSODIkSOxdetWPPfcczhy5Ah27NiBFStW4OjRo7CxsUFqaip69+6N8ePHY9GiRbh//z5mz56Np59+Gnv37tUr3uDgYHzwwQdlHtu6Ru+WA6NGjQKguVouKSmp2PnOBg0aGCYyolou2M8JnkqrUk/RyQB4KK0Q7Ff8i4mqWX428L6XYbf5vcRyhjduAZa25S6mVCrRpk0bREREoEOHDoiIiMCMGTMwb948ZGZmIi0tDVeuXEGPHj2KrWttbQ07OzuYm5uXeCpo9OjRGD58OADg/fffxxdffIETJ06UWJJhaWmp3UZycjLGjx+PsWPHYuzYsQA0CdicOXO03yWNGjXCu+++i9dee00naXr22Wd1Li4aPnw4Ro8ejUmTJgEAXnnlFRw7dgyffPJJuUnTrl27EBMTg4iICG1sCxYsQN++fYstO336dAwZMkRnWtH74E2dOhU7duzAr7/+iuDgYGRmZuK7777DmjVr0KdPHwDA6tWr4e3tXWZM169fh7u7u86pOVdXVygUClhbW5d5Su6bb75Bq1atMG3aNGzYsAFz585F+/btAQBfffUV2rZti/fff1+7/Pfffw8fHx9cunQJXl5ekuP18vJCfHw81Gq1Tpx1md5H4fLly+jWrRusra3h6+sLPz8/+Pn5oWHDhvDz86uKGIlqJbmZDBN7+Jc4r3BcITw0EHKz8kcZiACgR48eiIiIgBACBw8exJAhQ9C8eXMcOnQI+/fvh5eXl06tjFStWrXS/t/W1hYODg5ISkoqc538/HwMHToUvr6++Pzzz7XTT58+jfnz58POzk77eOGFF5CQkKA9vQQAHTp00Nne+fPn0aVLF51pXbp0wfnz58uN/+LFi/Dx8dFJRApHiR728H5VKhXeffddtGzZEk5OTrCzs8OOHTsQFxcHQFODlJeXh06dOmnXcXJyQtOmTcuM6f79+xU+Y1OvXj1899132tOVRU/Bnj59Gvv27dM5vs2aNdPGqk+81tbWUKvVJbYZqqv0HmkaPXo0zM3NsXXrVnh6epZ4JR0RSXP1rub0jMLcDLkF/43aeiitEB4aiAFBnsYKjYqysNGM+EiReEbaKNLY7YBHq/KXs5DeFb5nz574/vvvcfr0aVhYWKBZs2bo2bMnIiIicO/evRJHmaR4uBhaJpOVe1XVxIkTER8fjxMnTuhcOJSZmYl58+YVG80BoJNE2NqWP7pWFR7e78cff4zPP/8cixcvRsuWLWFra4vp06cjLy+vUvtxcXHBvXsV78V24MAByOVyJCQkICsrC/b2mntUZmZmIjQ0FB9++GGxdTw9PXHlyhXJ+0hJSYGtrW2NuWCgOuidNEVFReHUqVPazJWIKibtfj5+/Vtz6fny59vD0lyOpIwcuNlrTslxhMmEyGSSTpEBAMwlfsGYW0vfpkSFdU2LFi3SJkg9e/bEBx98gHv37pVaxwNoTqsVFpFX1meffYZff/0VR44cgbOz7pWf7dq1w8WLF9G4cWO9ttm8eXMcPnxYe1oPAA4fPozAwMBy123atCni4+Nx+/ZtuLu7A9DUD0lx+PBhDBo0SFtAr1arcenSJe1+/f39YWFhgePHj2vLU+7du4dLly6VmaS2bdsWiYmJuHfvHurVqycplkJHjhzBhx9+iC1btmD27NmYMmUKVq9eDUBzfNevX4+GDRvqJKuF9Ik3Ojoabdu21Su22k7vpCkwMFCnCp+IKubXk/HIzlOhqbs9ujdx5agtVVq9evXQqlUrrF27Fl999RUATfPEp59+Gvn5+WV+iTds2BCxsbGIioqCt7c37O3toVAo9I5h9+7deO211/D111/DxcUFiYmJADSnepRKJd555x088cQTaNCgAZ566imYmZnh9OnTiI6OxnvvvVfqdmfNmoWnn34abdu2RUhICLZs2YINGzZoC9fL0rdvX/j7+2PUqFH46KOPkJGRgbfeegsAyv29CwgIwO+//44jR46gXr16+Oyzz3D79m1t0mRnZ4dx48Zh1qxZcHZ2hpubG958881ya4Datm0LFxcXHD58GE888US5z6FQRkYGnn/+eUybNg0DBw6Et7c3OnbsiNDQUDz11FOYPHkyvv32WwwfPhyvvfYanJyccOXKFfz8889YsWKFXvEePHgQ/fr1kxxbXaB3TdOHH36I1157DREREUhOTkZ6errOg4jKV6BSY9WRawCAsV0bMmGqTWycNX2YymKu0CxXBXr06AGVSoWePXsC0NSrBAYGwsPDo8w6m6FDh2LAgAHo1asXXF1di11WL9WhQ4egUqkwYcIEeHp6ah8vv/wyAKB///7YunUrdu7ciY4dO+KRRx7BokWL4OvrW+Z2w8LC8Pnnn+OTTz5BixYt8M0332DlypXa51kWuVyOTZs2ITMzEx07dsT48ePx5ptvAkC5dUVvvfUW2rVrh/79+6Nnz57w8PBAWFiYzjIff/wxunXrhtDQUISEhKBr167awuyyYhozZgzWrl1bbvxFvfzyy7C1tdUWerds2RLvv/8+XnrpJdy8eRNeXl44fPgwVCoV+vXrh5YtW2L69OlwdHTUJkZS4r158yaOHDnCO308RCaE0KvdcOFBf/hDvrBTuKGGdw0lPT0dSqUSaWlpcHBwMHY4RACAv/5NwKS1kXCytcSROb1hZSE3dkj0QE5ODmJjY+Hn51fx1iqp8WX3YbJxNmpjS9KcduvatSuuXLkCf/+SL8ioaomJiWjRogUiIyPLTRqr2+zZs3Hv3j0sX77c2KFUWlm/0/rmCHqfntu3b5++qxDRQ747FAsAGNGpAROm2sjRh0mRidm4cSPs7OwQEBCAK1eu4OWXX0aXLl2MljABmk7f3333HeLi4kwuaXJzc8Mrr7xi7DBMjt5JU0WvviAijaj4VJy6fg8WchlGPGJaH5RENdHatWvx0ksvlTjP19cXZ8+eRUZGBmbPno24uDi4uLggJCREUufuqvbwqT5TUdZFA3WZ3kkToCkO++abb3D16lX89ttvqF+/Pn788Uf4+fmha9euho6RqFb5/sEoU2hrL7g5sNs3UWU9+eSTOn2HiipslzBy5EiMHDmyOsOiWkjvpGn9+vV4/vnn8dxzzyEyMlLb9CotLQ3vv/8+/vrrL4MHSVRbJKTdx1//am4xMbYLm8ESGYK9vb22TxFRVdL76rn33nsPy5Ytw7fffqvT8KxLly6IjIw0aHBEtc0PR6+jQC3Qyc8JQfWV5a9AREQmQ++k6eLFi9obLxalVCqRmppqiJiIaqXsvAKsO6659cK4rhxlMnV6XlhMRCbKkL/LeidNHh4eJbZhP3ToEBo1amSQoIhqo/WRN5F2Px++zjbo09zd2OFQKQpH0IveB42Iaq7C3+WHbwdUEXrXNL3wwgt4+eWX8f3330Mmk+HWrVs4evQoZs6cibfffrvSARHVRmq1wMrDmgLw0Y825C1STJhcLoejo6P2hrQ2NjZsPkpUAwkhkJ2djaSkJDg6OkIur3x7F72Tpjlz5kCtVqNPnz7Izs5G9+7doVAoMHPmTEydOrXSARHVRvsv3cHVO1mwV5jjfx3Yv8fUeXh4AIA2cSKimsvR0VH7O11ZencEL5SXl4crV64gMzMTgYGBsLOzM0hAhsaO4GQKnv/uOA5evovxXf3w1hPl32CUTINKpUJ+fr6xwyCiCrKwsChzhKnKO4IXsrS0lHR3aaK67mJiBg5evgszGTDq0YbGDof0IJfLDTKkT0S1g6SkaciQIVi1ahUcHBwwZMiQMpfdsGGDQQIjqi0Km1kOCPKAj5ONkaMhIqKKkpQ0KZVKbSGkUsneMkRSJWfmYmPUTQBsZklEVNNJSppWrlxZ4v+JqGxrj8chr0CN1t5KtPetZ+xwiIioEvTu00RE0uQWqPDjsesAgLFd/XjZOhFRDSdppKlt27aSP/B5KxUija2nE3AnIxceDlZ4rKWnscMhIqJKkpQ0hYWFaf+fk5ODJUuWIDAwEJ07dwYAHDt2DGfPnsWkSZOqJEiimkYIge8eFICPfNQXFnIO6hIR1XSSkqbw8HDt/8ePH49p06bh3XffLbZMfHy8YaMjqqGOx6bgXEI6rCzM8GxwA2OHQ0REBqD3n7+//fYbRo4cWWz6iBEjsH79eoMERVTTFY4yDW3nDUcbSyNHQ0REhqB30mRtbY3Dhw8Xm3748GFYWVkZJCiimuza3SzsPn8bADCGbQaIiGoNvTuCT58+HRMnTkRkZCSCg4MBAMePH8f333/PG/YSAVh15BqEAHo2dUVjN9O8vRAREelP75GmOXPmYPXq1Th16hSmTZuGadOmITIyEitXrsScOXP02taBAwcQGhoKLy8vyGQybNq0SWe+EALvvPMOPD09YW1tjZCQEFy+fFnfkImqTXpOPn77W1PbN64rR5mIiGqTCl3S8/TTT+Pw4cNISUlBSkoKDh8+jKefflrv7WRlZaF169b4+uuvS5z/0Ucf4YsvvsCyZctw/Phx2Nraon///sjJyalI2ERV7teT8cjKU6GJux26NnYxdjhERGRAFb5hryEMHDgQAwcOLHGeEAKLFy/GW2+9hUGDBgEAfvjhB7i7u2PTpk0YNmxYdYZKVCqVWuBEbAoS0+5jWUQMAM0tU9jMkoiodtE7aVKpVFi0aBF+/fVXxMXFIS8vT2d+SkqKQQKLjY1FYmIiQkJCtNOUSiU6deqEo0ePlpo05ebmIjc3V/tzenq6QeIhKsn26ATM23IOCWn/jX7KZICNpdyIURERUVXQ+/TcvHnz8Nlnn+GZZ55BWloaXnnlFQwZMgRmZmaYO3euwQJLTEwEALi7u+tMd3d3184rycKFC6FUKrUPHx8fg8VEVNT26ARMXBOpkzABgBDAyz9HYXt0gpEiIyKiqqB30rR27Vp8++23ePXVV2Fubo7hw4djxYoVeOedd3Ds2LGqiFEvr7/+OtLS0rQPNtykqqBSC8zbcg6ijGXmbTkHlbqsJYiIqCbRO2lKTExEy5YtAQB2dnZIS0sDADzxxBP4888/DRaYh4cHAOD27ds602/fvq2dVxKFQgEHBwedB1FZVGqBozHJ2Bx1E0djkiUlOidiU4qNMBUlACSk5eBErGFOVxMRkfHpXdPk7e2NhIQENGjQAP7+/ti5cyfatWuHkydPQqFQGCwwPz8/eHh4YM+ePWjTpg0ATX3S8ePHMXHiRIPth+q2kmqSPJVWCA8NxIAg3Zvs5uSrEHn9Ho5dTcafZ6SdekvK4JWeRES1hd5J0+DBg7Fnzx506tQJU6dOxYgRI/Ddd98hLi4OM2bM0GtbmZmZuHLlivbn2NhYREVFwcnJCQ0aNMD06dPx3nvvISAgAH5+fnj77bfh5eWlcwNhoooqrEl6eFwpMS0HE9dE4vNhbeBqb4WjV5Nx7GoyouJSkadS67UPN3t2ySciqi1kQohKFV0cO3YMR44cQUBAAEJDQ/VaNyIiAr169So2fdSoUVi1ahWEEAgPD8fy5cuRmpqKrl27YsmSJWjSpInkfaSnp0OpVCItLY2n6khLpRbo+uHeMk+xlcTdQYHOjZzRqZETPtt5GXczc0usa5IB8FBa4dDs3pCbsfUAEZEp0jdH0Ctpys/Px0svvYS3334bfn41o9sxkyYqydGYZAz/tvwLFxytLdCjqSseaeSMzo2c4etso+2/VDhSBUAncSpMkZaOaFfsFB8REZkOfXMEvQrBLSwssH79+goHR2QqpNYazXuyBT4f1hbDgxugoYutTsPKAUGeWDqiHTyUuqfgPJRWTJiIiGohvWuawsLCsGnTJr3rl4hMidRaIzeHspcbEOSJvoEeOBGbgqSMHLjZWyHYz4mn5IiIaiG9k6aAgADMnz8fhw8fRvv27WFra6szf9q0aQYLjqiqBPs5wVNphcS0nDJrkoL9nMrdltxMhs7+zgaPkYiITIveheBl1TLJZDJcvXq10kEZEmuaqDTboxMw4UFNUlGsSSIiqhv0zRH0HmmKjY2tUGBEpmZAkCcea+mJv/7V7bnkUUqfJiIiqtv0TpqIapNrd7MAABN6+KO5pz1rkoiIqFR6J02vvPJKidNlMhmsrKzQuHFjDBo0CE5O5deCEBnTzdT7OJeQDjMZ8EI3PzjbGa6jPRER1T56J03//PMPIiMjoVKp0LRpUwDApUuXIJfL0axZMyxZsgSvvvoqDh06hMDAQIMHTGQoe85r7mvY3rceEyYiIiqX3jfsHTRoEEJCQnDr1i2cOnUKp06dwo0bN9C3b18MHz4cN2/eRPfu3dmSgEzernOapCmkubuRIyEioppA76vn6tevj127dhUbRTp79iz69euHmzdvIjIyEv369cPdu3cNGmxF8Oo5KklGTj7avbsL+SqBPa/2gL+rnbFDIiKialalHcEBIC0tDUlJScWm37lzB+np6QAAR0dH5OXl6btpompz4NJd5KsEGrnYMmEiIiJJKnR6buzYsdi4cSNu3LiBGzduYOPGjRg3bhzCwsIAACdOnNDrprpE1W33g3qmkECemiMiImn0LgT/5ptvMGPGDAwbNgwFBQWajZibY9SoUVi0aBEAoFmzZlixYoVhIyUykAKVGnsvaEZLWc9ERERS6V3TVCgzM1Pb/btRo0awszPNUxysaaKHHbuajGHLj6GejQVOvhkCc7neA65ERFQLVHlH8EJ2dnZo1apVRVcnMprdD66a69XMjQkTERFJxm8MqlOEENj1oJ6pL0/NERGRHpg0UZ0ScycT15OzYSk3Q7cmrsYOh4iIahAmTVSn7DqnKQDv7O8MOwVvvUhERNIxaaI6ha0GiIioopg0UZ1xNzMXkXH3AAAhzd2MHA0REdU0TJqozth7IQlCAEH1HeCptDZ2OEREVMMwaaI6Yzdv0EtERJXApInqhJx8FQ5e1txAmkkTERFVBJMmqhOOxNzF/XwVPJVWaOHFzvBERKQ/Jk1UJxS2Gghp7g6ZTGbkaIiIqCZi0kS1nlotsIetBoiIqJKYNFGt9+/NNCRl5MLWUo5HGjkZOxwiIqqhmDRRrVfY0LJHU1cozOVGjoaIiGoqJk1U6+1iqwEiIjIAJk1Uq8WnZONCYgbMZECvpuwCTkREFcekiWq1wgLwDg2dUM/W0sjREBFRTcakiWq13ec1rQb68tQcERFVEpMmqrXSc/Jx7GoyALYaICKiymPSRLXW/ot3UKAW8He1hZ+LrbHDISKiGo5JE9Vau9nQkoiIDIhJE9VK+So19l1gPRMRERkOkyaqlU5eS0F6TgGcbC3RtkE9Y4dDRES1AJMmqpV2P7hBb+9mbpCb8Qa9RERUeUyaqNYRQmDX+UQA7AJORESGw6SJap3LSZmIT7kPS3MzdAtwMXY4RERUSzBpolqn8F5zXfydYaswN3I0RERUWzBpolqHrQaIiKgqmHTSNHfuXMhkMp1Hs2bNjB0WmbCkjBxExacCAPo0Y9JERESGY/LnLlq0aIHdu3drfzY3N/mQyYj2XUiCEEArbyU8lFbGDoeIiGoRk89AzM3N4eHhYewwqIbY9aDVAK+aIyIiQzPp03MAcPnyZXh5eaFRo0Z47rnnEBcXV+byubm5SE9P13lQ3XA/T4VDV+4AYNJERESGZ9JJU6dOnbBq1Sps374dS5cuRWxsLLp164aMjIxS11m4cCGUSqX24ePjU40RkzEdvnIXOflq1He0RnNPe2OHQ0REtYxMCCGMHYRUqamp8PX1xWeffYZx48aVuExubi5yc3O1P6enp8PHxwdpaWlwcHCorlBJIpVa4ERsCpIycuBmb4VgP6cKd/Ces/4Mfj4Zj1GdfTFvUJCBIyUiotomPT0dSqVSco5g8jVNRTk6OqJJkya4cuVKqcsoFAooFIpqjIoqant0AuZtOYeEtBztNE+lFcJDAzEgyFOvbanVArvPP6hnYqsBIiKqAiZ9eu5hmZmZiImJgaenfl+oZHq2Rydg4ppInYQJABLTcjBxTSS2RydI3pZKLbD2+HXczcyFlYUZOvg6GTpcIiIi006aZs6cif379+PatWs4cuQIBg8eDLlcjuHDhxs7NKoElVpg3pZzKOm8cOG0eVvOQaUu/8zx9ugEdP1wL97efBYAkJOvRu9PI/RKuoiIiKQw6aTpxo0bGD58OJo2bYqnn34azs7OOHbsGFxdXY0dGlXCidiUYiNMRQkACWk5+CPqJjJy8lFa2Z0hR6uIiIjKY9I1TT///LOxQ6AqkJRResJU1IxfTwMALOVmcLazhJOtJZztFHC2tYSjjQV+//tGqaNVMmhGq/oGelS4sJyIiKgok06aqHZytrWUtJzC3Ay5BWrkqdRISMspc3TqYYWjVSdiU9DZ37mCkRIREf2HSRNVqzM3UrFw2/kyl5EB8FBa4dDs3sgrUCM5KxfJmXlIycrD3cxcpGTl4XhsCvZeSCp3f1JHtYiIiMrDpImqRUZOPj7deQk/HL0GtQCsLcxwP18NGaBziq3wRFp4aCDkZjJYW8rhbWkD73o2Ottr5e0oKWlys+f954iIyDCYNFGVEkJgx9lEzP3jHBLTNaM+YW288ObjgTh1PaVYnyYPiX2agv2c4Km0QmJaTol1TYWjVcF+bD9ARESGwaSJqsyNe9mY+8dZbdNJX2cbvBcWhG4BmqsfBwR5om+gR4U6gsvNZAgPDcTENZHljlYREREZQo26jUpF6NsinfRT0m1QhBBYefgaPtt1CffzVbCQyzChhz8m92oMKwu5QfdvyK7iRERUt9Tq26iQaSkpYXG2tYS1hRw3Uu8DAIIbOmHB4CAEuFfNDXQrM1pFRESkDyZNVCGFjSUfHqZMzsoDANhYyjE3tAWeau8NsypOYORmMrYVICKiKmfSHcHJNJV1G5RC9lbmGFoNCRMREVF1YdJEeivvNigAcDs9FydiU6opIiIioqrHpIn0Fn0zTdJybCxJRES1CWuaSLI7Gbn4fM8lrDseJ2l5NpakOiM1HshOLn2+jTPg6FN98RBRlWDSROXKyi3AtwevYvmBq8jOUwH4775wJWFjyTqmricMqfHAV+2BgtzSlzFXAFNO1e7jQFQHMGmiUhWo1Pjl73gs2nUZdzM1XwitfRzxxsBmuJedh4lrIgGwsWSdxoRBkzCW9fwBzfzsZNM+BnU9+SWSgElTHVdSc0ozGbDr3G18sP0Crt7JAqDp5v1a/2Z4rKUHZDJNMrR0RLsK3waFaonakjDUdUx+iSRh0lSHldac0tHGAjEPkiUnW0tM690Yz3byhaW57nUDbCxJVEsw+SWShElTHVVWc8rkrDxYyGV4qbs/XurRCPZWFqVuh40lqU678TcQ8YGxo9Dg6TXj42tQ6zFpqoOkNKd0srHEjL5NOGpEhiFKvmgAQM37oinIBc5uBI5/A9yKNHY0GpU9vaZWVV1sdQVPcdYJTJrqGCEEfjoRV35zygxNc0qOIlGpVAXAv79JW/bHwUDgk0CzUKBRD82XB2A6XzRSEjczOfD398CpVUDWHc10uSXQqDdweXv5+8iR1t+sQqSeXos/DiREASmxwL1rwL3Cf69XXWzVyZgJOE9x1glMmmqwkoq4SxoZUqsFTsXdw7Z/E7HjbCJuPriZbnnYnJJKdfMUsGU6kHhG2vI5qUDkD5qHpR0Q0Bdo9gRg72n8LxopiZvMDIAMEA9GZBzqAx3GAu1HA2k3pCVNGycAIzcDrk0MEXXFrB9nvH2Xp7IJj6kk4FR5Jjz6zKSphiqpiNuzyJVrBSo1jl1NwfazCdhx9jbuZPz3QVJWj6Wi2JySislJA/a8C5xcAUBoEqC8zPLXe+xT4M554MKfQEaC5vTW2Y2AmQl8BEkZISg8vejbBQh+UZPwyR/Enn9f82Vc3jYybgHfhQBP/wA06lnpsCtE4QC4BAD1GgL1/AAnP83/8+4D654qf/3oDYB70H/P3VAMkfAYYqTHhL+s6wwTT35N4BOL9FVaEXdiWg4mrIlE50bOOJ+YjtTsfO08eytzhDR3x4AgD3Rt7IKQz/YjMS2nxLomNqesQarrQ14ITZKz/XUgM1EzrdUzwCOTge/7lv8B16Q/EDweGPixpg7owlbg/FYg+XLlY6suQ78DWpaQWDj6aD7Ay3odZGbAXzM1p8fWDAUe/1QzSmUIajVwZY+0ZUdtAbzaFJ9+K0ra+kc+B67sAgZ+CPh1lxph+Uzh1FZFvqyzUzSjrjdOAjF7qyauusYU3gtlYNJUw5RVxF047ehVzYd3PRsL9Av0wICWHuji76LTMiA8NBAT10RCBjanrLEM8ReZlKRLXaD5wr+yWzPNyR944rP/RkvKSxiKJm5mZoB3B80jZC5wbjPw68jS160OGQnSlnNuXPo8R5/yP8BH/gH8MRX491dgy8vA3ctA3/maWqmKitkH7A4HEk5XfBuA5jUqb7TMzBywsAWSzgGrQ4HAQUC/9wDHBpXbtz4S/9XUYGUkapL3jCKP9BvSthHxAVC/PVDPVzPK5ugL2LlJ/7L++3vN/m6cAJKv6P8cDn4GhIQDzv76rysFR8uqFJOmGuZEbEq5RdwA8NbjzTH60YYwl5d8T+YBQZ5sTlnTP1wq+xeZlKTLTA7IzAFVrqboudurQJfpgEWRU7dSEobSOPpKW27n20DwC0BAP919AxV7HdNvAWc3AdHrgZt/6xVyhVlYAUOWa06P7VsAHP0KSLkKDPkWUNjpt61bUcDuucDVfZqfzW2AguyKxyZltMzGGbC0Bfa9D/z9nSbhvbRD837o8rJm3ar+ffpjSuXWB4BL2zSPoixsAFs3aesf+kz3Z+fGgHdHwN4DOLSo/PXPbwYubAGaPwl0maZJ4Ayluv6QMuXPxSrGpKkGSUrPwU8npN0s19VeUWrCVKhON6c08fPm1UJK0qVWAVBpTsU8vghwKWO0pSpdO6B5KJRAYCjQ8n9Aw26a5Efq62hhDZzbpKnLuX4E/42xPjzeWoVkMqDHa4BTI2DTJODiX8DKAZpjKy+9H5r2iyrlKrD3PU2yBwBmFppkskl/4IdBlYtNavL7+CeaU4vbZgPXDwH7PwAiVwNZdwF1funrPfz7pFYDdy8B8ceAixIK6QHAzkMzOmTvrrmIwO7Bv/bumnq730aXv40OY4GCPM2IVep1TSF/fjaQek1aDPU7AP69NYmSdwfA5kEZw60oaUmTzyOa53xuk+bRsJsm8WzcR/P+qEzSUh1/SNX2z8VyMGkyIilXv+UWqLDnfBJ+P3UD+y/dgUot7cNdahF3nW1OaeLnzQ1qw0uaZMfB68GjvuaLRuol8L3eBLrP0nygG0urYcC1g0D6TeCfNZqHnYcmmZPyOv4+BrgZ+d/Vb4DmyytoqGakYM3gqo3/YS2f0oyy/Txcc8rpu5Cyl5crNLH++6vmdClkQKungV5vaJKI1PjyT6+ZKzRfuIbgEQSM3qqpc9v5trRTYwW5wOVdmt+p+OOa01v6tmF49peSa7IA6XVZ7UbpbqMgD0iL19SFbZtV/vqPf1pyDFJOcZorgKErgNx04MiXmpYd1w5qHm4tgLYjNCOIKiMlLcb+XExPMJ1msaVg0mQk5V39Fn0zDb+fuoHNUTdxr0hBd7sGjoi5k4X0+/ks4iZp7l7QPCoqoF/VJUxSv2h6v6VJ9uKOahKHs5s0NS3//iptPzdOav71bKNJPloM/u9DX+qXraH5dATG7wFWP1n+KIcqFzi9TvP/xiFAn3DAs9V/86WeXjPkF51MBgQNAZoMALbP0Yw2lefPGbo/m1trRmvq+WoSYWMwt9TUF+VmVG47+r4Gg5dp3tfHlmp6fyWdBXa8Xv5+CnL/6xOWGqf7uH1WWqxr/6f5A8rWBbBxefCvs+ZKUGPIvw8c+UozUpefZZwYJGLSZATlXf1W39Fap5eSu4MCQ9p546n23vB3tdOuzyLuOiw3AzjwsbRl+y3QJB7ptzSPjAf/pt0ACozci0vfL5qGXTSPgR8DMXuAE8ulXbXUcZzmSr+Sim+lJm6GGqUpqp4vELYEWPVY+cu6NAEe+0TTHLQklaktqwxLG80pLylJk60b0LAr4NMJaNDpQfsCC03iWtmkyZivYyF9XwOlN9B/AdB9pqbA/PCXQM698tdbEaI7aqqvrCTNw9iE0Jxq3hX+32ilW6DmYgMTxaSpmkm5+u1m6n1YmMnQr4UHnurgjW6NXXTqk1jEXUmqfE0BqxRV2cW5oq4f0TRKTJXYxblh11IuM/8HWN7TkJFVTEW+7M0tgaYDNacZpSRNbUeWfrWSMUZpirK0lbbc4OVA/bZVE0N1ee63yp3aKivhMfbrWBnW9TQXWfh207TwKI9QaerZlN6aqxcdG/x3UcW+98pfP2wpYO0EZN/V1KJl3wWykjU1c/HHyl//3980Mdcr5UIOKXVZGYmakbXCUWAHb6DvPMAnGPiqg3GT3zIwaapmUq9+WzKiHfoGepQ6v04XcVdUXpamI/WRr6RfnrxmqKbouON4wNuAV7lUREGupgj4yJcAhKamp7BnUoXwvaJlrFEafRizpqyqGSrhqczraAojVeaW0pZ79negce/i7SpuRUlLmtwCS+/XtbyUkcyijn6leXi00jR6bf6EZpuFhexSOuwXNoy1sAW6zQA6T9FcrAGYdPLLpKka5avU2HVO2pdcdl75Q681uojbEJe1St1GVrLmNM6Jb4D7D4a+repJGwZX52tqSU6vA7zaAh1f0NRxWFhX76W5CWeAjS/9N2zddgTwyBTg2x4m+xcZkV6MnbjWpJEqO9fK9feqLI82wO0zmtsoJZ4BIt7XdJhv/gTg0lR6h/02I4A+b2vaNRRl7PdCGZg0VYO45Gz8dDIOv/19A3czy3kzPVDltzAxxH2eKrq+oXqJlLcNuaVmlOjsRs0lxYDmF7vLNMAtSNow+KClQGyEZhu3/gE2TwJ2vgkEhgFRawFVXsWfgxSqAk0X5n0LNQmcrSsQ+gXQ7EENTGU+5E3hL2siU2LCX9aSVNfv9JOfa04NXtym6e4fs09z8+cjX0rfxpBvNVeA1jBMmiriQcKgEgJnb6YjJTsPTjaWaFHfAXKZDLBxRp5dfew6dxs/n4zDwct3tas621qiXsFtKPJSS9y0DIC5vUvVXv1W2aSlsusb4rJWKdtQ5WkSGwDwbK3phRI4SPMXmtRLtP26AW2f1RRT//OjplgzLR44tbLsfUt5DkDZyWfaDWD/R0Dig27PzZ4AQj/XXOlSqFKNJWvQX9alYeJnGvg6mIbK/k7r8zraugDtntc8cjM0LRsubAUu/CXtCjgXI964uhKYNOmrSMIgB9CqhEUKZJYYLPscZ7OVADSnebsFuOLZYB/08cqD2VfPQq4ofYRCVWAJeXqXquvYWtmkxdi9PPTh1Q7o847mlh9Fa0L0/XCxcwW6vaLpfHxph6YrcGEBY0VJST4BzU1xH/sEaD3M8HUtNf0vayZ+pqE2vA6mwFAF8dX9h5TCHmgRpnnEnyy/71gNxqRJXxISBnORB+SkwNXeDc908MEzHX3g42SjmXkrClCXcUoHgFydVzs6tmbdARKjgfspmhtbFv5756K09X8dpbmyyEz+4HYecs39r8zkQJ7EW0Y8saj0ZngV+XAxk2tOjTl4SSuY3Dpd0xvIqZHm6i2nRppThBZW0pJPAHjqe03HZyoZEz/TUNNfB1NgCu+Fyr6OZXW2rwWYNOlJJQSklN+9MbAZgrv0hkU5tzLRW0VHeYQA8jI1iYzUBmj7P9BcliqEpnBPqAGIsn+hi1pbwh3h9SH1tgam7NY/mocOmaZRo52rtG3YuRs8LDIxTDioEN8LJo1Jk57O3kwv8ZTcw4LOfgyLWysfnE6Raf6VmQH3U6Xt6J81wLVDmlEjcwVgbqUpbE6/JW39iA80SU7Wnf8e+jYyvLit/GXKZAbYOmsSLxunB//W0xQ3n/m5/NWfWKQZlREqzT3Q1KoH/y8Akq8Ce+dXMr5q0PMNTQF3ylUgOUbzb266puWB1LYHRERkEpg06Sklu+xTa4WUt48Btyuxo5PfVmJlFL+LdyELW835Zyn9fTq+ACjra5I9mRk0yZ+Z5v5AR78of/0X92ou03/YrShpSZNXu7LvM1UTkqYm/XWfg3gwUpdyVXPFScT7RguNiMjgakONXhmYNOnJyUZa87H4oEnw8W0MQGi+KAHNv6nx0hKOpgMBS3vN6JAqT/NvQZ6mJkjKfcQ6jtdcMWbr+uDhovnX0lZ6A7O2I0pvgCblObB5YnEy2YPXwkUzcsikiYhqE1Ooy6pCTJr01KK+g6TlvDo/XfItD6QmHD3mVK5ja9vnSx+lMTZD/CVi7L9mjL1/IiJTVYvrspg06Uku8ZJvqcsZRWW/8Cu7viH+EjH2XzPG3j8REVU7Jk36MnbCYQiV/cI3VNJT2YTC2H/NVHb/pvBeICIiyWpE0vT111/j448/RmJiIlq3bo0vv/wSwcHBxgmmSMJQVkfwKks4DPVFW9kvfGMnLLUBR6uIiGoUmRCFVcqm6ZdffsHIkSOxbNkydOrUCYsXL8Zvv/2Gixcvws3Nrdz109PToVQqkZaWBgcHafVIJq86bxRLRERUS+mbI5h80tSpUyd07NgRX331FQBArVbDx8cHU6dOxZw5c8pdv1YmTURERFRp+uYIBm5XbVh5eXk4deoUQkL+u4+NmZkZQkJCcPTo0RLXyc3NRXp6us6DiIiIqLJMOmm6e/cuVCoV3N11byPh7u6OxMSSmzMuXLgQSqVS+/Dx4WkqIiIiqjyTTpoq4vXXX0daWpr2ER8fb+yQiIiIqBYw6avnXFxcIJfLcfu27v1Ibt++DQ8PjxLXUSgUUCgU1REeERER1SEmPdJkaWmJ9u3bY8+ePdpparUae/bsQefOnY0YGREREdU1Jj3SBACvvPIKRo0ahQ4dOiA4OBiLFy9GVlYWxowZY+zQiIiIqA4x+aTpmWeewZ07d/DOO+8gMTERbdq0wfbt24sVh5emsKMCr6IjIiKiogpzA6ndl0y+T1Nl3bhxg1fQERERUani4+Ph7e1d7nK1PmlSq9W4desW7O3tIauCm+imp6fDx8cH8fHxbJ5ZQTyGhsHjaBg8jpXHY2gYPI6VV94xFEIgIyMDXl5eMDMrv8zb5E/PVZaZmZmk7LGyHBwc+KauJB5Dw+BxNAwex8rjMTQMHsfKK+sYKpVKydsx6avniIiIiEwFkyYiIiIiCZg0VZJCoUB4eDgbalYCj6Fh8DgaBo9j5fEYGgaPY+UZ+hjW+kJwIiIiIkPgSBMRERGRBEyaiIiIiCRg0kREREQkAZOmSvj666/RsGFDWFlZoVOnTjhx4oSxQ6pR5s6dC5lMpvNo1qyZscMyeQcOHEBoaCi8vLwgk8mwadMmnflCCLzzzjvw9PSEtbU1QkJCcPnyZeMEa6LKO4ajR48u9t4cMGCAcYI1UQsXLkTHjh1hb28PNzc3hIWF4eLFizrL5OTkYPLkyXB2doadnR2GDh2K27dvGyli0yTlOPbs2bPY+3HChAlGitg0LV26FK1atdL2Y+rcuTO2bdumnW+o9yKTpgr65Zdf8MorryA8PByRkZFo3bo1+vfvj6SkJGOHVqO0aNECCQkJ2sehQ4eMHZLJy8rKQuvWrfH111+XOP+jjz7CF198gWXLluH48eOwtbVF//79kZOTU82Rmq7yjiEADBgwQOe9+dNPP1VjhKZv//79mDx5Mo4dO4Zdu3YhPz8f/fr1Q1ZWlnaZGTNmYMuWLfjtt9+wf/9+3Lp1C0OGDDFi1KZHynEEgBdeeEHn/fjRRx8ZKWLT5O3tjQ8++ACnTp3C33//jd69e2PQoEE4e/YsAAO+FwVVSHBwsJg8ebL2Z5VKJby8vMTChQuNGFXNEh4eLlq3bm3sMGo0AGLjxo3an9VqtfDw8BAff/yxdlpqaqpQKBTip59+MkKEpu/hYyiEEKNGjRKDBg0ySjw1VVJSkgAg9u/fL4TQvO8sLCzEb7/9pl3m/PnzAoA4evSoscI0eQ8fRyGE6NGjh3j55ZeNF1QNVa9ePbFixQqDvhc50lQBeXl5OHXqFEJCQrTTzMzMEBISgqNHjxoxsprn8uXL8PLyQqNGjfDcc88hLi7O2CHVaLGxsUhMTNR5byqVSnTq1InvTT1FRETAzc0NTZs2xcSJE5GcnGzskExaWloaAMDJyQkAcOrUKeTn5+u8F5s1a4YGDRrwvViGh49jobVr18LFxQVBQUF4/fXXkZ2dbYzwagSVSoWff/4ZWVlZ6Ny5s0Hfi7X+3nNV4e7du1CpVHB3d9eZ7u7ujgsXLhgpqpqnU6dOWLVqFZo2bYqEhATMmzcP3bp1Q3R0NOzt7Y0dXo2UmJgIACW+NwvnUfkGDBiAIUOGwM/PDzExMXjjjTcwcOBAHD16FHK53NjhmRy1Wo3p06ejS5cuCAoKAqB5L1paWsLR0VFnWb4XS1fScQSAZ599Fr6+vvDy8sKZM2cwe/ZsXLx4ERs2bDBitKbn33//RefOnZGTkwM7Ozts3LgRgYGBiIqKMth7kUkTGc3AgQO1/2/VqhU6deoEX19f/Prrrxg3bpwRI6O6btiwYdr/t2zZEq1atYK/vz8iIiLQp08fI0ZmmiZPnozo6GjWJFZSacfxxRdf1P6/ZcuW8PT0RJ8+fRATEwN/f//qDtNkNW3aFFFRUUhLS8Pvv/+OUaNGYf/+/QbdB0/PVYCLiwvkcnmxyvvbt2/Dw8PDSFHVfI6OjmjSpAmuXLli7FBqrML3H9+bhtWoUSO4uLjwvVmCKVOmYOvWrdi3bx+8vb210z08PJCXl4fU1FSd5fleLFlpx7EknTp1AgC+Hx9iaWmJxo0bo3379li4cCFat26Nzz//3KDvRSZNFWBpaYn27dtjz5492mlqtRp79uxB586djRhZzZaZmYmYmBh4enoaO5Qay8/PDx4eHjrvzfT0dBw/fpzvzUq4ceMGkpOT+d4sQgiBKVOmYOPGjdi7dy/8/Px05rdv3x4WFhY678WLFy8iLi6O78UiyjuOJYmKigIAvh/LoVarkZuba9D3Ik/PVdArr7yCUaNGoUOHDggODsbixYuRlZWFMWPGGDu0GmPmzJkIDQ2Fr68vbt26hfDwcMjlcgwfPtzYoZm0zMxMnb8wY2NjERUVBScnJzRo0ADTp0/He++9h4CAAPj5+eHtt9+Gl5cXwsLCjBe0iSnrGDo5OWHevHkYOnQoPDw8EBMTg9deew2NGzdG//79jRi1aZk8eTLWrVuHzZs3w97eXlsbolQqYW1tDaVSiXHjxuGVV16Bk5MTHBwcMHXqVHTu3BmPPPKIkaM3HeUdx5iYGKxbtw6PPfYYnJ2dcebMGcyYMQPdu3dHq1atjBy96Xj99dcxcOBANGjQABkZGVi3bh0iIiKwY8cOw74XDXuBX93y5ZdfigYNGghLS0sRHBwsjh07ZuyQapRnnnlGeHp6CktLS1G/fn3xzDPPiCtXrhg7LJO3b98+AaDYY9SoUUIITduBt99+W7i7uwuFQiH69OkjLl68aNygTUxZxzA7O1v069dPuLq6CgsLC+Hr6yteeOEFkZiYaOywTUpJxw+AWLlypXaZ+/fvi0mTJol69eoJGxsbMXjwYJGQkGC8oE1QeccxLi5OdO/eXTg5OQmFQiEaN24sZs2aJdLS0owbuIkZO3as8PX1FZaWlsLV1VX06dNH7Ny5UzvfUO9FmRBCVDbDIyIiIqrtWNNEREREJAGTJiIiIiIJmDQRERERScCkiYiIiEgCJk1EREREEjBpIiIiIpKASRMRERGRBEyaiIiIiCRg0kREREQkAZMmIqp1Ro8ezXvtEZHBMWkiIiIikoBJExHVWL///jtatmwJa2trODs7IyQkBLNmzcLq1auxefNmyGQyyGQyREREAADi4+Px9NNPw9HREU5OThg0aBCuXbum3V7hCNW8efPg6uoKBwcHTJgwAXl5ecZ5gkRkUsyNHQARUUUkJCRg+PDh+OijjzB48GBkZGTg4MGDGDlyJOLi4pCeno6VK1cCAJycnJCfn4/+/fujc+fOOHjwIMzNzfHee+9hwIABOHPmDCwtLQEAe/bsgZWVFSIiInDt2jWMGTMGzs7OWLBggTGfLhGZACZNRFQjJSQkoKCgAEOGDIGvry8AoGXLlgAAa2tr5ObmwsPDQ7v8mjVroFarsWLFCshkMgDAypUr4ejoiIiICPTr1w8AYGlpie+//x42NjZo0aIF5s+fj1mzZuHdd9+FmRkH54nqMn4CEFGN1Lp1a/Tp0wctW7bE//73P3z77be4d+9eqcufPn0aV65cgb29Pezs7GBnZwcnJyfk5OQgJiZGZ7s2Njbanzt37ozMzEzEx8dX6fMhItPHkSYiqpHkcjl27dqFI0eOYOfOnfjyyy/x5ptv4vjx4yUun5mZifbt22Pt2rXF5rm6ulZ1uERUCzBpIqIaSyaToUuXLujSpQveeecd+Pr6YuPGjbC0tIRKpdJZtl27dvjll1/g5uYGBweHUrd5+vRp3L9/H9bW1gCAY8eOwc7ODj4+PlX6XIjI9PH0HBHVSMePH8f777+Pv//+G3FxcdiwYQPu3LmD5s2bo2HDhjhz5gwuXryIu3fvIj8/H8899xxcXFwwaNAgHDx4ELGxsYiIiMC0adNw48YN7Xbz8vIwbtw4nDt3Dn/99RfCw8MxZcoU1jMREUeaiKhmcnBwwIEDB7B48WKkp6fD19cXn376KQYOHIgOHTogIiICHTp0QGZmJvbt24eePXviwIEDmD17NoYMGYKMjAzUr18fffr00Rl56tOnDwICAtC9e3fk5uZi+PDhmDt3rvGeKBGZDJkQQhg7CCIiUzB69GikpqZi06ZNxg6FiEwQx5uJiIiIJGDSRERERCQBT88RERERScCRJiIiIiIJmDQRERERScCkiYiIiEgCJk1EREREEjBpIiIiIpKASRMRERGRBEyaiIiIiCRg0kREREQkAZMmIiIiIgn+D35Jg2DF51vpAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: the two grad-norm traces side by side\n",
"plt.figure(figsize=(6, 4))\n",
"plt.plot(broken_norms, \"o-\", label=\"no zero_grad (accumulates)\")\n",
"plt.plot(fixed_norms, \"s-\", label=\"with zero_grad (fixed)\")\n",
"plt.xlabel(\"step\"); plt.ylabel(\"gradient L2 norm\"); plt.title(\"zero_grad: the bug and the fix\")\n",
"plt.legend(); plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "12afc52d",
"metadata": {},
"source": [
"> **Key takeaways.** The four lines are `zero_grad -> forward -> backward ->\n",
"step`, in that order, every step. Verify the loss at init is near `log(n_classes)`. Run\n",
"a `randn` smoke test before training. When a loop misbehaves, look at the gradient norm:\n",
"a climbing norm with stable code almost always means a missing `zero_grad`. Eval goes\n",
"under `torch.no_grad()` with `model.eval()`.\n"
]
},
{
"cell_type": "markdown",
"id": "f0f23993",
"metadata": {},
"source": [
"## Safety lens\n",
"\n",
"PyTorch's defaults are not safety-neutral. Two of them bite in ways that matter for\n",
"anyone shipping or evaluating models.\n",
"\n",
"**Loading untrusted checkpoints is remote code execution.** `torch.load` of a pickle can\n",
"run arbitrary Python on load; malicious weights have circulated on model hubs. Modern\n",
"torch defaults `weights_only=True`, which restricts deserialization to safe types, and\n",
"the ecosystem has moved to the `safetensors` format, which has no code-execution\n",
"surface. Treat any checkpoint you did not produce as untrusted. The cell below\n",
"demonstrates a clean state-dict round-trip with `weights_only=True`.\n",
"\n",
"**Non-determinism hides bugs in eval.** Stochastic training is fine; a \"this model is\n",
"2% safer\" claim that lives inside run-to-run variance is noise. For numbers that will\n",
"appear in a report, fix the seeds (we do, in Setup), and on GPU add\n",
"`torch.use_deterministic_algorithms(True)` plus `CUBLAS_WORKSPACE_CONFIG=:4096:8` (the\n",
"price is roughly 10-30% slower training, the benefit is that the number means\n",
"something).\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "05b88693",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:59.310580Z",
"iopub.status.busy": "2026-06-10T19:01:59.310421Z",
"iopub.status.idle": "2026-06-10T19:01:59.315543Z",
"shell.execute_reply": "2026-06-10T19:01:59.315230Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] state_dict round-trip with weights_only=True reproduced every tensor\n"
]
}
],
"source": [
"# A safe state_dict round-trip: save weights only, reload with weights_only=True.\n",
"import tempfile\n",
"m = nn.Linear(4, 2)\n",
"before = {k: v.clone() for k, v in m.state_dict().items()}\n",
"with tempfile.NamedTemporaryFile(suffix=\".pt\", delete=False) as fh:\n",
" ckpt_path = fh.name\n",
"torch.save(m.state_dict(), ckpt_path) # save the state_dict, not the module object\n",
"m2 = nn.Linear(4, 2) # fresh random init\n",
"m2.load_state_dict(torch.load(ckpt_path, weights_only=True)) # restrict deserialization\n",
"for k in before:\n",
" assert torch.allclose(before[k], m2.state_dict()[k]), f\"{k} did not round-trip\"\n",
"os.remove(ckpt_path)\n",
"print(\"[ ok ] state_dict round-trip with weights_only=True reproduced every tensor\")"
]
},
{
"cell_type": "markdown",
"id": "62ceca2f",
"metadata": {},
"source": [
"> **Caveat:** `weights_only=True` rejects the small minority of legitimate\n",
"checkpoints that pickled custom objects, with a clear error. That is the right default;\n",
"opt out only for files you produced and trust.\n"
]
},
{
"cell_type": "markdown",
"id": "d03fb70b",
"metadata": {},
"source": [
"## Going further\n",
"- PyTorch, *Learn the Basics* tutorial — the official tensors -> autograd -> build-model -> optimization arc this notebook compresses.\n",
"- Stanford CS336 lecture 2 — PyTorch plus resource accounting; the \"predict the cost of a forward pass\" lecture.\n",
"- Karpathy, *A Recipe for Training Neural Networks* — read once now, re-read every six months; the source of \"verify the loss at init\" and \"fix the seed\".\n",
"- Karpathy, nanoGPT `train.py` — the same four-line loop at transformer scale; best read line by line.\n",
"- Howard & Gugger, *fastbook* ch 17 (*Foundations*) — builds `nn.Module` from scratch, a second view of Part 4.\n",
"\n",
"## What this enables\n",
"- **Ch 11 — Training Deep Networks**: the framework is no longer mysterious, so Ch 11 fixes what breaks at scale: vanishing gradients, dead ReLUs, the four diagnostic plots, hyperparameter search. All on this exact loop.\n",
"- **Ch 12 — CNNs**: same training loop, swap the model. We reached ~0.82 on FashionMNIST with a small MLP on a subset; a CNN on the full data clears ~0.90, and Ch 12 is why.\n",
"- **Ch 15 — Transformers from scratch**: nanoGPT is written in exactly this PyTorch style. You will read it and find no surprises.\n"
]
},
{
"cell_type": "markdown",
"id": "ecc5db52",
"metadata": {},
"source": [
"## Test yourself\n",
"\n",
"Three parts: concept self-checks with folded answers, two auto-checked problems, and a\n",
"capstone with a rubric and a reference solution. Try before you peek.\n"
]
},
{
"cell_type": "markdown",
"id": "46b0bc0d",
"metadata": {},
"source": [
"### Part A — Concepts\n",
"\n",
"Every answer is in this notebook; if unsure, re-run that section.\n",
"\n",
"1. What three fields distinguish a PyTorch tensor from a plain NumPy array? Answer`dtype`, `device`, and `requires_grad` (plus the `grad` field that autograd fills in). Part 1.\n",
"2. `torch.from_numpy(arr)` vs `torch.tensor(arr)` — which copies? Answer`torch.tensor` copies; `torch.from_numpy` shares memory with the array. Mutating the array changes the shared tensor, as the Part 1 assert showed.\n",
"3. You call `.view()` on the output of `.permute()` and get a `RuntimeError`. Why, and what are the two fixes? Answer`permute` makes the tensor non-contiguous, and `view` requires contiguity. Fix with `.reshape()` (copies if needed) or `.contiguous().view()` (explicit copy first). Part 2.\n",
"4. After two consecutive `.backward()` calls on `x**2` at `x=2`, what is `x.grad`? Answer8. Each backward adds `2x = 4`, and gradients accumulate until you call `x.grad.zero_()`. Part 3.\n",
"5. Why mark a learnable tensor as `nn.Parameter` and a causal mask as a buffer? Answer`nn.Parameter` attributes are discovered by `model.parameters()` so the optimizer updates them; a buffer is tracked (moves with `.to(device)`, serializes) but not updated. A mask should move and serialize but never be learned, so it is a buffer. Part 4.\n",
"6. The loss at the start of training a 10-class classifier should be near what number, and why does that matter? AnswerNear `log(10) = 2.303`: an untrained model spreads probability uniformly. If step 0 is far off, you have a bug before training even begins. Part 6 / Exercise 10.6.\n",
"7. Write the four-line training step in order, from memory. Answer`opt.zero_grad()` (clear), `logits = model(x)` (forward), `loss.backward()` (backward), `opt.step()` (update). Part 6.\n",
"8. Your loss diverges, the code looks right, and the gradient norm climbs every step. Most likely cause? AnswerA missing `opt.zero_grad()`. Gradients accumulate, so each step's effective gradient is the sum of all prior steps. The deliberate-failure diagnostic plot in Part 6 is exactly this.\n",
"9. Why wrap an eval loop in `torch.no_grad()`? AnswerTo skip building the autograd graph, saving memory and time; you are not going to call `.backward()` during evaluation. Pair it with `model.eval()` to fix dropout/batchnorm behavior. Part 6.\n",
"10. Why is `torch.load(..., weights_only=True)` the safe default for checkpoints you did not produce? AnswerA raw pickle load can execute arbitrary code; `weights_only=True` restricts deserialization to safe tensor types, closing the remote-code-execution surface. Safety lens.\n"
]
},
{
"cell_type": "markdown",
"id": "9c676e30",
"metadata": {},
"source": [
"### Part B1 — Top-1 accuracy from logits\n",
"`Difficulty 2/5 · ~8 min`\n",
"\n",
"Implement `accuracy(logits, targets)`: `logits` is `(batch, n_classes)`, `targets` is\n",
"`(batch,)` of integer labels. Return the fraction correct as a Python float, where\n",
"\"correct\" means the argmax class equals the target. The check uses a hand-built case\n",
"with a known answer plus a random shape/type smoke test.\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "3e9752df",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:59.316607Z",
"iopub.status.busy": "2026-06-10T19:01:59.316526Z",
"iopub.status.idle": "2026-06-10T19:01:59.320230Z",
"shell.execute_reply": "2026-06-10T19:01:59.319949Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] B1 accuracy: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def accuracy(logits, targets):\n",
" \"\"\"Fraction of rows whose argmax class matches the target. Return a Python float.\"\"\"\n",
" # TODO 1: predicted = argmax of logits along dim=1\n",
" # TODO 2: return the mean of (predicted == targets) as a float\n",
" result = None\n",
" attempted(result)\n",
" return result\n",
"\n",
"def _check_acc():\n",
" # Hand case: 3 rows, argmaxes are class 2, 0, 1; targets 2, 0, 0 -> 2 of 3 correct.\n",
" logits = torch.tensor([[0.1, 0.2, 0.9],\n",
" [0.8, 0.1, 0.1],\n",
" [0.1, 0.7, 0.2]])\n",
" targets = torch.tensor([2, 0, 0])\n",
" check_close(accuracy(logits, targets), 2 / 3, atol=1e-6, msg=\"2 of 3 rows correct\")\n",
" # smoke: all-correct case should be exactly 1.0\n",
" big = torch.randn(50, 10)\n",
" check_close(accuracy(big, big.argmax(1)), 1.0, atol=1e-6, msg=\"self-argmax targets -> 100%\")\n",
"\n",
"check(\"B1 accuracy\", _check_acc)"
]
},
{
"cell_type": "markdown",
"id": "bc099dcb",
"metadata": {},
"source": [
"Hint 1`logits.argmax(dim=1)` gives the predicted class per row; compare to `targets`, take the float mean.\n",
"Solution\n",
"\n",
"```python\n",
"def accuracy(logits, targets):\n",
" pred = logits.argmax(dim=1)\n",
" return (pred == targets).float().mean().item()\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "6fe89e05",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:59.321119Z",
"iopub.status.busy": "2026-06-10T19:01:59.321045Z",
"iopub.status.idle": "2026-06-10T19:01:59.332275Z",
"shell.execute_reply": "2026-06-10T19:01:59.331830Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] B1 accuracy\n",
"accuracy of the trained model on the test set: 0.816\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines accuracy; the check below re-verifies it.\n",
"def accuracy(logits, targets):\n",
" pred = logits.argmax(dim=1)\n",
" return (pred == targets).float().mean().item()\n",
"\n",
"check(\"B1 accuracy\", _check_acc, required=True)\n",
"print(\"accuracy of the trained model on the test set:\", f\"{accuracy(model(Xte), yte):.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "daf87c1c",
"metadata": {},
"source": [
"### Part B2 — Count a model's parameters\n",
"`Difficulty 1/5 · ~5 min`\n",
"\n",
"Implement `count_parameters(module)` returning the total number of learnable scalars in\n",
"a `nn.Module`. The check verifies it against a model whose count you can do by hand:\n",
"`nn.Linear(10, 20)` then `nn.Linear(20, 5)` is `(20*10+20) + (5*20+5) = 220 + 105 = 325`.\n"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "b2c6c2ae",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:59.333241Z",
"iopub.status.busy": "2026-06-10T19:01:59.333140Z",
"iopub.status.idle": "2026-06-10T19:01:59.336399Z",
"shell.execute_reply": "2026-06-10T19:01:59.336089Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] B2 count_parameters: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def count_parameters(module):\n",
" \"\"\"Total number of learnable scalars across all parameters of the module.\"\"\"\n",
" # TODO 1: sum p.numel() over module.parameters()\n",
" total = None\n",
" attempted(total)\n",
" return total\n",
"\n",
"def _check_count():\n",
" net = nn.Sequential(nn.Linear(10, 20), nn.ReLU(), nn.Linear(20, 5))\n",
" got = count_parameters(net)\n",
" assert got == 325, f\"got {got}; (20*10+20)+(5*20+5) = 220+105 = 325\"\n",
" # single linear sanity: nn.Linear(3,3) is 3*3+3 = 12\n",
" assert count_parameters(nn.Linear(3, 3)) == 12, \"nn.Linear(3,3) is 3*3+3 = 12\"\n",
"\n",
"check(\"B2 count_parameters\", _check_count)"
]
},
{
"cell_type": "markdown",
"id": "f07acdb9",
"metadata": {},
"source": [
"Hint 1`sum(p.numel() for p in module.parameters())`. `numel` is the element count of one tensor.\n",
"Solution\n",
"\n",
"```python\n",
"def count_parameters(module):\n",
" return sum(p.numel() for p in module.parameters())\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "8f8bf938",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:59.337134Z",
"iopub.status.busy": "2026-06-10T19:01:59.337060Z",
"iopub.status.idle": "2026-06-10T19:01:59.339577Z",
"shell.execute_reply": "2026-06-10T19:01:59.339107Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] B2 count_parameters\n",
"trained MLP parameter count: 101770\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines count_parameters; the check below re-verifies it.\n",
"def count_parameters(module):\n",
" return sum(p.numel() for p in module.parameters())\n",
"\n",
"check(\"B2 count_parameters\", _check_count, required=True)\n",
"print(\"trained MLP parameter count:\", count_parameters(model))"
]
},
{
"cell_type": "markdown",
"id": "746e9ffd",
"metadata": {},
"source": [
"### Part C — Capstone: the full-scale FashionMNIST classifier\n",
"\n",
"Redo Part 6 at full scale and push past 85%. This is the chapter's payoff: a real\n",
"classifier built entirely from pieces you now understand.\n",
"\n",
"**Deliverables**\n",
"1. Train on the *full* 60,000-image training set (not the `TRAIN_N` subset), evaluating on the full 10,000-image test set.\n",
"2. Use a slightly bigger model (`784 -> 256 -> 128 -> 10` with ReLUs) and `torch.optim.SGD` with momentum, or `Adam` at `3e-4`.\n",
"3. Report final test accuracy and a confusion-style observation: which two classes does the model confuse most?\n",
"\n",
"**Self-assessment** (pass / partial / fail):\n",
"(a) the training loop is the four flagged lines in the right order;\n",
"(b) you ran the `randn` smoke test and checked loss-at-init near `log(10)`;\n",
"(c) eval is under `torch.no_grad()` with `model.eval()`;\n",
"(d) final test accuracy clears 0.85;\n",
"(e) you name a real confusion (it will be shirt/T-shirt/pullover/coat, the visually similar tops), not a vague \"some errors\".\n",
"\n",
"My solution (reference, ~2-4 min on CPU)\n",
"\n",
"```python\n",
"# Full data. ToTensor + flatten, reusing the loaders' pattern.\n",
"def full_loader(split_train, n=None):\n",
" ds = datasets.FashionMNIST(root=\"data\", train=split_train, download=True, transform=to_tensor)\n",
" n = n or len(ds)\n",
" X = torch.stack([ds[i][0].view(-1) for i in range(n)])\n",
" y = torch.tensor([ds[i][1] for i in range(n)])\n",
" return FlatImageDataset(X, y)\n",
"\n",
"g = torch.Generator().manual_seed(SEED)\n",
"big_train = DataLoader(full_loader(True), batch_size=128, shuffle=True, generator=g, drop_last=True)\n",
"big_test = DataLoader(full_loader(False), batch_size=256, shuffle=False)\n",
"\n",
"torch.manual_seed(SEED)\n",
"big = nn.Sequential(nn.Linear(784, 256), nn.ReLU(),\n",
" nn.Linear(256, 128), nn.ReLU(),\n",
" nn.Linear(128, 10))\n",
"opt = torch.optim.SGD(big.parameters(), lr=0.1, momentum=0.9)\n",
"for epoch in range(5):\n",
" for xb, yb in big_train:\n",
" opt.zero_grad()\n",
" loss = F.cross_entropy(big(xb), yb)\n",
" loss.backward()\n",
" opt.step()\n",
" vl, va = evaluate(big, big_test)\n",
" print(f\"epoch {epoch}: val_loss {vl:.4f} val_acc {va:.4f}\")\n",
"\n",
"# Confusion: tally predicted-vs-true over the test set, find the worst off-diagonal pair.\n",
"import collections\n",
"conf = collections.Counter()\n",
"with torch.no_grad():\n",
" for xb, yb in big_test:\n",
" pred = big(xb).argmax(1)\n",
" for t, p in zip(yb.tolist(), pred.tolist()):\n",
" if t != p:\n",
" conf[(CLASSES[t], CLASSES[p])] += 1\n",
"print(\"most confused pair:\", conf.most_common(1))\n",
"```\n",
"The reference lands around 0.88 test accuracy after 5 epochs, and the most-confused pair\n",
"is almost always among Shirt / T-shirt / Pullover / Coat: the visually similar tops are\n",
"genuinely hard, which is the whole reason FashionMNIST exists as a harder drop-in for\n",
"MNIST.\n"
]
},
{
"cell_type": "markdown",
"id": "c156ca33",
"metadata": {},
"source": [
"## Reflection\n",
"\n",
"Write ~150 words, in the cell below, on the dumbest bug you hit in this notebook and how\n",
"you found it. The forgotten-`zero_grad` failure was staged for you; what was the first\n",
"real bug *you* hit (a shape error in `MyLinear`, a `gather` index dtype, a non-contiguous\n",
"`view`), and what was the diagnostic that pointed at it? Nobody grades this. Writing it\n",
"is how the bug becomes a reflex instead of a surprise the next time.\n"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "59ac239b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-10T19:01:59.340449Z",
"iopub.status.busy": "2026-06-10T19:01:59.340375Z",
"iopub.status.idle": "2026-06-10T19:01:59.342364Z",
"shell.execute_reply": "2026-06-10T19:01:59.341980Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(replace this string with ~150 words: the bug, the symptom, the diagnostic, the fix)\n"
]
}
],
"source": [
"reflection = \"\"\"\n",
"(replace this string with ~150 words: the bug, the symptom, the diagnostic, the fix)\n",
"\"\"\"\n",
"print(reflection.strip())"
]
},
{
"cell_type": "markdown",
"id": "c3a39980",
"metadata": {},
"source": [
"---\n",
"*Built top-to-bottom. If every check above printed `[ ok ]`, you've reproduced the chapter. Runtime stamp written by CI.*\n",
"\n",
"*Total running time: CI-written · Verified on: numpy 2.x, torch 2.x, Python 3.12 · 2026-06-11*\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "obvix-nb",
"language": "python",
"name": "obvix-nb"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"obvix": {
"title": "Ch 10 — PyTorch Foundations"
}
},
"nbformat": 4,
"nbformat_minor": 5
}