{
"cells": [
{
"cell_type": "markdown",
"id": "8c512636",
"metadata": {},
"source": [
"# Ch 19 — Reinforcement Learning and RLHF (notebook)\n",
"\n",
"`[← 18 generative-models]` · **this notebook** · `[20 agents →]`\n",
"\n",
"Runs top-to-bottom in ~6 min on free Colab CPU. Last verified 2026-06-11.\n",
"\n",
"**What you'll build**\n",
"- A 4x4 gridworld solver: value iteration and policy iteration, checked against the *exact* Bellman fixed point (residual zero, and the two methods agree to machine precision).\n",
"- Tabular Q-learning that learns the same optimal policy by interacting, plus the early-break bug that silently returns an all-zero value function, broken then fixed.\n",
"- The REINFORCE policy gradient by hand in NumPy, checked against finite differences, then PPO that balances a self-contained CartPole to 200+ steps in under a minute.\n",
"- A toy RLHF stack on a 6-token language model: a Bradley-Terry reward model, PPO fine-tuning, and the moment the policy collapses to `.......` when you drop the KL penalty.\n",
"- DPO and GRPO as one-function variants on the same preference data, and a reward-hacking gridworld where optimizing the proxy reward gets a true return of exactly zero.\n",
"\n",
"**How this notebook works.** Code cells with a `# TODO` are yours to fill in. Run the cell to grade yourself: `[ ok ]` passed, `[FAIL]` shows what went wrong, `[ -- ]` means not attempted yet. Every exercise has a hint ladder (open only as many as you need) and a folded solution below it. The notebook runs top-to-bottom even if you fill in nothing, because the solution cells redefine the functions the later cells need. See Ch 00 for the full protocol.\n"
]
},
{
"cell_type": "markdown",
"id": "16a9160e",
"metadata": {},
"source": [
"## Before you start\n",
"\n",
"1. The Bellman optimality equation $V^*(s) = \\max_a \\sum_{s'} P(s'\\mid s,a)[R + \\gamma V^*(s')]$ is a fixed-point equation. What does \"solving\" it mean? AnswerFinding the unique $V^*$ that the right-hand side maps to itself. Value iteration just applies the right-hand side repeatedly until $V$ stops changing; the map is a $\\gamma$-contraction, so it converges to that one fixed point regardless of where you start.\n",
"2. You optimize a policy against a *reward model* that approximates human preference. Where does this go wrong even if every line of your RL code is correct? AnswerThe optimizer finds the gap between the reward model and the true objective. The reward model has failure modes (length bias, sycophancy, confident-tone bias); a strong optimizer exploits them. This is reward hacking, and it is the central lesson of the chapter. The code being correct makes it worse, not better.\n",
"3. Predict before you run: in a corridor where a \"shiny\" cell pays a small bonus every time you step on it, and a distant goal pays a large reward once, what does a policy that maximizes the bonus-augmented (proxy) reward do? AnswerIt can get stuck oscillating around the shiny cell and never reach the goal. Part 6 builds exactly this and measures a true return of 0.0 for the proxy-optimal policy. The proxy and the true objective diverge.\n"
]
},
{
"cell_type": "markdown",
"id": "3dd0942a",
"metadata": {},
"source": [
"## Setup\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "017cbc1f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:40.275209Z",
"iopub.status.busy": "2026-06-11T20:07:40.275125Z",
"iopub.status.idle": "2026-06-11T20:07:41.172242Z",
"shell.execute_reply": "2026-06-11T20:07:41.171814Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"numpy 2.2.6 · torch 2.12.0+cpu\n",
"device cpu (this notebook is CPU-canonical; everything fits on CPU)\n"
]
}
],
"source": [
"import numpy as np\n",
"import torch\n",
"import matplotlib.pyplot as plt\n",
"print(f\"numpy {np.__version__} · torch {torch.__version__}\")\n",
"if np.__version__ < \"2.0\":\n",
" print(\"WARN: written for NumPy 2.x; older versions may shift the last digit\")\n",
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
"print(f\"device {device} (this notebook is CPU-canonical; everything fits on CPU)\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "6adc7140",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.173230Z",
"iopub.status.busy": "2026-06-11T20:07:41.173091Z",
"iopub.status.idle": "2026-06-11T20:07:41.181215Z",
"shell.execute_reply": "2026-06-11T20:07:41.180846Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"FAST=True QL=1500 PG=60 PPO=12 RLHF=20\n"
]
}
],
"source": [
"import os, random, math\n",
"import torch.nn.functional as F\n",
"from torch.distributions import Categorical\n",
"\n",
"SEED = 0\n",
"FAST = bool(os.environ.get('NB_FAST')) # CI smoke mode: ~10x fewer steps, same code paths\n",
"rng = np.random.default_rng(SEED) # the one RNG we thread through every stochastic cell\n",
"torch.manual_seed(SEED); random.seed(SEED)\n",
"\n",
"# Step budgets: FAST is the CI smoke setting; the second number is the full run.\n",
"QL_EPISODES = 1500 if FAST else 8000 # tabular Q-learning episodes\n",
"PG_EPISODES = 60 if FAST else 250 # REINFORCE episodes on CartPole\n",
"PPO_ITERS = 12 if FAST else 60 # PPO update iterations on CartPole\n",
"RLHF_STEPS = 20 if FAST else 60 # toy-LM RLHF PPO steps\n",
"print(f'FAST={FAST} QL={QL_EPISODES} PG={PG_EPISODES} PPO={PPO_ITERS} RLHF={RLHF_STEPS}')\n",
"\n",
"# ── house self-check harness (identical across all chapter notebooks) ──\n",
"import numpy as _np\n",
"\n",
"def check(label, test_fn, required=False):\n",
" \"\"\"Run one self-check. test_fn raises AssertionError (with a teaching\n",
" message) on failure, NotImplementedError if the stub is unfilled.\n",
" required=True is used only in solution cells; it is what CI grades.\"\"\"\n",
" try:\n",
" test_fn()\n",
" except NotImplementedError:\n",
" if required:\n",
" raise AssertionError(f\"{label}: reference solution incomplete\")\n",
" print(f\"[ -- ] {label}: not attempted yet — fill in the TODO above, then re-run.\")\n",
" return False\n",
" except AssertionError as e:\n",
" if required:\n",
" raise\n",
" print(f\"[FAIL] {label}: {e}\")\n",
" return False\n",
" print(f\"[ ok ] {label}\")\n",
" return True\n",
"\n",
"def attempted(*vals):\n",
" \"\"\"Treat None placeholders as 'not attempted'.\"\"\"\n",
" if any(v is None for v in vals):\n",
" raise NotImplementedError\n",
"\n",
"def check_shape(x, want):\n",
" assert tuple(x.shape) == tuple(want), \\\n",
" f\"shape {tuple(x.shape)}, expected {tuple(want)} — check your reshape/transpose order\"\n",
"\n",
"def check_close(got, want, atol=1e-5, rtol=1e-4, msg=\"\"):\n",
" g, w = _np.asarray(got, dtype=float), _np.asarray(want, dtype=float)\n",
" assert g.shape == w.shape, f\"shape {g.shape} vs expected {w.shape}. {msg}\"\n",
" bad = ~_np.isclose(g, w, atol=atol, rtol=rtol)\n",
" assert not bad.any(), \\\n",
" f\"{bad.mean():.2%} of values wrong (max diff {abs(g - w).max():.3g}). {msg}\""
]
},
{
"cell_type": "markdown",
"id": "e34e978e",
"metadata": {},
"source": [
"> **Note:** seeds make this notebook's printed numbers reproduce on CPU. The exact-DP asserts (value iteration, policy iteration, the Bellman residual, GAE limits, PPO clip values, the DPO identity) are deterministic and hold to machine precision. The *learning* numbers (Q-learning policy match, CartPole episode length, RLHF reward) are stochastic; we assert behavioral properties with margins, never bitwise equality. If your CartPole reaches 180 where the page says 210, you did nothing wrong.\n"
]
},
{
"cell_type": "markdown",
"id": "111f0107",
"metadata": {},
"source": [
"## The map\n",
"\n",
"> **Part 1 — MDPs and the Bellman equation.** Build a 4x4 gridworld as explicit transition and reward tensors. Solve it with value iteration; check the Bellman residual is zero.\n",
"> **Part 2 — Planning two ways.** Policy iteration with an exact linear solve for policy evaluation; prove it agrees with value iteration to machine precision. Then break value iteration with the early-break sentinel bug and fix it.\n",
"> **Part 3 — Learning without the model: Q-learning.** Estimate the value function by interacting; recover the value-iteration optimal policy. Watch why $\\epsilon$-greedy exploration is non-negotiable.\n",
"> **Part 4 — Policy gradients.** Derive REINFORCE from the log-derivative trick, check the gradient against finite differences, then run PPO with GAE and a clipped objective on a self-contained CartPole.\n",
"> **Part 5 — RLHF on a toy LM.** A Bradley-Terry reward model, PPO fine-tuning, and mode collapse when the KL penalty is off. DPO and GRPO as one-function variants.\n",
"> **Part 6 — Reward hacking.** A gridworld where the proxy reward and the true reward diverge. Measure the true return of the proxy-optimal policy: zero.\n"
]
},
{
"cell_type": "markdown",
"id": "7210df68",
"metadata": {},
"source": [
"## Part 1 — MDPs and the Bellman equation\n",
"\n",
"> **Objectives**\n",
"> - Encode a Markov Decision Process as transition and reward tensors $P$ of shape $(S, A, S')$ and $R$ of shape $(S, A, S')$.\n",
"> - Implement value iteration as repeated application of the Bellman optimality operator.\n",
"> - Verify the result is the true fixed point: the Bellman residual is zero.\n",
"\n",
"A Markov Decision Process is a tuple $\\langle \\mathcal{S}, \\mathcal{A}, P, R, \\gamma\\rangle$: states, actions, a transition kernel $P(s'\\mid s,a)$, a reward $R(s,a,s')$, and a discount $\\gamma\\in[0,1)$. A policy $\\pi(a\\mid s)$ maps states to action distributions; the agent wants the policy that maximizes expected discounted return $\\mathbb{E}_\\pi[\\sum_t \\gamma^t R_t]$.\n",
"\n",
"The *Bellman optimality equation* is the fixed-point characterization of the optimal value:\n",
"\n",
"$$V^*(s) = \\max_a \\sum_{s'} P(s'\\mid s,a)\\left[R(s,a,s') + \\gamma V^*(s')\\right]$$\n",
"\n",
"Our world is a 4x4 grid. State $s = 4r + c$. Actions are up/right/down/left. One goal cell pays $+1$, one trap pays $-1$, both absorbing; every other move costs $-0.04$ so dawdling is penalized. Deterministic transitions, so $P(s'\\mid s,a)$ is one-hot.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "3f2371eb",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.182022Z",
"iopub.status.busy": "2026-06-11T20:07:41.181894Z",
"iopub.status.idle": "2026-06-11T20:07:41.184914Z",
"shell.execute_reply": "2026-06-11T20:07:41.184565Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"from state 0 (top-left), right -> 1 | down -> 4\n"
]
}
],
"source": [
"N = 4 # grid side\n",
"S, A = N * N, 4 # 16 states, 4 actions (0=up,1=right,2=down,3=left)\n",
"GOAL, TRAP = 15, 5 # absorbing cells: +1 at the goal, -1 at the trap\n",
"TERMINAL = {GOAL, TRAP}\n",
"STEP_COST = -0.04 # small per-move cost: makes the agent prefer short paths\n",
"\n",
"def grid_step(s, a):\n",
" '''Deterministic next state. Terminal cells absorb (stay put).'''\n",
" if s in TERMINAL:\n",
" return s\n",
" r, c = divmod(s, N)\n",
" if a == 0: r = max(0, r - 1) # up\n",
" elif a == 1: c = min(N - 1, c + 1) # right\n",
" elif a == 2: r = min(N - 1, r + 1) # down\n",
" elif a == 3: c = max(0, c - 1) # left\n",
" return r * N + c\n",
"\n",
"print(\"from state 0 (top-left), right ->\", grid_step(0, 1), \"| down ->\", grid_step(0, 2))"
]
},
{
"cell_type": "markdown",
"id": "b431513b",
"metadata": {},
"source": [
"Now assemble the tensors. $P$ is one-hot over next states. $R$ pays $+1$ for *entering* the goal, $-1$ for entering the trap, and the step cost otherwise. Terminal states earn nothing further (they are absorbing).\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "1b9b941b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.185798Z",
"iopub.status.busy": "2026-06-11T20:07:41.185727Z",
"iopub.status.idle": "2026-06-11T20:07:41.188744Z",
"shell.execute_reply": "2026-06-11T20:07:41.188401Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"P (16, 4, 16) · R (16, 4, 16) · rows sum to 1: True\n"
]
}
],
"source": [
"def build_gridworld():\n",
" P = np.zeros((S, A, S)) # P[s,a,s'] transition prob\n",
" R = np.zeros((S, A, S)) # R[s,a,s'] reward for that transition\n",
" for s in range(S):\n",
" for a in range(A):\n",
" ns = grid_step(s, a)\n",
" P[s, a, ns] = 1.0\n",
" if s not in TERMINAL:\n",
" if ns == GOAL: R[s, a, ns] = 1.0\n",
" elif ns == TRAP: R[s, a, ns] = -1.0\n",
" else: R[s, a, ns] = STEP_COST\n",
" return P, R\n",
"\n",
"P, R = build_gridworld()\n",
"# every (s,a) row of P must be a probability distribution\n",
"assert np.allclose(P.sum(axis=2), 1.0), \"each (s,a) must define a distribution over next states\"\n",
"print(\"P\", P.shape, \"· R\", R.shape, \"· rows sum to 1:\", np.allclose(P.sum(2), 1.0))"
]
},
{
"cell_type": "markdown",
"id": "fca02dca",
"metadata": {},
"source": [
"> **Predict:** the Bellman optimality operator applied to $V$ is $(\\mathcal{T}V)(s) = \\max_a \\sum_{s'} P(s'\\mid s,a)[R + \\gamma V(s')]$. Starting from $V = 0$, what is $(\\mathcal{T}V)$ at a cell one step from the goal? AnswerIt is $\\max_a$ over the immediate rewards (since $V=0$): for the cell left of the goal, the \"right\" action enters the goal for $+1$, so $(\\mathcal{T}V) = 1$ there. Value flows outward from the goal one ring per iteration.\n",
"\n",
"The whole operator is one `einsum`. Reading it: for each $(s,a)$ we dot the next-state distribution $P[s,a,:]$ against $R[s,a,:] + \\gamma V[:]$, then take the max over $a$.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "628583c2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.189636Z",
"iopub.status.busy": "2026-06-11T20:07:41.189569Z",
"iopub.status.idle": "2026-06-11T20:07:41.191844Z",
"shell.execute_reply": "2026-06-11T20:07:41.191496Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"one backup, value left of goal (state 14): 1.0 (want 1.0)\n"
]
}
],
"source": [
"def bellman_operator(V, P, R, gamma):\n",
" # Q[s,a] = sum_s' P[s,a,s'] * (R[s,a,s'] + gamma * V[s'])\n",
" Q = np.einsum(\"sat,sat->sa\", P, R + gamma * V[None, None, :])\n",
" return Q\n",
"\n",
"V0 = np.zeros(S)\n",
"Q1 = bellman_operator(V0, P, R, 0.9)\n",
"print(\"one backup, value left of goal (state 14):\", round(Q1.max(1)[14], 3), \"(want 1.0)\")\n",
"assert abs(Q1.max(1)[14] - 1.0) < 1e-9, \"cell 14 steps right into the goal for +1 on the first backup\""
]
},
{
"cell_type": "markdown",
"id": "cf570a13",
"metadata": {},
"source": [
"### Exercise 19.1 — Value iteration\n",
"`Difficulty 2/5 · ~12 min`\n",
"\n",
"Fill in `value_iteration`. Apply the Bellman operator until the largest change in $V$ drops below `tol`, then read off the greedy policy. The convergence check goes *after* the backup, not before (Part 2 shows what happens if you get that wrong). Return `(V, pi)`.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "6b7179ff",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.192982Z",
"iopub.status.busy": "2026-06-11T20:07:41.192917Z",
"iopub.status.idle": "2026-06-11T20:07:41.197924Z",
"shell.execute_reply": "2026-06-11T20:07:41.197625Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 19.1 Bellman residual zero: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 19.1 greedy policy: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def value_iteration(P, R, gamma=0.9, tol=1e-9, max_iters=1000):\n",
" \"\"\"Return (V (S,), pi (S,)) for the optimal value and greedy policy.\"\"\"\n",
" Sn = P.shape[0]\n",
" V = np.zeros(Sn)\n",
" for _ in range(max_iters):\n",
" Q = bellman_operator(V, P, R, gamma)\n",
" # TODO 1: V_new = the max over actions of Q\n",
" V_new = None\n",
" attempted(V_new)\n",
" # TODO 2: if the largest absolute change |V_new - V| is below tol, accept V_new and stop\n",
" # (set V = V_new and break)\n",
" raise NotImplementedError # remove once TODO 1 and TODO 2 are done\n",
" Q = bellman_operator(V, P, R, gamma)\n",
" return V, Q.argmax(axis=1)\n",
"\n",
"def _vi_residual():\n",
" V, pi = value_iteration(P, R, gamma=0.9)\n",
" resid = np.abs(bellman_operator(V, P, R, 0.9).max(1) - V).max()\n",
" assert resid < 1e-6, f\"Bellman residual {resid:.2e} should be ~0; V is not the fixed point yet\"\n",
"\n",
"def _vi_policy():\n",
" V, pi = value_iteration(P, R, gamma=0.9)\n",
" # left-of-goal (14) goes right; above-goal (11) goes down\n",
" assert pi[14] == 1 and pi[11] == 2, \\\n",
" f\"greedy policy pi[14]={pi[14]} (want 1=right), pi[11]={pi[11]} (want 2=down)\"\n",
"\n",
"check(\"19.1 Bellman residual zero\", _vi_residual)\n",
"check(\"19.1 greedy policy\", _vi_policy)"
]
},
{
"cell_type": "markdown",
"id": "e9d1ac4e",
"metadata": {},
"source": [
"Hint 1 (conceptual)`Q` is shape $(S, A)$. The new value of a state is the best action's Q-value: `Q.max(axis=1)`. Convergence means the value stopped moving, so compare `np.abs(V_new - V).max()` to `tol`.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"V_new = Q.max(axis=1)\n",
"if np.abs(V_new - V).max() < tol:\n",
" V = V_new\n",
" break\n",
"V = V_new\n",
"```\n",
"Note the check is after computing `V_new` from a real backup. Checking before the backup is the bug in Part 2.\n",
"\n",
"Help — \"Bellman residual is 1.0, not 0\"You broke before any backup ran (the early-break bug) or you compared the wrong quantities. The residual is `|max_a Q(s,a) - V(s)|`. If it equals the goal reward, your `V` is still all zeros. Make sure `V = V_new` happens before `break`, and that the loop runs more than once.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "ee99d54e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.198793Z",
"iopub.status.busy": "2026-06-11T20:07:41.198724Z",
"iopub.status.idle": "2026-06-11T20:07:41.201777Z",
"shell.execute_reply": "2026-06-11T20:07:41.201533Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 19.1 Bellman residual zero\n",
"[ ok ] 19.1 greedy policy\n",
"V* on the grid:\n",
"[[0.427 0.519 0.621 0.734]\n",
" [0.519 0. 0.734 0.86 ]\n",
" [0.621 0.734 0.86 1. ]\n",
" [0.734 0.86 1. 0. ]]\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines value_iteration; the checks below re-verify the reference.\n",
"def value_iteration(P, R, gamma=0.9, tol=1e-9, max_iters=1000):\n",
" Sn = P.shape[0]\n",
" V = np.zeros(Sn)\n",
" for _ in range(max_iters):\n",
" Q = bellman_operator(V, P, R, gamma)\n",
" V_new = Q.max(axis=1)\n",
" if np.abs(V_new - V).max() < tol:\n",
" V = V_new\n",
" break\n",
" V = V_new\n",
" Q = bellman_operator(V, P, R, gamma)\n",
" return V, Q.argmax(axis=1)\n",
"\n",
"check(\"19.1 Bellman residual zero\", _vi_residual, required=True)\n",
"check(\"19.1 greedy policy\", _vi_policy, required=True)\n",
"\n",
"V_star, pi_star = value_iteration(P, R, gamma=0.9)\n",
"print(\"V* on the grid:\")\n",
"print(V_star.reshape(N, N).round(3))"
]
},
{
"cell_type": "markdown",
"id": "d548fef9",
"metadata": {},
"source": [
"Let us look at the value field and the policy. The arrows should all flow toward the goal and away from the trap.\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "4f8273dd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.202656Z",
"iopub.status.busy": "2026-06-11T20:07:41.202591Z",
"iopub.status.idle": "2026-06-11T20:07:41.331991Z",
"shell.execute_reply": "2026-06-11T20:07:41.331545Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAa4AAAGLCAYAAACflHYfAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQO1JREFUeJzt3Xl4E9X6B/DvJN1Xlu5QKftWoVikFtnUQgUu4MoqFFRcoIjWBeUixauCiGD9AYqigl5AuHJVvLJTRVmKyKayry0VaOlC6b5lzu+P0tjQtE3TpGEm38/z5NFMzuS8KUnenPecmZGEEAJEREQKobF1AERERPXBxEVERIrCxEVERIrCxEVERIrCxEVERIrCxEVERIrCxEVERIrCxEVERIrCxEVERIqi+sT17rvvolOnTpBl2SrPP2DAAAwYMMCiz3n8+HE4ODjg6NGjFn1eSwgJCcHEiRNtHYZFTZw4ESEhIbYOo0Y7d+6EJEnYuXOnfputYpZlGaGhoXj77bcbvW9bGT16NEaOHGnrMKiKRk9cw4cPh5ubG/Ly8mpsM27cODg5OSErK8tge30/qLm5uZg/fz5mzJgBjUY5ObpLly4YOnQoZs+ebetQiAx89dVXSE1NRWxsrK1DaTQzZszAf//7X/z++++2DoVuaPRv83HjxqGoqAjffvut0ccLCwuxYcMG3H///WjevDl27NgBnU5Xrd3WrVvr7Ovzzz9HeXk5xowZ0+C4G9szzzyDb7/9FufOnbN1KHQLWr58OU6dOtXo/S5YsACjR4+Gt7d3o/dtKz169EDPnj2xcOFCW4dCN9hkxOXp6Yk1a9YYfXzDhg0oKCjAuHHjIITA8uXLcdddd+l/7Vy4cAHR0dFYuHAh8vPza+1rxYoVGD58OFxcXCz+OqyhvLwcpaWlAICoqCg0bdoUX3zxhY2janwFBQW2DuGW5+joCGdn50bt8/Dhw/j9998tWjar7d+6sLDQYv001MiRI/HNN9/U+Z1DjaPRE5erqyseeughJCYm4urVq9UeX7NmDTw9PTF8+HBIkoR169Zh0aJFeOaZZ3DlyhU89NBDmDp1KrZt2wYPD48a+7lw4QL++OMPREVFVXtMlmV88MEHuP322+Hi4gJfX1/cf//9OHDggL5NeXk53nzzTbRt2xbOzs4ICQnBzJkzUVJSUudrvHr1Kp544gn4+/vDxcUF3bt3r5aAkpOTIUkS3nvvPSQkJOj7OX78OICKL6YBAwZgw4YNdfaXkpKCKVOmoGPHjnB1dUXz5s3x6KOPIjk52aDdypUrIUkS9uzZg7i4OPj6+sLd3R0PPvggMjIyDNoKIfDWW2+hZcuWcHNzwz333INjx47VGUulrKwsjB8/Hl5eXmjSpAliYmLw+++/Q5IkrFy5Ut9u4sSJ8PDwwLlz5zBkyBB4enpi3LhxACr+nRISEtC1a1e4uLjA398fTz/9NK5du1atv82bN6Nv375wd3eHp6cnhg4dajTe7777DqGhoXBxcUFoaGi1kb8QAiEhIRgxYkS1fYuLi+Ht7Y2nn3661tcuSRJiY2OxevVqdOzYES4uLggPD8cvv/xSre3hw4cxePBgeHl5wcPDA/fddx/27dtX6/MDxue46npf9+/fH927dzf6fB07dkR0dHStfX733XdwcnJCv379DLbX9/33888/Y8qUKfDz80PLli0BVMwVh4aG4uDBg+jXrx/c3Nwwc+ZMAKZ9nu644w489NBDBttuv/12SJKEP/74Q79t3bp1kCQJJ06cAADk5eXh+eefR0hICJydneHn54eBAwfi0KFDBs81cOBAFBQUYPv27bX+jahxONii03HjxuGLL77Af/7zH4NaeXZ2NrZu3YoxY8bA1dVVv12j0UCSJP39qv9fk7179wKoeEPf7IknnsDKlSsxePBgPPnkkygvL8euXbuwb98+9OzZEwDw5JNP4osvvsAjjzyCF198Eb/++ivmzZuHEydO1FjmBICioiIMGDAAZ8+eRWxsLFq3bo2vv/4aEydORE5ODqZPn27QfsWKFSguLsZTTz0FZ2dnNGvWTP9YeHg4NmzYgNzcXHh5edXY52+//Ya9e/di9OjRaNmyJZKTk/HRRx9hwIABOH78ONzc3AzaT5s2DU2bNkV8fDySk5ORkJCA2NhYrFu3Tt9m9uzZeOuttzBkyBAMGTIEhw4dwqBBg/QjwtrIsoxhw4Zh//79ePbZZ9GpUyds2LABMTExRtuXl5cjOjoaffr0wXvvvaeP9+mnn8bKlSsxadIkPPfcc7hw4QKWLFmCw4cPY8+ePXB0dAQA/Pvf/0ZMTAyio6Mxf/58FBYW4qOPPkKfPn1w+PBh/Rf8tm3b8PDDD6NLly6YN28esrKyMGnSJP2XJ1Dx3nrsscfw7rvvIjs72+Df43//+x9yc3Px2GOP1fk3+Pnnn7Fu3To899xzcHZ2xocffoj7778f+/fvR2hoKADg2LFj6Nu3L7y8vPDKK6/A0dERH3/8MQYMGICff/4ZERERdfZTVV3v6/Hjx2Py5Mk4evSoPgag4v1z+vRpzJo1q9bn37t3L0JDQ/V/96r71+f9N2XKFPj6+mL27NkGI66srCwMHjwYo0ePxmOPPQZ/f3+TP099+/bFV199pX+u7OxsHDt2DBqNBrt27UK3bt0AALt27YKvry86d+4MoKIkv379esTGxqJLly7IysrC7t27ceLECYPvji5dusDV1RV79uzBgw8+WJ9/FrIGYQPl5eUiMDBQREZGGmxftmyZACC2bt0qhBBClmUxduxY0bNnT3HkyBHRqlUrcf78eTFw4EAxcOBAkZeXV2Mfs2bNEgCqtfnxxx8FAPHcc89V20eWZSGEEEeOHBEAxJNPPmnw+EsvvSQAiB9//FG/rX///qJ///76+wkJCQKAWLVqlX5baWmpiIyMFB4eHiI3N1cIIcSFCxcEAOHl5SWuXr1q9DWsWbNGABC//vprja9TCCEKCwurbUtKShIAxJdffqnftmLFCgFAREVF6V+rEEK88MILQqvVipycHCGEEFevXhVOTk5i6NChBu1mzpwpAIiYmJha4/nvf/8rAIiEhAT9Np1OJ+69914BQKxYsUK/PSYmRgAQr776qsFz7Nq1SwAQq1evNti+ZcsWg+15eXmiSZMmYvLkyQbt0tLShLe3t8H2sLAwERgYqH+dQgixbds2AUC0atVKv+3UqVMCgPjoo48MnnP48OEiJCTE4G9iDAABQBw4cEC/LSUlRbi4uIgHH3xQv+2BBx4QTk5O4ty5c/ptly9fFp6enqJfv376bT/99JMAIH766Sf9tpiYGIOYTXlf5+TkCBcXFzFjxgyDx5977jnh7u4u8vPza31dLVu2FA8//HC17fV9//Xp00eUl5cbtO/fv78AIJYtW2aw3dTP09dffy0AiOPHjwshhPj++++Fs7OzGD58uBg1apR+327duhn8G3h7e4upU6fW+rordejQQQwePNiktmRdNllqp9VqMXr0aCQlJRmUE9asWQN/f3/cd999ACp+/U6cOBH79u3Tlzhat26Nbdu2IS4urtZSYVZWFhwcHKq1+e9//wtJkhAfH19tn8qR3KZNmwAAcXFxBo+/+OKLAICNGzfW2O+mTZsQEBBgsCDE0dERzz33HPLz8/Hzzz8btH/44Yfh6+tr9LmaNm0KAMjMzKyxPwAGo9OysjJkZWWhXbt2aNKkSbWSBwA89dRTBqPWvn37QqfTISUlBQCwY8cOlJaWYtq0aQbtnn/++VrjqLRlyxY4Ojpi8uTJ+m0ajQZTp06tcZ9nn33W4P7XX38Nb29vDBw4EJmZmfpbeHg4PDw88NNPPwEAtm/fjpycHIwZM8agnVarRUREhL7dlStXcOTIEcTExBgsLBg4cCC6dOli0HeHDh0QERGB1atX67dlZ2dj8+bNGDdunEkj/sjISISHh+vv33bbbRgxYgS2bt0KnU4HnU6Hbdu24YEHHkCbNm307QIDAzF27Fjs3r0bubm5dfZTyZT3tbe3N0aMGIGvvvoK4sb1Y3U6HdatW4cHHngA7u7utfaRlZWlf09WVd/33+TJk6HVaqttd3Z2xqRJkwy2mfp56tu3LwDoy7G7du3CnXfeiYEDB2LXrl0AgJycHBw9elTfFgCaNGmCX3/9FZcvX671tQMVn8e6PovUOGy2RrxyHqNykcZff/2FXbt2YfTo0QZv6oEDBxp9k99///1m9Xvu3DkEBQUZlIBulpKSAo1Gg3bt2hlsDwgIQJMmTfRf8DXt2759+2rL7ytLEzfv27p16xqfq/LLpa4vyqKiIsyePRvBwcFwdnaGj48PfH19kZOTg+vXr1drf9tttxncr/wyqpw7qoyxffv2Bu18fX2NfnHdLCUlBYGBgdVKRDf/PSs5ODgYlOsA4MyZM7h+/Tr8/Pzg6+trcMvPz9fPj545cwYAcO+991Zrt23bNn27ml4TUDG/c7MJEyZgz549+v2+/vprlJWVYfz48XW+/pr66dChAwoLC5GRkYGMjAwUFhYa7btz586QZRmpqakm9QWY9r4GKl7XxYsX9V/mO3bsQHp6usmvSxi5YHp93381vedbtGgBJycng22mfp78/f3Rvn17/evatWsX+vbti379+uHy5cs4f/489uzZA1mWDRLXu+++i6NHjyI4OBi9evXCnDlzcP78+Rpfuyk/Wsj6bDLHBVTM33Tq1AlfffUVZs6cqf8VWJnQjLl5src2zZs3R3l5OfLy8uDp6WlWjI3xJq36a/VmlYnEx8en1ueYNm0aVqxYgeeffx6RkZHw9vaGJEkYPXq00QOvjf0QAIx/KTUGZ2fnal9MsizDz8/PYNRTVeUotfL1/fvf/0ZAQEC1dg4O5r3FR48ejRdeeAGrV6/GzJkzsWrVKvTs2dNoolGS6Oho+Pv7Y9WqVejXrx9WrVqFgIAAo4uYbta8eXOjC2Pq+/6r6T1f22fBFH369EFiYiKKiopw8OBBzJ49G6GhoWjSpAl27dqFEydOwMPDAz169NDvM3LkSPTt2xfffvsttm3bhgULFmD+/Pn45ptvMHjwYIPnv3btmtEfJNT4bJa4gIpR1+uvv44//vgDa9asQfv27XHnnXda5Lk7deoEoGJ1YeXELAC0bdsWW7durTbxXlWrVq0gyzLOnDmj/2UHAOnp6cjJyUGrVq1q7LdVq1b4448/IMuywZfxyZMn9Y+b6sKFC9BoNOjQoUOt7davX4+YmBiD40yKi4uRk5Njcl9VVcZ45swZgzJWRkaG0S8uY/v/9NNPKCwsNBh1nT171uQY2rZtix07duDuu++u9Qutbdu2AAA/P79av3yrvqabGTseqlmzZhg6dChWr16NcePGYc+ePUhISDA5fmP9nD59Gm5ubvqk6+bmZrTvkydPQqPRIDg42OT+THlfAxU/WsaOHYuVK1di/vz5+O6772os3d2sU6dOuHDhQrXtln7/VVWfz1Pfvn2xYsUKrF27FjqdDr1794ZGo0GfPn30iat3797VXmtgYCCmTJmCKVOm4OrVq7jjjjvw9ttvGySu8vJypKamYvjw4Q1+TdRwNj2dROXoavbs2Thy5Eito636ioyMBACDJe5AxZySEAJvvPFGtX0qRxxDhgwBgGpfVIsWLQIADB06tMZ+hwwZgrS0NIMVeuXl5Vi8eDE8PDzQv39/k1/DwYMH0bVr1zoP9tRqtdVGS4sXLzZ64LYpoqKi4OjoiMWLFxs8r6lf3NHR0SgrK8Py5cv122RZxtKlS02OYeTIkdDpdHjzzTerPVZeXq7/UoyOjoaXlxfmzp2LsrKyam0rl/kHBgYiLCwMX3zxhUH5avv27fpDEG42fvx4HD9+HC+//LJ+XtZUSUlJBvM7qamp2LBhAwYNGgStVgutVotBgwZhw4YNBpWE9PR0rFmzBn369Kl1JenNTHlfV31d165dw9NPP438/HyTVkkCFZ+po0ePVjskxNLvv6rq83mqLAHOnz8f3bp1039u+vbti8TERBw4cMCgTKjT6aqVMv38/BAUFFTtNR4/fhzFxcXo3bt3g18TWYBNloRU0bt3b/0qrDNnzlj0uUNDQ8WYMWOqbR8/frwAIAYPHiw++OAD8f7774uHHnpILF68WN+mcrXbyJEjxdKlS/X3H3jgAYPnunlVYWFhoejcubNwcnISL774oli8eLF+xVTVVXaVqwoXLFhgNPbS0lLRrFkzMWvWrDpf54QJE4RWqxXTp08XH3/8sZg4caJo2bKlaN68ucEKwMpVXb/99pvB/sZWrb322msCgBgyZIhYsmSJeOKJJ0RQUJDw8fGpc1VheXm56NWrl9BqtSI2NlYsWbJEDBo0SISFhQkAYuXKlfq2MTExwt3d3ejzPP300/p/p/fff18sWbJETJ8+XQQFBYmvv/5a32716tVCo9GI0NBQ8dZbb4mPP/5Y/POf/xRhYWEGK8Y2b96sb7do0SIxa9Ys4e3tLbp27WqwQq9SSUmJaN68uT4GUwEQoaGhwsfHR/zrX/8S8+fPF61atRIuLi7i999/17c7evSocHd3Fy1atBBvv/22mD9/vmjTpo1wdnYW+/bt07czZVWhEKa9ryuFhoYKAKJz584mv64DBw4YrPqt1ND3nxAVn6OuXbtW227q56lSQECAACCmTZum31a5whGA2Llzp377tWvXhLu7u4iJiRGLFi0Sn3zyiRg5cqQAIBYuXGjwvO+9955wc3PTr2Ik27J54lq6dKkAIHr16mXx5160aJHw8PCotly3vLxcLFiwQHTq1Ek4OTkJX19fMXjwYHHw4EF9m7KyMvHGG2+I1q1bC0dHRxEcHCxee+01UVxcbPBcNycuIYRIT08XkyZNEj4+PsLJyUncfvvtBkvAhag7cW3evNnkZH7t2jV9fx4eHiI6OlqcPHlStGrVyuzEpdPpxBtvvCECAwOFq6urGDBggDh69Gi156xJRkaGGDt2rPD09BTe3t5i4sSJYs+ePQKAWLt2rb5dbYlLCCE++eQTER4eLlxdXYWnp6e4/fbbxSuvvCIuX75c7TVER0cLb29v4eLiItq2bSsmTpxosCRdiIql+p07dxbOzs6iS5cu4ptvvjGaBCpNmTJFABBr1qyp8zVXAiCmTp0qVq1aJdq3by+cnZ1Fjx49DP6+lQ4dOiSio6OFh4eHcHNzE/fcc4/Yu3dvtddmSuIy5X1d6d133xUAxNy5c01+XUJULCd/4oknDLY19P0nRM2JSwjTPk+VHn30UQFArFu3Tr+ttLRUuLm5CScnJ1FUVKTfXlJSIl5++WXRvXt34enpKdzd3UX37t3Fhx9+WO15IyIixGOPPVbbn4Yakc0TlzXl5OSIZs2aiU8//dTWodTbiBEjqo3ulO7bb78VAMTu3bttHYrJnn/+eeHp6SkKCgpM3qcycd3KEhIShCRJIiUlpV77ffnll8LT01Ncu3bNOoHdgg4fPiwkSRKHDx+2dSh0g3JOmW4Gb29vvPLKK1iwYIHVLmtiDSdOnMAPP/xgdH5HKYqKigzu63Q6LF68GF5eXkbPZnIrKi4uxqpVq/Dwww9XW9qvZEIIfPbZZ+jfv3+1QyPqMm7cONx22231mq9UunfeeQePPPIIwsLCbB3KLeeXX37BsGHDEBQUBEmS8N1339W5z86dO3HHHXfA2dkZ7dq1MzgFnKlsuqqwMcyYMQMzZsywdRj10rlzZ5SXl9s6jAaZNm0aioqKEBkZiZKSEnzzzTfYu3cv5s6d2+Blz9Z29epV7NixA+vXr0dWVla103QpVUFBAb7//nv89NNP+PPPP006D+bNNBrNLXmdOGtau3atrUO4ZRUUFKB79+54/PHHq50r0pgLFy5g6NCheOaZZ7B69WokJibiySefRGBgYJ3nyqxK9YmLbOPee+/FwoUL8cMPP6C4uBjt2rXD4sWLFXEdp+PHj2PcuHHw8/PD//3f/6nml3ZGRgbGjh2LJk2aYObMmVzarTLFxcUmnUu0NsLIQdbOzs41Xolg8ODB1Y53q82yZcvQunVr/aETnTt3xu7du/H+++8zcZHtjR07FmPHjrV1GGYZMGBAgw7Gbsi+1hQSEnLLxkYNU1xcjNatPJB2tWGHIHh4eFS7dEt8fDzmzJnToOetlJSUVO14y+joaJNPJ1eJiYuISOFKS0uRdlWHlIMh8PI0b+lCbp6MVuHJSE1NNTiG0JLXfUtLS4O/v7/BNn9/f+Tm5qKoqMjkaQQmLiIilfDwlODhad6p6mRU7Ofl5VWvg99twaTEJcsyLl++DE9PT55kkojIAoQQyMvLQ1BQULVzdZpLJ2TozKwG64T1V14HBAQgPT3dYFt6ejq8vLzqtWjLpMR1+fLlep03jYiITJOamlrt6ghqFRkZqb9sVKXt27frT9FnKpMSV+XZ1ds8PxsaZ5d6dUA1K/Fp+LncqDon/0Jbh6A6XQPSbB2C6pQVlGLzg6vNvnqFMTIEZJg35DJnv/z8fIOTZ1+4cAFHjhxBs2bNcNttt+G1117DpUuX8OWXXwKouOL0kiVL8Morr+Dxxx/Hjz/+iP/85z+1XuPQGJMSV2V5UOPsAi0Tl8VoXJm4rEHrppyDzZXC0d2p7kZkFktOv8iQYe6735w9Dxw4gHvuuUd/v/LiuzExMVi5ciWuXLmCixcv6h9v3bo1Nm7ciBdeeAEffPABWrZsiU8//bReS+EBLs4gIlINnRDQmXnIgzn71XXoiLGzYgwYMACHDx+ud19VqfqUT0REpD4ccRERqURjz3HZChMXEZFKyBDQMXEREZFS2MuIi3NcRESkKBxxERGpRGOvKrQVJi4iIpWQb9zM3VcpmLiIiFRC14DFGebuZwuc4yIiIkXhiIuISCV0Ag04O7xlY7EmJi4iIpWwlzkulgqJiEhROOIiIlIJGRJ0aNgVkJWAiYuISCVkUXEzd1+lYOIiIlIJXQNGXObuZwuc4yIiIkXhiIuISCXsZcTFxEVEpBKykCALMxdnmLmfLTBxERGphL2MuDjHRUREisIRFxGRSuiggc7M8YjOwrFYExMXEZFKiAbMcQnOcRERUWPjHBcREdEtiCMuIiKV0AkNdMLMOS6e8omIiBqbDAmymYU0WUFXQGbiIiJSCc5xERER3YI44iIiUomGzXGxVEhERI2sYo5L/ReSZKmQiIgUhSMuIiKVkBtwyiclrSrkiMsMkgQ8GNYF4be1gKSc0fUtr7V3U4zucjuaubjaOhTV0EDCw63CcEfzYAUVgshclXNc5t6UgiMuM/Rr1xrzHowGAGQVFGLTn6ew5dgZHEq9BAXNb95yPho8HJ2a+2LugEH49VIq/nf2JLacO4Ps4iJbh6ZY9wR1wNyewwEAmcX52PTXMWz+6zgOZ6Uq6Pc1mUqGhsdxkXGaKsOs5u5uGHNnN4y/qweTWANV/l01koReQS0R0SIYb/UfiH2XUvEDk5hZNFXGWT4uHhjbpicmtItgEiNFY+KyAAetFgCTmCVpNX//aowIaolIJjGLcNBUvFeZxNRJJyTozDzLu7n72QITl4UxiVleZRKTwCRmSUxi6tOw63Ep519bdYnr7ra3Yc4/oqCx4o+HFk29TWpXWxJ7e/NO6wVoYc5aLT4YOBShvv5W7aelV91/19qSWML+vdh/5S+rxmhJ/fzbIb7HYMCKyyZaujcxqV1NSeyH1KOY98c2q8VnaU4aR7RwDcCFglRbh2ITstBANnORhaygX9SqS1wxkeEIbmZaYmlMDlothBBo7u6GcRFhikpcXXz8cH/bDrYOo5qqSax3y9vw66VURSWuxztEoqV7U1uHUY2DpuK96uPigXFt71RU4nq54zPo0bQrHtn7jK1DIStSXeJ6atW3Vu/jng5t8NG4ESa11ckypBuLDg6lXMLGo6ew/cRZa4ZncYfTryBk6XtW72fbmIno0MzHpLZlsg6OGi2yiwrxvzOnsPHsKRxIu2TlCC1r4q5/W72PgUGdsCRypEltq75XD2SmYGPqMWy/fNKa4VlcJ6+2tg7BplgqJLNV/QI4fPGyPlll5hfaODJlqylZKanEcaup+l49mHVRn6yySgpsHBmZQ4b5iyxky4ZiVUxcFsJkZR1MVpbHZKVeDTuOiwcg2wVZFvp5dSYry2GysjydkCGByYrUgYnLDMlZ15CRV4CUrGtMVha0J/UifFzdmKws6HxeJjKK83EhL5PJyg407LImHHGp2oWsa+i38BMej2Vhb+z+Ef/a/aOCpohvfefyMtF34yL+Te0EL2tCtWLSsg7+WS2Pf1NSG464iIhUgqVCIiJSlIYdx8XERUREjUwWEmRzj+NS0El2lZNiiYiIwBEXEZFqyA0oFfIAZCIianQNOzs8ExcRETUyHSTozDwey9z9bEE5KZaIiAgccRERqQZLhUREpCg6mF/y01k2FKti4iIiUgl7GXEpJ1IiIiIwcRERqUbluQrNvZlj6dKlCAkJgYuLCyIiIrB///5a2yckJKBjx45wdXVFcHAwXnjhBRQXF9erTyYuIiKVEDcua2LOTZgxN7Zu3TrExcUhPj4ehw4dQvfu3REdHY2rV68abb9mzRq8+uqriI+Px4kTJ/DZZ59h3bp1mDlzZr36ZeIiIlKJxh5xLVq0CJMnT8akSZPQpUsXLFu2DG5ubvj888+Ntt+7dy/uvvtujB07FiEhIRg0aBDGjBlT5yjtZkxcRESkl5uba3ArKSkx2q60tBQHDx5EVFSUfptGo0FUVBSSkpKM7tO7d28cPHhQn6jOnz+PTZs2YciQIfWKkasKiYhUwhJnhw8ODjbYHh8fjzlz5lRrn5mZCZ1OB39/f4Pt/v7+OHnypNE+xo4di8zMTPTp0wdCCJSXl+OZZ56pd6mQiYuIFKmlayCeaz8JkvT3F7Wr1gUAsKD7Pw3aHsr+E1+lft+o8dmCJa7HlZqaCi8vL/12Z2dni8QGADt37sTcuXPx4YcfIiIiAmfPnsX06dPx5ptv4vXXXzf5eZi4iEiRwpp0QRuP24w+1trdcNTg5eBhF4nLEry8vAwSV018fHyg1WqRnp5usD09PR0BAQFG93n99dcxfvx4PPnkkwCA22+/HQUFBXjqqafwz3/+ExqNaUmXc1xEpEj7sg+b1E4ndNiTecDK0dwaKkuF5t7qw8nJCeHh4UhMTPy7f1lGYmIiIiMjje5TWFhYLTlptVoAgBDC5L454iIiRcosycbZ/BS0cQ+GRqr5N7hW0iIp62AjRmY7MjRmX1fLnP3i4uIQExODnj17olevXkhISEBBQQEmTZoEAJgwYQJatGiBefPmAQCGDRuGRYsWoUePHvpS4euvv45hw4bpE5gpmLiISLH2ZP6GNjeVBW+WXZKDM/nJjROQjemEBJ2ZizPM2W/UqFHIyMjA7NmzkZaWhrCwMGzZskW/YOPixYsGI6xZs2ZBkiTMmjULly5dgq+vL4YNG4a33367Xv0ycRGRYiVlHUJMyCM1Pq4TOuzO/K0RI7I/sbGxiI2NNfrYzp07De47ODggPj4e8fHxDeqTc1xEpFiV5UJZyEYft6cyIdC4c1y2xMRFRIq2p5YRlT2VCQFA3Dg7vDk3wbPDExE1jqSsQ0YXZ9hjmVAHqUE3pWDiIiJFq6lcqJW02GtHZUJ7wsRFRIpnrFyYVXINZ+2oTAgAsmjIPJetozcdExcRKd7N5UJ7Oui4KnPntxpy5WRbUE6kREQ1uLlcaK9lQnOvxVV5UwomLiJSharlQnssE9oTJi4iUoWq5UJ7LBMCf585w9ybUjBxEZEqVJYLAdhlmRCwnzkunvKJiFTjq5Tv0NGzrd2WCWU04EKSCprjqlfiKvHRQeOqs1Ysdqf9tF9tHYIqee9ubusQVGeE7xFbh2CiIygpAUb42jqOuhW5loNXCDMPR1xERCohGrA6UKh1xEVERLeuhpwslyfZJSIishKOuIiIVKIhqwO5qpCIiBqdvZQKmbiIiFSiIaduUtJyeOWMDYmIiMARFxGRarBUSEREisLERUREimIviYtzXEREpCgccRERqYS9jLiYuIiIVELA/GXtwrKhWBUTFxGRStjLiItzXEREpCgccRERqYS9jLiYuIiIVMJeEhdLhUREpCgccRERqYS9jLiYuIiIVEIICcLMBGTufrbAxEVEpBK8rAkREdEtiCMuIiKV4BwXEREpCue4iIhIUexlxMU5LiIiUhSOuIiIVIKlQiIiUhTRgFIhExcRETU6AUCYeWEtJV2Pi3NcRESkKBxxERGphAwJkh2cOYOJi4hIJbg4g4iIFEUWEiQex0VERHRr4YiLiEglhGjAqkIFLStk4iK9QJGPj7EdrtABAA7CD6+iLyBVKSEIgXewC+G4CgAoghbPIgqXJE9bhExEVdjLHBdLhaR3RfLAx+iuvx+OqxiBcwZtRuCcPmkBwCfoxqR1izN3lRnRrYqJiwxslNrgVwTo7z+BP9FC5AEAWog8PIk/9Y/thz9+kNo2eoxkuhauAfik5zt4o2scBvn3g7cjf2SoWeWIy9ybUrBUSNUsQjg+wXZ4oxSu0GEGfsOLoj9mYD9cbpQRc+GERehp40ipLi1dA9DUyRvejp7o4tUeT7YZjRO5Z7En8wB+zT6M62V5tg6RLMheVhUycVE12ZIr/k/cgdexDwDQGdlYjB/RFtf1bT5AD2RJrrYKkepJI1UUVyRI6OzV7qYk9hv2ZR9BLpOY4nFxBtm1X6SW2CFuQxQuAoBB0kpEMH6Rgm0VGjWQ8SQ2hkmMFIOJi2q0BGHogatojmL9tmw4YzF62DAqsqSaktjx3DPYm3mASUxhKkZc5q4qtHAwVqS6xLXyHw/js98PYFdqiq1DUTxfFMETpQbbPFGKABTiHJxsFJV6zOn6AkK9O9o6DL2qSayrVweEenfEpNajMGZfrI0jM11L9z5o6zUMP1+ZYetQbMJelsOrLnENaNUa7Zo2Q59/L7d1KIqmFTJm4Dc4QTbY7giBGdiPqeI+lElaG0WnDhsubYefsw8kyXpfGL7Ozeq9T7msg4NGi5zSXOzO/M0KUVnP3f7xcNLa78pJAfMvT6KgAZf6EhcAq34R2IsYHEM75Ojvb0Bb/TFdrZGLiTiG5ehmo+jU4XDOUUw59E+r9hHRLAwvd3qmznZVk9WezANIyjqIU3nnIRT1dQZI/DFlF1SZuKhhuohMjMRp/f3NCMESqQechA6DkQwAeBinkSQCcVTytVGU1FBqSVb0N5YKyS65iHK8ggPQ3vjyugI3fHTjbBofIgzdkYEgFEAL4BUcwFNiIIolvo2UgslK5eykVsgzZ5CBp/EHWiAfAKADsAB3okhyBAAUSw54F3dCd+MUQoEowLP43VahkgnkKt9GOaW52Jr2M2b9uQCTD8zAiuT/4GTeOSYtNWnIWTPMHHEtXboUISEhcHFxQUREBPbv319r+5ycHEydOhWBgYFwdnZGhw4dsGnTpnr1yZ/KpNdTpOEfOK+//w3a48+bSoHHJB+sEx0xFicBAENwAXtEEPZLgY0aK5nmYPYf+PDsl7hclM6RFVncunXrEBcXh2XLliEiIgIJCQmIjo7GqVOn4OfnV619aWkpBg4cCD8/P6xfvx4tWrRASkoKmjRpUq9+mbhI74AUgIF4pM52K6RQrEBoI0REDSVD4Mere20dBjWSxj5zxqJFizB58mRMmjQJALBs2TJs3LgRn3/+OV599dVq7T///HNkZ2dj7969cHSsqOSEhITUu1+WComIVMISJ9nNzc01uJWUlBjtq7S0FAcPHkRUVJR+m0ajQVRUFJKSkozu8/333yMyMhJTp06Fv78/QkNDMXfuXOh0unq9TiYuIiLSCw4Ohre3t/42b948o+0yMzOh0+ng7+9vsN3f3x9paWlG9zl//jzWr18PnU6HTZs24fXXX8fChQvx1ltv1StGlgqJiNSiAYssKvdLTU2Fl5eXfrOzs7MlIgMAyLIMPz8/fPLJJ9BqtQgPD8elS5ewYMECxMfHm/w8TFxERCphiTkuLy8vg8RVEx8fH2i1WqSnpxtsT09PR0BAgNF9AgMD4ejoCK327wPFO3fujLS0NJSWlsLJybRTybFUSESkFqKBt3pwcnJCeHg4EhMT9dtkWUZiYiIiIyON7nP33Xfj7NmzkOW/TyV3+vRpBAYGmpy0ACYuIiIyU1xcHJYvX44vvvgCJ06cwLPPPouCggL9KsMJEybgtdde07d/9tlnkZ2djenTp+P06dPYuHEj5s6di6lTp9arX5YKiYhUorFP+TRq1ChkZGRg9uzZSEtLQ1hYGLZs2aJfsHHx4kVoNH+Pj4KDg7F161a88MIL6NatG1q0aIHp06djxoz6nc2fiYuISE0a+Rjz2NhYxMYav/TNzp07q22LjIzEvn37GtQnExcRkUrYy0l2OcdFRESKwhEXEZFa2MnZ4Zm4iIhUQ7pxM3dfZWDiIiJSCzsZcXGOi4iIFIUjLiIitbCTERcTFxGRWljgJLtKwMRFRKQSjX0hSVtRdOJq7d0UbZo2rba9hacX7gtpo78vC2BnynkljYSJqA4SNAhyj4RUZTWco8YNANDSvY9B2+ulycgr+6tR4yPrUWzikgCse3AU/Nw9jD7+2dCHDO6/lLgZ608ea4TIiKgxtPd+EBF+Lxt97J6g9wzu55VdxnfJDxltqyp2Msel2FWFAkDy9RzIQq6zLQCkXM+xajxE1LhMHUHJQofc0otWjuYWUTnHZe5NIRSbuADgf2dOwpSD5rIKC3HgyiXrB0REjSat8ABKdfl1tpOgQUr+9kaIiBqLohPXlvOn62xTJuvw/dmTShoFE5EJBHS4mP8jZFFeZ7vU/F8aKSrbkkTDbkqh6MSVcWMkpZNrLhc6arTYdPZUI0ZFRI0lOT8RGqnmqXpZ6HCl8ABK5bxGjMqGGvEKyLak6MQFVJQLJanmciHLhETqVVe50O7KhJzjUobayoUsExKpW13lQnsqE9oTxSeu2sqFLBMSqV9N5UK7KxMCLBUqSU3lQpYJidSvpnKhBA2S8+yoTAgwcSmJsXIhy4RE9qGmcqGADn8V2FmZkIlLOYyVC1kmJLIfN5cL7bJMaEdUkbiA6uVClgmJ7MfN5UK7LBMCXFWoNFXLhSwTEtmXm8uFdlkmBA9AVpzKcqEQgmVCIjtUWS4UQrbfMiHnuJSnslzIMiGR/aksF0qSnZYJ7YiqEteW86chC8EyIZEdqloutMcyoT1R7PW4jMkoLESbDxfaOgwispGkq3ORdHWurcOwGQnmz1UpZ2lGPROXk38htG6mXf+K6ua9u7mtQ1ClEb5HbB2C6ozzzLJ1CKqTCxlPW/pJG7I6kKsKiYiIrENVpUIiIrvWkNWBCloYwMRFRKQWdpK4WCokIiJF4YiLiEglGnIGDCWdOYOJi4hILeykVMjERUSkFnaSuDjHRUREisIRFxGRSnCOi4iIlMVOzpzBxEVEpBac4yIiIrr1cMRFRKQSnOMiIiJlsZNSIRMXEZFaNGDEpaTExTkuIiJSFI64iIjUgqVCIiJSFDtJXCwVEhGRonDERUSkEvayHJ4jLiIiUhSOuIiI1IJzXERERLcejriIiFTCXua4mLiIiNREQQnIXExcRERqwTkuIiKiWw9HXEREKsE5LiIiUhY7KRUycRERqYS9jLg4x0VERIrCxEVEpBaigTczLF26FCEhIXBxcUFERAT2799v0n5r166FJEl44IEH6t0nExcRkVo0cuJat24d4uLiEB8fj0OHDqF79+6Ijo7G1atXa90vOTkZL730Evr27Vv/TsHERUSkGpVzXObeACA3N9fgVlJSUmN/ixYtwuTJkzFp0iR06dIFy5Ytg5ubGz7//PMa99HpdBg3bhzeeOMNtGnTxqzXycRFRER6wcHB8Pb21t/mzZtntF1paSkOHjyIqKgo/TaNRoOoqCgkJSXV+Pz/+te/4OfnhyeeeMLsGLmqkIhILSywHD41NRVeXl76zc7OzkabZ2ZmQqfTwd/f32C7v78/Tp48aXSf3bt347PPPsORI0fMDLICR1xm0EDCoyE9cJdvCLSSZOtwVKOFawCiA/qjiaNX3Y3JJBI0aO81Av6ud0Dix139LDDH5eXlZXCrKXHVV15eHsaPH4/ly5fDx8enQc/FEZcZ+ga0w1vhwwAAOaVF2PzXMWz56zh+y0yBTijoYIhbTFyHyWjl3gJPth6Nk3nnsCfzAPZlHUJOWa6tQ1Oslu59cJf/awCAYl0OUvISkZKfiKtFRyAg2zg6UjIfHx9otVqkp6cbbE9PT0dAQEC19ufOnUNycjKGDRum3ybLFe9BBwcHnDp1Cm3btjWpbyYuM2iqjLKaOLni0ZAeGNOmJ5NYA1X+XSVJQgfPNujo2RZPtB7FJNYgf4+yXLRN0N57BDo2eZhJTKUa8wBkJycnhIeHIzExUb+kXZZlJCYmIjY2tlr7Tp064c8//zTYNmvWLOTl5eGDDz5AcHCwyX0zcVmAg0YLgEnMkrTS31+4TGKWo5EqPvJMYirVyKd8iouLQ0xMDHr27IlevXohISEBBQUFmDRpEgBgwoQJaNGiBebNmwcXFxeEhoYa7N+kSRMAqLa9LkxcFsYkZnlMYtbBJKY+jX3Kp1GjRiEjIwOzZ89GWloawsLCsGXLFv2CjYsXL0KjsfzcqiRE3d+iubm58Pb2RtsvX4PWzcXiQVhSP/92eDt8mMGXnaU1d3Gv9z7lsg4OGq0+ic05vAm3B162QnSW56RxxIsdn0I7j1ZW7ce7nosyZCEDkCABOJl3Dmsvfo9juacxwveINcKzuJbufXCX32s3XoF1uDg0q/c+siiHRnJAsS4HyXnb8VvGQozzzLJCdFYguQIO7YGyP2wdSZ1y82Q07XAe169fN1jFZ9Zz3fiO7hw7F1pn876jdSXFOLFkpkXisTbVjbjGte0JP1dPW4dRjYNGC1kINHFyxciQOzDn8CZbh2SyVm4tEN70dluHUY2myo+Tzl7t0NmrHY7lnrZhRPXT0ftRuDo0t3UY1WgkBwghV4zEvEbgt4yFtg7JZFKTjyA53w05rb2tQ7ENnh1emZ7eu9bqfdwT2AHLeo82qa1OlqGRJMgQ2J+RjI2px7DjsvFjHG5VZ/KT8cjeZ6zez/thsxHsFmRS28oR7LXS69iTeQBJWQdxOu+ClSO0rMTL063eR7D7AAwIesektrLQ3VgyLyOt6CCS87bjYv7P1g3Q0hzDbB2BbTFxkbkMklXm38nqWmmRrUNTtJqSlVDSJ+4WUzVZpVdJVqUy5wyVSLpxM3dfpWDishDDZJWCjalHmawsgMnK8qonqx1Izf8ZJfJ1W4dGZBImrgbQCRkaMFlZ2t8LWXKxO/M3JisLYLKyEywVUk3O52XiSuF1JOdnY9Nfx7D90klcKy20dViKdyTnODwc3DmysqDrpRdQUHYVuWXJTFZ2wF6ugMzEZYaU/GwM2PyBrcNQnS+S1+OL5PW2DkNVcstS8E3ycFuHQY3FTkZcPOsmEREpCkdcRERqoqCRk7mYuIiIVMJe5rhYKiQiIkXhiIuISC3sZHEGExcRkUrYS6mQiYuISC3sZMTFOS4iIlIUjriIiFSCpUIiIlIWOykVMnEREamFnSQuznEREZGicMRFRKQSnOMiIiJlsZNSIRMXEZFKSEJAEuZlIHP3swXOcRERkaJwxEVEpBYsFRIRkZJwcQYR0a3MoT0k7/cBSPpNksa94r/NNxq2LdkJkb+gEYOzEY64iIhuYU53Q3LsaPQhybGDwX2h8QTsIXHZCS7OICJlKt5iUjMhyoHijXU3VIHKUqG5N6Vg4iIiZZLTIEp/hxByrc0kyQGiaFMjBWVjooE3hWDiIiLFEsU/1N1GdwUo/7MRoqHGwsRFRMpVvAWSVPPXmD2VCQGWComIbn11lAvtqkwIsFRIRKQEtZUL7bFMqPbRFsDERURKV0O5UIhyoKjuOTBSHiYuIlK2GsqFkuQAUbzZRkHZiBANuykEExcRKZ6xciHLhOotFzJxEZHy3VQutNsyIRdnEBEpxE3lQrssE9oRJi4iUoWq5UJ7LBMCgCQ37KYUTFxEpA5Vy4X2WCYEWCokIlKUG+VCAHZbJrSXxRm8rAkRqYbIfw/CsYddlgntSb0SV9eANDi6O1krFrszwveIrUNQpXGeWbYOQXWig8JsHYKJigEkAQizcRx1KxdlAM5b9kkbcjyWgo7j4oiLiEglGlLyY6mQiIgaX0MWWSgocXFxBhERKQpHXEREKsFSIRERKYudLM5gqZCIiBSFIy4iIpVgqZCIiJTFTlYVMnEREamEvYy4OMdFRESKwhEXEZFayKLiZu6+CsHERUSkFpzjIiIiJZHQgDkui0ZiXZzjIiIiReGIi4hILXjmDCIiUhJbXAF56dKlCAkJgYuLCyIiIrB///4a2y5fvhx9+/ZF06ZN0bRpU0RFRdXaviZMXEREaiEaeKundevWIS4uDvHx8Th06BC6d++O6OhoXL161Wj7nTt3YsyYMfjpp5+QlJSE4OBgDBo0CJcuXapXv0xcRERklkWLFmHy5MmYNGkSunTpgmXLlsHNzQ2ff/650farV6/GlClTEBYWhk6dOuHTTz+FLMtITEysV79MXEREKiEJ0aAbAOTm5hrcSkpKjPZVWlqKgwcPIioqSr9No9EgKioKSUlJJsVbWFiIsrIyNGvWrF6vk4mLiEgt5AbeAAQHB8Pb21t/mzdvntGuMjMzodPp4O/vb7Dd398faWlpJoU7Y8YMBAUFGSQ/U3BVIRER6aWmpsLLy0t/39nZ2Sr9vPPOO1i7di127twJFxeXeu3LxEVEpBJVS37m7AsAXl5eBomrJj4+PtBqtUhPTzfYnp6ejoCAgFr3fe+99/DOO+9gx44d6NatW71jZamQiEgtGnFVoZOTE8LDww0WVlQutIiMjKxxv3fffRdvvvkmtmzZgp49e9av0xs44iIiUotGPgA5Li4OMTEx6NmzJ3r16oWEhAQUFBRg0qRJAIAJEyagRYsW+nmy+fPnY/bs2VizZg1CQkL0c2EeHh7w8PAwuV8mLiIiMsuoUaOQkZGB2bNnIy0tDWFhYdiyZYt+wcbFixeh0fxd2Pvoo49QWlqKRx55xOB54uPjMWfOHJP7ZeIiIlIJW1xIMjY2FrGxsUYf27lzp8H95ORk8zq5CRMXEZFa2Mm5Cpm4iIhUQpIrbubuqxRcVUhERIrCERcRkVqwVEhERIpi5lne9fsqBBMXEZFKWOLMGUrAOS4iIlIUjriIiNSCc1xERKQoAvrLk5i1r0IwcRGRonk288DgJ+7FHfd1Q0hoMDybeQCShLzsfFw6cwUn959F0ve/4ejuk7YO1ersZY6LictMEiQIJf1EUQwJivrppwjq/ZsOmRyFZxbGwNWj+vWcmgc2RfPApujWrwtGvjQcIwMn41p6TuMHSRbHxGWGIFd/vNn1RaSXZGJ35gHsyzqE7NIcW4eleL18X0KI5yCk5O9ASt6PSC86DAGdrcNSNod2kJquAnQpEMU/AMVbAfmqraOyiEdfGo6n3h2vvy/LMn7/6RhO/HoGRflF8GzmibbdQxDapxOcXZ1sGGkjEmjAHJdFI7EqJi4zBLr4wdvJC56OHmjnEYLHW4/E6bzzTGIN5O96B5y1XmjnNRwdvB9CiS6XSayhtG0gaZtDaJpCcgwDPGcBZUcUn8Ru69QCT8wdq79/PTMXs0fMx/Gk09Xauri7IGp8P5QUlTZmiLbBxRlUF43099EE7TxCmMQsRCNVvC2ZxCxHqvJeFY7dFZ/EHnhuCLQOWv39D55dbjRpAUBxQTF+WLatsUKjRsDEZSFMYtbBJGZ5kvT3F75Sk1iPe0P1/5+bnY/d3/xqw2huITIqpjTN3VchVJm4JLP/5UyjqeO4bXUmMev+TU3pQ51JzNp/19rfq0pNYs1bNNP//6UzVyCqlLmCOwbh8xMfVNtn28qdWPD40kaJz1a4qlChZnWZhrAmXW0dhl7VJNbeozU6eLbB461H4pG9z9gwqvrxcQnF4OBPbR2GgapJrIP3Q+jg/RB+z1qOP7I/s3FkppOaroLkHGHrMPQMk1gPaJzugPB8DSK9iw2jMoGCvnCtjnNcyrTh0jY0d2oGjRV/ybZwC6j3PjpZB61Gi+zSHOzJPGCFqKwnu+QUkvO2o6lze6v24+0UUq/2stBBggYCMtIKf8OVwt+sE5iViIJlgNbHqn1IDm3rvY8Q5ZAkBwhdOlD0vRWiarisS9lo2SEIANCifaDBYzlXc/Hxy18CACbMGQlX9+pL5UnZVJe4/rx+Ci8cecOqfYQ3vR2vdZ5aZ7ubk9XezIM4m5+suOO/ZFGGXWmvW72fYbetQRPnNnXEYpiskvN3IDX/F5TKuVaPz+JKd0Nk3m/VLoTzIGia1l0eM0hWxf+DXLwFKPvdqrE1xOEfj+oTl1dzT0QO74mk7yt+EOZdy8f6hf8DAIye8YB9JS6OuMgcaklWtxJVJatbiNKSVVUblmzGkCfv068snP7RU7iakolzvyfbNjBbY+IiUzFZWR6TlXUoOVlVlXL8L6ycvRZPzB0HoOIsGUt/ewf7Nx/GmUPnoSvTIaC1H9y83GwcaSPjqkKqiVzllwmTleWIG58cWeiYrCzm71WWSk9WN1v7zncoLijB5PmPwcnFCVoHLSKH9UTksJ5G2+dm5zVyhGQtTFxmOJJzDB+c/hxpxRlMVha088oM+LuGITV/F5OVpZT8CDnnJUCXrIpkdbPvFm/GL18nYchTUbjjvm5o2TEInk3dUV6mw/WMXPx16jKO7j2JpO8P4NyRZFuHa3VcDk81EhDYlbnf1mGoTn7ZJeSXXbJ1GCojgOINtg7CqrLTcrDqX+ux6l/rbR2K7XGOi4iIFEUWgGRmApKVk7hqP6yeiIjoFsMRFxGRWrBUSEREytKAxKWgRWYsFRIRkaJwxEVEpBYsFRIRkaLIAmaX/BS0qpCJi4hILYRccTN3X4XgHBcRESkKR1xERGrBOS4iIlIUznEREZGi2MmIi3NcRESkKBxxERGphUADRlwWjcSqmLiIiNTCTkqFTFxERGohywDMPB5L5nFcREREVsERFxGRWrBUSEREisLERUREimInByBzjouIiBSFIy4iIpUQQoYw8yzv5u5nC0xcRERqIYT5JT8FzXGxVEhERIrCERcRkVqIBizOUNCIi4mLiEgtZBmQ1H8FZCYuIiK1sJMRF+e4iIhIUTjiIiJSCSHLEGaWCrkcnoiIGp+dlAqZuIiI1EIWgKT+xMU5LiIiUhSOuIiI1EIImH0hSQWNuJi4iIhUQsgCwsxSoVBb4qp8QWUFpVYNxt4UuZbbOgRVyjX3FyfVqFyU2ToE1SlHxd/UoglDyDB/xKWcz41JiSsvLw8AsPnB1VYNxt58b+sAVOppWwegSudtHYBq5eXlwdvb29ZhmG3p0qVYsGAB0tLS0L17dyxevBi9evWqsf3XX3+N119/HcnJyWjfvj3mz5+PIUOG1KtPkxJXUFAQUlNT4enpCUmS6tUBERFVJ4RAXl4egoKCLPecjVwqXLduHeLi4rBs2TJEREQgISEB0dHROHXqFPz8/Kq137t3L8aMGYN58+bhH//4B9asWYMHHngAhw4dQmhoqMn9SkJJhU0iIqomNzcX3t7eGIARcJAczXqOclGGndiA69evw8vLy6R9IiIicOedd2LJkiUAAFmWERwcjGnTpuHVV1+t1n7UqFEoKCjADz/8oN921113ISwsDMuWLTM5Vi6HJyJSiXKUoVyYebsx55abm2twKykpMdpXaWkpDh48iKioKP02jUaDqKgoJCUlGd0nKSnJoD0AREdH19i+JlxVSESkcE5OTggICMDutE0Neh4PDw8EBwcbbIuPj8ecOXOqtc3MzIROp4O/v7/Bdn9/f5w8edLo86elpRltn5aWVq84mbiIiBTOxcUFFy5cQGlpw1Z+CyGqrWNwdnZu0HNaAxMXEZEKuLi4wMXFpdH68/HxgVarRXp6usH29PR0BAQEGN0nICCgXu1rwjkuIiKqNycnJ4SHhyMxMVG/TZZlJCYmIjIy0ug+kZGRBu0BYPv27TW2rwlHXEREZJa4uDjExMSgZ8+e6NWrFxISElBQUIBJkyYBACZMmIAWLVpg3rx5AIDp06ejf//+WLhwIYYOHYq1a9fiwIED+OSTT+rVLxMXERGZZdSoUcjIyMDs2bORlpaGsLAwbNmyRb8A4+LFi9Bo/i7s9e7dG2vWrMGsWbMwc+ZMtG/fHt999129juECeBwXEREpDOe4iIhIUZi4iIhIUZi4iIhIUZi4iIhIUZi4iIhIUZi4iIhIUZi4iIhIUZi4iIhIUZi4iIhIUZi4iIhIUZi4iIhIUf4fQ96YzL8naEkAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: value heatmap with greedy-policy arrows\n",
"ARROWS = {0: (0, -0.3), 1: (0.3, 0), 2: (0, 0.3), 3: (-0.3, 0)} # dx,dy per action\n",
"fig, ax = plt.subplots(figsize=(4.4, 4.4))\n",
"im = ax.imshow(V_star.reshape(N, N), cmap=\"viridis\")\n",
"for s in range(S):\n",
" r, c = divmod(s, N)\n",
" if s == GOAL: ax.text(c, r, \"G\", ha=\"center\", va=\"center\", color=\"w\", fontsize=14, fontweight=\"bold\")\n",
" elif s == TRAP: ax.text(c, r, \"X\", ha=\"center\", va=\"center\", color=\"r\", fontsize=14, fontweight=\"bold\")\n",
" else:\n",
" dx, dy = ARROWS[int(pi_star[s])]\n",
" ax.arrow(c, r, dx, dy, head_width=0.12, color=\"w\")\n",
"ax.set_xticks([]); ax.set_yticks([]); ax.set_title(\"V* (color) and greedy policy (arrows)\")\n",
"plt.colorbar(im, fraction=0.046); plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "a2c05309",
"metadata": {},
"source": [
"> **Interpretation.** Value increases monotonically toward the goal cell and dips around the trap. Every arrow points along the gradient of $V^*$: acting greedily on the optimal value function *is* the optimal policy. That equivalence is the whole reason value iteration works.\n",
"\n",
"> **Key takeaways**\n",
"> - An MDP is four tensors and a scalar. The Bellman optimality operator is one `einsum`.\n",
"> - Value iteration is repeated application of that operator; it converges because the operator is a $\\gamma$-contraction.\n",
"> - The optimal policy is `argmax_a Q`. You never store a policy during value iteration; you read it off at the end.\n"
]
},
{
"cell_type": "markdown",
"id": "a0e8d84e",
"metadata": {},
"source": [
"## Part 2 — Planning two ways, and an early-break bug\n",
"\n",
"> **Objectives**\n",
"> - Implement policy iteration with an *exact* linear solve for policy evaluation.\n",
"> - Prove policy iteration and value iteration agree to machine precision (two independent routes to the same fixed point).\n",
"> - Stage and fix the early-break sentinel bug that silently returns a wrong value function.\n",
"\n",
"Policy iteration alternates two steps. *Policy evaluation*: given a fixed policy $\\pi$, compute its value $V^\\pi$. *Policy improvement*: act greedily with respect to $V^\\pi$ to get a new policy. Repeat until the policy stops changing. The clean part is that policy evaluation has a closed form. For a fixed $\\pi$, the Bellman expectation equation is linear:\n",
"\n",
"$$V^\\pi = r^\\pi + \\gamma P^\\pi V^\\pi \\quad\\Longrightarrow\\quad V^\\pi = (I - \\gamma P^\\pi)^{-1} r^\\pi$$\n",
"\n",
"where $P^\\pi$ is the $(S, S')$ transition matrix under $\\pi$ and $r^\\pi(s) = \\sum_{s'} P^\\pi(s,s') R(s,\\pi(s),s')$ is the expected immediate reward. We solve the linear system directly with `np.linalg.solve`.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "9f6f6c55",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.333056Z",
"iopub.status.busy": "2026-06-11T20:07:41.332973Z",
"iopub.status.idle": "2026-06-11T20:07:41.335714Z",
"shell.execute_reply": "2026-06-11T20:07:41.335428Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"exact eval of pi* agrees with V* from value iteration: True\n"
]
}
],
"source": [
"def policy_eval_exact(pi, P, R, gamma):\n",
" '''Exact value of a deterministic policy via the linear Bellman system.'''\n",
" idx = np.arange(S)\n",
" P_pi = P[idx, pi] # (S, S') transitions under pi\n",
" r_pi = np.einsum(\"st,st->s\", P_pi, R[idx, pi]) # (S,) expected immediate reward\n",
" V = np.linalg.solve(np.eye(S) - gamma * P_pi, r_pi)\n",
" return V\n",
"\n",
"# sanity: the value-iteration optimum must satisfy its own Bellman expectation equation\n",
"V_check = policy_eval_exact(pi_star, P, R, 0.9)\n",
"print(\"exact eval of pi* agrees with V* from value iteration:\",\n",
" np.allclose(V_check, V_star, atol=1e-6))"
]
},
{
"cell_type": "markdown",
"id": "77156c5d",
"metadata": {},
"source": [
"### Exercise 19.2 — Policy iteration\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"Fill in `policy_iteration`. Start from the all-\"up\" policy. Loop: evaluate the current policy exactly, then improve it greedily; stop when the policy is unchanged. The check that matters: the value it returns must equal value iteration's to machine precision. Two completely different algorithms, one fixed point.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "eb36fe2d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.336600Z",
"iopub.status.busy": "2026-06-11T20:07:41.336522Z",
"iopub.status.idle": "2026-06-11T20:07:41.340784Z",
"shell.execute_reply": "2026-06-11T20:07:41.340378Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 19.2 PI == VI value: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 19.2 PI optimal policy: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def policy_iteration(P, R, gamma=0.9, max_iters=1000):\n",
" \"\"\"Return (V (S,), pi (S,)). Exact policy evaluation + greedy improvement.\"\"\"\n",
" pi = np.zeros(S, dtype=int) # start: always \"up\"\n",
" for _ in range(max_iters):\n",
" # TODO 1: V = exact value of the current policy (use policy_eval_exact)\n",
" V = None\n",
" attempted(V)\n",
" # TODO 2: greedy improvement: Q = bellman_operator(V, P, R, gamma); new_pi = argmax over actions\n",
" new_pi = None\n",
" attempted(new_pi)\n",
" # TODO 3: if new_pi equals pi, we are done: return (V, pi). Otherwise set pi = new_pi.\n",
" raise NotImplementedError # remove once the TODOs are done\n",
" return policy_eval_exact(pi, P, R, gamma), pi\n",
"\n",
"def _pi_matches_vi():\n",
" Vpi, pipi = policy_iteration(P, R, gamma=0.9)\n",
" Vvi, pivi = value_iteration(P, R, gamma=0.9)\n",
" diff = np.abs(Vpi - Vvi).max()\n",
" assert diff < 1e-8, \\\n",
" f\"policy iteration and value iteration differ by {diff:.2e}; they must hit the same fixed point\"\n",
"\n",
"def _pi_optimal():\n",
" Vpi, pipi = policy_iteration(P, R, gamma=0.9)\n",
" assert pipi[14] == 1 and pipi[11] == 2, \\\n",
" f\"policy-iteration policy pi[14]={pipi[14]} (want 1), pi[11]={pipi[11]} (want 2)\"\n",
"\n",
"check(\"19.2 PI == VI value\", _pi_matches_vi)\n",
"check(\"19.2 PI optimal policy\", _pi_optimal)"
]
},
{
"cell_type": "markdown",
"id": "29d45665",
"metadata": {},
"source": [
"Hint 1 (conceptual)The loop body is two lines plus a stopping test. `V = policy_eval_exact(pi, ...)`; `new_pi = bellman_operator(V, ...).argmax(1)`. Compare arrays with `(new_pi == pi).all()`.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"V = policy_eval_exact(pi, P, R, gamma)\n",
"new_pi = bellman_operator(V, P, R, gamma).argmax(axis=1)\n",
"if (new_pi == pi).all():\n",
" return V, pi\n",
"pi = new_pi\n",
"```\n",
"\n",
"\n",
"Help — \"values differ by ~0.01, not ~1e-16\"You probably returned `bellman_operator(V,...).max(1)` as the value instead of the *exact* policy evaluation `V`. Policy iteration's value at convergence is the exact value of the converged policy. Return the `V` from `policy_eval_exact`, not a one-step backup of it.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "b1df8517",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.341499Z",
"iopub.status.busy": "2026-06-11T20:07:41.341423Z",
"iopub.status.idle": "2026-06-11T20:07:41.345147Z",
"shell.execute_reply": "2026-06-11T20:07:41.344697Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 19.2 PI == VI value\n",
"[ ok ] 19.2 PI optimal policy\n",
"max |V_policy_iteration - V_value_iteration| = 1.11e-16\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines policy_iteration; the checks below re-verify the reference.\n",
"def policy_iteration(P, R, gamma=0.9, max_iters=1000):\n",
" pi = np.zeros(S, dtype=int)\n",
" for _ in range(max_iters):\n",
" V = policy_eval_exact(pi, P, R, gamma)\n",
" new_pi = bellman_operator(V, P, R, gamma).argmax(axis=1)\n",
" if (new_pi == pi).all():\n",
" return V, pi\n",
" pi = new_pi\n",
" return policy_eval_exact(pi, P, R, gamma), pi\n",
"\n",
"check(\"19.2 PI == VI value\", _pi_matches_vi, required=True)\n",
"check(\"19.2 PI optimal policy\", _pi_optimal, required=True)\n",
"\n",
"V_pi, pi_pi = policy_iteration(P, R, gamma=0.9)\n",
"print(\"max |V_policy_iteration - V_value_iteration| =\",\n",
" f\"{np.abs(V_pi - V_star).max():.2e}\")"
]
},
{
"cell_type": "markdown",
"id": "9c825fd5",
"metadata": {},
"source": [
"> **Interpretation.** Two algorithms with nothing in common except the MDP land on the same value function to within $10^{-15}$. That is the practical meaning of \"the Bellman optimality equation has a unique solution.\" When two independent methods agree to machine precision, you have strong evidence both are correct, which is the cheapest correctness oracle in the chapter.\n"
]
},
{
"cell_type": "markdown",
"id": "04c1123f",
"metadata": {},
"source": [
"### A deliberate failure: the early-break sentinel\n",
"\n",
"Iterative algorithms live or die by their stopping rule. Here is the canonical RL version of the same bug that bites k-means: check convergence at the *top* of the loop against a `V_prev` that was initialized to the *same* zeros as `V`. On iteration one, `V` equals `V_prev` (both all-zero), the check fires, and the function returns before a single Bellman backup. The result looks like a value function. It is all zeros. Watch it happen, then read the fix.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "8132e08e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.346162Z",
"iopub.status.busy": "2026-06-11T20:07:41.346086Z",
"iopub.status.idle": "2026-06-11T20:07:41.349082Z",
"shell.execute_reply": "2026-06-11T20:07:41.348771Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"BROKEN: V is all zero: True · Bellman residual: 1.0\n",
"the residual equals the goal reward (1.0): the loop never ran a backup\n"
]
}
],
"source": [
"# This is intentionally WRONG. The convergence check fires on iteration 1.\n",
"def value_iteration_BROKEN(P, R, gamma=0.9, tol=1e-9, max_iters=1000):\n",
" V = np.zeros(S)\n",
" V_prev = np.zeros(S) # same zeros as V\n",
" for _ in range(max_iters):\n",
" if np.abs(V - V_prev).max() < tol: # BUG: true on iteration 1 (both all-zero)\n",
" break\n",
" V_prev = V.copy()\n",
" V = bellman_operator(V, P, R, gamma).max(1)\n",
" return V\n",
"\n",
"V_bad = value_iteration_BROKEN(P, R, gamma=0.9)\n",
"resid_bad = np.abs(bellman_operator(V_bad, P, R, 0.9).max(1) - V_bad).max()\n",
"print(\"BROKEN: V is all zero:\", np.allclose(V_bad, 0.0), \"· Bellman residual:\", round(resid_bad, 3))\n",
"print(\"the residual equals the goal reward (1.0): the loop never ran a backup\")"
]
},
{
"cell_type": "markdown",
"id": "4bdb7cf1",
"metadata": {},
"source": [
"> **Common confusion:** the broken function does not error. It returns an array of the right shape full of plausible-looking zeros, and a greedy policy derived from it would be garbage. Silent wrong answers are the dangerous kind. The only thing that catches this is a *property* check, here the Bellman residual, which is exactly why Exercise 19.1's check asserts the residual is zero rather than just checking the shape.\n",
"\n",
"The fix is to do at least one real backup before you are allowed to stop. Either check convergence *after* computing `V_new` from a backup (as in Exercise 19.1), or seed `V_prev` with something that cannot equal `V` on the first pass.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "45099c1a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.349982Z",
"iopub.status.busy": "2026-06-11T20:07:41.349917Z",
"iopub.status.idle": "2026-06-11T20:07:41.352856Z",
"shell.execute_reply": "2026-06-11T20:07:41.352506Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"FIXED: matches V* : True · Bellman residual: 0.00e+00\n"
]
}
],
"source": [
"def value_iteration_FIXED(P, R, gamma=0.9, tol=1e-9, max_iters=1000):\n",
" V = np.zeros(S)\n",
" for _ in range(max_iters):\n",
" V_new = bellman_operator(V, P, R, gamma).max(1) # backup FIRST\n",
" if np.abs(V_new - V).max() < tol: # then test\n",
" V = V_new\n",
" break\n",
" V = V_new\n",
" return V\n",
"\n",
"V_fixed = value_iteration_FIXED(P, R, gamma=0.9)\n",
"resid_fixed = np.abs(bellman_operator(V_fixed, P, R, 0.9).max(1) - V_fixed).max()\n",
"print(\"FIXED: matches V* :\", np.allclose(V_fixed, V_star, atol=1e-6),\n",
" \"· Bellman residual:\", f\"{resid_fixed:.2e}\")\n",
"assert resid_fixed < 1e-6, \"the fixed version must reach the true fixed point\""
]
},
{
"cell_type": "markdown",
"id": "6cd89a14",
"metadata": {},
"source": [
"> **Key takeaways**\n",
"> - Policy evaluation for a fixed policy is a linear system; you can solve it exactly, no iteration.\n",
"> - Value iteration and policy iteration converge to the same unique fixed point; their agreement is a free correctness check.\n",
"> - Convergence checks belong *after* a real update. The early-break sentinel returns a wrong answer with no error; only a property check catches it.\n"
]
},
{
"cell_type": "markdown",
"id": "37f46338",
"metadata": {},
"source": [
"## Part 3 — Learning without the model: Q-learning\n",
"\n",
"> **Objectives**\n",
"> - Estimate $Q(s,a)$ from sampled transitions, with no access to $P$ or $R$ as tensors.\n",
"> - Recover the value-iteration optimal policy by interaction alone.\n",
"> - See concretely why $\\epsilon$-greedy exploration is required even in a fully observable world.\n",
"\n",
"Value iteration needed the full $P$ and $R$. The agent rarely has those. *Q-learning* (Watkins, 1989) learns $Q(s,a)$ from sampled transitions $(s, a, r, s')$ with the temporal-difference update:\n",
"\n",
"$$Q(s,a) \\leftarrow Q(s,a) + \\alpha\\left[r + \\gamma \\max_{a'} Q(s',a') - Q(s,a)\\right]$$\n",
"\n",
"The bracket is the *TD error*: the gap between the bootstrapped estimate $r + \\gamma \\max_{a'} Q(s',a')$ and the current $Q(s,a)$. It is *off-policy*: the target uses $\\max_{a'}$ regardless of which action the behavior policy actually takes next. We behave $\\epsilon$-greedily (random with probability $\\epsilon$, greedy otherwise) and decay $\\epsilon$ over training.\n",
"\n",
"First, a sampler that returns one transition. This is the only interface Q-learning gets to the world.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "18586d10",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.353739Z",
"iopub.status.busy": "2026-06-11T20:07:41.353670Z",
"iopub.status.idle": "2026-06-11T20:07:41.355896Z",
"shell.execute_reply": "2026-06-11T20:07:41.355519Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"sample (state 14, right): (15, 1.0, True) (into goal, reward 1, done)\n"
]
}
],
"source": [
"def grid_sample(s, a, rng):\n",
" '''Sample one transition (next_state, reward, done) from the true MDP.'''\n",
" ns = grid_step(s, a)\n",
" if s in TERMINAL:\n",
" return ns, 0.0, True\n",
" if ns == GOAL: r = 1.0\n",
" elif ns == TRAP: r = -1.0\n",
" else: r = STEP_COST\n",
" return ns, r, ns in TERMINAL\n",
"\n",
"print(\"sample (state 14, right):\", grid_sample(14, 1, rng), \"(into goal, reward 1, done)\")"
]
},
{
"cell_type": "markdown",
"id": "102ad874",
"metadata": {},
"source": [
"### Exercise 19.3 — The Q-learning TD update\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"Fill in `td_update`: given the table `Q`, a transition, and hyperparameters, return the new value for `Q[s,a]`. Use a bootstrap target of $r + \\gamma \\max_{a'} Q(s',a')$ for non-terminal `s'`, and just $r$ when `done` (no future to bootstrap from). The hand-computed check below pins the arithmetic.\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "f06c859f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.356832Z",
"iopub.status.busy": "2026-06-11T20:07:41.356764Z",
"iopub.status.idle": "2026-06-11T20:07:41.360486Z",
"shell.execute_reply": "2026-06-11T20:07:41.360242Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 19.3 TD update (bootstrap): not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 19.3 TD update (terminal): not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def td_update(Q, s, a, r, s_next, done, alpha, gamma):\n",
" \"\"\"Return the updated scalar Q[s,a] after one TD step (does not mutate Q).\"\"\"\n",
" # TODO 1: bootstrap = 0.0 if done else gamma * max over actions of Q[s_next]\n",
" bootstrap = None\n",
" attempted(bootstrap)\n",
" # TODO 2: target = r + bootstrap ; td_error = target - Q[s,a]\n",
" td_error = None\n",
" attempted(td_error)\n",
" # TODO 3: return Q[s,a] + alpha * td_error\n",
" raise NotImplementedError # remove once the TODOs are done\n",
"\n",
"def _td_toy():\n",
" Q = np.array([[0.0, 0.0], [1.0, 0.5]])\n",
" # s=0,a=0,r=0, s'=1 (max Q[1]=1.0), gamma=0.9, alpha=1.0 -> target 0.9, new value 0.9\n",
" got = td_update(Q, 0, 0, 0.0, 1, False, alpha=1.0, gamma=0.9)\n",
" assert abs(got - 0.9) < 1e-9, f\"got {got}, expected 0.9 = 0 + 0.9*max(Q[1])=0.9*1.0\"\n",
"\n",
"def _td_done():\n",
" Q = np.array([[0.2, 0.0]])\n",
" # done -> no bootstrap; target is just r=1.0; alpha=0.5 -> 0.2 + 0.5*(1.0-0.2)=0.6\n",
" got = td_update(Q, 0, 0, 1.0, 0, True, alpha=0.5, gamma=0.9)\n",
" assert abs(got - 0.6) < 1e-9, f\"got {got}, expected 0.6: terminal target is r only, no gamma*max\"\n",
"\n",
"check(\"19.3 TD update (bootstrap)\", _td_toy)\n",
"check(\"19.3 TD update (terminal)\", _td_done)"
]
},
{
"cell_type": "markdown",
"id": "d7675193",
"metadata": {},
"source": [
"Hint 1 (conceptual)The only subtlety is the terminal case. When `done`, there is no next state to bootstrap from, so the target is exactly `r`. Otherwise add `gamma * Q[s_next].max()`.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"bootstrap = 0.0 if done else gamma * Q[s_next].max()\n",
"td_error = (r + bootstrap) - Q[s, a]\n",
"return Q[s, a] + alpha * td_error\n",
"```\n",
"\n",
"\n",
"Help — \"the agent learns but never quite reaches the goal value\"You are bootstrapping through terminal states. If `done` is True and you still add `gamma * Q[s_next].max()`, the absorbing state leaks a nonzero value back and inflates every estimate. Gate the bootstrap on `not done`.\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "e776f204",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.361341Z",
"iopub.status.busy": "2026-06-11T20:07:41.361274Z",
"iopub.status.idle": "2026-06-11T20:07:41.363337Z",
"shell.execute_reply": "2026-06-11T20:07:41.363076Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 19.3 TD update (bootstrap)\n",
"[ ok ] 19.3 TD update (terminal)\n",
"[ ok ] TD arithmetic pinned by two hand-computed values\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines td_update; the checks below re-verify the reference.\n",
"def td_update(Q, s, a, r, s_next, done, alpha, gamma):\n",
" bootstrap = 0.0 if done else gamma * Q[s_next].max()\n",
" td_error = (r + bootstrap) - Q[s, a]\n",
" return Q[s, a] + alpha * td_error\n",
"\n",
"check(\"19.3 TD update (bootstrap)\", _td_toy, required=True)\n",
"check(\"19.3 TD update (terminal)\", _td_done, required=True)\n",
"print(\"[ ok ] TD arithmetic pinned by two hand-computed values\")"
]
},
{
"cell_type": "markdown",
"id": "c69555e5",
"metadata": {},
"source": [
"Now the training loop. Re-seed at the top so re-running this cell in isolation reproduces the same run. Start each episode from a random non-terminal state, act $\\epsilon$-greedily, update with `td_update`, decay $\\epsilon$.\n",
"\n",
"> **Runtime:** this cell takes ~5-15 s on CPU (fewer episodes under `NB_FAST`).\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "806a9774",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.364328Z",
"iopub.status.busy": "2026-06-11T20:07:41.364263Z",
"iopub.status.idle": "2026-06-11T20:07:41.396918Z",
"shell.execute_reply": "2026-06-11T20:07:41.396581Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"trained 1500 episodes · final eps 0.223\n",
"policy match vs value-iteration optimum: 100% of non-terminal states\n"
]
}
],
"source": [
"# scale-up via QL_EPISODES in CONFIG; re-seed so the cell is reproducible standalone\n",
"rng = np.random.default_rng(SEED)\n",
"Q = np.zeros((S, A))\n",
"alpha, gamma, eps = 0.5, 0.9, 1.0\n",
"for ep in range(QL_EPISODES):\n",
" s = int(rng.integers(0, S))\n",
" while s in TERMINAL:\n",
" s = int(rng.integers(0, S))\n",
" for _ in range(50): # episode step cap\n",
" a = int(rng.integers(0, A)) if rng.random() < eps else int(Q[s].argmax())\n",
" s_next, r, done = grid_sample(s, a, rng)\n",
" Q[s, a] = td_update(Q, s, a, r, s_next, done, alpha, gamma)\n",
" s = s_next\n",
" if done:\n",
" break\n",
" eps = max(0.05, eps * 0.999) # decay exploration\n",
"print(f\"trained {QL_EPISODES} episodes · final eps {eps:.3f}\")\n",
"\n",
"pi_q = Q.argmax(1)\n",
"nonterminal = [s for s in range(S) if s not in TERMINAL]\n",
"match = np.mean([pi_q[s] == pi_star[s] for s in nonterminal])\n",
"print(f\"policy match vs value-iteration optimum: {match:.0%} of non-terminal states\")\n",
"assert match >= 0.9, \"Q-learning should recover almost all of the optimal policy\""
]
},
{
"cell_type": "markdown",
"id": "de9ce654",
"metadata": {},
"source": [
"> **Interpretation.** With no access to $P$ or $R$ as tensors, sampling transitions alone, Q-learning recovers the value-iteration policy on essentially every state. The optimal policy was learnable from interaction. The value-iteration answer from Part 1 is the ground-truth oracle that tells us the learned policy is right.\n",
"\n",
"Why does exploration matter? A purely greedy agent ($\\epsilon = 0$) commits to whatever looks best early and never discovers that a different action was better. Let us measure it.\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "1af06335",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.397735Z",
"iopub.status.busy": "2026-06-11T20:07:41.397667Z",
"iopub.status.idle": "2026-06-11T20:07:41.438467Z",
"shell.execute_reply": "2026-06-11T20:07:41.437937Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"explore (eps->0.05) -> 100% optimal | greedy (eps=0) -> 71% optimal\n",
"the explored policy matches the optimum more often than the greedy-collapsed one\n"
]
}
],
"source": [
"# deeper: greedy-only (eps=0) Q-learning gets stuck; eps-greedy explores out of it\n",
"def train_q(epsilon, episodes, seed=SEED, decay=False):\n",
" g = np.random.default_rng(seed)\n",
" Q = np.zeros((S, A)); eps = (1.0 if decay else epsilon)\n",
" for ep in range(episodes):\n",
" s = int(g.integers(0, S))\n",
" while s in TERMINAL:\n",
" s = int(g.integers(0, S))\n",
" for _ in range(50):\n",
" a = int(g.integers(0, A)) if g.random() < eps else int(Q[s].argmax())\n",
" s_next, r, done = grid_sample(s, a, g)\n",
" Q[s, a] = td_update(Q, s, a, r, s_next, done, 0.5, 0.9)\n",
" s = s_next\n",
" if done: break\n",
" if decay:\n",
" eps = max(epsilon, eps * 0.999)\n",
" return Q.argmax(1)\n",
"\n",
"eps_episodes = QL_EPISODES\n",
"pi_explore = train_q(0.05, eps_episodes, decay=True) # anneal from 1.0 down to a 0.05 floor: keeps exploring\n",
"pi_greedy = train_q(0.0, eps_episodes, decay=False) # eps=0 the entire run: truly greedy from the start, never explores\n",
"m_explore = np.mean([pi_explore[s] == pi_star[s] for s in nonterminal])\n",
"m_greedy = np.mean([pi_greedy[s] == pi_star[s] for s in nonterminal])\n",
"print(f\"explore (eps->0.05) -> {m_explore:.0%} optimal | greedy (eps=0) -> {m_greedy:.0%} optimal\")\n",
"print(\"the explored policy matches the optimum more often than the greedy-collapsed one\")\n",
"assert m_greedy < m_explore, \"greedy-from-the-start must underperform the exploring agent\"\n",
"assert m_explore - m_greedy > 0.1, \"the explore-vs-greedy gap should be large enough to make the point\""
]
},
{
"cell_type": "markdown",
"id": "a40363a5",
"metadata": {},
"source": [
"> **Common confusion:** \"the world is fully observable, so why explore?\" Observability is about the *state*, not the *values*. You can see exactly where you are and still have wrong $Q$ estimates because you never tried the better action. Exploration is about coverage of the action space under uncertainty, not about hidden state.\n",
"\n",
"> **Key takeaways**\n",
"> - Q-learning learns $Q$ from sampled transitions with a one-line TD update; no model needed.\n",
"> - The bootstrap target uses `max` over next actions (off-policy), and drops to just `r` at terminal states.\n",
"> - $\\epsilon$-greedy exploration is required even in a fully observable MDP: greedy-from-the-start collapses onto early mistakes.\n"
]
},
{
"cell_type": "markdown",
"id": "be4c69ed",
"metadata": {},
"source": [
"## Part 4 — Policy gradients: REINFORCE and PPO\n",
"\n",
"> **Objectives**\n",
"> - Derive the REINFORCE gradient from the log-derivative trick and verify it against finite differences.\n",
"> - Build a self-contained CartPole (no gym) and train it with REINFORCE.\n",
"> - Implement GAE and the PPO clipped objective; train CartPole past 200 steps.\n",
"\n",
"Value methods learn $Q$ and act greedily. *Policy-gradient* methods parameterize the policy $\\pi_\\theta(a\\mid s)$ directly and push $\\theta$ toward high-return actions. The policy gradient theorem:\n",
"\n",
"$$\\nabla_\\theta J(\\theta) = \\mathbb{E}_{\\tau\\sim\\pi_\\theta}\\left[\\sum_t \\nabla_\\theta \\log \\pi_\\theta(a_t\\mid s_t)\\, G_t\\right]$$\n",
"\n",
"where $G_t = \\sum_{t'\\geq t}\\gamma^{t'-t} r_{t'}$ is the return from step $t$. The derivation is the *log-derivative trick*: $\\nabla p = p\\,\\nabla\\log p$, applied to the trajectory distribution. This is REINFORCE (Williams, 1992): unbiased, high variance.\n",
"\n",
"Before any training, we verify the gradient itself. For a one-layer logistic policy with a binary action, $\\frac{\\partial}{\\partial\\,\\text{logit}}\\log\\pi(a) = a - p$ where $p = \\sigma(\\text{logit})$. We check that against a finite-difference estimate. If the analytic gradient is wrong, no amount of training will save you.\n"
]
},
{
"cell_type": "markdown",
"id": "e4f7a2e3",
"metadata": {},
"source": [
"### Exercise 19.4 — The policy gradient by hand\n",
"`Difficulty 3/5 · ~15 min`\n",
"\n",
"For a linear logistic policy $\\pi(a{=}1\\mid s) = \\sigma(w^\\top s)$, fill in `logprob_grad`: return $\\nabla_w \\log\\pi(a\\mid s)$. The identity is $\\nabla_w \\log\\pi(a\\mid s) = (a - p)\\,s$ where $p = \\sigma(w^\\top s)$. The check compares your analytic gradient to a central finite difference of $\\log\\pi$; they must match to $10^{-6}$.\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "85909b86",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.439429Z",
"iopub.status.busy": "2026-06-11T20:07:41.439348Z",
"iopub.status.idle": "2026-06-11T20:07:41.443551Z",
"shell.execute_reply": "2026-06-11T20:07:41.443072Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 19.4 log-prob gradient vs finite diff: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def logprob_grad(w, s, action):\n",
" \"\"\"Gradient wrt w of log pi(action | s) for a linear logistic policy.\n",
" w: (d,), s: (d,), action in {0,1}. Returns (d,).\"\"\"\n",
" p = 1.0 / (1.0 + np.exp(-(s @ w))) # P(action=1)\n",
" # TODO 1: the gradient of log pi(action) wrt the logit is (action - p)\n",
" # TODO 2: chain through the logit = w @ s, whose gradient wrt w is s\n",
" grad = None\n",
" attempted(grad)\n",
" return grad\n",
"\n",
"def _grad_vs_finite_diff():\n",
" g = np.random.default_rng(1)\n",
" w = g.normal(size=4); s = g.normal(size=4); action = 1\n",
" def logpi(w_):\n",
" p = 1.0 / (1.0 + np.exp(-(s @ w_)))\n",
" return np.log(p if action == 1 else 1 - p)\n",
" fd = np.zeros(4); eps = 1e-6\n",
" for i in range(4):\n",
" wp = w.copy(); wp[i] += eps\n",
" wm = w.copy(); wm[i] -= eps\n",
" fd[i] = (logpi(wp) - logpi(wm)) / (2 * eps)\n",
" err = np.abs(logprob_grad(w, s, action) - fd).max()\n",
" assert err < 1e-6, f\"analytic gradient disagrees with finite differences by {err:.2e}\"\n",
"\n",
"check(\"19.4 log-prob gradient vs finite diff\", _grad_vs_finite_diff)"
]
},
{
"cell_type": "markdown",
"id": "09198d1b",
"metadata": {},
"source": [
"Hint 1 (conceptual)The logit is $z = w^\\top s$. $\\frac{d\\log\\pi(a)}{dz} = a - p$. Then $\\frac{dz}{dw} = s$. Multiply: $(a - p)\\, s$.\n",
"\n",
"Hint 2 (the line)`grad = (action - p) * s`. That single product is the whole policy gradient for one logistic action.\n",
"\n",
"Help — \"off by a sign\"Check the direction: increasing $\\log\\pi(a{=}1)$ should pull the logit up, so the gradient for $a{=}1$ is $(1 - p)\\,s$, positive in the direction of $s$. If yours is $(p - a)\\,s$ you flipped the sign; that is the negative gradient.\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "dbaed6b1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.444599Z",
"iopub.status.busy": "2026-06-11T20:07:41.444523Z",
"iopub.status.idle": "2026-06-11T20:07:41.446771Z",
"shell.execute_reply": "2026-06-11T20:07:41.446329Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 19.4 log-prob gradient vs finite diff\n",
"[ ok ] the from-scratch policy gradient matches finite differences to 1e-6\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines logprob_grad; the check below re-verifies it.\n",
"def logprob_grad(w, s, action):\n",
" p = 1.0 / (1.0 + np.exp(-(s @ w)))\n",
" return (action - p) * s\n",
"\n",
"check(\"19.4 log-prob gradient vs finite diff\", _grad_vs_finite_diff, required=True)\n",
"print(\"[ ok ] the from-scratch policy gradient matches finite differences to 1e-6\")"
]
},
{
"cell_type": "markdown",
"id": "5c256486",
"metadata": {},
"source": [
"> **Interpretation.** The gradient that drives every policy-gradient method in this chapter is $(a - p)\\,s$, and it is provably correct against a numerical derivative. Karpathy's *Pong from Pixels* is this exact gradient applied for six million frames. We now hand the same computation to PyTorch autograd and trust it, because we checked the primitive.\n",
"\n",
"Now a self-contained CartPole. Cart on a track, pole hinged on top; push left (0) or right (1) each step; reward $+1$ per step alive; episode ends if the pole tips past 12 degrees, the cart leaves the track, or 500 steps elapse. The physics are the classic Barto-Sutton-Anderson equations, no gym install.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "fa22ab57",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.447451Z",
"iopub.status.busy": "2026-06-11T20:07:41.447384Z",
"iopub.status.idle": "2026-06-11T20:07:41.453878Z",
"shell.execute_reply": "2026-06-11T20:07:41.453345Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"random policy survives ~23 steps on average (out of 500)\n"
]
}
],
"source": [
"class CartPole:\n",
" '''Self-contained CartPole. Classic continuous physics, discrete actions.'''\n",
" g, mc, mp, length, fmag, dt = 9.8, 1.0, 0.1, 0.5, 10.0, 0.02\n",
" x_thresh, theta_thresh = 2.4, 12 * np.pi / 180\n",
"\n",
" def __init__(self, rng):\n",
" self.rng = rng\n",
" def reset(self):\n",
" self.state = self.rng.uniform(-0.05, 0.05, size=4) # x, x_dot, theta, theta_dot\n",
" self.steps = 0\n",
" return self.state.copy()\n",
" def step(self, action):\n",
" x, x_dot, theta, theta_dot = self.state\n",
" force = self.fmag if action == 1 else -self.fmag\n",
" ct, st = np.cos(theta), np.sin(theta)\n",
" total = self.mc + self.mp\n",
" temp = (force + self.mp * self.length * theta_dot**2 * st) / total\n",
" theta_acc = (self.g * st - ct * temp) / (self.length * (4/3 - self.mp * ct**2 / total))\n",
" x_acc = temp - self.mp * self.length * theta_acc * ct / total\n",
" x += self.dt * x_dot; x_dot += self.dt * x_acc\n",
" theta += self.dt * theta_dot; theta_dot += self.dt * theta_acc\n",
" self.state = np.array([x, x_dot, theta, theta_dot]); self.steps += 1\n",
" done = bool(abs(x) > self.x_thresh or abs(theta) > self.theta_thresh or self.steps >= 500)\n",
" return self.state.copy(), 1.0, done\n",
"\n",
"env = CartPole(np.random.default_rng(SEED))\n",
"lengths = []\n",
"for _ in range(20):\n",
" env.reset(); done = False; L = 0\n",
" while not done:\n",
" _, _, done = env.step(int(env.rng.integers(0, 2))); L += 1\n",
" lengths.append(L)\n",
"print(f\"random policy survives ~{np.mean(lengths):.0f} steps on average (out of 500)\")"
]
},
{
"cell_type": "markdown",
"id": "c1415c37",
"metadata": {},
"source": [
"A random policy lasts about 20 steps. Now REINFORCE: sample a full episode, compute returns-to-go, normalize them (a baseline that reduces variance without biasing the gradient), and step in the direction of $\\nabla\\log\\pi \\cdot G$.\n",
"\n",
"> **Runtime:** this cell takes ~5-15 s on CPU.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "38aac385",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:41.454678Z",
"iopub.status.busy": "2026-06-11T20:07:41.454604Z",
"iopub.status.idle": "2026-06-11T20:07:42.805394Z",
"shell.execute_reply": "2026-06-11T20:07:42.804697Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"REINFORCE: first-20 mean length 38, last-20 mean length 139\n"
]
}
],
"source": [
"# re-seed so the run reproduces standalone\n",
"torch.manual_seed(SEED)\n",
"pg_env = CartPole(np.random.default_rng(SEED))\n",
"\n",
"class PolicyNet(torch.nn.Module):\n",
" def __init__(self):\n",
" super().__init__()\n",
" self.net = torch.nn.Sequential(\n",
" torch.nn.Linear(4, 64), torch.nn.Tanh(), torch.nn.Linear(64, 2))\n",
" def forward(self, x): return self.net(x)\n",
"\n",
"def returns_to_go(rewards, gamma=0.99):\n",
" out, g = [], 0.0\n",
" for r in reversed(rewards):\n",
" g = r + gamma * g\n",
" out.insert(0, g)\n",
" return out\n",
"\n",
"policy = PolicyNet()\n",
"opt = torch.optim.Adam(policy.parameters(), lr=1e-2)\n",
"pg_lengths = []\n",
"for ep in range(PG_EPISODES):\n",
" s = pg_env.reset(); logps, rews = [], []; done = False\n",
" while not done:\n",
" dist = Categorical(logits=policy(torch.tensor(s, dtype=torch.float32)))\n",
" a = dist.sample(); logps.append(dist.log_prob(a))\n",
" s, r, done = pg_env.step(int(a.item())); rews.append(r)\n",
" G = torch.tensor(returns_to_go(rews), dtype=torch.float32)\n",
" G = (G - G.mean()) / (G.std() + 1e-8) # variance-reducing baseline\n",
" loss = -(torch.stack(logps) * G).sum() # the policy-gradient loss\n",
" opt.zero_grad(); loss.backward(); opt.step() # forward / backward / update\n",
" pg_lengths.append(len(rews))\n",
"\n",
"window = 20\n",
"print(f\"REINFORCE: first-{window} mean length {np.mean(pg_lengths[:window]):.0f}, \"\n",
" f\"last-{window} mean length {np.mean(pg_lengths[-window:]):.0f}\")\n",
"assert np.mean(pg_lengths[-window:]) > np.mean(pg_lengths[:window]) + 10, \\\n",
" \"REINFORCE should clearly improve episode length over training\""
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "2da4aee2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:42.806507Z",
"iopub.status.busy": "2026-06-11T20:07:42.806369Z",
"iopub.status.idle": "2026-06-11T20:07:42.864511Z",
"shell.execute_reply": "2026-06-11T20:07:42.864091Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAEiCAYAAAAPh11JAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAgNBJREFUeJztnXd4U2X7x79p2qZ70klbWqALZJSyypC9RGQoAqIMURygDMHN/mkFREBBQN9XhqKgKPgqyhApIJsiu7RQCi3QRUt305E+vz+enow2SZM2aZJyf64rV8Y5Oec5p2nON/d9P99bxBhjIAiCIAiCIOrEytQDIAiCIAiCsBRIOBEEQRAEQegICSeCIAiCIAgdIeFEEARBEAShIyScCIIgCIIgdISEE0EQBEEQhI6QcCIIgiAIgtAREk4EQRAEQRA6QsKJIAiCIAhCR0g4EQRBEAbl9u3bEIlE2LJli6mHQhAGh4QTQZgBW7ZsgUgkkt+sra3RvHlzTJkyBffu3au1ft++fVXWV75FRETU2u65c+fkry1evBgikQg+Pj4oKSmpte3g4GA8+eSTKq9p2pevr2+t9x8/fhyjR4+Gj48PJBIJgoOD8corryA1NbXWusJYhJuNjQ2Cg4Px5ptvIi8vT+252r17N4YNG4ZmzZrB1tYW/v7+ePbZZ/H333/L14mLi9M4ZpFIhB07dqjdtqnQ5ZgMwbVr17B48WLcvn271rKanykPDw906dIF33zzDaqqqgw6DoKwZKxNPQCCIBQsXboUISEhkEqlOHXqFLZs2YJ//vkHV65cgZ2dncq6AQEBiI2NrbUNV1dXnfaVlZWFDRs24K233tJp/UGDBmHSpEkqr9nb26s8/+KLLzBr1iy0bNkSb7zxBvz8/JCQkID//Oc/2LlzJ/744w/06NGj1rY3bNgAJycnFBcX49ChQ/jiiy9w/vx5/PPPP/J1GGN48cUXsWXLFkRFRWHu3Lnw9fVFeno6du/ejQEDBuD48eMq23/zzTfRpUuXWvuLiYnR6ZiNTX2OqSFcu3YNS5YsQd++fREcHFxrufJnKjs7G9u2bcO0adOQlJSETz75xCBjIAiLhxEEYXI2b97MALCzZ8+qvP7OO+8wAGznzp0qr/fp04e1bdu2XttdtGgRA8A6duzIfHx8WElJicp7WrRowYYPH67yGgA2Y8YMrfv6559/mJWVFevduzcrLi5WWXbz5k3m4+PD/Pz8WG5ubq2xZGdnq6w/btw4BoCdPn1a/trKlSsZADZ79mxWVVVVa//btm2Tr3/48GEGgP30009ax2xq9DmmhlBaWspkMhn76aefGAB2+PDhWuuo+0wVFxezgIAA5ujoyMrLy3XeX0pKCgPANm/e3MCRE4T5Qak6gjBjevfuDQBITk42+LYXLlyIzMxMbNiwwSDbW7ZsGUQiEbZu3QoHBweVZa1atcKKFSuQnp6OTZs21bmtmsddWlqK2NhYRERE4NNPP4VIJKr1nhdeeAFdu3Y1wJFwfvrpJ0RHR8Pe3h7NmjXD888/XyttOmXKFDg5OeHevXsYNWoUnJyc4OXlhXnz5kEmk2ndvr7HlJubi3nz5qFdu3ZwcnKCi4sLhg0bhosXL6q8R0hT7tixAx9++CGaN28OBwcHfP755xg7diwAoF+/fvKUXFxcnMYxOjg4oHv37iguLkZ2djYA4NatWxg7diw8PDzky/fu3Vvn+QSA69ev45lnnoGHhwfs7OzQuXNn/O9//9PpvQRhLpBwIggzRqhFcXd3r7VMJpPhwYMHtW7FxcU6bbt3797o378/VqxYgdLS0jrXl0qltfZVVlYGACgpKcGhQ4fQu3dvhISEqH3/uHHjIJFI8Pvvv9e5r5rH/c8//yA3NxfPPfccxGKxTscHAIWFhWrPEWNM6/u2bNmCZ599FmKxGLGxsXj55Zfxyy+/oFevXrVqr2QyGYYMGQJPT098+umn6NOnD1atWoWvvvpK6z70PaZbt25hz549ePLJJ/HZZ59h/vz5uHz5Mvr06YP79+/XWn/ZsmXYu3cv5s2bh48//hiDBw/Gm2++CQB4//338e233+Lbb79FZGRknfsVi8Vwc3NDZmYmevTogf379+P111/HRx99BKlUiqeeegq7d+/Wup2rV6+ie/fuSEhIwLvvvotVq1bB0dERo0aNqvO9BGFWmDrkRRCEIqX2119/sezsbJaWlsZ27drFvLy8mEQiYWlpaSrr9+nThwFQe3vllVdqbVddqi47O5sdOXKEAWCfffaZfLmmVJ26m5CKuXDhAgPAZs2apfU427dvzzw8PGqNJTExkWVnZ7Pbt2+zb775htnb2zMvLy95ym/t2rUMANu9e7dO51NI1Wm6paena3xveXk58/b2Zo899hgrLS2Vv/77778zAGzhwoXy1yZPnswAsKVLl6psIyoqikVHR2sdo77HJJVKmUwmU3ktJSWFSSQSlf0Lx96yZctaadi6UnUREREsOzubZWdns4SEBPbmm28yAGzEiBGMMcZmz57NALBjx47J31dYWMhCQkJYcHCwfHzqUnUDBgxg7dq1Y1KpVP5aVVUV69GjBwsNDdXpHBCEOUDF4QRhRgwcOFDleXBwML777jsEBATUWjc4OBhff/11rdfVrauJxx9/HP369cOKFSvw6quv1ir2VmbkyJGYOXOmymtt27YFwCM7AODs7Kx1f87OzigoKKj1enh4uMrzdu3aYfPmzfKUn/CeurZfk4ULF8rTfsp4eHhofM+5c+eQlZWFxYsXqxTkDx8+HBEREdi7dy+WLFmi8p5XX31V5Xnv3r3x7bffah2bvsckkUjkj2UyGfLy8uDk5ITw8HCcP3++1vqTJ0/W+vdUx/Xr1+Hl5SV/LhKJMHz4cHzzzTcAgD/++ANdu3ZFr1695Os4OTlh+vTpeO+993Dt2jU89thjtbabm5uLv//+G0uXLkVhYaH88wIAQ4YMwaJFi3Dv3j00b95cr/EShCkg4UQQZsT69esRFhaG/Px8fPPNNzh69KjKBVMZR0fHWkKrPixevBh9+vTBxo0bMWfOHI3rBQQEaNyfcPFXviCqo7CwUK1Q+Pnnn+Hi4oLs7Gx8/vnnSElJUbnou7i46LT9mrRr107vc3Tnzh0AtcUcAERERKjM9AMAOzs7FbEB8BTjw4cPte5H32OqqqrC2rVr8eWXXyIlJUWlhsrT07PW+ppSptoQxLhIJIKdnR1CQ0Ph7e0tX37nzh1069at1vuEdN+dO3fUCqebN2+CMYYFCxZgwYIFavedlZVFwomwCEg4EYQZ0bVrV3Tu3BkAMGrUKPTq1QvPPfccEhMT4eTkZJR9Pv744+jbt6886lQfWrduDWtra1y6dEnjOmVlZUhMTJQfX80xNGvWDAAwYsQItGvXDhMnTkR8fDysrKzk3lSXL1/GqFGj6jVGY6FPzZUy+h7Txx9/jAULFuDFF1/EsmXL4OHhASsrK8yePVutz5K+0SbAcGK8JsL45s2bhyFDhqhdp3Xr1gbfL0EYAyoOJwgzRShMvn//PtatW2fUfS1evBgZGRk6zXhTh6OjI/r164ejR4/KIzY1+fHHH1FWVlbLXLMmTk5OWLRoES5cuIAff/wRANCrVy+4u7vjhx9+qHO2WkNp0aIFACAxMbHWssTERPnyhqLvMe3atQv9+vXDf//7X4wfPx6DBw/GwIEDNRqFqkPdzD19aNGihdrzcv36dflydbRs2RIAYGNjg4EDB6q96ZuGJQhTQcKJIMyYvn37omvXrlizZg2kUqnR9tOnTx/07dsXy5cvr/d+PvzwQzDGMGXKlFqz9FJSUvD222/Dz88Pr7zySp3bmjhxIgICArB8+XIAfFr8O++8g4SEBLzzzjtqZ8V99913OHPmTL3Grkznzp3h7e2NjRs3ymcNAsCff/6JhIQEDB8+vMH7APQ/JrFYXGudn376Sa2zvCYcHR0BQC+xpcwTTzyBM2fO4OTJk/LXiouL8dVXXyE4OBht2rRR+z5vb2/07dsXmzZtQnp6eq3lgtUBQVgClKojCDNn/vz5GDt2LLZs2aKSSsvPz8d3332n9j3PP/+83vtZtGgR+vXrV+9xPv744/j0008xd+5ctG/fHlOmTIGfnx+uX7+Or7/+GlVVVfjjjz/UWivUxMbGBrNmzcL8+fOxb98+DB06FPPnz8fVq1exatUqHD58GM888wx8fX2RkZGBPXv24MyZMzhx4oTKdo4dO6ZWCLZv3x7t27fXuO/ly5dj6tSp6NOnDyZMmIDMzEysXbsWwcHBWuvA9EWfY3ryySexdOlSTJ06FT169MDly5exfft2eTRHFzp27AixWIzly5cjPz8fEokE/fv3V6lj0sa7776LH374AcOGDcObb74JDw8PbN26FSkpKfj5559hZaX5t/j69evRq1cvtGvXDi+//DJatmyJzMxMnDx5Enfv3q3lR0UQZotJ5/QRBMEY0+wczhhjMpmMtWrVirVq1YpVVlYyxrTbESj/W9dlR1ATYbv1cQ4XOHr0KBs5ciRr1qwZs7GxYUFBQezll19mt2/frrWutrHk5+czV1dX1qdPH5XXd+3axQYPHsw8PDyYtbU18/PzY+PGjWNxcXHydeqyI1i0aFGdx7Fz504WFRXFJBIJ8/DwYBMnTmR3795VWWfy5MnM0dFR43Hpii7HJJVK2VtvvcX8/PyYvb0969mzJzt58iTr06ePyjmqyzX966+/Zi1btmRisVjFmkBXN/rk5GT2zDPPMDc3N2ZnZ8e6du3Kfv/9d5V1NDmHJycns0mTJjFfX19mY2PDmjdvzp588km2a9cu3U4UQZgBIsbqcIIjCIIgCIIgAFCNE0EQBEEQhM6QcCIIgiAIgtAREk4EQRAEQRA6QsKJIAiCIAhCR0g4EQRBEARB6AgJJ4IgCIIgCB0hA0zwPkr379+Hs7Nzg1sSEARBEARhWTDGUFhYCH9/f61GrgAJJwDA/fv3ERgYaOphEARBEARhQtLS0hAQEKB1HRJOgLy5ZFpaGlxcXEw8GoIgCIIgGpOCggIEBgbq1GyahBMUHcNdXFxIOBEEQRDEI4ou5TpUHE4QBEEQBKEjJJwIgiAIgiB0hIQTQRAEQRCEjlCNk45UVVWhvLzc1MMgCIvAxsYGYrHY1MMgCIIwOCScdKC8vBwpKSmoqqoy9VAIwmJwc3ODr68veaMRBNGkIOFUB4wxpKenQywWIzAwsE5jLIJ41GGMoaSkBFlZWQAAPz8/E4+IIAhzIPMB4OwIONibeiQNg4RTHVRWVqKkpAT+/v5wcHAw9XAIwiKwt+ffjFlZWfD29qa0HUE84jzMB05dBDzdgF7Rph5Nw6DwSR3IZDIAgK2trYlHQhCWhfBDo6KiwsQjIQjC1BSV8vu8AoAx046loZBw0hGq0yAI/aD/GYIgBKRl/F5WpXhsqZBwIgiCIAjCqCiLpcIS043DEJBwIjSyZcsWuLm5GXUfwcHBWLNmTb3ff/v2bYhEIly4cMFgY2oIU6ZMwahRo0w9DIIgCLOiTMnNp6jYdOMwBCScCI2MGzcOSUlJph6GWWJugo0gCMKcURZOhRYunGhWHaERe3t7+ewogiAIgqgvKqk6CxdOFHFqolRVVSE2NhYhISGwt7dHhw4dsGvXLvnyuLg4iEQi7N27F+3bt4ednR26d++OK1euyNepmaq7ePEi+vXrB2dnZ7i4uCA6Ohrnzp2TL//555/Rtm1bSCQSBAcHY9WqVSpjysrKwogRI2Bvb4+QkBBs37691rjz8vLw0ksvwcvLCy4uLujfvz8uXryo17FfuXIFw4YNg5OTE3x8fPDCCy/gwYMH8uV9+/bFm2++ibfffhseHh7w9fXF4sWLVbZx/fp19OrVC3Z2dmjTpg3++usviEQi7NmzBwAQEhICAIiKioJIJELfvn1V3v/pp5/Cz88Pnp6emDFjBs0sIwjikUYlVUc1To8elTLT3PQhNjYW27Ztw8aNG3H16lXMmTMHzz//PI4cOaKy3vz587Fq1SqcPXsWXl5eGDFihMaL/MSJExEQEICzZ88iPj4e7777LmxsbAAA8fHxePbZZzF+/HhcvnwZixcvxoIFC7Blyxb5+6dMmYK0tDQcPnwYu3btwpdffik3SRQYO3YssrKy8OeffyI+Ph6dOnXCgAEDkJubq9Nx5+XloX///oiKisK5c+ewb98+ZGZm4tlnn1VZb+vWrXB0dMTp06exYsUKLF26FAcPHgTALShGjRoFBwcHnD59Gl999RU++OADlfefOXMGAPDXX38hPT0dv/zyi3zZ4cOHkZycjMOHD2Pr1q3YsmWLynkgCOLRRCYD8otMPYrGRyYDKioVz8vKgXIL/i1JqTo9qZQBe+NMs+/hfQFrHXwEy8rK8PHHH+Ovv/5CTEwMAKBly5b4559/sGnTJvTp00e+7qJFizBo0CAAXEwEBARg9+7dtYQGAKSmpmL+/PmIiIgAAISGhsqXffbZZxgwYAAWLFgAAAgLC8O1a9ewcuVKTJkyBUlJSfjzzz9x5swZdOnSBQDw3//+F5GRkfJt/PPPPzhz5gyysrIgkUgA8MjNnj17sGvXLkyfPr3OY1+3bh2ioqLw8ccfy1/75ptvEBgYiKSkJISFhQEA2rdvj0WLFsmPY926dTh06BAGDRqEgwcPIjk5GXFxcfD19QUAfPTRR/LzBABeXl4AAE9PT/k6Au7u7li3bh3EYjEiIiIwfPhwHDp0CC+//HKd4ycIouly5QZw+x7QPhwICTD1aBoPIdoktgKsrfnz4hLA1tW046ovFHFqgty8eRMlJSUYNGgQnJyc5Ldt27YhOTlZZV1BWAGAh4cHwsPDkZCQoHa7c+fOxUsvvYSBAwfik08+UdlWQkICevbsqbJ+z549cePGDchkMiQkJMDa2hrR0QrL2IiIiFqpwKKiInh6eqqMOyUlpda4NXHx4kUcPnxY5f2C0FPeRvv27VXe5+fnJ49+JSYmIjAwUEUQde3aVaf9A0Dbtm1VnLKVt00QxKMJY8D96q+Ba8mA9BHqGS8cq8SWt1wBLNuSgCJOemIt5pEfU+1bF4qKeCx47969aN68ucoyIZJTHxYvXoznnnsOe/fuxZ9//olFixZhx44dGD16dL23qUxRURH8/PwQFxdXa5mutghFRUUYMWIEli9fXmuZcs80IcUoIBKJDNbE2ZjbJgjCMikoUqSnKiuBazeBTm1MOyZ9uZAAONgBYSH6vU+IONlJuHB68NCyLQlIONUDXQWMqWjTpg0kEglSU1NV0nLqOHXqFIKCggAADx8+RFJSkkr6rCZhYWEICwvDnDlzMGHCBGzevBmjR49GZGQkjh8/rrLu8ePHERYWJk9ZVVZWIj4+Xp6qS0xMRF5ennz9Tp06ISMjA9bW1ggODq7XsXfq1Ak///wzgoODYW1dv493eHg40tLSkJmZCR8fHwDA2bNnVdYRWvAILXkIgiC0kZnD750ceHF0WjoQ3BzwsJB0VVEJcOc+IBIBocH8XleEGXUSW378wvYsFUrVNUGcnZ0xb948zJkzB1u3bkVycjLOnz+PL774Alu3blVZd+nSpTh06BCuXLmCKVOmoFmzZmoNHEtLSzFz5kzExcXhzp07OH78OM6ePSsXWW+99RYOHTqEZcuWISkpCVu3bsW6deswb948AFyMDB06FK+88gpOnz6N+Ph4vPTSSyp2BwMHDkRMTAxGjRqFAwcO4Pbt2zhx4gQ++OADldl72pgxYwZyc3MxYcIEnD17FsnJydi/fz+mTp2qs8gZNGgQWrVqhcmTJ+PSpUs4fvw4PvzwQwCKNiLe3t6wt7eXF5/n5+frtG2CIB5Nsqvnt4QEAC38+eNLiZbTt02IljGmOkNOF5QjTk5Cqs6CI04knJooy5Ytw4IFCxAbG4vIyEgMHToUe/fulU+jF/jkk08wa9YsREdHIyMjA7/99pvahsZisRg5OTmYNGkSwsLC8Oyzz2LYsGFYsmQJAB7p+fHHH7Fjxw489thjWLhwIZYuXYopU6bIt7F582b4+/ujT58+GDNmDKZPnw5vb2/5cpFIhD/++AOPP/44pk6dirCwMIwfPx537tyRR37qwt/fH8ePH4dMJsPgwYPRrl07zJ49G25ubrCy0u3jLhaLsWfPHhQVFaFLly546aWX5LPq7OzsAADW1tb4/PPPsWnTJvj7+2PkyJE6bZsgiEePShmQm8cfe3kAka0AG2sgv5AXi1sC5UpiqVSq33uVI07O1RGn4lLAUisYRIxZit41HgUFBXB1dUV+fj5cXFxUlkmlUqSkpCAkJER+0WwKxMXFoV+/fnj48KHR26o0BY4fP45evXrh5s2baNWqlamHYxE01f8dgtCXzBzg1AXAXgIM7sVfS7nLI0421sCAGC4qzJnU+8C/1fOGurQD/L21r6/M6YtAxgOgYwQQ5A/8cYSLyf7dFcXipkabDqgJ1TgRhBp2794NJycnhIaG4ubNm5g1axZ69uxJookgCL3Jqq5v8vZUvBbcnNcM5RfyQvEoMy8UL1PyXWpIxEkkAhwd+HEXlZiPcNIHStURhBoKCwsxY8YMREREYMqUKejSpQt+/fVXUw+LIAgLRKhv8vJQvCYScT8nAEhNB3LNvExS2bCytEzzeuqQKtU4AYoCcUutc6KI0yNK3759QVlazUyaNAmTJk0y9TAIgrBwpGUKgaAsnAA+oy7IjwunS4lAny76zVZrTOpb46RcTC4IJyHKZKkz6yjiRBAEQRBGQkjTubsAtja1l0e25m7a+YU8dWeuKEecpHpEnMorFDMHheOXCycLjTiRcCIIgiAII5GlJk2njJ0tENmSP752U/+p/o1FeT1rnIQ0na0NIExslqfqKOJEEARBEIQAY4r6JuXC8JqEBAAuTrwR7vVbjTM2fVGJOJXr7j9VM00HAI72PCVZWWmZrWdIOBEEQRCEERDarFiLeapOE8qF4rfv8feZG8rCSR8TTOUZdQJiMW/dAlhmuo6EE0EQBEEYAaHNSjN3RZpKE55uinSeuc2wY0whnMTVx6Fruk4QTnY12qRacusVEk4EQRAEYQTU2RBoQ3DVLik1znjqi3K0ycWJ3+tqSSBEpmoafFpy6xUSTsQjTVxcHEQikUqzYYIgiIai3GZFW32TMkLrzmIzFU421oB9dYpN74hTDeHkTBGn+hEbG4suXbrA2dkZ3t7eGDVqFBITE1XWkUqlmDFjBjw9PeHk5ISnn34amZmZKuukpqZi+PDhcHBwgLe3N+bPn4/KysrGPBSz4+jRoxgxYgT8/f0hEomwZ8+eWutMmTIFIpFI5TZ06NDGH6wJ6dGjB9LT0+HqaiEtygmCsAhy8oAqxmt5hLRUXTjoKUoaC0E4SWyVhJOBIk5U46QnR44cwYwZM3Dq1CkcPHgQFRUVGDx4MIqLFWdyzpw5+O233/DTTz/hyJEjuH//PsaMGSNfLpPJMHz4cJSXl+PEiRPYunUrtmzZgoULF5rikMyG4uJidOjQAevXr9e63tChQ5Geni6//fDDD400QvPA1tYWvr6+EJmr61wDqaioqHslgiB0IvMBsPeIbo15Bf8mXdN0AJ9tBgAlZiqcbG14vz1Ad3GnblYdoBCTJVJAJmv4GBsTkwqnffv2YcqUKWjbti06dOiALVu2IDU1FfHx8QCA/Px8/Pe//8Vnn32G/v37Izo6Gps3b8aJEydw6tQpAMCBAwdw7do1fPfdd+jYsSOGDRuGZcuWYf369Sgvt8B5jgZi2LBh+L//+z+MHj1a63oSiQS+vr7ym7u7e53bvnLlCoYNGwYnJyf4+PjghRdewIMHD+TL+/bti5kzZ2LmzJlwdXVFs2bNsGDBAq1O5YsXL0bHjh3xzTffICgoCE5OTnj99dchk8mwYsUK+Pr6wtvbGx999JHK+1JTUzFy5Eg4OTnBxcUFzz77rDwimZSUBJFIhOvXr6u8Z/Xq1fKeczVTdVu2bIGbmxv279+PyMhIODk5ycWlQGVlJd588024ubnB09MT77zzDiZPnoxRo0ZpPL6cnBxMmDABzZs3h4ODA9q1a6ciUr/66iv4+/ujqka78JEjR+LFF1+UP//111/RqVMn2NnZoWXLlliyZIlKdFUkEmHDhg146qmn4OjoiI8++ggymQzTpk1DSEgI7O3tER4ejrVr16rsR5djqqqqQmxsrHw7HTp0wK5duzQeM0E0NTIe8Cn0lxOBvALt6+piQ1ATIZpTVm5eYqJcyYtJGKOuJphSDREnia3CENPS0nVmVeOUn8+nEnh4cIkeHx+PiooKDBw4UL5OREQEgoKCcPLkSQDAyZMn0a5dO/j4+MjXGTJkCAoKCnD16lW1+ykrK0NBQYHKTVcY44V7prgZo0NKXFwcvL29ER4ejtdeew05OTla18/Ly0P//v0RFRWFc+fOYd++fcjMzMSzzz6rst7WrVthbW2NM2fOYO3atfjss8/wn//8R+u2k5OT8eeff2Lfvn344Ycf8N///hfDhw/H3bt3ceTIESxfvhwffvghTp8+DYBfyEeOHInc3FwcOXIEBw8exK1btzBu3DgAQFhYGDp37ozt27er7Gf79u147rnnNI6jpKQEn376Kb799lscPXoUqampmDdvnnz58uXLsX37dmzevBnHjx9HQUGB2lSoMlKpFNHR0di7dy+uXLmC6dOn44UXXsCZM2cAAGPHjkVOTg4OHz4sf09ubi727duHiRMnAgCOHTuGSZMmYdasWbh27Ro2bdqELVu21BKTixcvxujRo3H58mW8+OKLqKqqQkBAAH766Sdcu3YNCxcuxPvvv48ff/xRr2OKjY3Ftm3bsHHjRly9ehVz5szB888/jyNHjmg9doJoKghioYoB565w3yV1lEp50bNIxGfU6YqtDXcRB8wr6lRWz4iTTMaFJlA74gQoHMQtrkCcmQkymYwNHz6c9ezZU/7a9u3bma2tba11u3Tpwt5++23GGGMvv/wyGzx4sMry4uJiBoD98ccfave1aNEiBqDWLT8/v9a6paWl7Nq1a6y0tJRvu4SxoL6muRWX1O/cAmC7d++u9foPP/zAfv31V3bp0iW2e/duFhkZybp06cIqKys1bmvZsmW1zndaWhoDwBITExljjPXp04dFRkayqqoq+TrvvPMOi4yM1LjdRYsWMQcHB1ZQUCB/bciQISw4OJjJZDL5a+Hh4Sw2NpYxxtiBAweYWCxmqamp8uVXr15lANiZM2cYY4ytXr2atWrVSr48MTGRAWAJCQmMMcYOHz7MALCHDx8yxhjbvHkzA8Bu3rwpf8/69euZj4+P/LmPjw9buXKl/HllZSULCgpiI0eO1Hh86hg+fDh766235M9HjhzJXnzxRfnzTZs2MX9/f/nxDxgwgH388ccq2/j222+Zn5+f/DkANnv27Dr3PWPGDPb000/rfExSqZQ5ODiwEydOqGxn2rRpbMKECWr3UfN/hyAsncOnGdvzF2O/HuL3Zy+rX+/OPb78yBn99/H3Kf7ejAcNG6shuZzEx3Q5ibFSqeIcKH3Fq6WohK/729/ql/97jS9PSDb8mPUlPz9fow6oidlEnGbMmIErV65gx44dRt/Xe++9h/z8fPktLS3N6Ps0R8aPH4+nnnoK7dq1w6hRo/D777/j7NmziIuLAwB5Os7JyQlt27YFAFy8eBGHDx+Wv+7k5ISIiAgAPGIk0L17d5W6oZiYGNy4cQMyLfHn4OBgODs7y5/7+PigTZs2sFIyQPHx8UFWVhYAICEhAYGBgQgMDJQvb9OmDdzc3JCQkCA/xtu3b8tTu9u3b0enTp3kY1aHg4ODPJUHAH5+fvJ95ufnIzMzE127dpUvF4vFiI6O1rg9gNfiLVu2DO3atYOHhwecnJywf/9+pKamyteZOHEifv75Z5SVlcnHOn78ePnxX7x4EUuXLlU59y+//DLS09NRUqKIdXfu3LnW/tevX4/o6Gh4eXnByckJX331lXzfuhzTzZs3UVJSgkGDBqnsf9u2bSp/d4JoyghRlvbhPJp0LxNIVdNfrq42K9oQCsTNyZKgQiniJLHlx66LCaYmDycBS7UksDb1AABg5syZ+P3333H06FEEBATIX/f19UV5eTny8vLg5uYmfz0zMxO+vr7ydYR0h/JyYZk6JBIJJBINf8k6sLcDEv6o11sbjJBbNhYtW7ZEs2bNcPPmTQwYMAD/+c9/UFrK/3ttbHgyuqioCCNGjMDy5ctrvd/Pz69B+xf2ISASidS+VrMOSBu+vr7o378/vv/+e3Tv3h3ff/89XnvtNb3HwRqYJ125ciXWrl2LNWvWoF27dnB0dMTs2bNV6vBGjBgBxhj27t2LLl264NixY1i9erV8eVFREZYsWaIyOULAzk7x4XB0dFRZtmPHDsybNw+rVq1CTEwMnJ2dsXLlSnnKUxeKiriV8d69e9G8eXOVZfX9XyIIS0ImUxRJ+3vzxwnJwKVEwM1F4W+ka5sVTciFkxml6uSz6my4aLKz5bPqSqWaRRGgeUadgKVaEphUODHG8MYbb2D37t2Ii4tDSEiIyvLo6GjY2Njg0KFDePrppwEAiYmJSE1NRUxMDAAeyfjoo4+QlZUFb29vAMDBgwfh4uKCNm3aGHzMIhHgYG/wzZoFd+/eRU5OjlwA1bxAAkCnTp3w888/Izg4GNbWmj8+NS/Kp06dQmhoKMRiscHGGxkZibS0NKSlpcmjTteuXUNeXp7K337ixIl4++23MWHCBNy6dQvjx4+v9z5dXV3h4+ODs2fP4vHHHwfAo0nnz59Hx44dNb7v+PHjGDlyJJ5//nkAvD4rKSlJZZx2dnYYM2YMtm/fjps3byI8PBydOnWSL+/UqRMSExPRunVrvcZ8/Phx9OjRA6+//rr8NeUokS7H1KZNG0gkEqSmpqJPnz567Z8gmgJCkbPYikdeQlsADx5ykXTuCtCnC28lomubFU0I1xdzijjJZ9VVCyB7u2rhVAZoK+HSNeJUXMIFp6VMbjZpqm7GjBn47rvv8P3338PZ2RkZGRnIyMiQRzlcXV0xbdo0zJ07F4cPH0Z8fDymTp2KmJgYdO/eHQAwePBgtGnTBi+88AIuXryI/fv348MPP8SMGTMe6V/CRUVFuHDhAi5cuAAASElJwYULF+TpmaKiIsyfPx+nTp3C7du3cejQIYwcORKtW7fGkCFDNG53xowZyM3NxYQJE3D27FkkJydj//79mDp1qkoaLjU1FXPnzkViYiJ++OEHfPHFF5g1a5ZBj3HgwIFo164dJk6ciPPnz+PMmTOYNGkS+vTpo5KuGjNmDAoLC/Haa6+hX79+8Pf3b9B+33jjDcTGxuLXX39FYmIiZs2ahYcPH2q1NAgNDcXBgwdx4sQJJCQk4JVXXqnlRwZwkbd3715888038qJwgYULF2Lbtm1YsmQJrl69ioSEBOzYsQMffvih1vGGhobi3Llz2L9/P5KSkrBgwQKcPXtWr2NydnbGvHnzMGfOHGzduhXJyck4f/48vvjiC2zdulXXU0cQFouQphMi/yIR0Kktj6YUFgNXbvDX9Wmzog5BOJmTl5OyHQGgEEJ1jbGuiJODHWAlAmRV5hVhqwuTCqcNGzYgPz8fffv2hZ+fn/y2c+dO+TqrV6/Gk08+iaeffhqPP/44fH198csvv8iXi8Vi/P777xCLxYiJicHzzz+PSZMmYenSpaY4JLPh3LlziIqKQlRUFABg7ty5iIqKkvtbicViXLp0CU899RTCwsIwbdo0REdH49ixY1oFp7+/P44fPw6ZTIbBgwejXbt2mD17Ntzc3FRqkSZNmoTS0lJ07doVM2bMwKxZszB9+nSDHqNIJMKvv/4Kd3d3PP744xg4cCBatmyp8vkB+EV/xIgRuHjxYi0xUh/eeecdTJgwAZMmTUJMTAycnJwwZMgQlXRZTT788EN06tQJQ4YMQd++feHr66vWvqB///7w8PBAYmJirZl/Q4YMwe+//44DBw6gS5cu6N69O1avXo0WLVpoHe8rr7yCMWPGYNy4cejWrRtycnJUok+6HtOyZcuwYMECxMbGIjIyEkOHDsXevXtrRYoJoilSqiZ6YmfLxRPAvZ3uZzUsTQeYZ6quTMmOANDdBFOTa7iASAQ4WmC6TsQaWrzRBCgoKICrqyvy8/Ph4qIaW5VKpUhJSUFISIjWCyOhoG/fvujYsSPWrFlj6qE0ClVVVYiMjMSzzz6LZcuWmXo4BsEQx0T/O0RTIuk2r2kK9AM61agCuXYTuHGHWwlUybhdwYAY3R3DlSmvAP48yh8P78tTfqakqgr4rdolZWhvHj1KTuURNn9voEs7ze89dYFH4DpGAC1qV34AAM5e5oLzsVCgVZDBh68z2nRATcyiOJwgLIk7d+7gwIED6NOnD8rKyrBu3TqkpKRo9YYyd5riMRGEIRGiJ/ZqAvIRLXmLlVxuRahXm5WaCF5OlZW8zkkoOjcV5UpeVTUjTnWZYAp1YdoKyAUvJ0uKOJmNHQFBWApWVlbYsmULunTpgp49e+Ly5cv466+/EBkZaeqh1ZumeEwEYUi0FTpbWQHRbXkTXKD+aToBc+pZp+waLpRx6mqCqandijKCwLQkSwKKOBEGR/CBaqoEBgbi+PHjph6GQWmKx0QQhkReHK5BBDjYA53bATduAy0D1a+jK472fHaeOdQ51SwMB5QiTuWaZ8Mp+zxpKg4HLDPiRMKJIAiCIOpAKITW5qfn7cFvDcXejArE5R5OSuKnpgmmuohSWYWiTZg24SQUh5eV833Z2mhe11ygVB1BEARBaKGqSre0k6EwJ/dwdREnwQQT0JyuU442afNnshYroniWEnUi4aQjNPmQIPRDH4d3gjBnhPomK5H26ImhMCcTzPIaVgQCdVkSCOdMl/MlGGEWWUidE6Xq6sDGxgYikQjZ2dnw8vLSanJIEAT/kVFeXo7s7GxYWVnB1rYRrjQEYUR0SdMZErlwMoNUXZmaiBNQtwlmWR2u4co4O3D/q0ILiTiRcKoDsViMgIAA3L17F7dv3zb1cAjCYnBwcEBQUJCKMSpBWCKCOGiMNB2gSF2VVwCVMtN6OdVstyJQZ8RJh8JwgSYZcZo7d67OG/zss8/qPRhzxcnJCaGhoagQWkQTBKEVsVgMa2tritASTYLGjjjZ2nBrgwoz8HKSC6caaqEuS4KyOlzDlXGyMPdwnYTTv//+q/L8/PnzqKysRHh4OAAgKSkJYrEY0dHRhh+hmSAWiw3aoJYgCIKwDKR1WBEYAwd7IL+QCxOTCqc6apw0mWDqE3ESLAmKS3khvrkHqXUSTocPH5Y//uyzz+Ds7IytW7fC3Z33RX748CGmTp2K3r17G2eUBEEQBGEidHHANjQOdlw4FZu4QLyi2jm8VqquroiTHufMTqJwSy8qMb1bel3oretWrVqF2NhYuWgCAHd3d/zf//0fVq1aZdDBEQRBEISpkZtfNmLLRXPxctJUHF7TBLPW+/SIOAG8QBxQn64rrwCycoGkFCA9W7ftGRO9i8MLCgqQnV175NnZ2SgsLDTIoAiCIAjCXCjV0qfOWDhWz6wzZduVqioeBQJqC6e6TDC1tahRh5Mj8LCAO6bbSYC8Av48r0BVTPl7A35e9TseQ6G3cBo9ejSmTp2KVatWoWvXrgCA06dPY/78+RgzZozBB0gQBEEQpqKqSkkENGLEyRxMMIXCcJFI0YdPQDDBLC3j4k5ZIFXK+A3QPeIkFIgnpvBbTRztATcXwzizNxS9hdPGjRsxb948PPfcc/JZZtbW1pg2bRpWrlxp8AESBEEQhKkQUk4iESBpxHYg9mbg5aScplM3Qdberlo4lQHuSq8LQtNaXFtwacLTTfFYYgu4u3Ch5OYCuDk3jvGorugtnBwcHPDll19i5cqVSE5OBgC0atUKjo6OBh8cQRAEQZgS5TRdY7prCBGn8gpeoK2rADEkmmbUCWgywdS3vgngwql/9+oWLI0Y2asP9Z70l56ejvT0dISGhsLR0ZFakhAEQRBNDlMUhgNcKAliyVRRJ3V96pTRZIKpb32TgLOj+YsmoB7CKScnBwMGDEBYWBieeOIJpKenAwCmTZuGt956y+ADJAiCIAhTUVpPEWAIhNYrpSaqc6pTOBkw4mRJ6C2c5syZAxsbG6SmpsLBwUH++rhx47Bv3z6DDo4gCIIgTEljt1tRxtHEdU66RpxqmmBK9XANt0T0zpoeOHAA+/fvR0BAgMrroaGhuHPnjsEGRhAEQRCmRmoCKwIBU3s51VXjRBEnHSkuLlaJNAnk5uZCIjHBJ4sgCIIgjIS0kfvUKSMUiJvKy6lcg2u4gCYTTGM5re87xt3UTY3ewql3797Ytm2b/LlIJEJVVRVWrFiBfv36GXRwBEEQBGFK5MXhpqhxqhYmpmq7UleqrqYJpoA84mSgc/YwH3hjGfDKQmDRF4bZZkPQO1W3YsUKDBgwAOfOnUN5eTnefvttXL16Fbm5uTh+/LgxxkgQBEEQjQ5jiuiJSSJOQo2TqYSTIIA0CCdNJpiGrHE6dBJ451MgOxcQWwHNfUzfCFhv4fTYY48hKSkJ69atg7OzM4qKijBmzBjMmDEDfn5+xhgjQRAEQTQ6ZdUpKJHINPU6glirqDSNl1NdESegtgkmY4r3NSTiVFAELF0P/FQ956x1C+Czd4EOEfXfpqGo15/B1dUVH3zwgaHHQhAEQRBmg3xGnW3jml8K2Fhz0VJewQvEXZ0ad/9y4aRFNNY0wVQRm/V0Wj96Fnh7JW/oKxIBLz8LvPWi+czS0zvY1bp1ayxevBg3btwwxngIgiAIwiwoNWFhuIC8QLyR03UypX5zdUWcAMW5KlOaiaev2CwuBT5YDbzwNhdNLfyBn9YCH7xqPqIJqIdwmjFjBvbu3Yvw8HB06dIFa9euRUZGhjHGRhAEQRAmw5QeTgJCnVNjF4hra/CrTE1Lgvq4hpeUAtv/Bwx+Efjuf/y1yaOBff8BurTTb9yNQb0MMM+ePYvr16/jiSeewPr16xEYGIjBgwerzLYjCIIgCEvGlK7hAqbyctKlvgmobYKpj4fT3Qzg441A93HA+6v58+Y+wPZPgaVvKkSjuVHvuvSwsDAsWbIESUlJOHbsGLKzszF16lRDjo0gCIIgTIYpPZwEBPfwxvZyKtNVOGmKOGkQTowBpy8Cry4Cek8ENu3k3kxB/sDCGcCBb4Be0Q0fvzFpUI3+mTNn8P3332Pnzp0oKCjA2LFjDTUugiAIgjAppnQNF3AwVcSpDtdwgZommJoiTowBvx4CvvoRuKpUIt2zE/Di00C/boBYbJixGxu9I05JSUlYtGgRwsLC0LNnTyQkJGD58uXIzMzEjh079NrW0aNHMWLECPj7+0MkEmHPnj0qy6dMmQKRSKRyGzp0qMo6ubm5mDhxIlxcXODm5oZp06ahqKhI38MiCIIgCBXk5pcmjDjJU3WNXONUUe0aXlfKraYJpibX8B17gVkfcdFkJwGee5JHl75fBQzsYTmiCahHxCkiIgJdunTBjBkzMH78ePj4+NR758XFxejQoQNefPFFjBkzRu06Q4cOxebNm+XPa7Z1mThxItLT03Hw4EFUVFRg6tSpmD59Or7//vt6j4sgCIJ4tGHMTCJO1am6ikped1RXBMhQ6FrjVNMEU13E6c497skEAFPHALMmAe6uhh9zY6G3cEpMTERoaKhBdj5s2DAMGzZM6zoSiQS+vr5qlyUkJGDfvn04e/YsOnfuDAD44osv8MQTT+DTTz+Fv7+/QcZJEARBPFqUVQBV1f3XTNms1lqs8HIqlTaicNIxVQeommDWnFUnkwFzP+Gpxu4deB2TKV2/DYHewzeUaNKVuLg4eHt7Izw8HK+99hpycnLky06ePAk3Nze5aAKAgQMHwsrKCqdPn27UcRIEQRBNB6mSFYGpL/SmqHPStTgcUDXBFISTIDa//hE4dwVwcgA+fdf059IQ6BRx8vDwQFJSEpo1awZ3d3eItLha5ebmGmxwQ4cOxZgxYxASEoLk5GS8//77GDZsGE6ePAmxWIyMjAx4e3urvMfa2hoeHh5avaXKyspQVlYmf15QUGCwMRMEQRCWT6kB+601FAd7IK+wceucdHENFxDqsAqLAVkVf2wnARKSgVXVlTaLZgKB6pNHFodOwmn16tVwdnaWP9YmnAzJ+PHj5Y/btWuH9u3bo1WrVoiLi8OAAQPqvd3Y2FgsWbLEEEMkCIIgmiBCYbg5eAnJm/02YsRJ1xonQFEDllfI763FPEU352O+nUE9gLFDNb/f0tBJOE2ePFn+eMqUKcYaS520bNkSzZo1w82bNzFgwAD4+voiKytLZZ3Kykrk5uZqrIsCgPfeew9z586VPy8oKEBgYKDRxk0QBEFYFnX5ETUm8rYrjSmchCJvHWucAN6YF+DRpjVbgYRbgKcb8Mk80/T6MxZ6ZxsHDhyILVu2mCS9dffuXeTk5MDPzw8AEBMTg7y8PMTHx8vX+fvvv1FVVYVu3bpp3I5EIoGLi4vKjSAIgiAE5Kk6E1oRCAjCqTHbrtQn4sSqi+lv3wM2VrsTfTwXaOZu+PGZEr2FU9u2bfHee+/B19cXY8eOxa+//oqKiop67byoqAgXLlzAhQsXAAApKSm4cOECUlNTUVRUhPnz5+PUqVO4ffs2Dh06hJEjR6J169YYMmQIACAyMhJDhw7Fyy+/jDNnzuD48eOYOXMmxo8fTzPqCIIgiHoj93AyoRWBQGO3XamUKWqV9KlxArgdwaYdQFUV8MwQYGhv44zRlOgtnNauXYt79+5hz549cHR0xKRJk+Dj44Pp06fjyJEjem3r3LlziIqKQlRUFABg7ty5iIqKwsKFCyEWi3Hp0iU89dRTCAsLw7Rp0xAdHY1jx46peDlt374dERERGDBgAJ544gn06tULX331lb6HRRAEQRByzKHdioDQdqWyUhEJMiZCLMRKxOuV6kIwwQSA3QeBzBzec27RTOON0ZSIGBOCa/VDKpXit99+w0cffYTLly9DJpMZamyNRkFBAVxdXZGfn09pO4IgCAK//c19nAb2UAgXU7LvGI/m9O0KuDobd1/5hUDcGV6rNKSXbu858A9w7iqPNgHAD58BPaKMN0ZDo48OaFCvuoyMDOzYsQPfffcdLl26hK5duzZkcwRBEARhcsrKFeaX5pCqA3idU1k5T9dpEk73s3gdVGiLhu1LHw8ngXuZwPe/8cdjh1qWaNIXvYVTQUEBfv75Z3z//feIi4tDy5YtMXHiROzcuROtWrUyxhgJgiAIotFQNnE0F8NGezvgYYH6AnHGgCs3gFtp/Hkzt4a1NNHHNVxaDqzeDHy1k4tNPy/gjRfqv29LQG/h5OPjA3d3d4wbNw6xsbEqrt0EQRAEYemUKrmGmwuCl1NpDeFUUcmdubMUTTXwsKCBwknHiNOFBOCt5cDNO/x5l3bAmMGAWxOveNFLODHG8Pnnn2PixIlwcHAw1pgIgiAIwmQIVgQOZlAYLqCu7UpxKXD6InfsFltxwZKTpzCirC+CcNLUo6+snPs0bayePeflAbz5giKFaMrefo2BXkFIxhhmzJiBe/fuGWs8BEEQBGFSajaqNQdquofn5AFHz3LRZCcBekUDrYP4srwG2ixqizhdvA4Mnw58+T0XTaMGAn9t5kX0AJ9dp4tppiWjV8TJysoKoaGhyMnJafRmvwRBEATRGJSao3BSijil3ucCpooBbs5Atw58rILgKyrhXky6WAmoo0xNjRNjPMr0xbfc46mZOze3FGbdCXYEdrZNyyVcHXqXvX3yySeYP38+rly5YozxEARBEIRJkZtfmmGqrrIS+DeBiyZ/bx5pEgSenYTfGOOWAvVFXcTpv7u4cJJVAU/1Bw5uVrUqcHUG2rQGOkTWf7+Wgt7F4ZMmTUJJSQk6dOgAW1tb2NurGlzk5uYabHAEQRAE0djIzS/NKOIkFvPaISEaFB7CbzWjO+4uQHo2T9d5utVvX3LhVF2rdPIC8PFG/vjD14CXn1X/vobaIFgKegunNWvWGGEYBEEYEsb4zBo3Z/OZTk0Q5sDDfMDFiQsRTZjjrDoA8PHkXk0dIoAADX3s3ZyrhZMBIk4SG76tmUt5pGn0IOClsfXfblNBb+E0efJkY4yDIAgDcvsecCkRiGwJhIWYejQEYR5k5gCnLvBITM9O6mtxyisUfdrMKeIEAFFtgPbh2kWfa7UVQEMKxAUfJzDgtUXAg4dAm1ZA7NymX7+kC3oLp9TUVK3Lg4KC6j0YgiAMg1DfkF9k2nEQhDnx4CG/z8kD7twHgpvXXkcoDLe10S5QTEVdY3KrtgQoKuEeTzZ6XuUrKhWu6bFf83oqV2dg01LzqvkyJXoLp+DgYIi0SE5L7FVHEE0NwV1YqIcgCEK1YPrqTcC3We10nNRM03S6IrHlheQlUn68zdz1e7+Qpjt9Efjhdx5hWvsBEORv+LFaKnoLp3///VfleUVFBf7991989tln+Oijjww2MIIg6o9Qo0HCiSAUCMJJKLK+nMTdrpUxR/NLfXFz4cLpYYH+wqmigtsd7PyTP58zBejXzeBDtGj0Fk4dOnSo9Vrnzp3h7++PlStXYsyYMQYZGEEQ9YMxhXASZgcRxKOOtIxHU0QioGt74J94Xmidns37qymvB1huxAngqbX7WfWrc8rM4dYDlZXc1PKN5w0/PkvHYPNtwsPDcfbsWUNtjiCIeqLc2b1SxmsWCOJRR4g2OTkAHq4Kl+1Liar/I3IPJwsWTu7VBeL6ejnJZMD7n/FIla8XsPo9mpWrDr0jTgUFqhKWMYb09HQsXryY3MQJwgyo2T29rFz/AlGCaGoIIsLFid+Hh/BoU1EJkJDMZ6sBSq7hFpyqE3rGFZfyKFtdzXoBXjD/2WbeMNjWBnjnJcW5IlTR++vUzc2tVnE4YwyBgYHYsWOHwQZGEET9UG4CCvDUgxP15CYecYQZpoKoEIu5WDrxL5Byl/siebg2jYiTrQ3/ny8q4X5O3h7q1ysqAQ78A+w5BPxzTmHD8NyTj46ZZX3QWzgdPnxY5bmVlRW8vLzQunVrWFvTz1qCMDWlNSJOVOdEEEBBDeEEAF4eQJAfkJoOXEgA+nYFpNUTKiy5xgngx1lUwuuclIWTtByIOw38egg4dFJ1AsljocCgnkBIgMI1nKiN3kqnT58+xhgHQRAGombEiWbWEY86FZVcRAC1009tQ3lBdGExcC2ZF0UDlu9Z5OYC3MtUpCjvZwHrt3PBVFisWK9VIPDUAN5/rmUgcPYyX1eX9N6jit7CaevWrWjWrBmGDx8OAHj77bfx1VdfoU2bNvjhhx/QogXF9wjClAjCycaaXzCkJJyIRxxBKNhJALsakRRbG6BdGK/tSa72d7axBqzN0PxSHwQjzJQ04LfD3JNJ8GjybaYQS4+FqrqBCz+0SDhpRu96+Y8//lje2PfkyZNYt24dVqxYgWbNmmHOnDkGHyBBEPoh1Gi4u/J7StURjzpC1MVVQ7Fzcx/eB07A0tN0AP/R9MsB4P3VwNbdXDR17wBs/xQ4uRP44FUuGGv6Wcsb/JJw0ojeEae0tDS0bt0aALBnzx4888wzmD59Onr27Im+ffsaenwEQegBY0BJdY2ThyuQlWO8VB1j1LeKsAzkwslZ8zodIoC/T3ELD0tO0+XmA5t2AFv3KH5EdYgA3p0O9Iiq+/1y4UQ1ThrRO+Lk5OSEnJwcAMCBAwcwaNAgAICdnR1Ka1alEgTRqAgeTiIRr3EAjBNxSkwB/jiqKLglCHOmphWBOuzteL0TAHi6Gn9MhqaqCtjwA9BrArBxBxdNoS2A1yYAK+brJpoAhXCSUMRJI3pHnAYNGoSXXnoJUVFRSEpKwhNPPAEAuHr1KoKDgw09PoIg9KBEaSq1MJ3aGMLpfhYvon3wkLxeiMblYT5PpekaFWJMUeOkLeIE8Ka/vs14SxZLorgUmBsL7DvGnz8WCsydCrTw5z35dDXCLK/g5wsg7zdt6B1xWr9+PWJiYpCdnY2ff/4Znp48MRwfH48JEyYYfIAEQeiOkKazt1PUaVRU8l+jxthPzRl8BGFMCouBo+eAUxf1e4+sihd7O9rXvb6dxLJS0KnpwJiZXDTZ2gCfzAN+3wQMiFHUOebpIZwAfq7EFl4cb0zqZYC5bt26Wq8vWbLEIAMiCKL+CELGwZ5/iVqJeOpOWsZfMwTScl4HAihqKAiiMRBSwwVF3F5AF2NX4T0uTpYliHThxL/A64t5ixQvd2DTMiC6rWK5qzM/ZmkZv9VV9F5BheE6QV1oCKIJIQgZobO78EVpSEuCEqVSRhJORGOinHbOfKDbe2o6hjcFGAO27QGen8dFU/tw4LdNqqIJ4JEjQVzq0vCXZtTpBgkngmhClGgQToacWScYCQKKvl4E0Rgop4Yzc3R7T11WBJZGeQVvxLtgLU9BjhoI/LQW8PNSv77Q8FeXdF0ZzajTCSr/IogmhHKNE6AocjVkgbhyxElaxuunqIM60Rgof45zHvKUcV1GlbpYEVgKDx4Cry7i7t4iEbcYeGWc9hSkmwuvg6KIk+GgrzuCaEKUKtU4AUqpOgMKp+IariMUdSIaC+XUcBUDHuRqX19axsWASAQ4Oxp3bMbmn3hg2MtcNLk4AptjgVfH1123JTiI6xJxKifXcJ3QWziVlpaipEQRq79z5w7WrFmDAwcOGHRgBEHoh7Sch+5FIoUVgRBxMmSqrrhE9XkJ2bcRjYQg0oX0U13pOiHa5ORgubPEyiuA2E3A8/O5oW3rFsCeL4F+3XR7v1AUX1Zed00ieTjpht7CaeTIkdi2bRsAIC8vD926dcOqVaswcuRIbNiwQa9tHT16FCNGjIC/vz9EIhH27NmjspwxhoULF8LPzw/29vYYOHAgbty4obJObm4uJk6cCBcXF7i5uWHatGkoKiJXPuLRQxAwdraK1JmdEVJ1QsRJEGVUIE40BlVVis9xcHN+X1eBuC7Gl+bMrTRuNbBxBy8If/4p4PeNQKsg3bchFiuOv66oE7mG64bewun8+fPo3bs3AGDXrl3w8fHBnTt3sG3bNnz++ed6bau4uBgdOnTA+vXr1S5fsWIFPv/8c2zcuBGnT5+Go6MjhgwZAqlU8U09ceJEXL16FQcPHsTvv/+Oo0ePYvr06foeFkFYPDXTdIDhZ9VVVCq+XD3dqvdLqTqiERCiplYiwN8HEFvxz54293pLnVHHGLDzD2D4dOByEq9T+moZ8NGc+rWDkafr6qhzohon3dC7OLykpATOzvyvcODAAYwZMwZWVlbo3r077ty5o9e2hg0bhmHDhqldxhjDmjVr8OGHH2LkyJEAgG3btsHHxwd79uzB+PHjkZCQgH379uHs2bPo3LkzAOCLL77AE088gU8//RT+/v76Hh5BWCzFNQrDAcPXOJUoRZtcnLiDOKXqiMZAmFFnJ+EF4c3ceaouM0dzRKnAAoVTfiGfNfd7HH8eEwWseQ/w1TBrThfcXIA797l1gTbKqMZJJ/SOOLVu3Rp79uxBWloa9u/fj8GDBwMAsrKy4OLiYrCBpaSkICMjAwMHDpS/5urqim7duuHkyZMAgJMnT8LNzU0umgBg4MCBsLKywunTpzVuu6ysDAUFBSo3grB0ano4AYp0mnIrhYYgWBE42isEGrmHE42B8PkWPnc+zfi9pnRdpUzxebWUVN2FBF4A/nscF4fvTge2r2yYaAIo4mRo9BZOCxcuxLx58xAcHIyuXbsiJiYGAI8+RUXp2EVQBzIyMgAAPj4+Kq/7+PjIl2VkZMDb21tlubW1NTw8POTrqCM2Nhaurq7yW2BgoMHGTRCmoqaHE8CFk0jERZMhCsSFqJajg2I/xuiFRxA1EdLNQhTVm3f7Qm4+TyHXRIg22UkUtX7mTFoGMPld4F4mr+H6ZR1v0GuIonYXJ57irKisPStWoKhEcR6pxkk7egunZ555BqmpqTh37hz2798vf33AgAFYvXq1QQdnLN577z3k5+fLb2lpaaYeEkE0GCFlplzjJBIpfj0aRDipiziVGiaaRRDaKK2Rina057PlGAOy1dgSWJLxpbQMeGUhjwi1Dwf2fgV0iDDc9q2slArEa0SdSqU80vX3KX4ura0BW3J41Eq9To+vry98fX3lgiMwMBBdu3Y16MB8fX0BAJmZmfDz85O/npmZiY4dO8rXycrKUnlfZWUlcnNz5e9Xh0QigURSR9MegrAwaqYyBOwkXDRJyxpe6yGPONkrLA+qGHcctoRf9YTlIkxCsFf66vZpBhSl8nSdv2ryQaVHnTnDGPDeKuDqDT7hYtNS3Xrw6YubC59Vl1cINPfhEbwbt4Hbd/n/MAD4NgMiW5GhbV3ofXoqKyuxYMECuLq6Ijg4GMHBwXB1dcWHH36ICqFDoAEICQmBr68vDh06JH+toKAAp0+flqcHY2JikJeXh/j4ePk6f//9N6qqqtCtm44mF4RGZDIePjakBxBhHAQPJ0A1VQcY1pJAWThZWSnSJqVUIE4YGXU/DHyq03WZObWjnpbiGL51D/DLQT5LcN3C2gLQUAh1Trl5QEIycOgEtzuoYlyw9e4MdOtg/kLTHNA74vTGG2/gl19+wYoVK+QC5uTJk1i8eDFycnL08nIqKirCzZs35c9TUlJw4cIFeHh4ICgoCLNnz8b//d//ITQ0FCEhIViwYAH8/f0xatQoAEBkZCSGDh2Kl19+GRs3bkRFRQVmzpyJ8ePH04w6A5CaDlxKBEICePiYMF8E4WInqf1r0VCWBDKZQnwJ6UAHO/5aiRRwd23Y9glCG0LESfmHgacbL6IuK+dCya16fhJjljGj7swlYFm1G897rwA9DFcmXAu36v/P3Hx+A7iYimwNeHsYb79NEb2F0/fff48dO3ao2Ai0b98egYGBmDBhgl7C6dy5c+jXr5/8+dy5cwEAkydPxpYtW/D222+juLgY06dPR15eHnr16oV9+/bBzk7xn7N9+3bMnDkTAwYMgJWVFZ5++mm9/aQI9RQV8/vCYtOOg6gboTDc0b72MkO5hwvRJmtrxTbt7QDkk5cTYVyqqhSfXzulVJ2VFeDlAaRn86iTIJyKSngE1lqs/n/CHMh8ALy+mM/+e6o/8NJY4+7P2YHXO5ZX8PYzka00NwYmtKO3cJJIJAgODq71ekhICGz1LMXv27cvmJaqUpFIhKVLl2Lp0qUa1/Hw8MD333+v134J3RAuhuQMbf6UaKhvAgzn5VSilKYTEH79U6qOMCbCd5GVSCHaBXw8uXDKygHCQ/hryo7hdfVyMwXlFcCri4Hsh0BES2D5POOP08oK6NGJfw94e5jnebEU9K5xmjlzJpYtW4ayMsW3cFlZGT766CPMnDnToIMjTItwMS6V0qwpc0edh5OAIJwMFXFSFk7k5UQ0BpomPgAKP6fcfMVn3Nwdw5esA85f5cLuq6WqM2GNiasTF5okmhqG3hGnf//9F4cOHUJAQAA6dOgAALh48SLKy8sxYMAAjBkzRr7uL7/8YriREo2O8GVVVe0BZEcTEc0WuRWBmguLxEDF4eqEkwMJJ6IRECJO6r6D7CRcgBQUAVm5QKCvefeo2/kH8N3/uHj5/EOgRXNTj4jQF72Fk5ubG55++mmV18hAsulRKVO4yAL8wkjCyXyRp+rU/HI11Kw6QTgpT5UWIgCUziWMibaIE8CjKAVFvG5IWTiZm4fTpURgwRr+eO5UoB9N/rZI9BZOmzdvNsY4CDOj5oWwpBTwoFlTZosuqboqxsVwfdspCOaXDmpSdRWV/GZDxnmEEZCq8XBSxqcZcOMOr3MqlfLPuUhkXhGn9GzgpQ+559mgHsDMiaYeEVFf6mVzVVlZib/++gubNm1CYSGX9vfv30dRkZY21YRFUXOWFKVizJeych4hBNRfWKysFIKmvlGnqirFZ0A54mRjrdg2RZ0IY1FXxMnDlX8OKyp5M1uAp5QN0a7EEBSXAi++xyNiYcHA6vfJZNKS0fv34Z07dzB06FCkpqairKwMgwYNgrOzM5YvX46ysjJs3LjRGOMkGpmas6Toomi+CH8bO4nmC4WdhF9UpOVAfVpxl5bxCQJiq9qzmuztgIoiPg5z+oWvjcJiwMaG3M4tBXWu4cqIRLx33b1MbuoImE9huEwGvLEMuJYMNHMHNsdyOwDCctFb886aNQudO3fGw4cPYa9UUDF69GgVl2/CshG+qMTVnxCKOJkv6pr71qShM+uE+iYH+9ozcuSWBBbi5VQqBeJOAyfOm3okhK6UaqnhExBm1wmNas1FOP3fBuDQSf6D4z//BwRo7gZGWAh6R5yOHTuGEydO1PJsCg4Oxr179ww2MMK0CF9U7q7Ag4cUcTJntHk4CTR0Zp1yc9+ayGfWWYiXU14hr/cqLOY3+vVv3siUJqpoijgB3AhTGXMQTlt3A9/8zB+vfh+IamPa8RCGQe+IU1VVFWQyWa3X7969C2dnM/ikEgZBuBh7uvF7Ek7mS6kWKwKBhppgKkecam3bwiwJlJ3ws3NNNw5CN5Sj39omNtjZAu5KeWhTp43jzgCL1/HH77wMDO9j2vEQhkNv4TR48GCsWbNG/lwkEqGoqAiLFi3CE088YcixESZEuMB6uPH7Shk1+zVX5Kk6LWmMhloSCBEndV3bBcFmiCbCjUFRieJxVo7pxkHoRl2F4coI6To7iWnr167fAmYs4ZMqnh0GvDbBdGMhDI/ewmnVqlU4fvw42rRpA6lUiueee06eplu+fLkxxkiYACHt4mRvOANFwjgI0SBdUnWGqHGqib2FpeqKlCJODx7yixthvmgzv6xJgC//rJuyjigrF5j6HhfoMVHAR3PIqbupoXeNU0BAAC5evIidO3fi4sWLKCoqwrRp0zBx4kSVYnHCcpGW8xoQgH9Z2Uv4BbdEah51A4Qq6rrG16QhxeGMKQlpbRGnci5CzH2adWF1xEkk4o1gc/P5bCfCPJHqMPlBwNEeGNrbuOPRRqkUeOkD4H4W0CoQ2LSk/r5phPmit3A6evQoevTogYkTJ2LiRIWDV2VlJY4ePYrHH3/coAMkGh/hi8pOwi+CDva8oNZSIgqPEuUVQGX1LCJj1TiVlXOBIRKpL86V2PLmq1WMb7+x+m7VB2kZP18iEe8Mfz+Lp+tIOJkv+kScTElFJTBzGXDxOq+12hxLPzSbKnr/NuzXrx9yc2tXVObn56Nfv34GGRRhWmpOb6dGruaLUP8hsdVu9iek6iplCrNMXRFqghzs1EeTRCLL+YwIheGO9oCvF39MBeLmjT41Tqaiqgp46xPgrxP8f+3r/6MedE0ZvYUTYwwiNQnbnJwcODrSvN6mgLKhIqDk02PmF8VHkWIdZtQB3FXZulpY6Rt1KtFS3yQg71ln5nVwRUpF7l7VUaa8Qpr4YM7IzS/NVDgxBixYC/x6iP+PbVwCdGln6lERxkTnVN2YMWMA8Fl0U6ZMgUSiiJvKZDJcunQJPXr0MPwIiUanpi+Qg4VEEx5FdPFwErCTcOEgLVNfq6QJdc19ayIX12aezhUKw50c+flwceLNYR88BJr7mHZshHrkESczTdWt/A/w3f945HXN+0D/7qYeEWFsdBZOrq68wytjDM7OziqF4La2tujevTtefvllw4+QaHSkNYqNhT81RZzMD0GoqDOmrInElgsnfaMrRWqa+9bEYlJ11cfiXC0CvTy4cMrKIeFkjlTKFE7g5hhx2vADsP57/vjjOcCI/qYdD9E46CycNm/eDIA7hM+bN4/Sck2YmjUFwi+98gr+RWZtJo0zCf0iTvW1lSjRQZxZSlRSOeIE8P5myalU52SuCN9F1mJFM2lzYfv/gE++4o/fewV4boRpx0M0HnrXOL399tsqNU537tzBmjVrcODAAYMOjDAdNYvDbW0A6+ovLZpZZ17oYn4pIJ9Zp2/ESQfhZG8BdXCVMkW9jHAsnq58RmBpmaqjOGEemOuMuv/9DXywhj+e8Rzw6niTDodoZPQWTiNHjsS2bdsAAHl5eejatStWrVqFkSNHYsOGDQYfING4VFUpUjnKX1amaORaVaWoryHUo0uDX4H6RJzKyhV2B5YunAT3c1sbxbkQiwHP6iJxchE3P6R6RFQbi0MngTkf86Lw558C5r9k6hERjY3ewun8+fPo3Zs7jO3atQu+vr64c+cOtm3bhs8//9zgAyQaF+W+UBKllgWNPbOOMeDURT69N4vSKGrR1cNJoD4mmEKE0U6i3e5ASOfKqsx3hpoQUarZ1Nfbk99Tus780MXctTE5exl4bTGPXo4cACybRa7gjyJ6C6eSkhJ5M98DBw5gzJgxsLKyQvfu3XHnzh2DD5BoXIQLZc1feI3dViP1vuJClnircfZpaejq4SRQHxPMYh2Lz8VihdA216hTkYZ+e14e/J7ar5gf5uThlJXLRVNZOTAgBlj1rvm75BPGQe8/e+vWrbFnzx6kpaVh//79GDx4MAAgKysLLi4udbybMHc0eaY05qwpaRlw5abieW4+v6gRqsgLw3Ws/7CrR786XYUTYP4F4oU1CsMFXJ246BParxDmg7zGyYQNe4FqV/Al/MdcWDCwboH5FasTjYfewmnhwoWYN28egoOD0a1bN8TExADg0aeoqCiDD5BoXDT1hZL3I2uEGqfLSTwF5eYMtPDnryXdNv5+LQ1djCmVESJC5RW6R1b0EU7mXuckRJxqpuoARdSJ6pzMC3nEycRtfFb8Bzh9iUcrNy4x77ZChPHRWzM/88wz6NWrF9LT09GhQwf56wMGDMDo0aMNOjii8dE0vd2hkVJ1Gdm8f5hIBHRsw6chp6bzX3p5BYAbBTXl6DOjDlDtKVdWrlv6Q25FoINhpjl7OTGmZEWg5li8PYG7GVTnZG7II+AmnFX351Hgq5388cq3gVZBphsLYR7UK9jo6+sLX19flde6du1qkAERpkXTF5Xwi09azqMVxsjtV1QCFxP549ZBPIUCcGPCuxk86tS1veH3a6mUaogOakNiy//G0jLdhJMQpdEnVdcYUUl9KZFywSi2Un++hIiT0H5FYuLUEMG/D4TJD6ayI7iVBsxbzh+//CzwRB/TjIMwL6i0jVBBU3G4xIZHKxgzniVBQjK/6DraA+EhitdDW/D79Gzu8kxw9LEiENDHy6lSpqiH0iWq1VhRyfogRJscHdTPgrKz5e1XAIo6mQty80tr09QTlZQCry7iPx66tQfend74YyDMExJOhApSDcXhIpHi4mmMGpbcfCDlLn/cIUJ1lpiLE+DvzR/foImbcjSJXG3oM7NOqG+yteG3umhoqq6ikk8CuHGHT/u+nMSFuiEo1DCjThkh6kTCyTwwZZqOMeDdVUBiCv9cfLGQOiYQCmheACFHaKkCqP+ysrfjv74MXcNSVQVcSOCPg/wUFzBlQlvw2qd7mUBES91SR02ZikpFDy99ClX18XIq1iNNByiEU3kFIJNpt0hgDMgvBB4W8Nq1hwXqnbv9vQFPN932r42iGh5OxaU8gplXwMeRVwCkZQLXbgBlFTxNbGvDRbuLE+DiyO9dnRWvhQQYJ4WUmQNcSeI/IJq5G377loIpzS+//RX49RBP7X65CPDxbPwxEOYLCSdCjvALT5MvkLFMMG/c4RdNWxugTWv167i5cEGVnQvcvMMvKo8ygni1tdHvl7A+7uHFGmbtXU4EVm3mBqVeHlzcNPfmtWg5eQpR0TqIX/TUpcZO/KveYsJeAri7AvlFQEEh/3sbRDhVi8CsHOCbn4E9B7lAagi2NkDHSF53170D0KltwwU9Y1w0FZXw/4tHWTiZyvzy32vA0vX88XuvUF0lURuzFk6LFy/GkiVLVF4LDw/H9evXAQBSqRRvvfUWduzYgbKyMgwZMgRffvklfHyozXl9KBVSPxp+RRvDBLOwGEhK4Y/bhWkvyg0L5hfS1Pu8BqquX/v5hTz6oS6CZenkF/J7fS8qdnoIJ+HvLKS3ElO4YNp/TLFO6n1+q8mGH/i9xBbwcAXcXbggcnfhn6OCIl63IqsCysuBkjIeFcovAh7m88+FWAwENwf6dOECJaoNEOirv1NzVRUXan+d4Mcg4OIIuLly2ws3Zy7OS6S8lq9lII8uFRTVvuUXcYGYXwicucRv677jArZdGL/QduvAx+7tyc+frmO+n6UQedm5/POrS5q0KSK3ImjEVF1aBje5rKgEhj0OvDS28fZNWA5mLZwAoG3btvjrr7/kz62tFUOeM2cO9u7di59++gmurq6YOXMmxowZg+PHj5tiqBaP/Beehl/Nhp5uzhhw8Tqf7eTtCQT4al+/mTuPPuTkATdTgcdCNW83MYXPwmMM6BFlfuJJJuMRBZGIX7DdnLWLRsZ4Oik9m9+Ei6umv9X1WzyyEtMRGN5XcfGVR5x0SNUJ+3iQB7yxDPjtMB+HSASMGghMe5pv516mIo169QaQ8YCLihIpTwkKY9YXmQxITuU3AU83LqA6RgIRIfwz0cwD8HKvLaRLSoGfDwD/3aWon7OyAob2Bl58Guj8WG1BczOVH4OPJ9C9o+axMQbcvgecvshvZy4BdzOBfxP4bdNOxbr2doC3B/+MC/dCJKlSVp12rU6T377HBYO1GOgZze05gvz1P3dNgcZs8FtWDnz9I/DFd/xHRctAbj1A7VQIdZi9cLK2tq5lfQAA+fn5+O9//4vvv/8e/fv3BwBs3rwZkZGROHXqFLp3797YQ7V4BEGk6YvK0NPNb9/jIshaDHQI1+09oS34e+7c449rio1SKRB/la8jcO0m0MfM3DKuJfOpzso42nMR5e7C712ceO1PRrXwUD7vViIuGMJCUIt9x3gT0hIpsPMP4ONNvBnpxBH61Til3gf2/MULtWXVhplP9AHmTOHRP4Eu7RSPL17nf9ewYH7Bz80H8vL5fW4+P44LCTwNKIgIIRLl4cofCxGqP4/xv520jIvgazf53/WvE/xWEycHLpCbufP3n76kiMzZSYDe0cCiN3jUShM1269ost0QiXg6MiQAGD+cv3Y3gwuo05eA+CsKgVsqBe7c5zd9OHkByHwALJzxaF7AG6vdStwZYPEXCnHdvQMXTeqMUgkCsADhdOPGDfj7+8POzg4xMTGIjY1FUFAQ4uPjUVFRgYEDB8rXjYiIQFBQEE6ePEnCqR7U9UWlPN1ciDzUl+xcPmsK4MXeuhY4+zTjKZT8Qv5FF9FSsSw9m9cnVFTyKcxtWvGLbV4hj4gIM/NMzYOHCtHk58XTUkUlXEwUl/LIjTqsxVxo+Hnx81BzijZj/Bfzqm/4844RwP1sXtfz2WaeThreB2gdzIvwlf+GjPFzdOE6cDGBCyBlwTQgBpg7VXOUT0A5Kuloz2/KQuVCAq+F8vOqu3akXSh/f3BzXtMmLefRoAsJ/O+clg5kPwQe5PJ6paISfhMugAAXb6MG8GhmoJ920QQo2q+UlXORpk+kMsCX38YMVrxWUsp7nGXlKN1y+WfAyor/DW2s+ec1I5un5rw9eZTtxL88anj7HvDpO4ap9bIkNM3wNRR3M4ClXypSz96ewAev8ua9j6JQJXTHrIVTt27dsGXLFoSHhyM9PR1LlixB7969ceXKFWRkZMDW1hZubm4q7/Hx8UFGRobW7ZaVlaGsTPHzvaCgwBjDtzikdRRj2kn4F4rgPF3fEHpBEXDmMr9YN/fhYXF9CG0BnLvCxUerIB59uXYTuFV9wXRzBjq34xfdsnKetktI5hdrU38hVlTyiz7A28l0jOSPyyu4wMvL5/cPC/jfQ2IL+DbjY/fy0BwBKZVyo77f4/jzqWOAD1/nUZM/jgCbf+aiaHd11rtVIK/VyX3ITUcvXldfrB3REoh9C+jURrfj0zaBQFrOxQ6gm/uylwf/mwptUOxsgei2/KYMY0BBMRdQDx7yW1YuF4d9uwIJt7gQ0TWC4O1ZLcpyG57idagWfsHNta/34CFw/Dz/LA/qyVOr768GfvwT+PsUMGQasPo9oHfnho3HEEjLeWrRmBGZumb4NoSycuCrH/kPCWkZnzk39Wlg9mSKMhG6YdbCadiwYfLH7du3R7du3dCiRQv8+OOPsG9A86LY2NhaReeEki+Qhi8qKyt+8Sot4xGF+ggnaRlw6gJ3BPZ04xdkfcWMvzdPyxSVcEGUk6cwxmwVxCNNgsBoFcQjEEUlPPXUoo4LmLG5eoOfOwc7oK1S9MbWpjp1pXShLivnr9d1fu5nAS9/CFy5waMXy2YDE6rTRxDzeqRRA7lg++ZnLq6S04APV6tux1oMhLfkkarg5lwgh4foLpoA7f3qUtL4Nj1cdYueNPPgQqJEyv9+mjyYRCIeKXJ1Ui/IaloR1IWXBxdO6dlAZKvGEdtCL8Ygf8X/1ZTRXPx99z8ehXx+PjB9HDB/mukKxhkD/jnHI6MdI4z3/yTUN9lYa7e10AfGuAhd9qVqWm7pLFXDXYKoC7MWTjVxc3NDWFgYbt68iUGDBqG8vBx5eXkqUafMzEy1NVHKvPfee5g7d678eUFBAQID9Qx7NDEYUxQMa9OkDvb8S61UCsBVv31UyvgU9tIyfhHs2r5+rVtEIh51+jdB8QVoa8Ong9f0W7Gx5vU2V27wyFOAr+G+iPUl84GiziWqTd1uyLq0/Yi/CryygKesPN14A1JNKbCoNsAXbYB+3YF9R/nFuHULXl/WIRJo21px0b6VxlOp2gwj1SGPOJWppgIrZYq/la69vqzFvOYpJ49Hf/Qdi4BQ5K7r+32r06BFJVyUNjfyJN28An58IhH/ewj4V1s8vDWVR1i//533TDtxHvj8Q9P0TMvJU9hUXLgOVMi47YShMbSHU1IKT8sdO8efe3kAH75GaTmifliUcCoqKkJycjJeeOEFREdHw8bGBocOHcLTTz8NAEhMTERqaipiYmK0bkcikUAiMWHXSDNEWn2hsxLx9iqaqO/MOsaAc5d5bZKtDZ+x1JBfzQG+XAiVSHkxcHRbzRGw4OY8VVNaBqTcM84XfV2UV/ALDcBTkw3155HJ+IyxD1bzbUe2BP7zUd0zEwGe9hveV3vEoK5Ze5oQ0rmMqfbDS73P05RODnz/uuLtoRBOIQH6jQXg50n4rDrpGHGyseai5Pot/hnz9zbuxVWINjX3UfWBcnFSRFZnTAT6dgPeXsl/BAyZxlPSNjbVLUnE1Y+r7x3s+Oe+ZSC/tQ7i22/ojwah/s5Owv++V2/w6LFyraEhMJSHU24+r+/7/jder2drA0x7hp9PSssR9cWshdO8efMwYsQItGjRAvfv38eiRYsgFosxYcIEuLq6Ytq0aZg7dy48PDzg4uKCN954AzExMVQYXg9KlH7habtI1NcE83ISd0QWW3GPm4YaBVpZAT078cJqb0/tYxaL+Rf7vwnAjdu8tqixe19dSuQXGicHnkrUFcZ4jc/1W0DibSDxFn98445iZtyQ3rz+RddzKng5aZpZV1is8GYSGi3rikjEU70lUn6zt+PHIBTDtwzUT4R4e/Iapew6Zrlporh6IoO1teK4daFlIBfbhcXGjToVFCmsGkJb1F7u782F1f0sYEgvHh2cE8sLx7PV1KQpc/y86nOJDRAcwI/tsVBg8mj9xIMwgQDg0cu8fP63SUzhovixUMMJzIbOqKuoBLbtAdZsVaTxh/YG3n/F9Ol6wvIxa+F09+5dTJgwATk5OfDy8kKvXr1w6tQpeHnxn6yrV6+GlZUVnn76aRUDTEJ/hC+quuqW6mOCeTNVkabp1JbXuBgCB3vdIyKBflxsFJXwC6KhfyFr414mv4lEPDKm7Ve/TMZF1rF4fnFMSOapHHU42AGvjAfefEE/QaGt0a9MxtNCsioeFauPYLC346JJ+Ezdz+ICxtaG1+zog6szf195BS+Y13dmmRA5c9Yzzaccdbp+y3hRJ6H3or+3osmwMv4+XDhlPuBiwNcL+H4VT/mWlNb2gRIeFxbzwvpbafzzfvsun3mYmMJvfx4Ftv8GfDwX6NdNt7FmP1QYcnq582igtTX/UXQrjUeeOkYa5jxp8nAqrwB2H+TF9EIPRVtbLgqF50WlwLpveR0fwH+oLJzJPc0IwhCYtXDasWOH1uV2dnZYv3491q9f30gjaroIEae6hIi+qbr7WTycD/BfpKayBBCJeDuXM5f4hSQ4QL8IRH2RlnMhBPBaKzeX2uukZfCC26PneJRA8B4SsLLiaaqIEF7EGt6SC7/AetZraWv0e/Um/4Vua8NFXn0ugjU/IzerxUFIgP7jFYm4gLufVb/2K0JhuK5pOmWEqFNRCRe+uqRB9UHZekJdtAngET8hXZeVw4WsSFT3LL2ayGTAvSyFoejWPTyqOOVd4JkhwILX1X82lblXPVlZWUS2DOTi6UICkJrOxVt02/rVLiqjLuJ09jJvvCt8nuqimTswbxrw7FDT1TUSTROzFk5E4yH3TKkj4qRc/FsXD/O5ESAAtAwwTTGrMn5e3BjxYQFwIwVop6PpZkO4mMB/Jbs6K0wj84v4zMJj54B/4lV9hwDeCqRHJz71vGMkLxg2pMiTaEjVpWerRgbrazehnM598JDbK4it6lejBPDIhiCc9I0UFupZGK6MjTWvDRLSUYJoMRQ37/D0l5eHdtHi58UjUw1JGYrFPNoX5McjTM89CXz6DZ9luWs/cOQs8NEcng5UR1WVIqVYcwxBfry2Kv4KH2OlDOjarmFiRfn7KL8QiN0E/LCXv+bpBvTvzqNr5RXVt3J+X1bBI1+9OwOvP0d1TIRxIOFEAFD8wqurGFNYXllZdx+thFt8+rlvM+CxMMOMs6FEtuIpsNv3gJZBhmnKqom0dN5+RCbj5+vTb4Dj8cClJH4hEhBb8ZqRxzsDvbsA7cP1a9yrL+oiTiWlCn+p1i0a1g1eWTgJ7VIC/XSbJagO7+qxPCzQv3ebvlYENQkJ5KlmQ0edpGWKOjJlF3Z1NPfhwinzARclhvhsONhzR/LhfYH5K/jfafoCYEQ/YMmbtSN7WTlcqAhNpXPyeMpdEJL+3oB1R+DMRb7uXye1j9PbU3tNVKlUYR+w/GuFx9j44cB70+uOjhGEMSHhRABQardSh3ASixU1J6VSzRexklIeIQB441NzmfLr5cFv2bm8dqWmmaKuVFQCR88qamgEyiv4tgW36OTq+q6a0Z1WQby4vVc0r71QV99iLGo2+mWM2xpUVPKIXGQD678EO4vcfL5NoGHRRns7RbrqwUPd072M6W9FUBNjRZ1upvIfFZ5udc+wdHXmAr+4lH+mDJnujm4L/PE18PlWYOMO3o/wn3jg1fHcakBwO0+5yz/XhcWKv6nEhtdc+XkBft783sWJ1+S5OAEebpp/iBWV8B8MbVrXXlZWzve580/+Pwrwz0/sXD6xhCBMDQknAoB+ncgd7LhAKJHyL3V1pFXXQzRz139Ku7Fp0wo4kstbLrRuof/MMcZ4LdLpi3ymYHa1SMrO5WkpdXh5cJHUqxNv3qrPlHxDI0R+qhj/OyancpFjbQ1EP9bw+hThMyRcYP286i9cBLw9+MU2O1d34SAt4xEakahhkUXlqNPdzLrbttRFWTnvtQhorm2qiZ83T+3dyzR8naCdLfD2y8Cwx3n0KeEWEPtV3e8rq6i7B5+zIx+vnxe/9/fhNgoZ2bz2Mf4q/9tIy6pv1U2jd/6hiHDNnAi8OqH+EUuCMDQknAg+E6f6IqfL9F8Hey4QNFkSMKZIQ7Qww87ubi78S/x+FnA9Wfuv2If5wPXqmUiJtxSPi0s0v8fNmU95btEciIrggik02HyibmIxj6RUVHLxKPgIdYxoeOoSqB1laK2jONCGt6dq+xVdEOqbHO0bJgZtrPkxJCRzI8WABkadUu5yQefqzHsO6kLzauGUlcNTv8Yodm4XDvxvI/DNLt4SycOVn3eJLY8ieXkAT/blr1lZ8TR0Rjb/P8rI5n0Rhef3s3g6r7BYMZNPXyJaAl8uMn1tJEHUhIQTIRdAQsPRuqhrZt2Dh3yZtbVpIyvaiGjJi10zHvCoUc2anrQMYMxMzRdqsRUXXx0ieEQipLni3t1AdgvGxE7ChdOV6hmPwc0N51WknM71cDWM/YSnu27tV5RpaH2TMiEBihl2DYk6lVcopsnXVdukjJsLF6QlUp4GNtb/la0Nj+68OkHx2tnLXAi1CuItYQSEYnNNFJfymXhpGVyg383kdX93M1SjgXYSfjwOdvxxeQU/3uF9SDQR5gkJJ0I+Q05Xs7m6TDBTqxu5BhjAqdhYODvyi+GtNODSdd6GRLmY1ccTyM3jjwP9qm0AQvh7cvL4r+8+XRruAG4qJLY8GsAYPxePhdb9Hn1wdeZpNV1TUXVRn/YrDa1vUkbwdWpo1Ck5lU8UcHHSX/z4efP338tsvB8klTJelA7wY9YHR3sgLITf1FFWzusES6Q8itW9Az+n127yYviGuoYThLEg4UToPKNOQJsJZnkFkF7tLmyOaTplIlvx1EKJlF8Q2ynN/LO14WmLFv6qF974q9UNeT0tVzQBipl1Yiug82OGF7id2vCIg76+S9rw9tSv/UphAzyc1KHs65SWob+ZZ3kFTzcCXITXp7l1ciqPkBorXVeTjGxuhupob/iZbBJb3lfx2DmFO35kKyXzSxJOhJnSwDJQoimgb3sDBy2puvtZ/IvW2dH8pwxbi4H2Efxxyt3aDt1tW6uKpvwinmYA9GubYo74ePJUaodI48zos5MYVjQBvEAcULRfqYv6uoZrwlqsqNdKStFuRaGOhkSbAD7j0V7CtyHMWDU2QosVY7WccXXmn0GA19qlZ+s3UYUgTAEJJ0JhRaDjF5UgsMor+C9fZYTZQuYebRLw8eQXBcZ4LzttF+SEm/ze31vzbEJLIcAXeOLxhs8Qa0yE9iuVldzTSRsVlQq7BUcDCSeAR7psbXg0LS1d9/eVlSv69UW0rF+aTyTi6TqAu4DXhDEu7u/c47MANfUi1JWKSkWazt9Iwgngn8GWgfzx+auK3nL17VNHEMaGUnVNmIpKXi/g48n9VjShb6pOMMGrrO48LxTfFhTx2XYikeHbUxiTx8J4qqCgiF9w1BXt5uTxFIlIxNMJTQFzmeWnK/q0XxFmPUps9TPMrAtrMZ8hefUGb0+jq91GcqpiJl1D6pP8vbkAy3jAU5H5hVxE5hfyiKlMSfgnpfDataB6/ohJz+aWFU4O+lt26Evb1vwYcvIUr1HEiTBXKOLUhLl+iztkx19V35dMQKpncbjyusrpOqEo3M/LsjxX7GwVzuZJKbVNLQFeAwXUrnkiGhdh9mNdqaoCob7JCH+rkObccqK8gk/brxl1rUlZuaKVTbiGQmld8XDlkeHKSu6qHX+VC6mcPC6arK25mHN25D+c/k3ghpZCvZc+3K/uo9cYP4KsrIDO7VTFUn1b/hCEsSHh1EQpKlF8WVfKFNPOa8KY/hEn5XWl1e+tqlKkLiwp/SMQ5MdnysmqgIvXVZdlZPMLk9iq4Rc+omF4Vdc5Ce1XNGFIK4KaiMW8qNnWhkdJLlzXvr6hok0Aj7oJaXArEa97CgngLXv6d+fp156dgL5duSu32Ip/duNOcy8lXWrDgGr37mpx2liNue2qi8Wtxfy4GmrEShDGglJ1TZRrN7kocnXmX+73MrlXT82ZYGXlPBwv+KnoipCeECJOmTn8Qiax1d3Uz9zoEAEcPsV9qFLv8xQHY9xJGeA+TfQr2LTo2n7FkFYEmsbRpR3ve3g3g0eg1HkOKUebIgwkusNDuEWGvUSzuLCy4lYQ/t7ApUTFrLW7GbxxdF2F+xnZ/LPv4tS4jXLdXICBPYzbq5EgGgpp+ibIg4e8PkEk4tPCg5vz1y8l1v7FKUSb7Gz1q3kRBIQgnASn8CA/y6udEXC0B8Kr+7RducHbP9zN5LVP1taG8yQiGoa3Duk6ITXlbMTanGbuvDYH4PVOQiNaZW5WR5vcnLXXGeqD0EJGl4iMoz3vhRjdlv+oKSrhqbt/r/FIlKaZgfeMPJtOGxJb8/V/IwiAIk5NDsZU3aBdnHgx8/0sfjFJuav6y1gQPvr2k1M2wZSW8YgTwH8JWzKtg3h0Lr8QuJzI7wEumgxZZEzUH28PXtdzN1N9PRrAZ70BhrMi0ESrID4h4m4Gd9ju21VR/yctB1KqZ9KZOsUb4MsF57WbvLdcajq/SWwB32Z8tp6XOxdj0nKFCDSFcCIIc4eEUxMjLYNf7K2tFV/Wtjbcd+jCdd5rrbkvjzABSmZzeqag5Km6Ur5PVt3pvTHD+sZAJOI9246eU3jYSGwV06UJ0+Ppzj/flZXqozwCEtvGSa12jFDMcDtzifcmFIt5bZOsyrDRpoZga8PTdEH+XHhm5lQ3HK5u1GttzUWptZj/P7s5G6Z3IUE0NUg4NSEqZYrZX2HBqjPbgvz5DLu8Qj6VOrotf70+heHK60vLVdN0TQE3F4VLNMAFKNVcmA/WYuDxzopooCbcXRsnbSwWA13bAXFn+P/XpUQgsrUi2hTR0vhj0Aehf2BVFfAgj9czpWfzyPF9JX+o5hY4yYMgGgMSTk2I5FT+5edoD7Ss0ZJCJOLFz0fO8rRCcHMeIdLXNVxAYstn9VQxni6xFivM+ZoCES15rzorK8sx83yUcHY0r+imgz2fTn/qAk+B5RXyaJO7i/lOlrCy4hEmbw/ebiivgAuo9Gy+3JK82AiiMSHhZEKk5VzEMMbrDBpyIZCWATdu88eRrdQXV7q5cBFw5z7/Vdy3a/3bG4hEXGwJtST+3rwRalPBWgw83sXUoyAsCW8P/r937abC/TrczKJNmhCJeITO3ZXbGBAEoZkmdKmzHHLyeJF2ehaP2AD8y9bJgfu8+HlxkaNPmiEhmf/C9XDVXtApFIoXFPExCMXh9vWoZVAWTvV1JyaIpkRoCx65uZ9VHW3yNPWICIIwNCScGolKGXAvg3dHF36NAlzoWFsDD3J5yuvGHX6zk3AB5esFNHPTPvU4v1Dh2t02VPs4JLZcPF1K5P5ElZX89fq0NxDqnJwcDN/QlSAslU5tuFEniSaCaJqQcDIygoN3arpCpIiteFSoZaCiWazQUDM9m5vVScv4+1LucmHl48nTYd6etQuVBfuB5j5ciNVFcHOerhOKa63F9Ztq7+XBj6u1GuM/gnhUEYsV3mkEQTQ9SDgZkcJi3k9KwNEeCA7gs89qChUba16MGeDLZ7tkP+SpvIwHfMrwvUx+sxJx8eTbDPDxAvLy+ZRsKxG3HNAFkQhoHw4cO8ef17cLeYAvL3xtSrVNBEEQBKENuuQZEWdH7oUiseXtOrw9dKtbsrLiESYfT144npvPI1EZ2bymKOMBv+G6IvrUKkg/E0sPVy7gUtMb5tVCookgCIJ4lKDLnpHp3blhzSpFIl4/5OkGPBbK66MysoH72TzVVinjwiw0WP9tPxbGa6kaq4knQRAEQVg6JJyMjKE7fLs48VtYCLcSyM7lM/DqE/mxseaF4gRBEARB6AYJJwvG3o5sAAiCIAiiMTFwPIQgCIIgCKLpQsKJIAiCIAhCR0g4EQRBEARB6AgJJ4IgCIIgCB0h4UQQBEEQBKEjJJwIgiAIgiB0hOwIADDGAAAFBQUmHglBEARBEI2NcP0X9IA2SDgBKCzk3W4DAwNNPBKCIAiCIExFYWEhXF1dta4jYrrIqyZOVVUV7t+/D2dnZ4h0aSanBwUFBQgMDERaWhpcXFwMuu2mDp27hkHnr/7Quas/dO7qD527htGQ88cYQ2FhIfz9/WFVR8sPijgBsLKyQkBAgFH34eLiQv8I9YTOXcOg81d/6NzVHzp39YfOXcOo7/mrK9IkQMXhBEEQBEEQOkLCiSAIgiAIQkdIOBkZiUSCRYsWQSKRmHooFgedu4ZB56/+0LmrP3Tu6g+du4bRWOePisMJgiAIgiB0hCJOBEEQBEEQOkLCiSAIgiAIQkdIOBEEQRAEQegICScjs379egQHB8POzg7dunXDmTNnTD0ks+Po0aMYMWIE/P39IRKJsGfPHpXljDEsXLgQfn5+sLe3x8CBA3Hjxg3TDNbMiI2NRZcuXeDs7Axvb2+MGjUKiYmJKutIpVLMmDEDnp6ecHJywtNPP43MzEwTjdh82LBhA9q3by/3fImJicGff/4pX07nTXc++eQTiEQizJ49W/4anT/NLF68GCKRSOUWEREhX07nTjv37t3D888/D09PT9jb26Ndu3Y4d+6cfLmxrxkknIzIzp07MXfuXCxatAjnz59Hhw4dMGTIEGRlZZl6aGZFcXExOnTogPXr16tdvmLFCnz++efYuHEjTp8+DUdHRwwZMgRSqbSRR2p+HDlyBDNmzMCpU6dw8OBBVFRUYPDgwSguLpavM2fOHPz222/46aefcOTIEdy/fx9jxowx4ajNg4CAAHzyySeIj4/HuXPn0L9/f4wcORJXr14FQOdNV86ePYtNmzahffv2Kq/T+dNO27ZtkZ6eLr/9888/8mV07jTz8OFD9OzZEzY2Nvjzzz9x7do1rFq1Cu7u7vJ1jH7NYITR6Nq1K5sxY4b8uUwmY/7+/iw2NtaEozJvALDdu3fLn1dVVTFfX1+2cuVK+Wt5eXlMIpGwH374wQQjNG+ysrIYAHbkyBHGGD9XNjY27KeffpKvk5CQwACwkydPmmqYZou7uzv7z3/+Q+dNRwoLC1loaCg7ePAg69OnD5s1axZjjD53dbFo0SLWoUMHtcvo3GnnnXfeYb169dK4vDGuGRRxMhLl5eWIj4/HwIED5a9ZWVlh4MCBOHnypAlHZlmkpKQgIyND5Ty6urqiW7dudB7VkJ+fDwDw8PAAAMTHx6OiokLl/EVERCAoKIjOnxIymQw7duxAcXExYmJi6LzpyIwZMzB8+HCV8wTQ504Xbty4AX9/f7Rs2RITJ05EamoqADp3dfG///0PnTt3xtixY+Ht7Y2oqCh8/fXX8uWNcc0g4WQkHjx4AJlMBh8fH5XXfXx8kJGRYaJRWR7CuaLzWDdVVVWYPXs2evbsicceewwAP3+2trZwc3NTWZfOH+fy5ctwcnKCRCLBq6++it27d6NNmzZ03nRgx44dOH/+PGJjY2sto/OnnW7dumHLli3Yt28fNmzYgJSUFPTu3RuFhYV07urg1q1b2LBhA0JDQ7F//3689tprePPNN7F161YAjXPNoCa/BNFEmDFjBq5cuaJSK0FoJzw8HBcuXEB+fj527dqFyZMn48iRI6YeltmTlpaGWbNm4eDBg7CzszP1cCyOYcOGyR+3b98e3bp1Q4sWLfDjjz/C3t7ehCMzf6qqqtC5c2d8/PHHAICoqChcuXIFGzduxOTJkxtlDBRxMhLNmjWDWCyuNRMiMzMTvr6+JhqV5SGcKzqP2pk5cyZ+//13HD58GAEBAfLXfX19UV5ejry8PJX16fxxbG1t0bp1a0RHRyM2NhYdOnTA2rVr6bzVQXx8PLKystCpUydYW1vD2toaR44cweeffw5ra2v4+PjQ+dMDNzc3hIWF4ebNm/TZqwM/Pz+0adNG5bXIyEh5qrMxrhkknIyEra0toqOjcejQIflrVVVVOHToEGJiYkw4MssiJCQEvr6+KuexoKAAp0+fpvMIPu125syZ2L17N/7++2+EhISoLI+OjoaNjY3K+UtMTERqaiqdPzVUVVWhrKyMzlsdDBgwAJcvX8aFCxfkt86dO2PixInyx3T+dKeoqAjJycnw8/Ojz14d9OzZs5blSlJSElq0aAGgka4ZBikxJ9SyY8cOJpFI2JYtW9i1a9fY9OnTmZubG8vIyDD10MyKwsJC9u+//7J///2XAWCfffYZ+/fff9mdO3cYY4x98sknzM3Njf3666/s0qVLbOTIkSwkJISVlpaaeOSm57XXXmOurq4sLi6Opaeny28lJSXydV599VUWFBTE/v77b3bu3DkWExPDYmJiTDhq8+Ddd99lR44cYSkpKezSpUvs3XffZSKRiB04cIAxRudNX5Rn1TFG508bb731FouLi2MpKSns+PHjbODAgaxZs2YsKyuLMUbnThtnzpxh1tbW7KOPPmI3btxg27dvZw4ODuy7776Tr2PsawYJJyPzxRdfsKCgIGZra8u6du3KTp06ZeohmR2HDx9mAGrdJk+ezBjj00sXLFjAfHx8mEQiYQMGDGCJiYmmHbSZoO68AWCbN2+Wr1NaWspef/115u7uzhwcHNjo0aNZenq66QZtJrz44ousRYsWzNbWlnl5ebEBAwbIRRNjdN70paZwovOnmXHjxjE/Pz9ma2vLmjdvzsaNG8du3rwpX07nTju//fYbe+yxx5hEImERERHsq6++Ullu7GuGiDHGDBO7IgiCIAiCaNpQjRNBEARBEISOkHAiCIIgCILQERJOBEEQBEEQOkLCiSAIgiAIQkdIOBEEQRAEQegICSeCIAiCIAgdIeFEEARBEAShIyScCIIgCIIgdISEE0EQTZotW7bAzc3NqPsIDg7GmjVrjLoPgiDMAxJOBEE0acaNG4ekpCRTD4MgiCaCtakHQBAEYUzs7e1hb29v6mEQBNFEoIgTQRBmTVVVFWJjYxESEgJ7e3t06NABu3btAgDExcVBJBJh7969aN++Pezs7NC9e3dcuXJF/v6aqbqLFy+iX79+cHZ2houLC6Kjo3Hu3Dn58p9//hlt27aFRCJBcHAwVq1apTKerKwsjBgxAvb29ggJCcH27dtrjTkvLw8vvfQSvLy84OLigv79++PixYsGPjMEQZgCijgRBGHWxMbG4rvvvsPGjRsRGhqKo0eP4vnnn4eXl5d8nfnz52Pt2rXw9fXF+++/jxEjRiApKQk2Nja1tjdx4kRERUVhw4YNEIvFuHDhgny9+Ph4PPvss1i8eDHGjRuHEydO4PXXX4enpyemTJkCAJgyZQru37+Pw4cPw8bGBm+++SaysrJU9jF27FjY29vjzz//hKurKzZt2oQBAwYgKSkJHh4exjtZBEEYH0YQBGGmSKVS5uDgwE6cOKHy+rRp09iECRPY4cOHGQC2Y8cO+bKcnBxmb2/Pdu7cyRhjbPPmzczV1VW+3NnZmW3ZskXt/p577jk2aNAgldfmz5/P2rRpwxhjLDExkQFgZ86ckS9PSEhgANjq1asZY4wdO3aMubi4MKlUqrKdVq1asU2bNul3AgiCMDso4kQQhNly8+ZNlJSUYNCgQSqvl5eXIyoqSv48JiZG/tjDwwPh4eFISEhQu825c+fipZdewrfffouBAwdi7NixaNWqFQAgISEBI0eOVFm/Z8+eWLNmDWQyGRISEmBtbY3o6Gj58oiIiFqpwKKiInh6eqpsp7S0FMnJyfqdAIIgzA4STgRBmC1FRUUAgL1796J58+YqyyQSSb2EyOLFi/Hcc89h7969+PPPP7Fo0SLs2LEDo0ePNtiY/fz8EBcXV2uZsW0RCIIwPiScCIIwW9q0aQOJRILU1FT06dOn1nJBOJ06dQpBQUEAgIcPHyIpKQmRkZEatxsWFoawsDDMmTMHEyZMwObNmzF69GhERkbi+PHjKuseP34cYWFhEIvFiIiIQGVlJeLj49GlSxcAQGJiIvLy8uTrd+rUCRkZGbC2tkZwcHADzwBBEOYGCSeCIMwWZ2dnzJs3D3PmzEFVVRV69eqF/Px8HD9+HC4uLmjRogUAYOnSpfD09ISPjw8++OADNGvWDKNGjaq1vdLSUsyfPx/PPPMMQkJCcPfuXZw9exZPP/00AOCtt95Cly5dsGzZMowbNw4nT57EunXr8OWXXwIAwsPDMXToULzyyivYsGEDrK2tMXv2bBW7g4EDByImJgajRo3CihUrEBYWhvv372Pv3r0YPXo0OnfubPwTRxCE8TB1kRVBEIQ2qqqq2Jo1a1h4eDizsbFhXl5ebMiQIezIkSPy4vDffvuNtW3bltna2rKuXbuyixcvyt+vXBxeVlbGxo8fzwIDA5mtrS3z9/dnM2fOZKWlpfL1d+3axdq0acNsbGxYUFAQW7lypcp40tPT2fDhw5lEImFBQUFs27ZtrEWLFvLicMYYKygoYG+88Qbz9/dnNjY2LDAwkE2cOJGlpqYa9VwRBGF8RIwxZmrxRhAEUR/i4uLQr18/PHz4kOqHCIJoFMgAkyAIgiAIQkdIOBEEQRAEQegIpeoIgiAIgiB0hCJOBEEQBEEQOkLCiSAIgiAIQkdIOBEEQRAEQegICSeCIAiCIAgdIeFEEARBEAShIyScCIIgCIIgdISEE0EQBEEQhI6QcCIIgiAIgtAREk4EQRAEQRA68v85KeUKBfJwCQAAAABJRU5ErkJggg==",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: REINFORCE learning curve\n",
"fig, ax = plt.subplots(figsize=(6, 3))\n",
"ax.plot(pg_lengths, alpha=0.35, color=\"#1E40FF\", label=\"episode length\")\n",
"smooth = np.convolve(pg_lengths, np.ones(15)/15, mode=\"valid\")\n",
"ax.plot(range(14, 14 + len(smooth)), smooth, color=\"#1E40FF\", label=\"15-ep moving average\")\n",
"ax.set_xlabel(\"episode\"); ax.set_ylabel(\"steps survived\"); ax.set_title(\"REINFORCE on CartPole\")\n",
"ax.legend(); plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "73edb259",
"metadata": {},
"source": [
"> **Interpretation.** Episode length climbs from ~20 toward the low hundreds. The curve is noisy because REINFORCE is a high-variance estimator: a single lucky or unlucky episode moves the gradient a lot. The next step is to cut that variance with an advantage estimate and a trust region. That is PPO.\n",
"\n",
"GAE (Schulman et al., 2015) interpolates between one-step TD and Monte-Carlo return. With TD residual $\\delta_t = r_t + \\gamma V(s_{t+1}) - V(s_t)$:\n",
"\n",
"$$\\hat A_t^{\\text{GAE}} = \\sum_{l\\geq 0}(\\gamma\\lambda)^l \\delta_{t+l}$$\n",
"\n",
"$\\lambda = 0$ gives one-step TD (low variance, biased); $\\lambda = 1$ gives Monte-Carlo (unbiased, high variance). We verify both limits exactly before trusting the general case.\n"
]
},
{
"cell_type": "markdown",
"id": "33925c9d",
"metadata": {},
"source": [
"### Exercise 19.5 — Generalized Advantage Estimation\n",
"`Difficulty 3/5 · ~12 min`\n",
"\n",
"Fill in `compute_gae`. Walk the trajectory backward accumulating $\\hat A_t = \\delta_t + \\gamma\\lambda(1-\\text{done}_t)\\hat A_{t+1}$. `values` has length $T+1$ (the bootstrap value of the state after the last step is at the end). The checks pin the two limits: with zero values, $\\lambda=1$ gives discounted returns and $\\lambda=0$ gives the rewards themselves.\n"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "1bb8efdd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:42.865680Z",
"iopub.status.busy": "2026-06-11T20:07:42.865575Z",
"iopub.status.idle": "2026-06-11T20:07:42.870787Z",
"shell.execute_reply": "2026-06-11T20:07:42.870494Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 19.5 GAE = Monte Carlo at lambda 1: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 19.5 GAE = one-step TD at lambda 0: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 19.5 GAE respects done boundaries: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def compute_gae(rewards, values, dones, gamma=0.99, lam=0.95):\n",
" \"\"\"rewards (T,), values (T+1,), dones (T,) in {0,1}. Returns advantages (T,).\"\"\"\n",
" T = len(rewards)\n",
" adv = np.zeros(T)\n",
" gae = 0.0\n",
" for t in reversed(range(T)):\n",
" # TODO 1: delta = rewards[t] + gamma * values[t+1] * (1 - dones[t]) - values[t]\n",
" delta = None\n",
" attempted(delta)\n",
" # TODO 2: gae = delta + gamma * lam * (1 - dones[t]) * gae ; then adv[t] = gae\n",
" raise NotImplementedError # remove once the TODOs are done\n",
" return adv\n",
"\n",
"def _gae_mc():\n",
" r = np.array([1.0, 1.0, 1.0]); v = np.zeros(4); d = np.zeros(3)\n",
" # lambda=1, gamma=1, zero values -> Monte Carlo returns: [3,2,1]\n",
" check_close(compute_gae(r, v, d, gamma=1.0, lam=1.0), [3.0, 2.0, 1.0],\n",
" msg=\"GAE at lambda=1 with zero values is the Monte-Carlo return\")\n",
"\n",
"def _gae_td():\n",
" r = np.array([1.0, 1.0, 1.0]); v = np.zeros(4); d = np.zeros(3)\n",
" # lambda=0 -> one-step TD; zero values -> just the rewards: [1,1,1]\n",
" check_close(compute_gae(r, v, d, gamma=1.0, lam=0.0), [1.0, 1.0, 1.0],\n",
" msg=\"GAE at lambda=0 with zero values is the one-step reward\")\n",
"\n",
"def _gae_done():\n",
" # a done in the middle must stop the bootstrap from leaking across the boundary\n",
" r = np.array([1.0, 1.0]); v = np.array([0.0, 0.0, 5.0]); d = np.array([1.0, 0.0])\n",
" adv = compute_gae(r, v, d, gamma=1.0, lam=1.0)\n",
" assert abs(adv[0] - 1.0) < 1e-9, f\"adv[0]={adv[0]}; a done at t=0 must cut the future off (just r=1)\"\n",
"\n",
"check(\"19.5 GAE = Monte Carlo at lambda 1\", _gae_mc)\n",
"check(\"19.5 GAE = one-step TD at lambda 0\", _gae_td)\n",
"check(\"19.5 GAE respects done boundaries\", _gae_done)"
]
},
{
"cell_type": "markdown",
"id": "09484b24",
"metadata": {},
"source": [
"Hint 1 (conceptual)Two lines in the loop. The TD residual `delta` uses `values[t+1]` for the bootstrap (masked by `1 - dones[t]`). The accumulator `gae` multiplies the previous `gae` by `gamma * lam * (1 - dones[t])`.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"delta = rewards[t] + gamma * values[t+1] * (1 - dones[t]) - values[t]\n",
"gae = delta + gamma * lam * (1 - dones[t]) * gae\n",
"adv[t] = gae\n",
"```\n",
"\n",
"\n",
"Help — \"the lambda=1 limit gives [3,3,3] not [3,2,1]\"You forgot to subtract `values[t]` in `delta`, or you are accumulating forward instead of backward. GAE is a *backward* recursion: `for t in reversed(range(T))`. With zero values, `delta[t] = rewards[t]`, and the backward accumulation gives the cumulative future reward, which decreases as `t` increases.\n"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "6eb9a731",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:42.871773Z",
"iopub.status.busy": "2026-06-11T20:07:42.871687Z",
"iopub.status.idle": "2026-06-11T20:07:42.875401Z",
"shell.execute_reply": "2026-06-11T20:07:42.875031Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 19.5 GAE = Monte Carlo at lambda 1\n",
"[ ok ] 19.5 GAE = one-step TD at lambda 0\n",
"[ ok ] 19.5 GAE respects done boundaries\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines compute_gae; the checks below re-verify it.\n",
"def compute_gae(rewards, values, dones, gamma=0.99, lam=0.95):\n",
" T = len(rewards)\n",
" adv = np.zeros(T)\n",
" gae = 0.0\n",
" for t in reversed(range(T)):\n",
" delta = rewards[t] + gamma * values[t + 1] * (1 - dones[t]) - values[t]\n",
" gae = delta + gamma * lam * (1 - dones[t]) * gae\n",
" adv[t] = gae\n",
" return adv\n",
"\n",
"check(\"19.5 GAE = Monte Carlo at lambda 1\", _gae_mc, required=True)\n",
"check(\"19.5 GAE = one-step TD at lambda 0\", _gae_td, required=True)\n",
"check(\"19.5 GAE respects done boundaries\", _gae_done, required=True)"
]
},
{
"cell_type": "markdown",
"id": "25fad0b9",
"metadata": {},
"source": [
"Next, the PPO clipped objective. With probability ratio $r_t(\\theta) = \\pi_\\theta(a_t\\mid s_t)/\\pi_{\\theta_\\text{old}}(a_t\\mid s_t)$:\n",
"\n",
"$$\\mathcal{L}^{\\text{CLIP}} = \\mathbb{E}_t\\left[\\min\\!\\big(r_t \\hat A_t,\\ \\text{clip}(r_t, 1-\\epsilon, 1+\\epsilon)\\hat A_t\\big)\\right]$$\n",
"\n",
"When $\\hat A_t > 0$, the clip caps how far up you can push the ratio; when $\\hat A_t < 0$, it caps how far down. The `min` takes the pessimistic side. We pin all four corners of this geometry with hand-computed values before trusting it.\n"
]
},
{
"cell_type": "markdown",
"id": "010db6be",
"metadata": {},
"source": [
"### Exercise 19.6 — The PPO clipped surrogate\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"Fill in `ppo_clip_term(ratio, adv, eps)` returning the per-sample surrogate $\\min(r\\hat A,\\ \\text{clip}(r,1-\\epsilon,1+\\epsilon)\\hat A)$ (the objective we *maximize*; the loss negates it). The four checks pin the corners: positive advantage with ratio above the band clips to $1+\\epsilon$; negative advantage with ratio above the band does *not* clip because the `min` picks the more pessimistic unclipped value.\n"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "6d52af5f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:42.876290Z",
"iopub.status.busy": "2026-06-11T20:07:42.876213Z",
"iopub.status.idle": "2026-06-11T20:07:42.879837Z",
"shell.execute_reply": "2026-06-11T20:07:42.879624Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 19.6 PPO clip corners: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def ppo_clip_term(ratio, adv, eps=0.2):\n",
" \"\"\"Per-sample PPO surrogate min(r*A, clip(r,1-eps,1+eps)*A). Scalars or arrays.\"\"\"\n",
" ratio = np.asarray(ratio, dtype=float); adv = np.asarray(adv, dtype=float)\n",
" # TODO 1: surr1 = ratio * adv\n",
" surr1 = None\n",
" # TODO 2: surr2 = clip(ratio, 1-eps, 1+eps) * adv (use np.clip)\n",
" surr2 = None\n",
" attempted(surr1, surr2)\n",
" # TODO 3: return the elementwise minimum of surr1 and surr2\n",
" raise NotImplementedError # remove once the TODOs are done\n",
"\n",
"def _ppo_corners():\n",
" f = lambda r, a: float(ppo_clip_term(r, a, 0.2))\n",
" # A>0, r above band -> clipped at 1.2: 0.2 below band -> unclipped (min picks smaller)\n",
" assert abs(f(2.0, 1.0) - 1.2) < 1e-9, \"A>0, r=2 clips the upside to 1+eps=1.2\"\n",
" assert abs(f(0.5, 1.0) - 0.5) < 1e-9, \"A>0, r=0.5: min picks the smaller unclipped 0.5\"\n",
" # A<0, r above band -> min picks the more pessimistic unclipped -2.0\n",
" assert abs(f(2.0, -1.0) + 2.0) < 1e-9, \"A<0, r=2: unclipped -2.0 is more pessimistic than -1.2\"\n",
" assert abs(f(0.5, -1.0) + 0.8) < 1e-9, \"A<0, r=0.5: clip floors r at 0.8 -> -0.8\"\n",
"\n",
"check(\"19.6 PPO clip corners\", _ppo_corners)"
]
},
{
"cell_type": "markdown",
"id": "f0c49df0",
"metadata": {},
"source": [
"Hint 1 (conceptual)Two products and a minimum. `surr1` is unclipped; `surr2` clips the ratio into $[1-\\epsilon, 1+\\epsilon]$ first. Return `np.minimum(surr1, surr2)`.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"surr1 = ratio * adv\n",
"surr2 = np.clip(ratio, 1 - eps, 1 + eps) * adv\n",
"return np.minimum(surr1, surr2)\n",
"```\n",
"\n",
"\n",
"Help — \"the A<0 corner is -1.2 not -2.0\"You clipped the advantage or took a `max`. The `min` of $(-2.0, -1.2)$ is $-2.0$. With a negative advantage, the unclipped term is the more pessimistic one, so PPO keeps it; the clip does not protect you from pushing a bad action's probability further down. That asymmetry is the point of the objective.\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "897ffcf4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:42.880771Z",
"iopub.status.busy": "2026-06-11T20:07:42.880702Z",
"iopub.status.idle": "2026-06-11T20:07:42.883684Z",
"shell.execute_reply": "2026-06-11T20:07:42.883364Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 19.6 PPO clip corners\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines ppo_clip_term; the check below re-verifies it.\n",
"def ppo_clip_term(ratio, adv, eps=0.2):\n",
" ratio = np.asarray(ratio, dtype=float); adv = np.asarray(adv, dtype=float)\n",
" surr1 = ratio * adv\n",
" surr2 = np.clip(ratio, 1 - eps, 1 + eps) * adv\n",
" return np.minimum(surr1, surr2)\n",
"\n",
"check(\"19.6 PPO clip corners\", _ppo_corners, required=True)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "f25e6e80",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:42.884680Z",
"iopub.status.busy": "2026-06-11T20:07:42.884595Z",
"iopub.status.idle": "2026-06-11T20:07:43.006560Z",
"shell.execute_reply": "2026-06-11T20:07:43.005964Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3EAAAFJCAYAAAA1wqSlAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAcpdJREFUeJzt3XmcjeX/x/HXmTP7MDPMjH2syZZdZAojY0tFylLKEmmjLBX6FqK+QiFLKWFQ1hItImT4JluWkhDCCGM3q9nOuX9/zM/JaWzDzNxzZt7Px+M8OPe57nPe556Zc92fc9/3dVkMwzAQERERERERl+BmdgARERERERG5eSriREREREREXIiKOBEREREREReiIk5ERERERMSFqIgTERERERFxISriREREREREXIiKOBEREREREReiIk5ERERERMSFqIgTERERERFxISriJN+JjIzEYrFw5MgRs6OIiIiYRv2hSP6lIk4kB/3xxx+MHDlSHehN2r9/PwMHDiQsLAxvb2/tfIiI5BPqD52pv5PbpSJOJAf98ccfvPXWW/pgvkmbNm1i8uTJxMfHU61aNbPjiIhINlF/6Ez9ndwuFXEikivKly/PyJEjr9vm4Ycf5uLFi+zevZtu3brlTjAREZFcpv5ObpeKOMlzjh49ygsvvECVKlXw8fEhKCiITp06XfXbuz179nD//ffj4+NDmTJlePvtt7Hb7U5tHnzwQSpWrHjV12rcuDENGjRw3J89ezb3338/xYoVw8vLi+rVq/PRRx9lWq98+fI8+OCD/PTTTzRs2BBvb28qVqzI3LlzHW0iIyPp1KkTAM2bN8disWCxWIiKigJg+fLltGvXjlKlSuHl5UWlSpUYPXo0Npst0+tNmzaNihUr4uPjQ8OGDfnf//5HeHg44eHhTu1SUlIYMWIEd9xxB15eXoSGhvLaa6+RkpJy1fef1xQtWpTChQubHUNEJE9Qf5h/+0P1d3K73M0OIPJv27Zt4+eff6Zr166UKVOGI0eO8NFHHxEeHs4ff/yBr68vADExMTRv3pz09HSGDh2Kn58fn3zyCT4+Pk7P16VLF7p37862bdu4++67HcuPHj3K5s2bGT9+vGPZRx99RI0aNXj44Ydxd3fnm2++4YUXXsBut/Piiy86Pe/Bgwd57LHH6N27Nz169GDWrFn07NmT+vXrU6NGDZo2bcpLL73E5MmTef311x2nS1z+NzIykkKFCjFo0CAKFSrEjz/+yPDhw4mLi8uUqV+/fjRp0oSBAwdy5MgROnToQJEiRShTpoyjnd1u5+GHH+ann36ib9++VKtWjd27dzNx4kT+/PNPli1bdt3tnpSURFJS0g1/PlarlSJFitywnYiI3B71h+oPRa7JEMljkpKSMi3btGmTARhz5851LBswYIABGFu2bHEsO336tBEQEGAAxuHDhw3DMIzY2FjDy8vLGDx4sNNzjhs3zrBYLMbRo0ev+9qtW7c2Klas6LSsXLlyBmBs2LDB6bX//TpLliwxAGPdunU39T6fffZZw9fX10hOTjYMwzBSUlKMoKAg4+677zbS0tIc7SIjIw3AaNasmWPZvHnzDDc3N+N///uf03NOnz7dAIyNGzdmer0rjRgxwgBueCtXrtx1n+daypUrZ4wYMeKm248fP97p5ygiUtCoP8yf/eG/qb+TW6EjcZLnXPnNYVpaGnFxcdxxxx0EBgayY8cOnnrqKQBWrFjBPffcQ8OGDR3tQ0JC6NatGx9++KFjmb+/P23btmXx4sWMHz8ei8UCwKJFi7jnnnsoW7bsVV87NjaWtLQ0mjVrxqpVq4iNjSUgIMDxePXq1WnSpInTa1epUoW//vory+8zPj6elJQUmjRpwscff8y+ffuoXbs2v/zyC+fOnWPMmDG4u//z59qtWzcGDhzo9HxLliyhWrVqVK1albNnzzqW33///QCsW7eOsLCwa+bp3r079913X5ZyX0tKSgrx8fFOy+x2O0lJSU7ZAIKDg2/4fCIiBZH6Q9fvD0Vyioo4yXMuXbrEmDFjmD17NsePH8cwDMdjsbGxjv8fPXqURo0aZVq/SpUqmZZ16dKFZcuWsWnTJsLCwjh06BDbt29n0qRJTu02btzIiBEj2LRpU6ZTKf7daV3Z2V1WpEgRLly4cFPvc8+ePbzxxhv8+OOPxMXFZXqty+8R4I477nB63N3dnfLlyzstO3DgAHv37iUkJOSqr3f69Onr5qlYseI1r5XIqgULFtCrV69My8ePH+90agzg9PMVEZF/qD907f7w0qVLTj8ngBIlStz284qAijjJg/r378/s2bMZMGAAjRs3JiAgAIvFQteuXTNdpH2zHnroIXx9fVm8eDFhYWEsXrwYNzc3x4XWAIcOHaJFixZUrVqVCRMmEBoaiqenJytWrGDixImZXttqtV71tW6mKLl48SLNmjXD39+fUaNGUalSJby9vdmxYwdDhgy5pfdpt9upWbMmEyZMuOrjoaGh110/ISGBhISEG76O1Wq9Zsd4WevWrVm9erXTsieffJJWrVrRvXv3G76GiIioP3T1/nDRokWZvtDUF5eSXVTESZ7zxRdf0KNHD95//33HsuTkZC5evOjUrly5chw4cCDT+vv378+0zM/PjwcffJAlS5YwYcIEFi1aRJMmTShVqpSjzTfffENKSgpff/2107eK69atu+X3cvlUlX+Liori3LlzLF26lKZNmzqWHz582KlduXLlgIyLxps3b+5Ynp6ezpEjR6hVq5ZjWaVKlfj1119p0aLFNV/3et577z3eeuutG7YrV67cDef5KVmyJCVLlnRadnnEsoiIiCxnExEpiNQf/sMV+8OrfaEpkl1UxEmeY7VaM31TNWXKlExDDT/wwANMmjSJrVu3Oq4DOHPmDJ9//vlVn7dLly4sXryYTz/9lF9//dXpOoHLrwtkOl1l9uzZt/xe/Pz8ADJ1uFd7rdTU1EyZGjRoQFBQEDNmzKBXr16O6wA+//zzTKepdO7cmRUrVjBjxgz69u3r9NilS5ew2+2OPFejawBERPIW9Yf/cMX+8GpfaIpkFxVxkuc8+OCDzJs3j4CAAKpXr86mTZtYs2YNQUFBTu1ee+015s2bR5s2bXj55ZcdQyqXK1eO3377LdPzPvDAAxQuXJhXXnkFq9XKo48+6vR4q1at8PT05KGHHuLZZ58lISGBGTNmUKxYMU6ePHlL76VOnTpYrVbGjh1LbGwsXl5e3H///YSFhVGkSBF69OjBSy+9hMViYd68eZk6a09PT0aOHEn//v25//776dy5M0eOHCEyMpJKlSo5fcP41FNPsXjxYp577jnWrVvHvffei81mY9++fSxevJhVq1Y5zQH0b9l5Tdytio2NZcqUKUDG9RgAU6dOJTAwkMDAQPr162dmPBGRXKX+8B/5rT9Ufye3zYwhMUWu58KFC0avXr2M4OBgo1ChQkbr1q2Nffv2GeXKlTN69Ojh1Pa3334zmjVrZnh7exulS5c2Ro8ebcycOfOaQ/V269bNAIyIiIirvvbXX39t1KpVy/D29jbKly9vjB071pg1a1am5ytXrpzRrl27TOs3a9bMaZhjwzCMGTNmGBUrVjSsVqvT8MobN2407rnnHsPHx8coVaqU8dprrxmrVq266hDMkydPNsqVK2d4eXkZDRs2NDZu3GjUr1/faNOmjVO71NRUY+zYsUaNGjUMLy8vo0iRIkb9+vWNt956y4iNjb3qe84tNzPFwOHDh3N8KGcREVeh/jB/9oeGof5Obp/FMHSFpYirsdvthISE0LFjR2bMmGF2HBEREVOoP5SCys3sACJyfcnJyZlOK5k7dy7nz58nPDzcnFAiIiK5TP2hyD90JE4kj4uKimLgwIF06tSJoKAgduzYwcyZM6lWrRrbt2/H09PT7IgiIiI5Tv2hyD80sIlIHle+fHlCQ0OZPHky58+fp2jRonTv3p13331XHZaIiBQY6g9F/qEjcSIiIiIiIi5E18SJiIiIiIi4EJc4ndJut3PixAkKFy7sNA+IiIjkP4ZhEB8fT6lSpXBzK9jfNar/ExEpOLLS/7lEEXfixAlCQ0PNjiEiIrno2LFjlClTxuwYplL/JyJS8NxM/+cSRVzhwoWBjDfk7+9vchrJ79JtcO4iWC1g5hffNns6e377HzVqNcHqZu6fqmGAzYCgQHC3mhpFCoC4uDhCQ0Mdn/0Fmfo/yU15pf+DvNMHqv+T3JSV/s8lirjLp5D4+/urE5Mcl26DVFvGh7WZH9g2Wzp+fn4UCfTHajX3TzXdlnHz91cnJrlHpw+q/5PclVf6P8g7faD6PzHDzfR/BftiAxERERERERejIk5ERERERMSFuMTplNdjGAbp6enYbDazo0g2sVqtuLu761QqERERKfC0r5v/ZMe+rksXcampqZw8eZKkpCSzo0g28/X1pWTJknh6epodRURERMQU2tfNv253X9dlizi73c7hw4exWq2UKlUKT09PHbnJBwzDIDU1lTNnznD48GEqV65c4OeJEhERkYJH+7r5U3bt62apiBszZgxLly5l3759+Pj4EBYWxtixY6lSpcp111uyZAlvvvkmR44coXLlyowdO5YHHnggy2GvlJqait1uJzQ0FF9f39t6LslbfHx88PDw4OjRo6SmpuLt7W12JBERAKZNm8b48eOJiYmhdu3aTJkyhYYNG16zfU70fyJSMGhfN//Kjn3dLJV969ev58UXX2Tz5s2sXr2atLQ0WrVqRWJi4jXX+fnnn3n88cfp3bs3O3fupEOHDnTo0IHff/89y2GvRkdp8if9XEUkr1m0aBGDBg1ixIgR7Nixg9q1a9O6dWtOnz591fY53f+JSMGgfaL86XZ/rhbDMIxbXfnMmTMUK1aM9evX07Rp06u26dKlC4mJiXz77beOZffccw916tRh+vTpV10nJSWFlJQUx/3LE9/FxsY65slJTk7m8OHDVKhQQUdq8iEzf77pNjh7wfx5cmy2dHZuj6Ju/fA8M09ccBHNkyM5Ly4ujoCAAKfP/LygUaNG3H333UydOhXA8Q15//79GTp0aKb2OdX/3YqEhATsdnue2p6S9+SV/g/yTh9oZv+nfd387Wo/36z0f7dVAsbGxgJQtGjRa7bZtGkTERERTstat27Npk2brrnOmDFjCAgIcNxCQ0NvJ6ZL69mzJ5MmTQJg+vTpjB8/Ptdee+rUqfTs2fOm2l7+LuBy+39/N9CjRw/8/f2ve9RWRFyT3W5n69at/Prrr2ZHyTGpqals377dqT9zc3MjIiLimv1ZXur/Vq9ezaeffsq+ffuy5flERLKL9nVvzS0XcXa7nQEDBnDvvfdy1113XbNdTEwMxYsXd1pWvHhxYmJirrnOsGHDiI2NddyOHTt207lSU1NJTU112rA2m43U1FTS09NvuW1e8Nxzz/Hqq6+aHSOT9PR0Ro0axbRp00hPT2f+/Pn079/f8XhcXBzffPMNtWvXZsmSJSYmFZGcsG/fPtatW8ePP/5IQkKC2XFyxNmzZ7HZbFnqz3K7/7uW1NRUkpKSsNls1/3SVURcg/Z1c19e3Ne95SLuxRdf5Pfff2fhwoXZmQcALy8v/P39nW43a+LEiUycOJFLly45lm3ZsoWJEyeyevVqp7ZTp05l4sSJxMXFOZbt2LGDiRMn8v33399SdovFwsWLFx33g4ODOXLkCADly5dn+PDhNG7cmAoVKvD222872h0/fpzHHnuMmjVrUqtWLd58881Mzz1y5EgGDBgAQGRkJPfffz8PP/ww1atXp2nTpo7Xud5jAO+99x4NGzakXr16tGnThqNHjwIQHx9Ply5dqFKlCvfddx+7d+++5vssX748Q4YMoWHDhvTo0YMRI0bg5ubG559/zm+//eY43QhgwYIFREREMGjQIGbOnJnFLSoieV21atWoVKkS999/P35+fmbHcWm30/9di6enJ0888QTdunWjWLFijuUnT57MkztxInJ92tfVvi7cYhHXr18/vv32W9atW0eZMmWu27ZEiRKcOnXKadmpU6coUaLErby0y7t48SKbNm1i27ZtjB8/nuPHjwPw5JNPUr9+fXbv3s1vv/3GSy+9dMPn2rhxI2PHjuWPP/7gwQcfpG/fvjd8bP78+ezfv59NmzaxY8cOunXrxgsvvADAqFGj8PLyYt++fXz33Xds2LDhuq9/7tw5tmzZwueff87o0aOx2Wx069aNmjVr8vLLLzvazZw5k6effpoHH3yQAwcOsH///ixvNxHJO6Kjo/n222+x2+1ARof+2GOPUbt27Xw7/HVwcDBWqzVL/Vle6v8sFgulSpVy3I+Pj2fhwoXMnj3baedOROR2aV83d/Z1s3SlqGEY9O/fn6+++oqoqCgqVKhww3UaN27M2rVrHVU1ZJyb37hx4yyHvRkDBw4EwMPDw7GsUaNGNGjQINMoMP369cvUtl69etSuXTvHRgJ64okngIwdgooVK3L48GECAgL46aefWLVqlaNdSEjIDZ8rLCyMatWqAdC3b1/eeOMNbDbbdR9btmwZ27Zto379+gCO9gBr165l4sSJWCwWAgICeOKJJzh06NA1X79nz56OHbY33ngDi8XCL7/8Qrdu3Rzvc/fu3Zw8eZJWrVrh5ubGk08+yaxZsxg7duxNbzMRyTtSUlL48ssvSU1NpUyZMtSpU8fsSLnC09OT+vXrs3btWjp06ABkXFawdu1aR1/yb7nd/2VFbGwsHh4eeHl5UahQIbPjiEgWaF9X+7qQxSLuxRdfZP78+SxfvpzChQs7zusPCAjAx8cHgO7du1O6dGnGjBkDwMsvv0yzZs14//33adeuHQsXLuSXX37hk08+yea3kuFqs55brVas1sxDCmWl7c2yWq1OvyzJyclOj185upDVas31U1kMw2DYsGFO32Rcy42+Ub+y47/cNjIy0un+zJkziY+Pp2LFigCkpaVht9t55513cHd32bnmRQosLy8vwsPDOX36tKPzLCgGDRpEjx49aNCgAQ0bNmTSpEkkJibSq1cvwPz+LyvKlClDnz59SE5OduzIGYbBsWPHKFu2rMnpROR6tK97fQVlXzdLJfhHH31EbGws4eHhlCxZ0nFbtGiRo010dDQnT5503A8LC2P+/Pl88skn1K5dmy+++IJly5ZddzAUV3bHHXewZcsWAJYuXXpTI9QUKlSIpk2b8v777zuWnTlz5obrbdq0yTHS2Keffkrz5s0df5TXeqxDhw5Mnz6d8+fPAxm/aDt37gQgIiKC2bNnYxgGcXFxLFiwIAvvPLPU1FQ+++wzNm/ezJEjRzhy5AjHjx+nbNmyfPfdd7f13CKSO+Lj41m+fLnTZ1LdunVp3bo1Xl5eJibLfV26dOG9995j+PDh1KlTh127drFy5UrH4CWu1v95e3sTGBjouL9r1y4WLFhwy9fJiEjBoH3df5i5r5vl0ylvJCoqKtOyTp060alTp6y8lMuaOHEiL730Em+88Qbt2rUjKCjoptabN28e/fv3p0aNGnh4eNC+fXveeuut664TFhbGkCFDOHjwIEFBQcydO/eGj3Xr1o1z587RvHlzIGO0naeffpq6devy5ptv0qdPH6pWrUpISAj33Xef03xFWbVs2TLKlStH1apVnZZ369aNmTNn0r59+1t+bhHJHVFRUezbt4+EhASeeOKJfHvN283q16/fNU+fdPX+LykpCYvF4jT4iYjIv2lf9x9m7uve1mTfueVqE98V9AkQIyMjWbZsGcuWLcvSY65Ck33nnYlOQZN9F2Tx8fF899133H///bm2c59XJ/s2Q25vizNnzhAUFOQ4xfLChQtYLBanI3aSf+WV/g/yTh+oyb7NUxD3dbPyma+LkkREBMi4+Hvbtm3Y7XbCwsIAKFy4MF27djU5meSWKwcaMAyD7777jlOnTvHQQw9x5513mphMRESulDPD0kiO69mz5zW/fbjeYyIi13LkyBHWr1/Pxo0buXDhgtlxxGTJyclYrVbc3NwK7LRAImIe7eten47EiYgIAJUqVaJWrVqEhobq9DnBx8eHrl27cu7cOafTeo4cOULp0qWdhiwXEZHcpSNxIiIF1JEjR1i8eLHT8M9t27blrrvuKvADmEgGi8VCcHCw4/7Zs2dZsmQJs2fPJikpycRkIiIFm4o4EZECKD09ne+++47Dhw+zbds2s+OIi0hKSsLX15egoCDH/LAiIpL7dDqliEgB5O7uTsuWLYmOjqZevXpmxxEXUbZsWfr06UN6errjaG16ejpHjhyhUqVKOoIrIpJLdCRORKQAiIuL44svviA6Otqx7M477yQiIqLATdott8fLyws/Pz/H/U2bNvHll1+yatUqE1OJiBQsKuKyWfny5alSpQp16tShevXqTJs2Dci49sRqtVKnTh1q165N/fr1WbdunWO9n3/+mWbNmlG5cmUqVqzI448/zsmTJ816G9lm0qRJxMTEmB1DpMDbunUrhw4dYvXq1bjA9KDiQtzc3HBzc6N8+fJmRxGRXKB9XWdm7evmm9MpDQMuJefsa/h4w82cKbJo0SLq1KnD0aNHqVWrFk2aNMHf35/ChQuza9cuAJYuXUrnzp05ffo0u3fv5uGHH2bRokW0aNECgLFjxxIeHs7OnTvx9fXN9vdis9mwWv+ZtTI9PR139+z/dZg0aRLh4eEanlrEZE2aNCE+Pp4mTZrolDfJVvfeey81atQgICDAsSwmJgYPDw+CgoJMTCaSv2hfN2vy+75uviniLiVDtQdy9jX2rgDfLFzHXa5cOapUqcKff/5JgwYNnB5r06YNZ8+e5dy5c4wbN46nn37a8UsNMGTIEL788ksWLlzI008/7bTugQMH6NmzJwkJCdjtdtq3b8/bb7/NyJEjuXjxIpMmTQJg6tSp/PLLL0RGRhIZGcmcOXMoWrQof/75J5988glhYWEMHz6cFStWEB4ezquvvspzzz3HgQMHMAyD/v378+yzzwIZ35688MIL2Gw27r77brZv384HH3xAeHg4EyZMYMGCBaSlpeHh4cHkyZNp3Lgxo0aN4sSJE3Tp0gUfHx8iIyOpUaMGb775Jj/++COpqanceeedfPzxxxQpUuTWfigikkl6ejqbN28mKSmJVq1aARmnwD3yyCMmJ5P86sopKdLT0/nmm2+IjY3lkUceoVKlSuYFE8lHtK+rfd0r5ZsiLi/avXs3+/bto3bt2pkeW7BgAWXLliU4OJgdO3bw6KOPZmrTuHFjtm/fnukXe+rUqTz44IMMGzYMgPPnz99Uni1btrBz506qVKniWGa1Wh0j03Xp0oUqVaqwdOlSTp8+Tf369alduzb16tWjS5cuzJ07l+bNm7Nu3Tpmz57teI6nnnqKQYMGAbB582Z69uzJvn37GD58OLNmzXJ8WwPw3//+Fz8/P7Zu3QrA6NGjeeONNxyH4kXk9p06dYqNGzcCULt2bYoXL25yIilIUlNTCQgIICUlhdKlS5sdR0RykPZ1zdvXzTdFnI93xrcHOf0aN+NyNe7r68usWbOoXLkyR44cIT4+3vEDLl26NF9//fUt5WjatCmvvvoqCQkJNGvWjIiIiJtaLywszOmXGnD6o1mzZg3bt28HoFixYnTs2JE1a9bg6+uLu7s7zZs3B6B58+ZO36zu3LmTd955h3PnzuHu7s7+/fu5dOnSVYefXrZsGbGxsXz55ZdARmev6yhEbp9hGI7TJEuXLk3jxo0JCQmhWLFiJieTgsbX15dOnToRHx+Pt/c/Hee+ffuoWLEinp6eJqYTcV3a172xgrSvm2+KOIsla4d/c9KV1fiVrjxP+Er16tVj06ZNmU512rRpk+MQ75UeffRRwsLCWL16NVOnTmXSpEmsWLECd3d3bDabo11ysvOJ04UKFcr0XFdbdtn1rpu5/FhqaiodO3Zk3bp13H333cTFxTm+gb3aL7ZhGEyZMsVxipeI3L5Dhw6xceNGOnXq5Pi7a9q0qcmppCCzWCz4+/s77h87dozly5cTEBBAr169NCKqyC3Qvq72da+k0SnzgFdeeYWZM2eydu1ax7Jx48Zx4cIFHn/88UztDxw4QPHixenevTvjxo1j8+bNANxxxx388ssv2Gw2kpKSHN8A3KyIiAhmzJgBwJkzZ1i6dCktW7akSpUqpKWlsX79egDWr1/PwYMHgYw/ntTUVMqWLQvAlClTnJ7T39+f2NhYx/0OHTowceJEkpKSgIyJY/fs2ZOlnCLyD7vdzrp16zh58qTjs0AkrzEMA39/f8qXL68CTqQA0r5u9u/r5psjca6sTp06LF++nGHDhvHcc8+Rnp5Ow4YNiYqKuupoPV988QWfffYZnp6e2O12pk+fDkDHjh1ZsmQJ1apVo0yZMtStW9fxC3QzJk+ezPPPP0/NmjUxDIP//Oc/NGrUCICFCxfy4osvYrfbqV+/PlWqVCEwMBB/f3/efvttGjZsSHBwMF27dnV6zpdeeolnnnkGX19fIiMjGTJkCCkpKTRq1MjxDceQIUOoUaPGrW4+kQLn8hQBFosFNzc32rRpw59//sl9991ncjKRqytbtiy9e/d2mt4iOTmZw4cPU7VqVY2YKpLPaV83+/d1LYYLTBh0+bBlbGys4/SMyx/+FSpUcDrnXnJGfHw8hQsXBmDbtm08/PDDHDp0KEeGhAVzf77pNjh7AdytGTez2Gzp7NweRd364Vit5n7fkm7LuAUXMXebCFy8eJFVq1ZRq1YtqlWrZnacHHG1z/yCKj9vi5UrV/Lrr79Sv379m77eRXJWXun/IO/0gWb2f9rXzV15YV83K5/5OhInN+XLL79k4sSJGIaBu7s78+bNy7FfahG5tj179nDkyBHOnz9PlSpVcHPTWfHiei6fXmm1WjMNQiAiYgZX29dVESc3pWfPnvTs2dPsGCIF0pUjTzZq1Ii4uDgaNWqkAk5clsViISwsjNq1a+Pn5+dYfvToUXx9fQkJCTExnYgURK62r+vyRZzdbjc7guQA/VxFMiZN/vnnnzl79iyPPPIIFosFd3d32rZta3Y0kWxxZQGXnJzMN998w6VLl+jcuTPlypUzMZlI3qF9ovzpdn+uLlvEeXp64ubmxokTJwgJCcHT01MXRucDhmGQmprKmTNncHNz03xCUqDFxsaydetWbDYbx44dc4yMJZIf2Ww2SpYsyfnz5zVJuAja182vsmtf12WLODc3NypUqMDJkyc5ceKE2XEkm/n6+lK2bFmdLiYFzpWnTgYFBdG8eXMKFy6sAk7yPT8/Pzp27EhycjLu7hm7J4Zh8Ouvv1KtWjVNTSAFjvZ187fb3dd12SIOMr6hKFu2LOnp6U4T/4lrs1qtuLu769smKXD+/PNPoqKi6NKlCwEBAQDUr1/f5FQiucdisThNnrt//35WrVrFli1b6N27t6O4EykotK+bP2XHvq7LfxpaLBY8PDzw8PAwO4qIyC0zDINt27Zx4cIFNm3aRJs2bcyOJGI6X19fAgMDqV69ugo4KbC0rytXo09EERGTXDlpt8VioU2bNuzZs4ewsDCTk4nkDWXLluXpp592+rY6Li6Oo0ePctddd+mMDREpsFTEiYiY4Pz586xatYrKlSvToEEDIOMauKZNm5qcTCRvufLog2EY/PDDDxw6dIgzZ85w//33m5hMRMQ8GjVCRMQE0dHRREdHs2nTJtLS0syOI+IyQkND8fLyombNmmZHERExjY7EiYjkEpvNhtVqBaB27dpcvHiROnXq6DoHkZtksVho1KgRderUcRqtct++fRQpUoTixYubmE5EJPfoSJyISA5LTU3lxx9/ZP78+Y7JPS0WC+Hh4QQGBpobTsQFXVnAxcXF8f333zNnzhyOHz9uYioRkdyjIk5EJIelpqby22+/ceLECf766y+z44jkK1arlQoVKlCqVClKlSpldhwRkVyh0ylFRHLAladOFipUiFatWuHl5UWlSpVMTiaSv/j5+dGhQwdSU1Mdo1Xa7Xa2bNlC3bp18fb2NjmhiEj205E4EZFstnfvXqZPn87p06cdy6pXr64CTiQHeXp6Ov6/c+dONmzYwLx58xynMIuI5Ccq4kREstn+/ftJSEhgy5YtZkcRKZCKFStG0aJFadCgAW5u2tURkfxHp1OKiNwmwzCw2+2O0ycjIiIoXrw4d999t8nJRAqm0NBQevXq5fibBDhz5gzHjx+ndu3amiRcRFyeijgRkdtw9uxZVq5cSZkyZQgPDwcyroFr3LixucFECjh39392cex2O99//z0nT54kPj6eJk2amJhMROT26RwDEZHbcOHCBY4fP86vv/5KSkqK2XFE5BqqV69OoUKFqFOnjtlRRERum47EiYhkUVpammOC7sqVK9O8eXOqVq3qNHeViOQdbm5uNGjQgDp16jgdodu1axfFixenZMmSJqYTEck6HYkTEblJqamprFmzhtmzZ5OWluZY3rBhQ/z9/U1MJiI348oC7vTp06xevZp58+Zx5swZE1OJiGSdijgRkZtkGAZ//vknFy5c4ODBg2bHEZHbUKhQIapWrUrlypUJCQkxO46ISJbodEoRketITU11zD/l5eXFAw88gGEYVKhQweRkInI7fH19eeihh7DZbI5laWlp/PTTT9xzzz34+PiYmE5E5Pp0JE5E5Br27NnD9OnTOXLkiGNZ+fLlVcCJ5CNXTkPw888/s3XrVhYuXIhhGCamEhG5PhVxIiLXcOLECS5dusSOHTvMjiIiueCOO+4gODiYe++9V3PJiUieptMpRUT+n91uJz093XH6ZNOmTQkMDKRevXomJxOR3FC6dGl69eqFm9s/33FHR0dz5swZ6tWrp8JORPIMFXEiIsCZM2f4/vvvCQoKol27dkDGNXB33323yclEJDddWcClpaWxcuVKLly4QHp6Oo0aNTIxmYjIP7J8OuWGDRt46KGHKFWqFBaLhWXLll23fVRUFBaLJdMtJibmVjOLiGS71NRUTp48yYEDB0hISDA7juQx58+fp1u3bvj7+xMYGEjv3r1v+HsSHh6eqe977rnncimxZAd3d3caNGhA0aJFNUm4iOQpWT4Sl5iYSO3atXn66afp2LHjTa+3f/9+p3mUihUrltWXFhHJVpcuXXKMQFe6dGkeeOABypcvT6FChUxOJnlNt27dOHnyJKtXryYtLY1evXrRt29f5s+ff931nnnmGUaNGuW47+vrm9NRJRtZLBbq1atHnTp1nI7Q/fzzz5QtW5YyZcqYmE5ECrIsF3Ft27albdu2WX6hYsWKERgYmOX1RESyW2pqKj/++CMHDhygd+/ejh3rmjVrmpxM8qK9e/eycuVKtm3bRoMGDQCYMmUKDzzwAO+99x6lSpW65rq+vr6UKFEit6JKDrmygDt69Cj/+9//sFgs9O3bV/s2ImKKXBudsk6dOpQsWZKWLVuycePG67ZNSUkhLi7O6SYikl2sVivHjx8nKSmJQ4cOmR1H8rhNmzYRGBjoKOAAIiIicHNzY8uWLddd9/PPPyc4OJi77rqLYcOGkZSUdN326v/yvmLFilGzZk3q1KmjAk5ETJPjA5uULFmS6dOn06BBA1JSUvj0008JDw9ny5Yt1xzxbcyYMbz11ls5HU1ECpCkpCR8fHywWCxYrVbatWtHamoqZcuWNTua5HExMTGZLgFwd3enaNGi172++4knnqBcuXKUKlWK3377jSFDhrB//36WLl16zXXU/+V9Pj4+PPDAA07zyF26dIkNGzZw33334efnZ2I6ESkocryIq1KlClWqVHHcDwsL49ChQ0ycOJF58+ZddZ1hw4YxaNAgx/24uDhCQ0NzOqqI5FO7d+9m7dq1tGzZkho1agDoFDdh6NChjB079rpt9u7de8vP37dvX8f/a9asScmSJWnRogWHDh2iUqVKV11H/Z/ruHK6gXXr1rF7927Onj1Lt27dTEwlIgWFKVMMNGzYkJ9++umaj3t5eeHl5ZWLiUQkP0tISCAlJYU//vjDUcSJDB48mJ49e163TcWKFSlRogSnT592Wp6ens758+ez9GXA5eHpDx48eM0iTv2fa6pXrx5nzpwhPDzc7CgiUkCYUsTt2rWLkiVLmvHSIlIA2Gw2UlNTHSNPNmzYED8/P+666y6Tk0leEhISQkhIyA3bNW7cmIsXL7J9+3bq168PwI8//ojdbs/SvGG7du0CUP+XD5UoUYLu3bs7HZ3bt28fcXFxNGjQwGlgFBGR7JDlIi4hIYGDBw867h8+fJhdu3ZRtGhRypYty7Bhwzh+/Dhz584FYNKkSVSoUIEaNWqQnJzMp59+yo8//sgPP/yQfe9CROT/nTlzhm+//RZfX186d+7suAauVq1aZkcTF1WtWjXatGnDM888w/Tp00lLS6Nfv3507drVMTLl8ePHadGiBXPnzqVhw4YcOnSI+fPn88ADDxAUFMRvv/3GwIEDadq0qX4X86krC7jk5GRWr15NUlISHh4e1K1b18RkIpIfZbmI++WXX2jevLnj/uVz93v06EFkZCQnT54kOjra8XhqaiqDBw/m+PHj+Pr6UqtWLdasWeP0HCIi2cVqtXLu3Dni4uKIjY3V6HGSLT7//HP69etHixYtcHNz49FHH2Xy5MmOx9PS0ti/f79j9ElPT0/WrFnDpEmTSExMJDQ0lEcffZQ33njDrLcgucjLy4umTZvy+++/q2gXkRxhMa4cXimPiouLIyAggNjYWKcJw0VyQroNzl4Ad2vGzSw2Wzo7t0dRt344VqspZz47pNsybsFFzN0m1xIfH0/hwoUd9w8cOECpUqU0SpyL0mf+P7QtXJthGI4jdIZhsGbNGqpUqZJnR8XNK/0f5J0+MK/3f5K/ZOUzXydpi4jLSktLY8WKFcyYMYOLFy86lleuXFkFnIiY7t/XyO3YsYPFixeTmJhoYioRyQ9UxImIy3J3dyc2Npa0tDQOHz5sdhwRkWuqUKECderUISwsTF8yichtM/ccLRGRLIqPj8fPzw83NzcsFgtt2rQhMTGRMmXKmB1NROSavL29ad26tdMk4bGxsWzYsIHw8HCnU8JFRG5ER+JExGX89ttvfPrpp2zbts2xrEiRIirgRMRlXHmK5Zo1a/jjjz9YtWqViYlExBWpiBMRl5Kamsrhw4dxgTGZRESuq0mTJpQpU0YjdotIlul0ShHJs2w2G4mJiY4RmmrWrImXlxd33nmn07fZIiKuqFixYnTr1s1p2Y4dO0hJSaFhw4ZYrRoOUUSuTkWciORJZ8+eZfny5bi5udG9e3esVisWi4UqVaqYHU1EJEfEx8cTFRVFWloaAQEBVK9e3exIIpJH6XRKEcmTfHx8SEhIICEhgQsXLpgdR0QkxxUqVIjWrVtz5513Uq1aNbPjiEgepiNxIpJnnDt3jqCgIAD8/Pzo2LEjwcHB+Pj4mJxMRCTnWSwWatSoQY0aNRzL7HY7X3/9NbVr16ZChQomphORvERH4kTEdDabja+//pqZM2cSExPjWB4aGqoCTkQKtF27drF//36WL19OSkqK2XFEJI/QkTgRMd2VF+8fP36cEiVKmJhGRCTvqFGjBufPnyckJAQvLy+z44hIHqEiTkRMERsbi4+PD56engC0aNGCu+++m5IlS5qcTEQk7/Dy8iIiIsJp2alTp9i4cSMRERGO0XtFpGDR6ZQikut2797NzJkz+emnnxzL/Pz8VMCJiNyAYRisXr2aAwcOsGHDBrPjiIhJVMSJSK7z9fUlLS2N06dPY7fbzY4jIuIyLBYLbdq0oVKlSoSHh5sdR0RMotMpRSTHpaenExsb6xh5slKlSnTp0oVy5cpp0m4RkSwKDg7msccec1q2YcMG3NzcuOeee3B31+6dSH6nv3IRyVHnzp1j6dKlpKen07t3b8c1cOXLlzc3mIhIPnH+/Hk2b96MYRiULl1aUxGIFAA6nVJEclThwoVJT0/Hbrdr0m4RkRxQpEgRHnroIerWrasCTqSA0JE4Ecl2J0+edAxS4unpSceOHQkICMDb29vkZCIi+Y/FYqFatWpUq1bNsSwtLY0vvviChg0bUqlSJRPTiUhO0JE4Eck2hmGwfPly5s6dy6FDhxzLixcvrgJORCQXbdu2jejoaFatWkV6errZcUQkm+lInIhkG4vFQuHChbFYLJw/f17f/oqImOTuu+8mJSWFsmXLOg10YhiGBpQSyQdUxInIbTl//jze3t74+voCcN9993HXXXdRrFgxk5OJiBRcHh4eNG/e3GnZ4cOH2bp1K61btyYwMNCcYCKSLXQ6pYjcst9//53Zs2ezdu1axzJPT08VcCIieYxhGKxdu5YjR46wfft2s+OIyG1SEScitywoKAibzUZSUpKuuRARycMsFgsdO3akevXq3HfffY7lhmGYmEpEbpVOpxSRm5aWlsaZ02cdI0+WLFmSp556ihIlSugaCxGRPK5o0aI89NBDTstWrlyJr68vYWFhWNw8TEomIlmlIk5Ebkp83EWWfbGIlJRk+vTpg5+fH4CjoBMREddy6tQpfvvtNwDuvPNOQorp81zEVaiIE5HrikuAtZthy6/+7D/wIDabnf8ddMfLy+xkklc1rgP9njQ7hVw2MRJ++d3sFJI3FScpqR+pqWn89n4gxYPhrsrQvJFBEX+dXSGSl6mIE5GrstsNFn5zhhlfhnA+1kLGJbSlATh62tRokscVDTA7gVxp71/wk8axkGvyc7r35Sp4f+Yl+nW9yOMdSpmUSURuREWciGQSc8Zg4Nsx/HEk49SasiXh/nugVlWwajgkuYHSxc1OIFfq/Sg80NTsFJLX2Q3Y/xd8sTKJsxd9eXe2L1v3Ggzra6FYkNnpROTfVMSJiIPdDktWwgdzLSReKombxUb7Zid57fkyuLlBcBFwt5qdUkSyolFtsxOIq0i3QZe2noz7OJofNofy42YL23bDgB4Gj0SA1apTLEXyCn2nLiIA7NpzgadeS+e/H0PiJahVxWDW2/GMHFgGTw1YJiJSIPj6uvP2K2VZONFCjcoQnwijP7TQ8YXT/L7votnxROT/qYgTKeDS0uC/007T643C/H7AHV9vg2F9Yc67FureFWh2PBERMcGd5WHeWBjcy46HNY0jMcXp8bo/M7+ANE0LKmI6FXEiBdiv+6DLIFj0QzHsdneqlj3BoompdG0Hbvp0EBEp0KxW6N7Bjc/GXaJa+bOk29yYPA+6vQK7/7SbHU+kQNM1cSIF0IXYFN6fmcK3G/wxDCgSAP2fSOCRViVxc9M1DyIi8o+qd/izYBJ8GwXjZ8L+w/DUa9CiQTQjXi6Bf2FPsyOKFDj6rl2kgFnzUxIPP5fKN+szCriH74dlU+HRNoVUwImIyFVZLPBQc/hqKoTfnYhhuLFmW1k6D7SyaZfZ6UQKHh2JEykgzsfC+E9hxQZfAAJ84xj6TCoP3B9scjIREXEVQYHwwRt+LF1xkqkLgzh5xpPnRmR8ITiwh52igTo+IJIb9Jcmks/Z7QazFp3kkRcNVmzIuNbt8XapfPuJtwo4ERG5JR0fKMk30z15vF3GUbqvf4R2fVOIXHIcu90wO55IvqcjcSL52PFTMPDtM+yPzpi0+87yMLIf1Kis6xdEROT2+PnC0L7Qtim8Ni6emHOFmfhZabbvM/jPc1AixOyEIvmXjsSJ5EM2G8xbDh37w/7oYljd0unU4hjz34calc1OJyIi+UntqrD8Qx86NIvG3Wqw4RcLj/SDBd/asdl0VE4kJ6iIE8lntu46x+OD03hvFiSnQP0aBvPGJvPGS6F46Ni7iIjkAG9vd94aVJbFkyzUqQpJyfDuDDfaP3uWnbsvmB1PJN/RLp1IPpGSCuM+PsfStYHYDSuFfA0G9bLwSIQFN7dCZscTEZECoFJZmD0GFq6wMWG2jWNnQug9wk7fzvD0o+DpYXZCkfxBR+JE8oHte6DTy/DFmiDshpW7Khxn0YQUHm2lSbtFRCR3ubnBEw9aWTQhhVp3nMNmc+OjBdBlIOz4I93seCL5go7EibiwcxdSePfjJH7YVASAkCIwsMcl2jUvbXIyEREp6CqVK8zc92DVTzB2Bvx1DHoNs9Ks7jHeGliMIgFeZkcUcVn6jl7ERa36XwoPP5fmKOAebQVLp0K75j4mJxMREclgsUCbJrBsGrRoFAdYWL8zlC4DPdjwi9npRFyXjsSJuJgz5zO+0Vz9sxfgRZFCFxnWN43WzTSWs4iI5E0BhWHC6/58s/okU+YHceqcJ/1HZ0xPMKhnGsWCdLGcSFboSJyIi7DbDT6ad5xH+hms/hmsbtCzQzrffuKnAk5ERFzCQy1LsuwjT3p0yLh27vsN8NCz6cyY/zeGZiMQuWlZLuI2bNjAQw89RKlSpbBYLCxbtuyG60RFRVGvXj28vLy44447iIyMvIWoIgVX9Eno3P8c078oTXyiheqVYMEEGNjLnUJ++vZSJKe98847hIWF4evrS2Bg4E2tYxgGw4cPp2TJkvj4+BAREcGBAwdyNqiIC/D1hkG94PPxUCo4luQ0H6YuKsNzI+HvU2anE3ENWS7iEhMTqV27NtOmTbup9ocPH6Zdu3Y0b96cXbt2MWDAAPr06cOqVauyHFakIJq3HB57CQ78HYy7WxpPtI5m3nioUsHsZCIFR2pqKp06deL555+/6XXGjRvH5MmTmT59Olu2bMHPz4/WrVuTnJycg0lFXEf1O+CrDwvRKeIYnh4Gm3fBo/0hcqmNtDS72fFE8jSLYdz6wWuLxcJXX31Fhw4drtlmyJAhfPfdd/z++++OZV27duXixYusXLnypl4nLi6OgIAAYmNj8ff3v9W4Ijcl3QZnL4C7NeNmlo1bT+HFbl4cH05yqjv31IFXe13ijvK5P3BJui3jFlzE3G0iBUNe/syPjIxkwIABXLx48brtDMOgVKlSDB48mFdeeQWA2NhYihcvTmRkJF27dr2p18vL20LyHzP7v6MnYNQ0+OX/dxcrlDjFG713U7d+OFareUM4qP+T3JSVz/wcvyZu06ZNREREOC1r3bo1mzZtuuY6KSkpxMXFOd1ECopLKTDyg4sMHp8x6mRhP4PRL8P0kZhSwIlI1h0+fJiYmBin/i8gIIBGjRqp/xO5inKl4NO34fVn0/ByT+Hk+SAAPl4IKakmhxPJg3K8iIuJiaF48eJOy4oXL05cXByXLl266jpjxowhICDAcQsNDc3pmCJ5wpZfM06d/OrHQIz///Oc/d9kHr4/Y5hmEXENMTExAFft/y4/djXq/6Qgs1igywMeLPkgnXrVLgIwZxl0ehk27VQlJ3KlPDk65bBhw4iNjXXcjh07ZnYkkRx16uwlXhp1jr7D4e8YKB4E7wxIAaBUcR19E8kJQ4cOxWKxXPe2b9++XM2k/k8EypXxY+qIQACCAzNOtXxupCcvvHGMcxdSTM0mklfk+EnGJUqU4NQp56GGTp06hb+/Pz4+V9859fLywsvLK6ejiZjOMOD7DemMnmaQlBKExWLQ5QELLz0J3l5e7NxudkKR/Gvw4MH07Nnzum0qVqx4S89dokQJIKO/K1mypGP5qVOnqFOnzjXXU/8n4uyz9+C/H13gh01F2Lg7lM4DDV5/Flo0NjuZiLlyvIhr3LgxK1ascFq2evVqGjfWX58UbKfOwjvTYf02d8CdIP8LvPFcOvffmzHnm81mbj6R/C4kJISQkJyZY7FChQqUKFGCtWvXOoq2uLg4tmzZkqURLkUKOv9CMH5oEVpEnWLy50U5ftqDQe9mFHGDe6ZQuoS+9JCCKcunUyYkJLBr1y527doFZFy8vWvXLqKjo4GMU0G6d+/uaP/cc8/x119/8dprr7Fv3z4+/PBDFi9ezMCBA7PnHYi4GJvNYNLMY3ToZ7B+G7i7Q9/Odr6bUdhRwIlI3hIdHe3o62w2m6MfTEhIcLSpWrUqX331FZAxevOAAQN4++23+frrr9m9ezfdu3enVKlS1x3RWUSurk14cb6a5kGfxzJGiVy7CTq8CNPmaJJwKZiyfCTul19+oXnz5o77gwYNAqBHjx5ERkZy8uRJR0EHGd9GfvfddwwcOJAPPviAMmXK8Omnn9K6detsiC/iWv46BoP/e5G/TmQMVlCrCozoB3eUdSOPXqIqIsDw4cOZM2eO437dunUBWLduHeHh4QDs37+f2NhYR5vXXnuNxMRE+vbty8WLF7nvvvtYuXIl3t7euZpdJL/w8oT+T0Gr+2DQOxf4+0wRPllahh1/wvAXM0a4FCkobmueuNyieXIkN+XEPDlpaTBrKcxYDGnp4GFN5Ym2MbzUKxR396sPO2mzpbNze5Tpc+SA5smR3KXP/H9oW0huyivzpMKN+8C0NDuTZp3gi7WlSU6x4OkBz3RKp3sHC95e2Rde/Z/kpjw1T5xIQRf182ke6ZfCh/MzCrgm9WHpFINBz5S9ZgEnIiIi1+bh4carz5bhyykW7qkDqWkwbb47Dz97kZ9/OW92PJEcZ+7X+yL5WNIlGDM9jq+jQgALgf52hj7jRpsmYLHoQmwREZHbVaY4TB8JX65KZvynBqcuBPHi2wZPtofnHwdfnb0s+ZSOxInkgJ+2Q8f+8HWUP2Dh7qp/s3hCCm2batJuERGR7GSxwGNtvFk6xc59dWOxGxbmLoPHXoL1Wy6ZHU8kR+hInEg2OnEqibemJLJ5d8Yok6WKwet902lydxmTk4mIiORvpUv6MW0kbPgF3vkIjp+Cl/7rQ8NqxxjzWnGCi3qaHVEk2+hInEg2MAz4+kc7HftZ2Lw7BIvF4Kn28OUUaHK3visRERHJLU0bwNIp0Pbec4DB1r2hdBrgwcr/oekIJN/Q3qXIbTp+Ct7+CH7e6Qb4UCzwPMNfsNOkUbDZ0URERAokP19497UgWm08w6TPinD0hDtD3oPvogwG9UyiQqif2RFFbouKOJFblJZmZ+Knf/Plj6Ekp2YMb9y3s52n2gdk6/DGIiIicmvuvzeE+xrCrC9hxhLY8IuFzbvcefKBY/TvFYqbzkkTF6VfXZFb8OcReLR/Ap+vLEtyqoX6NQyWTIJnOrupgBMREclDPD3gua6weCKULX6O1HQvZn0dSq9hcCja7HQit0ZFnEgWpKTC1M/g8UFw9KQ/nu4p9HnkGDNGQ3mNXSIiIpJnVSoLyz4qSu8OJ/H1Nti1D7oMhKnzUklOtpkdTyRLVMSJ3KTVG07T4YVkZiyBdBs0bwTfTHenf89QrFbNGyAiIpLXWa0WXupVkq+mWmjSANLSYcYXnjzYN44NWzRJuLgOXRMncgPxifDux4l8u74YAEUD7PzneTciGgPo1EkRERFXUyIEprwBy1YnMnaGhTOxRXhpjEHXB6D/kxkDo4jkZToSJ3Id67bAI/3g2/UZo1iF3XWMJZNS/7+AExEREVdlscAjrfxY/qGFFo3iMQwLC76Djv3hhw2JZscTuS4diRO5ir9PJPL2h4ns3J9x9K1sSfjPczbuqRNqcjIRERHJTsVDfJjwOmzaBaM/zJg66NX3/Zj/9d+MGVKcIgEeZkcUyURH4kSuYBiwaIXBE6+4s3N/MdwsBk8/Cks+gHvq6NRJERGR/KpxHfhiMrRrchoLdnYeKEOXge6sWK9JwiXvUREn8v+OHIfHB8PrEyykpHlRsuhZPnrzPC93B28vs9OJiIhITvP1hv++Uoyp/znPHWVtxMZbeGsq9BxqsOfPOLPjiTioiJMCLzXNzuvjomnZy2DTzoyCrf9TBks/Kso99YPMjiciIiK57L6GwSycaKXfkxnzzP203cLDL/jw1qTj2DQbgeQBKuKkQNv9J7TskcTn35clNc3CvfUMvp8BTz5swdNDfx4iIiIFlYc79OoIn403qFjyNOk2D2YtL80j/WDvIbPTSUGnvVQpkC4lw3+nQ/vn4cjJQnh7JjOw2zE+Gw9lS5mdTkRERPKKcqUtrIwMYVD30xT2M/h1Hzz4LIyemkxiUrrZ8aSAUhEnBc7S70/RtFsyHy8Cmx0eag7/m+/FgD6huLlp0m4RERFxZrVaeLlXMdZGWmjTBNJt8OmX3tzXNYFV6y+aHU8KIBVxUmBcjIP+o5IZOK44p897U6xoOjPfganDoViQijcRERG5vuLB8PEoGP9KHH7eiZyPD6TvyEBenwBxCWank4JERZzke4YB30VBRE/4ep03Fgya149m5QwbEWFmpxMRERFX07mdP1HzrLRvnjEp+OffQEQvWPJdrMnJpKBQESf52p9/xfNw31O88BacuQB3lIMlkyHyvbIEFdW8ASIiInJrigV7M3m4H4smQoUycOosvPJeAA/3Oc7J0xrCUnKWijjJl+x2mLvM4MHnvPjtYHGsbnYG9IAVn8DdNXXqpIiIiGSPe+rAyk/h0RYxuFls/HqoNK16W1n4nSYJl5yjIk7ynYPR0OllePMDCylpnoSGnGH++IsM7AlenmanExERkfzG2wsmvFGCee/GcldlG3EJMOQ96DrQzq49OsVSsp+KOMk3LiXbeOWdaNr0tvPL7+DrDSP7Q9T8YO6pV9TseCIiIpLP3dewKMs/svLG8xmF3eZf3ej4ki/D3z9BmmYjkGykIk7yhZ1/QMteKSxZU5a0dDea3W1nTWTGJJ3u7jp9UkRERHKHuxWe6QyrPrVTuUwMNrsHc74txcPPw+79ZqeT/EJFnLi0xEvw1lR4pB8ci/HF1zuZIb3+JvJdC6WLm51ORERECqryZdz4YU5x/tP3HIH+8MdBePgFeGNCEnHxaWbHExenIk5c1sLlJ7mv6yVmfZlx4fCjreCnBV680L2MJu0WERER07m5Wej7eBBrI+Hh+zMGXpv3jS9NHk9ixbo4s+OJC3M3O4BIVp27CG9OSuW79SUBKB6UxvjXPGjWEEDFm4iIiOQtwUVgypsQ3uAiI6a4czExgOdHQaet8MbzEOhvdkJxNToSJy7DMGDpamjRA75b74nFYtCqUTSrZhr/X8CJiIiI5F2Ptg1k/ecedGlzCYsFlqyEFj1hzpfnsNs1H4HcPBVx4hL2Hoijba9TDPwvXIiDqhVh2TQLM94tS5EAzRsgIiIiriGoiBfjhvjwxWS4oxycvQDDpwbxYJ8YTpxWISc3R0Wc5Gk2G8z6Ah5+wYe9R4vjbrXxam/49mOoU83sdCIiIiK3psFdsOIT6NLqBG4WG3sOl6RlLwtzl2VcOydyPSriJM/a9xd07AdvTYPUdA8qlDzN4glx9HsSPHQ1p4iIiLg4L08YN6wUC9+Po251OwlJ8OYH0LGfjW27LpgdT/IwFXGS5yQmpfPSiKM88IzBrn1Q2A/GDIIfPytG/VpFzI4nIiIikq0a1S3C0ilujHoJ/Hxg514rnQcXZti4GFI1G4FchYo4yVO2/gYte6WzfEM5bHYLLe5JZ81seOIhcNNvq4iIiORTbm7Q4xFY+Wk61cqdxG53Z/73JWjXF3b8YXY6yWu0Wyx5QlwC/GcidHoZjp/2prBvMm/2Pc6n71gpEWJ2OhEREZHcUbaUOytmleCdl2IJCoQ/j2RcXvLKmHguxKaaHU/yCBVxYro5S05w3+OX+OzrjPtd28HGhd70eby0Ju0WERGRAsfNzcKTjwSwNhIea50xzdKSHwrT9IkUvlkbb3Y8yQM0PISY5vR5eHNiOit/KgVAyeAUJrzuRVhdk4OJiIiI5AFFAuD9odC03nnenOxObKI//d6G1ZtgRD8ICjQ7oZhFR+Ik1xkGLFqRMWn3yp/ccXMzaHfvUVbPdlMBJyIiIvIv7VsVZcN8b7q3T8HNDZavhRY9DKZ/dkqThBdQKuIkV/36RywRT53itfEZ18HdVRm+/djCh2+Xo3AhD7PjiYiIiORJgf6ejB7gxfIPoVpFuBBnYczM4rTtdZrokyrkChoVcZIr0m0wfQE8+lIhDh4vjod7Oq8/B8s/ghp3mJ1ORERExDXUqgLffAzd2v6N1S2dfdHFafW0hZlfgM1mdjrJLSriJMft/hMefh7GfAJpNiuVy5xm6eREnu0C7laz04mIiIi4Fg93+O9rZVg6OYlGtQ0uJcOoafDQc+ls3HbO7HiSC1TESY6Ji0/j+f8c5eHnDfYcgIDC8N4QWD23GLWqBZgdT0RERMSl1anhz8IJFt4dDIX9DPYcdOfJIYG89u4ZkjUbQb6mIk5yxMYd0Lq3wYqfy2G3W2hzXxprI6FTG7Bo1gARERGRbOHmBo8/CCs+SaVWpRPYDSuLVoXQtg9s+dXsdJJTVMRJtoqNh1fHwROD4cQZT4oUvsTofif4eLQHIUXNTiciIiKSP5Ut5cU3n5bi/dcSKRYEfx2DzgOg38hYzp5PMTueZLNbKuKmTZtG+fLl8fb2plGjRmzduvWabSMjI7FYLE43b2/vWw4seZPdbvDJ58cJ63KJxd9nLOveAf63wIfuj5YyNZuIyO165513CAsLw9fXl8DAwJtap2fPnpn6vzZt2uRsUBEp8B5r68eaSHi8Xcb9b9YH0KxbGstXXzI1l2SvLBdxixYtYtCgQYwYMYIdO3ZQu3ZtWrduzenTp6+5jr+/PydPnnTcjh49eluhJW+JOQPPvGHwzqelSbjkQ+mQS3wxGUa/DIX9zE4nInL7UlNT6dSpE88//3yW1mvTpo1T/7dgwYIcSigi8o+AQvDuKzDtjXMULXyRhORCvPRfH54fCafPm51OsoN7VleYMGECzzzzDL169QJg+vTpfPfdd8yaNYuhQ4dedR2LxUKJEiVu+jVSUlJISfnnsG9cXFxWY0ousNkMFnxn4d1PID7RDXerwUNNonnn1dL4+ZqdTkQk+7z11ltAxtklWeHl5aX+T0RM82CLIJo1SmPyvDRmfuHBivXw03aDp9uf5OWnS+LmpoEKXFWWjsSlpqayfft2IiIi/nkCNzciIiLYtGnTNddLSEigXLlyhIaG0r59e/bs2XPd1xkzZgwBAQGOW2hoaFZiSi745bcL3N/tDP+ZCPGJULcarJhhYdKIcvj5Zvm7ARGRfCkqKopixYpRpUoVnn/+ec6du/7Q3+r/RCS7FS7kwX+e9+Cbj6HmnRCXYGHS56Vo1eMsR46bnU5uVZaKuLNnz2Kz2ShevLjT8uLFixMTE3PVdapUqcKsWbNYvnw5n332GXa7nbCwMP7+++9rvs6wYcOIjY113I4dO5aVmJKDUtNg8jzoMiiAI6eK4emexsh+Bl9OgSoVzE4nIpJ3tGnThrlz57J27VrGjh3L+vXradu2LbbrzMar/k9EckqNO2DZh9DjwWO4u6Vx4O8QWj0NHy2AdE0S7nJy/JBJ48aNady4seN+WFgY1apV4+OPP2b06NFXXcfLywsvL6+cjiZZtOMPg6HvWdh/GMCN6hVOM2GYN9Uq+5sdTUQky4YOHcrYsWOv22bv3r1UrVr1lp6/a9eujv/XrFmTWrVqUalSJaKiomjRosVV11H/JyI5yd0KowaH0vXhBN752J2ftmdcFvPVD2m83jeW8MbBZkeUm5SlIi44OBir1cqpU6eclp86deqmz/n38PCgbt26HDx4MCsvLSa6EJvKK+/EsHZbKAZQNABG9IP2LYppzjcRcVmDBw+mZ8+e121TsWLFbHu9ihUrEhwczMGDB69ZxImI5IbqlQvx2Xj4chWM+tBg/xEPev6nKI9FnGP0oCB8NJB8npelIs7T05P69euzdu1aOnToAIDdbmft2rX069fvpp7DZrOxe/duHnjggSyHldwXtRWGjHcj5mxZANo1TebtQd4UDTA5mIjIbQoJCSEkJCTXXu/vv//m3LlzlCxZMtdeU0TkWiwWeKwN3FM7hf4jz7LjzzIsWR3Elj0wZhDcV9/shHI9WZ5iYNCgQcyYMYM5c+awd+9enn/+eRITEx2jVXbv3p1hw4Y52o8aNYoffviBv/76ix07dvDkk09y9OhR+vTpk33vQrLduYsGA/4LPYZAzFl3ggOTGTvwJB++pQJORAqe6Ohodu3aRXR0NDabjV27drFr1y4SEhIcbapWrcpXX30FZAzo9eqrr7J582aOHDnC2rVrad++PXfccQetW7c2622IiGRSpqQ3X31chg+HJ1MyBKJPQLdXoM+wC5w8rbnl8qosXxPXpUsXzpw5w/Dhw4mJiaFOnTqsXLnSMdhJdHQ0bm7/1IYXLlzgmWeeISYmhiJFilC/fn1+/vlnqlevnn3vQrKN3W7w4dzjTFsYRFKKD25u0KsjDH7aGz8ffXssIgXT8OHDmTNnjuN+3bp1AVi3bh3h4eEA7N+/n9jYWACsViu//fYbc+bM4eLFi5QqVYpWrVoxevRoXfMmInlSu+beNG0I42fC3GUGqzcXYWP3RP47KIUOLb10CU0eYzEMwzA7xI3ExcUREBBAbGws/v4aRCOn/B0DwybY2bAtowgPLZ7I1BF+1KlmcrBclm6DsxcyLv51t5qXw2ZLZ+f2KOrWD8dqNXfahnRbxi24iLnbRAoGfeb/Q9tCclNe6f8g7/SBBbX/W7X+LK9PsHI2rggALcNg9AAomXtnoBdIWfnMz/LplJL/pKcbzPzCoGUv2LDNDQ93g84RR1kd6V3gCjgRERGRgq51s2A2LChMv27peLjD6p+hRU+DMdP+xmbL88d/CgQVcQXcpu3nafbEWUZNs5CUDA1rwcqZFsb/pxw+3gXoKycRERERcfDzdefVPu6s+ATqVofEJAvTvyhDRPcLHIw2O52oiCugUlLh/VnQ7bVA/j4TgpdHKu8MNFg0Ee4oa3Y6EREREckL7qwAX06GZx45hoc1lb9OFKVtH/hgLqSmmZ2u4FIRVwBt+dVO22dg8jyw2d2oc+dpVnySypMPW3DTb4SIiIiIXMFqhTdeCuWHWXaaN8oo3ibMhpa9Ulm1/rTZ8Qok7bIXIOfOp9Bj0DE6D3DjUDSEFIWPRsKy6cW4o3whs+OJiIiISB5Wsaw3s8fA5DegaIDBkeOe9B0ZwsC3L5Ko2QhylYq4AmL1RmjzjDtRO0MBeCg8ibWR8EAzNGSsiIiIiNwUiwXat4AVn6TQqPrfgIWlawNp2QvWbTE7XcGhIi6fizlj44WR0OcNOH3eSomgZCa+doqpI3wJKGx2OhERERFxRSWLebN4WhlmvZNKmRJw/BT0HApPDTrH3yeTzI6X76mIy6fsdoP3PzlG027pfLcerG7w/OOw/nNvOrYtbnY8EREREckHWoR5snoW9OkEbhaDDTuDiOhpYcn36eT92ahdl4q4fOjocej2CkxeEEpKmhflSsTx9XQY2he8vcxOJyIiIiL5ia8PvPkCfDrqHMWLnOdSqg+vjHOnxxA4FmN2uvxJRVw+kppm56MFdlr1hp93WvDyNHjygaOsmVuIuyqbnU5ERERE8rMW9wWzYUEAg3vZ8PKA9dugZU+DkROjSUuzmx0vX1ERl09s2HyOpl0v8O4nbiSnwL31YPUsC++8Wg5PD/2YRURERCTneXtZeam7le8/hYa1DC6lWJj9dVkiesSx7y+z0+Uf2rt3cckpMOZj6PGfopw8H4S3RzJjX7Hz+XtQrrTZ6URERESkIKpUFhZOgBc6HcPTPYUjJwNp1xfemwXJqWanc30q4lzYhm02Wj0N0xeC3W6h0V1nWDnTRtd2bpo2QERERERMZbVaGPJCKOvmudH6Pki3wZR50KJ7Ct+uOWV2PJemIs4FnTqTzOMvHeOp16wcPQElguHTt2HxlBAqhPqZHU9ERERExKFMCQ8+GQ3T34LgInb+PuXFi+8U56VR8cQlmJ3ONamIcyGGAd+th7Z9Pfl5d8ak3R3uj2dNJLS819xsIiIiIiLX07YpfP9JKvfVPgbA8nWFadkLVm80OZgLcjc7gNycYyfTGDXNgx82ArgRWiKFYX1iadeimNnRRERERERuSrFgbz6fFMr/frHxxiQrR45DnzegYY0zjBviq7PKbpKOxOVxNpvBmGnR3N/dzg8bwd0KLz0Fa+Z4qYATEREREZfUpIGVVTPhhSfAzc1g654QWve2suBbuyYJvwkq4vKwQ9HQdZCF6V+UJTXdiwolL/LdJzD4afD2NDudiIiIiMit8/aCIc/A3P+ep1TQWVLSvBn6vhtPDIYjx81Ol7epiMuDLiXb+GCOjTZ9YOtv4ONt0Lt9NKvn+FO1otnpRERERESyT5NGQaxfUJT/PGfg7QU/74RWvQxeHxdNqiYJvyoVcXnMmv+dpenjsUyItJKaBs0awprZFoYPKIuHJu0WERERkXzI08ONvl0srJ4F99U3SEmz8Pn3ZWnRPYHdf5qdLu9RVZBHJF6CUdOgz4ggTl8sio/nJd4fks6cd6FMCbPTiYiIiIjkvLKlYN44GNDtGN6eyUTH+NP+eRjzMVxKNjtd3qEiLg9YszGVVr1g5hdgGBaa1TvH2kiDx9q4a9JuERERESlQ3NwsDOwTyobPPXmoOdjsMH0hhD+ZzNLvNUk4aIoBU504lczLo86y9Y8yAJQpDu8MgvCGQSYnExERERExV/FgN6YOhw4R8PoEGzHnvBk4zpsftybxziBfAgqbndA8OhJnAsOAZWvggWe9/r+AM3g04iI/zIbwhmanExERERHJOyLC4PtP0mlePxoLBt9E+dKiB3wXRYGdjkBH4nLZwSPJjJ7uTdQWAAsVyqTyn2diadk0xOxoIiIiIiJ5UlBRLyLfK8vW3+wMm2Dh4FF44S2oU/k044f6cGfFgnVYTkficonNBqMmRdOmjxtRW8DTA17pDT/M8lQBJyIiIiJyExrWcmPFJ/Byd7Ba7ew6UIx2z3oxd5mBvQDNRqAiLhfsPwyP9oeZy8uSZvOkUulzfP8p9H8yo5gTEREREZGb4+UJg3rB52MvEhpyhtR0T978wELnAXAw2ux0uUNFXA5KTEpn3Ix02vWFnXvBz9fghU5/s3pOUe4oa3Y6ERERERHX1bh+UaLmBzOyP/h6w7bd0Ka3wStvH+VSss3seDlKRVwO+X7dWZo8Hs+0+e6kpUPLMPgx0sKQF8pgtWreABERERGR2+XubqFXR1gTCc3utpOWbmHJ2nJE9Ehm5x9mp8s5KuKyWXwivDEJnhsVzLm4Ivh6JTLp9RRmvA0ldOmbiIiIiEi2K10cIt+1MKTX3/h6JfP3aT8e6QdvTYXES2any34q4rLRtz8mE9ET5i3PuN867AJR86w80tJLk3aLiIiIiOQgNzcLL3Qvw08LvXi0Vcb0A7O+hKZPJLNw+Umz42UrTTGQDf4+mUy/EefYeaA0AOVKwZjBcG+9IiYnExEREREpWIICLUwYBu1bwGvj04k5682QSSVZvyOFdwZ5UTTA7IS3T0fiboNhwOLvMybt3nmgNBaLnU6tzrFqJtxbz+x0IiIiIiIFV7OGsPJTO60aRWOxGKzY4EWLHvDVatefJFxF3C3a82cC3V4xeHUcxMZbqFI+jch3zvPesCB8vM1OJyIiIiIiRQI8mfFuWb6aClUrwvlYGPBfeODp0+w9EGd2vFumIi6L0m3w5nvHePh5LzbusODlCcOehRWfehDeONjseCIiIiIi8i91q1v49mN4tTe4W238caQYD7/gw8wvDGwuOBuBirgs+P0AtH8e5n4XSrrdgztDz/DDLHiuK7hbzU4nIiIiIiLX4uEO/Z6ExRPiqFDiFKnpHoyaZuHR/rD/sNnpskZF3E2IT0hj9LRUHn4uo5DzL2Qw6KkTrIoMpnxps9OJiIiIiMjNql+rCGs/K8aYQVDYD3buhbZ9DF4eeZTEpHSz490UFXE38M3qszR5PIlPv/DEZod2zWDtHAsvP10KNzfNGyAiIiIi4mqsVgtPPARrZkOLxjZsdgvL1pejZa80tu02O92NqYi7hth4eG089PtvMBcSAijkncDk1y/x4UgoVtTsdCIiIiIicrtKhMDMd6y82fc4hX0vcfy0D4+9BP+ZCPGJZqe7NhVx/2K3G3zxfSItesKiFRnLHg6PZf3nHrRv6WNqNhERERERyV4WC/R5vDQbF/rQtV3Gss++hvu6JjNnyQlzw12DirgrHD2ewsPPnGTwOD/OnIdKofDFZJgyIoDgol5mxxMRERERkRwSUBjGvgILJkCpYqlcTPBm+IeleOaNNE6fNzudMxVxgN0On38N7Z71ZPdfpXCz2Oja5gwrPoW7a5qdTkREREREcktYXfhhpoV290bj5mbww0YPWvTIOEsvr0wSXuCLuJ2/x9J5gMHrEyE+0ULNyunMHx/L2CEheHuanU5ERERERHJb4UIefPh2Wb6ZbuGuyhCXkDFeRsvup/ltb6zZ8QpuEZeWDkPHHufRl/3YttuCrzeMeBGWf+RO4/oauUREREREpKC7qzIs/whefw483NM58HcxOr7kx/SFkG7iJOG3VMRNmzaN8uXL4+3tTaNGjdi6det12y9ZsoSqVavi7e1NzZo1WbFixS2FzS679sKDz8KClaWx2d2pWvYUq2YZPP0YWDVpt4iIXOHIkSP07t2bChUq4OPjQ6VKlRgxYgSpqanXXS85OZkXX3yRoKAgChUqxKOPPsqpU6dyKbWIiGQXdys82wWWTk6kcplTpKW7M+ZjaP98xhzSZshyEbdo0SIGDRrEiBEj2LFjB7Vr16Z169acPn36qu1//vlnHn/8cXr37s3OnTvp0KEDHTp04Pfff7/t8Fl1MS6VNyam8Eg/2PcXFPE3eL3PKb6fXYyyJTXnm4iIZLZv3z7sdjsff/wxe/bsYeLEiUyfPp3XX3/9uusNHDiQb775hiVLlrB+/XpOnDhBx44dcym1iIhkt1rVAlg9tzjvDckYBOX3A/DQcwbP/yea+IS0XM1iMYysXZ7XqFEj7r77bqZOnQqA3W4nNDSU/v37M3To0Eztu3TpQmJiIt9++61j2T333EOdOnWYPn36Tb1mXFwcAQEBxMbG4u/vn5W4DstWnWf4FHdiEzPWf6QlvPkCBAXe0tNJPpZug7MXMr51cTfxyKzNls7O7VHUrR+O1epuXhAytkm6DYKLmLtNpGDIjs/8nDZ+/Hg++ugj/vrrr6s+HhsbS0hICPPnz+exxx4DMorBatWqsWnTJu65556beh1X2BaSf+SV/g/yTh+o/k+u5cx5ePODdL7fkPH7WTI4hQmvexFW99afMyuf+Vk6Epeamsr27duJiIj45wnc3IiIiGDTpk1XXWfTpk1O7QFat259zfYAKSkpxMXFOd1ux/7D8PK7RYlN9MffN55pb8Qz6XUVcCIicmtiY2MpWvTa109v376dtLQ0p/6vatWqlC1bNlf7PxERyRkhRWH6W+6MfvEEgYUucfKsF08MhsN/587rZ6mIO3v2LDabjeLFizstL168ODExMVddJyYmJkvtAcaMGUNAQIDjFhoampWYmVSpAJ3bQufWCWyY78WDLQrf1vOJiEjBdfDgQaZMmcKzzz57zTYxMTF4enoSGBjotDy3+z8REclZ3R8rxU8LfXiqPXR7CCqUyZ3XzZOjUw4bNozY2FjH7dixY7f9nONehfFDC1EkQPMGiIgIDB06FIvFct3bvn37nNY5fvw4bdq0oVOnTjzzzDPZnikn+j8REclZhf3g7QEZt9ySpZOMg4ODsVqtmUbXOnXqFCVKlLjqOiVKlMhSewAvLy+8vLyyEu2GLBq3RERErjB48GB69ux53TYVK1Z0/P/EiRM0b96csLAwPvnkk+uuV6JECVJTU7l48aLT0Tgz+j8REckduVlvZOlInKenJ/Xr12ft2rWOZXa7nbVr19K4ceOrrtO4cWOn9gCrV6++ZnsREZHcEBISQtWqVa978/TMOHvj+PHjhIeHU79+fWbPno2b2/W7z/r16+Ph4eHU/+3fv5/o6Gj1fyIictuyfDrloEGDmDFjBnPmzGHv3r08//zzJCYm0qtXLwC6d+/OsGHDHO1ffvllVq5cyfvvv8++ffsYOXIkv/zyC/369cu+dyEiIpJDLhdwZcuW5b333uPMmTPExMQ4Xdt2/Phxqlat6pg3NSAggN69ezNo0CDWrVvH9u3b6dWrF40bN77pkSlFRESuJctjtnbp0oUzZ84wfPhwYmJiqFOnDitXrnQMXhIdHe30DWVYWBjz58/njTfe4PXXX6dy5cosW7aMu+66K/vehYiISA5ZvXo1Bw8e5ODBg5Qp43zF+uVZetLS0ti/fz9JSUmOxyZOnIibmxuPPvooKSkptG7dmg8//DBXs4uISP6U5XnizKB5ciQ35ZV5cvLKHDmgeXIkd+kz/x/aFpKb8kr/B3mnD1T/J7kpx+aJExEREREREXOpiBMREREREXEhKuJERERERERciLkX2tyky5ftxcXFmZxECoJ0G8THg9Vi7vyCNns6iYmJXLgYh9XN3D9VwwCbAZ554DoJyf8uf9a7wCXbOU79n+SmvNL/Qd7pA9X/SW7KSv/nEkVcfHw8AKGhoSYnERGR3BIfH09AQIDZMUyl/k9EpOC5mf7PJUantNvtnDhxgsKFC2O5xa+G4uLiCA0N5dixY3l+hC9XygqulVdZc44r5VXWnJMdeQ3DID4+nlKlSt1wUu38rqD1f+BaeZU157hSXmXNOa6UN7f7P5c4Eufm5pZpbp5b5e/vn+d/CS5zpazgWnmVNee4Ul5lzTm3m7egH4G7rKD2f+BaeZU157hSXmXNOa6UN7f6v4L9FaeIiIiIiIiLUREnIiIiIiLiQgpMEefl5cWIESPw8vIyO8oNuVJWcK28yppzXCmvsuYcV8tbELjaz8SV8iprznGlvMqac1wpb25ndYmBTURERERERCRDgTkSJyIiIiIikh+oiBMREREREXEhKuJERERERERciIo4ERERERERF+LSRdy0adMoX7483t7eNGrUiK1bt163/ZIlS6hatSre3t7UrFmTFStWOD1uGAbDhw+nZMmS+Pj4EBERwYEDB3I964wZM2jSpAlFihShSJEiREREZGrfs2dPLBaL061Nmza5njUyMjJTDm9vb6c2Oblds5o3PDw8U16LxUK7du0cbXJq227YsIGHHnqIUqVKYbFYWLZs2Q3XiYqKol69enh5eXHHHXcQGRmZqU1W/w5yIuvSpUtp2bIlISEh+Pv707hxY1atWuXUZuTIkZm2a9WqVXM9a1RU1FV/B2JiYpza5YXterXfRYvFQo0aNRxtcmq7jhkzhrvvvpvChQtTrFgxOnTowP79+2+4npmfswWJ+r+c6f+ymtfsPlD9n/mf0+r/ci6v+sDrc9kibtGiRQwaNIgRI0awY8cOateuTevWrTl9+vRV2//88888/vjj9O7dm507d9KhQwc6dOjA77//7mgzbtw4Jk+ezPTp09myZQt+fn60bt2a5OTkXM0aFRXF448/zrp169i0aROhoaG0atWK48ePO7Vr06YNJ0+edNwWLFhwWzlvJStkzEx/ZY6jR486PZ5T2/VW8i5dutQp6++//47VaqVTp05O7XJi2yYmJlK7dm2mTZt2U+0PHz5Mu3btaN68Obt27WLAgAH06dPHqXO4lZ9XTmTdsGEDLVu2ZMWKFWzfvp3mzZvz0EMPsXPnTqd2NWrUcNquP/30023lvJWsl+3fv98pS7FixRyP5ZXt+sEHHzhlPHbsGEWLFs30+5oT23X9+vW8+OKLbN68mdWrV5OWlkarVq1ITEy85jpmfs4WJOr/cuYz+lbygnl9oPq/vPE5rf4v5/KqD7wBw0U1bNjQePHFFx33bTabUapUKWPMmDFXbd+5c2ejXbt2TssaNWpkPPvss4ZhGIbdbjdKlChhjB8/3vH4xYsXDS8vL2PBggW5mvXf0tPTjcKFCxtz5sxxLOvRo4fRvn3728p1NVnNOnv2bCMgIOCaz5eT2/VW8v7bxIkTjcKFCxsJCQmOZTm1ba8EGF999dV127z22mtGjRo1nJZ16dLFaN26teP+7b7/7Mp6NdWrVzfeeustx/0RI0YYtWvXzrZcV3MzWdetW2cAxoULF67ZJq9u16+++sqwWCzGkSNHHMtyY7sahmGcPn3aAIz169dfs42Zn7MFifq/nPuMdqU+UP1f3vycNgz1fzdDfWCG2/k8cMkjcampqWzfvp2IiAjHMjc3NyIiIti0adNV19m0aZNTe4DWrVs72h8+fJiYmBinNgEBATRq1Oiaz5lTWf8tKSmJtLQ0ihYt6rQ8KiqKYsWKUaVKFZ5//nnOnTt3yzlvJ2tCQgLlypUjNDSU9u3bs2fPHsdjObVdbyfvlWbOnEnXrl3x8/NzWp7d2/ZW3Oh3Njvef06x2+3Ex8dn+p09cOAApUqVomLFinTr1o3o6GiTEkKdOnUoWbIkLVu2ZOPGjY7leXm7zpw5k4iICMqVK+e0PDe2a2xsLECmn+mVzPqcLUjU/2XIic9oV+oD1f/l3c9p9X85R32gM5cs4s6ePYvNZqN48eJOy4sXL57pvN7LYmJirtv+8r9Zec6cyvpvQ4YMoVSpUk4/9DZt2jB37lzWrl3L2LFjWb9+PW3btsVms+Vq1ipVqjBr1iyWL1/OZ599ht1uJywsjL///hvIue16q3mvtHXrVn7//Xf69OnjtDwntu2tuNbvbFxcHJcuXcqW362c8t5775GQkEDnzp0dyxo1akRkZCQrV67ko48+4vDhwzRp0oT4+PhczVayZEmmT5/Ol19+yZdffkloaCjh4eHs2LEDyJ6/2Zxw4sQJvv/++0y/r7mxXe12OwMGDODee+/lrrvuumY7sz5nCxL1fzn3Ge1KfaD6v7z5OQ3q/3KK+sDM3LO8huSqd999l4ULFxIVFeV0sXTXrl0d/69Zsya1atWiUqVKREVF0aJFi1zL17hxYxo3buy4HxYWRrVq1fj4448ZPXp0ruW4FTNnzqRmzZo0bNjQaXle2bauav78+bz11lssX77c6Tz7tm3bOv5fq1YtGjVqRLly5Vi8eDG9e/fOtXxVqlShSpUqjvthYWEcOnSIiRMnMm/evFzLkVVz5swhMDCQDh06OC3Pje364osv8vvvv2fLdQYiNyuv93/gun2g+r+cof4v56gPzMwlj8QFBwdjtVo5deqU0/JTp05RokSJq65TokSJ67a//G9WnjOnsl723nvv8e677/LDDz9Qq1at67atWLEiwcHBHDx40JSsl3l4eFC3bl1HjpzarrebNzExkYULF97UH3h2bNtbca3fWX9/f3x8fLLl55XdFi5cSJ8+fVi8eHGmUwr+LTAwkDvvvDPXt+vVNGzY0JEjL25XwzCYNWsWTz31FJ6entdtm93btV+/fnz77besW7eOMmXKXLetWZ+zBYn6v8yy6zPalfpA9X9573Na/V/OUR94dS5ZxHl6elK/fn3Wrl3rWGa321m7dq3TN2JXaty4sVN7gNWrVzvaV6hQgRIlSji1iYuLY8uWLdd8zpzKChmj14wePZqVK1fSoEGDG77O33//zblz5yhZsmSuZ72SzWZj9+7djhw5tV1vN++SJUtISUnhySefvOHrZMe2vRU3+p3Njp9XdlqwYAG9evViwYIFTkNWX0tCQgKHDh3K9e16Nbt27XLkyGvbFTJGyTp48OBN7XRl13Y1DIN+/frx1Vdf8eOPP1KhQoUbrmPW52xBov4vs+z6jHalPlD9X976nFb/l7PUB147pEtauHCh4eXlZURGRhp//PGH0bdvXyMwMNCIiYkxDMMwnnrqKWPo0KGO9hs3bjTc3d2N9957z9i7d68xYsQIw8PDw9i9e7ejzbvvvmsEBgYay5cvN3777Tejffv2RoUKFYxLly7latZ3333X8PT0NL744gvj5MmTjlt8fLxhGIYRHx9vvPLKK8amTZuMw4cPG2vWrDHq1atnVK5c2UhOTs7VrG+99ZaxatUq49ChQ8b27duNrl27Gt7e3saePXuc3k9ObNdbyXvZfffdZ3Tp0iXT8pzctvHx8cbOnTuNnTt3GoAxYcIEY+fOncbRo0cNwzCMoUOHGk899ZSj/V9//WX4+voar776qrF3715j2rRphtVqNVauXHnT7z+3sn7++eeGu7u7MW3aNKff2YsXLzraDB482IiKijIOHz5sbNy40YiIiDCCg4ON06dP52rWiRMnGsuWLTMOHDhg7N6923j55ZcNNzc3Y82aNY42eWW7Xvbkk08ajRo1uupz5tR2ff75542AgAAjKirK6WealJTkaJOXPmcLEvV/OfMZfSt5zewD1f/ljc9p9X85l/cy9YFX57JFnGEYxpQpU4yyZcsanp6eRsOGDY3Nmzc7HmvWrJnRo0cPp/aLFy827rzzTsPT09OoUaOG8d133zk9brfbjTfffNMoXry44eXlZbRo0cLYv39/rmctV66cAWS6jRgxwjAMw0hKSjJatWplhISEGB4eHka5cuWMZ555Jlv+wLKadcCAAY62xYsXNx544AFjx44dTs+Xk9s1q3kNwzD27dtnAMYPP/yQ6blyctteHtr337fL+Xr06GE0a9Ys0zp16tQxPD09jYoVKxqzZ8/O9LzXe/+5lbVZs2bXbW8YGcNDlyxZ0vD09DRKly5tdOnSxTh48GCuZx07dqxRqVIlw9vb2yhatKgRHh5u/Pjjj5meNy9sV8PIGH7Yx8fH+OSTT676nDm1Xa+WE3D6Hcxrn7MFifq/nOn/sprX7D5Q/Z/5n9Pq/3Iur2GoD7wey/8HFRERERERERfgktfEiYiIiIiIFFQq4kRERERERFyIijgREREREREXoiJORERERETEhaiIExERERERcSEq4kRERERERFyIijgREREREREXoiJORERERETEhaiIE7lC+fLlmTRp0m09R2RkJIGBgddtM3LkSOrUqeO437NnTzp06OC4Hx4ezoABA24rR27Iju0lIiLmU/+XNer/xGwq4kRM8Morr7B27dprPr506VJGjx7tuG92Z3Gtjnnbtm307ds39wOJiIhLUv8nkj3czQ4gkhtSU1Px9PQ0O4ZDoUKFKFSo0DUfL1q0aK7kuN3tEhISko1pMqSlpeHh4ZHtzysiUhCp/7s69X/i6nQkTlxOeHg4/fr1o1+/fgQEBBAcHMybb76JYRiONuXLl2f06NF0794df39/x7dlX375JTVq1MDLy4vy5cvz/vvvZ3r++Ph4Hn/8cfz8/ChdujTTpk1zenzChAnUrFkTPz8/QkNDeeGFF0hISMj0PMuWLaNy5cp4e3vTunVrjh075njs36eTXO09Xj6dJDw8nKNHjzJw4EAsFgsWi4XExET8/f354osvMr2mn58f8fHx1912AwYMIDg4mNatW9/wPUVFRdGrVy9iY2Mdrz9y5EjHdr7yG9Lo6Gjat29PoUKF8Pf3p3Pnzpw6deqa7/PIkSNYLBYWLVpEs2bN8Pb25vPPP79mexGRgkz9n/o/kctUxIlLmjNnDu7u7mzdupUPPviACRMm8Omnnzq1ee+996hduzY7d+7kzTffZPv27XTu3JmuXbuye/duRo4cyZtvvklkZKTTeuPHj3esN3ToUF5++WVWr17teNzNzY3JkyezZ88e5syZw48//shrr73m9BxJSUm88847zJ07l40bN3Lx4kW6du16S+916dKllClThlGjRnHy5ElOnjyJn58fXbt2Zfbs2U5tZ8+ezWOPPUbhwoWvu+08PT3ZuHEj06dPv+F7CgsLY9KkSfj7+zte/5VXXsn0vHa7nfbt23P+/HnWr1/P6tWr+euvv+jSpcsN3+Pl7bx3715HxyoiIpmp/1P/JwKAIeJimjVrZlSrVs2w2+2OZUOGDDGqVavmuF+uXDmjQ4cOTus98cQTRsuWLZ2Wvfrqq0b16tWd1mvTpo1Tmy5duhht27a9Zp4lS5YYQUFBjvuzZ882AGPz5s2OZXv37jUAY8uWLYZhGMaIESOM2rVrOx7v0aOH0b59e6f3+PLLLzvlmjhxotPrbtmyxbBarcaJEycMwzCMU6dOGe7u7kZUVNQ1szZr1syoW7fuNR+/3nsKCAjI1O7KXD/88INhtVqN6Ohox+N79uwxAGPr1q1XfZ3Dhw8bgDFp0qQbZhIRKejU/2VQ/ydiGDoSJy7pnnvuwWKxOO43btyYAwcOYLPZHMsaNGjgtM7evXu59957nZbde++9mdZr3LixU5vGjRuzd+9ex/01a9bQokULSpcuTeHChXnqqac4d+4cSUlJjjbu7u7cfffdjvtVq1YlMDDQ6XluV8OGDalRowZz5swB4LPPPqNcuXI0bdr0uuvVr18/07KbeU83snfvXkJDQwkNDXUsq169+k2973//rERE5OrU/6n/EwGdTin5mJ+fX7Y/55EjR3jwwQepVasWX375Jdu3b3dcM5Camprtr3cjffr0cZwOM3v2bHr16uXUuV/Nv7dLXnhPOfGzEhEpqNT/XZ36P8lPVMSJS9qyZYvT/c2bN1O5cmWsVus116lWrRobN250WrZx40buvPNOp/U2b96c6bmrVasGwPbt27Hb7bz//vvcc8893HnnnZw4cSLTa6Wnp/PLL7847u/fv5+LFy86nierPD09nb4tvezJJ5/k6NGjTJ48mT/++IMePXpk+blv5j1d6/WvVK1aNY4dO+Z0Afsff/zBxYsXqV69epZziYhIZur/Mqj/k4JORZy4pOjoaAYNGsT+/ftZsGABU6ZM4eWXX77uOoMHD2bt2rWMHj2aP//8kzlz5jB16tRMFylv3LiRcePG8eeffzJt2jSWLFnieO477riDtLQ0pkyZwl9//cW8efMcF0dfycPDg/79+7Nlyxa2b99Oz549ueeee2jYsOEtvd/y5cuzYcMGjh8/ztmzZx3LixQpQseOHXn11Vdp1aoVZcqUyfJz38x7Kl++PAkJCaxdu5azZ89e9TSTiIgIatasSbdu3dixYwdbt26le/fuNGvWTKeLiIhkE/V/GdT/SUGnIk5cUvfu3bl06RINGzbkxRdf5OWXX77hpJv16tVj8eLFLFy4kLvuuovhw4czatQoevbs6dRu8ODB/PLLL9StW5e3336bCRMmOEaMql27NhMmTGDs2LHcddddfP7554wZMybTa/n6+jJkyBCeeOIJ7r33XgoVKsSiRYtu+f2OGjWKI0eOUKlSpUxz0/Tu3ZvU1FSefvrpW3rum3lPYWFhPPfcc3Tp0oWQkBDGjRuX6XksFgvLly+nSJEiNG3alIiICCpWrHhb71tERJyp//uH+j8pyCyGccXkIiIuIDw8nDp16jjNz1KQzZs3j4EDB3LixIk8NaGriIhkL/V/ztT/SUHmbnYAEbk1SUlJnDx5knfffZdnn31WHZiIiBQI6v9EdDqliMsaN24cVatWpUSJEgwbNszsOCIiIrlC/Z+ITqcUERERERFxKToSJyIiIiIi4kJUxImIiIiIiLgQFXEiIiIiIiIuREWciIiIiIiIC1ERJyIiIiIi4kJUxImIiIiIiLgQFXEiIiIiIiIuREWciIiIiIiIC/k/n3RVyALGuyIAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: the clipped objective as a function of the ratio, both advantage signs\n",
"ratios = np.linspace(0.0, 2.0, 200)\n",
"fig, ax = plt.subplots(1, 2, figsize=(9, 3.4))\n",
"for axi, A in zip(ax, [1.0, -1.0]):\n",
" axi.plot(ratios, ratios * A, ls=\":\", color=\"#888\", label=\"unclipped r*A\")\n",
" axi.plot(ratios, ppo_clip_term(ratios, np.full_like(ratios, A), 0.2),\n",
" color=\"#1E40FF\", label=\"PPO surrogate\")\n",
" axi.axvspan(0.8, 1.2, color=\"#1E40FF\", alpha=0.07)\n",
" axi.axvline(1.0, color=\"#bbb\", lw=0.8)\n",
" axi.set_xlabel(\"probability ratio r\"); axi.set_title(f\"advantage = {A:+.0f}\")\n",
" axi.legend(fontsize=8)\n",
"plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "f7955967",
"metadata": {},
"source": [
"> **Interpretation.** For positive advantage (left), the objective tracks the unclipped line until $r = 1.2$, then flattens: no reward for pushing the action's probability up past the trust region. For negative advantage (right), the objective flattens on the low side at $r = 0.8$, but the `min` lets the loss keep falling for $r > 1.2$ so a bad action that the policy made *more* likely still gets penalized. The flat regions are the trust region in action.\n",
"\n",
"Now assemble PPO: collect a batch of transitions with the current policy, compute GAE advantages, then take several epochs of clipped updates on minibatches. CartPole reaches a few hundred steps quickly.\n",
"\n",
"> **Runtime:** this cell takes ~10-30 s on CPU.\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "d8e3f28b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:43.007616Z",
"iopub.status.busy": "2026-06-11T20:07:43.007517Z",
"iopub.status.idle": "2026-06-11T20:07:45.080380Z",
"shell.execute_reply": "2026-06-11T20:07:45.079995Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"PPO greedy episode length: start 14, end 500 (out of 500)\n"
]
}
],
"source": [
"# re-seed so the run reproduces standalone\n",
"torch.manual_seed(SEED)\n",
"ppo_env = CartPole(np.random.default_rng(SEED))\n",
"\n",
"class ActorCritic(torch.nn.Module):\n",
" def __init__(self):\n",
" super().__init__()\n",
" self.body = torch.nn.Sequential(\n",
" torch.nn.Linear(4, 64), torch.nn.Tanh(),\n",
" torch.nn.Linear(64, 64), torch.nn.Tanh())\n",
" self.actor = torch.nn.Linear(64, 2)\n",
" self.critic = torch.nn.Linear(64, 1)\n",
" def forward(self, x):\n",
" h = self.body(x)\n",
" return self.actor(h), self.critic(h).squeeze(-1)\n",
"\n",
"ac = ActorCritic()\n",
"opt = torch.optim.Adam(ac.parameters(), lr=3e-3)\n",
"\n",
"def collect(steps=800):\n",
" obs, acts, rews, dones, vals, lps = [], [], [], [], [], []\n",
" s = ppo_env.reset()\n",
" for _ in range(steps):\n",
" st = torch.tensor(s, dtype=torch.float32)\n",
" with torch.no_grad():\n",
" logits, v = ac(st); dist = Categorical(logits=logits); a = dist.sample()\n",
" obs.append(s); acts.append(int(a)); vals.append(float(v)); lps.append(float(dist.log_prob(a)))\n",
" s, r, done = ppo_env.step(int(a)); rews.append(r); dones.append(float(done))\n",
" if done: s = ppo_env.reset()\n",
" with torch.no_grad():\n",
" _, v_last = ac(torch.tensor(s, dtype=torch.float32))\n",
" return obs, acts, rews, dones, vals + [float(v_last)], lps\n",
"\n",
"def episode_length():\n",
" e = CartPole(np.random.default_rng(SEED + 99)); s = e.reset(); done = False; L = 0\n",
" while not done:\n",
" with torch.no_grad():\n",
" logits, _ = ac(torch.tensor(s, dtype=torch.float32))\n",
" s, _, done = e.step(int(logits.argmax())); L += 1\n",
" return L\n",
"\n",
"ppo_curve = []\n",
"for it in range(PPO_ITERS):\n",
" obs, acts, rews, dones, vals, lps = collect(800)\n",
" adv = compute_gae(np.array(rews), np.array(vals), np.array(dones))\n",
" ret = adv + np.array(vals[:-1])\n",
" obs_t = torch.tensor(np.array(obs), dtype=torch.float32); acts_t = torch.tensor(acts)\n",
" oldlp = torch.tensor(lps); adv_t = torch.tensor(adv, dtype=torch.float32)\n",
" ret_t = torch.tensor(ret, dtype=torch.float32)\n",
" adv_t = (adv_t - adv_t.mean()) / (adv_t.std() + 1e-8)\n",
" for _ in range(4): # PPO epochs\n",
" idx = torch.randperm(len(obs_t))\n",
" for start in range(0, len(obs_t), 64):\n",
" b = idx[start:start + 64]\n",
" logits, v = ac(obs_t[b]); dist = Categorical(logits=logits)\n",
" ratio = (dist.log_prob(acts_t[b]) - oldlp[b]).exp()\n",
" surr1 = ratio * adv_t[b]\n",
" surr2 = torch.clamp(ratio, 0.8, 1.2) * adv_t[b]\n",
" policy_loss = -torch.min(surr1, surr2).mean()\n",
" value_loss = F.mse_loss(v, ret_t[b])\n",
" entropy = dist.entropy().mean()\n",
" loss = policy_loss + 0.5 * value_loss - 0.01 * entropy\n",
" opt.zero_grad(); loss.backward()\n",
" torch.nn.utils.clip_grad_norm_(ac.parameters(), 0.5); opt.step()\n",
" ppo_curve.append(episode_length())\n",
"\n",
"print(f\"PPO greedy episode length: start {ppo_curve[0]}, end {ppo_curve[-1]} (out of 500)\")\n",
"assert max(ppo_curve) > 120, \"PPO should comfortably clear 120 steps; it usually nears the cap\""
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "ddc7e1d4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.081620Z",
"iopub.status.busy": "2026-06-11T20:07:45.081499Z",
"iopub.status.idle": "2026-06-11T20:07:45.139833Z",
"shell.execute_reply": "2026-06-11T20:07:45.139537Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAEiCAYAAAAPh11JAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAXX1JREFUeJzt3Xl8TNf7wPHPZCerpFkEiaglaIhdLFWEUFVrW+VLqrTfaqzBT32rtrYotbW1txVtaVRbpZbatxK72LeS1pbFlkSQdc7vj5FhmiATE5PE83695uXOuWfufWaGO49zzj1Ho5RSCCGEEEKIx7IwdwBCCCGEEEWFJE5CCCGEEHkkiZMQQgghRB5J4iSEEEIIkUeSOAkhhBBC5JEkTkIIIYQQeSSJkxBCCCFEHkniJIQQQgiRR5I4CSGEEELkkSROQghRTEVERKDRaPj777/NHYoQxYYkTkI847J/XLMfdnZ2VK5cmf79+xMfH6+vt3XrVoN61tbWVKhQgV69enH+/Pkcx71+/TrDhw+nSpUq2NnZ4erqSkhICKtWrXqaby/PUlNTmT59Og0aNMDZ2dngczhz5oxJz7VmzRrGjh2b674HP2MLCwu8vb1p3bo1W7duNWkMQoj8sTJ3AEKIwmH8+PH4+fmRmprKn3/+yZw5c1izZg3Hjh2jZMmS+noDBw6kXr16ZGRkcPDgQebPn8/q1as5evQo3t7eAJw+fZqWLVty9epVevfuTd26dUlMTGTx4sW0b9+eYcOGMWXKFHO91RyuXbtGmzZtOHDgAK+88grdu3fHwcGB06dPExkZyfz580lPTzfZ+dasWcOsWbMemjy1atWKXr16oZQiJiaG2bNn06JFC1avXk3btm1NFocQIh+UEOKZtnDhQgWoffv2GZSHh4crQC1ZskQppdSWLVsUoJYtW2ZQ74svvlCAmjBhglJKqfT0dPXCCy+okiVLqt27dxvUzczMVG+88YYCVGRkZAG+K+O0a9dOWVhYqJ9//jnHvtTUVDV06FCTnCclJUUppVRYWJh62OUXUGFhYQZlR44cUYBq3bq1UefL/m5jYmLyFa8QIifpqhNC5KpFixYAxMTEGFXvl19+4dixY3zwwQc0aNDAoK6lpSXz5s3DxcXloa0tD8rMzOTjjz/m+eefx9bWlvLly/O///2PtLQ0g3rly5fnlVde4c8//6R+/frY2dlRoUIFvvvuu8eeY8+ePaxevZo+ffrQpUuXHPttbW35/PPP9c+PHDnCW2+9RYUKFbCzs8PLy4u3336b69evG7xu7NixaDQaTpw4Qffu3SlVqhRNmjThrbfeYtasWYBht9yjBAQE8Nxzzxl8F5s3b6Zp06bY29vj4uJChw4dOHny5GPfL8DatWv1r3V0dKRdu3YcP348T68V4lknXXVCiFydO3cOADc3N6Pq/f777wD06tUr1/rOzs506NCBRYsW8ddff1GxYsWHHrtv374sWrSIrl27MnToUPbs2cPEiRM5efIky5cvN6j7119/0bVrV/r06UNoaCjffvstb731FnXq1KF69eoPPcfKlSsB6Nmz5yPfZ7YNGzZw/vx5evfujZeXF8ePH2f+/PkcP36c3bt350iCXnvtNSpVqsSECRNQSlGrVi2uXLnChg0b+P777/N0zps3b3Lz5k39Z7Vx40batm1LhQoVGDt2LHfv3uXLL7+kcePGHDx4kPLlyz/0WN9//z2hoaGEhITw2WefcefOHebMmUOTJk04dOjQI18rhEC66oR41mV352zcuFFdvXpVXbx4UUVGRio3NzdVokQJdenSJaXU/a66b7/9Vl29elVduXJFrV69WpUvX15pNBp9V19gYKBydnZ+5DmnTZumALVy5cqH1omOjlaA6tu3r0H5sGHDFKA2b96sL/P19VWA2r59u74sISFB2draPrabrVOnTgpQN2/efGS9bHfu3MlR9uOPP+Y4/5gxYxSg3nzzzRz1H9dV16dPH3X16lWVkJCg9uzZo1q2bKkANXXqVKWU7jP28PBQ169f17/u8OHDysLCQvXq1Utf9u+uulu3bikXFxf1zjvvGJwzLi5OOTs75ygXQuQkLU5CCACCg4MNnvv6+rJ48WLKlCljUP72228bPHd3d2fRokXUrVsXgFu3buHo6PjIc2XvT05OfmidNWvWABAeHm5QPnToUD7//HNWr15N8+bN9eXVqlWjadOmBnFVqVIl1zv+HpQdw+NizlaiRAn9dmpqKikpKTRs2BCAgwcPGsQA8N577+XpuA/65ptv+Oabb/TP7ezsCA8PZ/DgwcTGxhIdHc3//d//4erqqq9To0YNWrVqpf/ccrNhwwYSExN58803uXbtmr7c0tKSBg0asGXLFqNjFeJZI4mTEAKAWbNmUblyZaysrPD09KRKlSpYWOQcBjl69GiaNm2KpaUlzz33HFWrVsXK6v6lxNHR0eBHOTe3bt3S132Yf/75BwsLixxdeV5eXri4uPDPP/8YlPv4+OQ4RqlSpbh58+YjY3FyctLH5OLi8si6ADdu3GDcuHFERkaSkJBgsC8pKSlHfT8/v8ce8986dOhA//790Wg0ODo6Ur16dezt7QH077tKlSo5Xle1alXWrVvH7du39fUfdPbsWeD+uLR/y/4shBAPJ4mTEAKA+vXr61uNHiUgICBH69SDqlatSnR0NBcuXMg1mQHdAGvQtRI9zuMGTmeztLTMtVwp9cjX+fv7A3D06NEcrUW5ef3119m1axfDhw8nMDAQBwcHtFotbdq0QavV5qj/YAtVXpUtW/aRn3F+Zcf3/fff4+XllWP/gwmwECJ38q9ECGFSr7zyCj/++CPfffcdo0aNyrE/OTmZFStW4O/v/8iB4b6+vmi1Ws6ePUvVqlX15fHx8SQmJuLr62uSeNu3b8/EiRP54YcfHps43bx5k02bNjFu3DhGjx6tL89uycmrvCaDucl+36dPn86x79SpUzz33HO5tjYBPP/88wB4eHgUSGImxLNApiMQQphU165dqVatGpMmTWL//v0G+7RaLf369ePmzZuMGTPmkcd5+eWXAZgxY4ZB+bRp0wBo166dSeINCgqiTZs2fP311/z222859qenpzNs2DDgfqvWv1ux/h3j42QnNomJiUbHW7p0aQIDA1m0aJHB648dO8b69ev1n1tuQkJCcHJyYsKECWRkZOTYf/XqVaPjEeJZIy1OQgiTsrGx4eeff6Zly5Y0adLEYObwJUuWcPDgQYYOHUq3bt0eeZyaNWsSGhrK/PnzSUxMpFmzZuzdu5dFixbRsWNHg4HhT+q7776jdevWdO7cmfbt29OyZUvs7e05e/YskZGRxMbG8vnnn+Pk5MSLL77I5MmTycjIoEyZMqxfv/6xc139W506dQDdLOwhISFYWlo+9vN40JQpU2jbti1BQUH06dNHPx2Bs7PzI+fHcnJyYs6cOfTs2ZPatWvTrVs33N3duXDhAqtXr6Zx48Z89dVXRr0XIZ455r6tTwhhXg+bOfzfHjZz+MMkJCSo8PBwVbFiRWVra6tcXFxUcHDwI6cg+LeMjAw1btw45efnp6ytrVW5cuXUyJEjVWpqqkE9X19f1a5duxyvb9asmWrWrFmeznXnzh31+eefq3r16ikHBwdlY2OjKlWqpAYMGKD++usvfb1Lly6pTp06KRcXF+Xs7Kxee+01deXKFQWoMWPG6OtlT0dw9erVHOfKzMxUAwYMUO7u7kqj0RhMTUAuM4fnZuPGjapx48aqRIkSysnJSbVv316dOHHCoM7DZg7fsmWLCgkJUc7OzsrOzk49//zz6q233lL79+/P02clxLNMo9RjRk4KIYQQQghAxjgJIYQQQuSZJE5CCCGEEHkkiZMQQgghRB5J4iSEEEIIkUeSOAkhhBBC5JEkTkIIIYQQeSQTYKKbzfjKlSs4Ojo+0VIIQgghhCh6lFLcunULb2/vXBc3f5AkTsCVK1coV66cucMQQgghhBldvHiRsmXLPrKOJE6Ao6MjoPvAnJyczByNEEIIIZ6m5ORkypUrp88HHkUSJ+6vVO7k5CSJkxBCCPGMystwHRkcLoQQQgiRR5I4CSGEEELkkVkTp7Fjx6LRaAwe/v7++v2pqamEhYXh5uaGg4MDXbp0IT4+3uAYFy5coF27dpQsWRIPDw+GDx9OZmbm034rQgghhHgGmH2MU/Xq1dm4caP+uZXV/ZCGDBnC6tWrWbZsGc7OzvTv35/OnTuzc+dOALKysmjXrh1eXl7s2rWL2NhYevXqhbW1NRMmTHjq70UIIYQQxZvZEycrKyu8vLxylCclJfHNN9+wZMkSWrRoAcDChQupWrUqu3fvpmHDhqxfv54TJ06wceNGPD09CQwM5OOPP2bEiBGMHTsWGxubp/12hBCiyIm9CjGXwK8slHY3dzQPV1TihKITa1GJEwpPrGZPnM6ePYu3tzd2dnYEBQUxceJEfHx8OHDgABkZGQQHB+vr+vv74+PjQ1RUFA0bNiQqKoqAgAA8PT31dUJCQujXrx/Hjx+nVq1aRsWSnp6OUko/qj4rK4usrCwsLCwMWsLS09MBsLa2NmndjIwMlFJYWVnpJ+DSarVkZmai0WiwtrYuVHUzMzPRarVYWlpiaWlpdF2lFBkZGQAGSa4p6+b2uRtT15zffXH+e/I0vvsn/XvytL57c18jFq+E0V9kAAqtsqJ9cwvqVAeltCiViQYNGov7x1XaDBQKjcYKjcbi3mf5sLqZKLRoNJZoNJb5qKtQSvcdHTppw8rNoBRYWmTSvrmWWtVyr2thYZPn42o09z93pbJQKsu4uligsbj/HWm16USfhBWbrVFKg0YDr7bIolbV3Ovm9bi5f+7G1M35uR84Dqu33vvutVa0b2HK7/5J/54Yfp8HjsPKzWChycTSUsv4gZa8+YpprxF5ZdbEqUGDBkRERFClShViY2MZN24cTZs25dixY8TFxWFjY4OLi4vBazw9PYmLiwMgLi7OIGnK3p+972HS0tJIS0vTP09OTgZg1qxZDB8+nJIlSwKwZ88eduzYQY0aNWjbtq2+/ldffUVGRgbvvfcezs7OABw8eJDNmzdTrVo12rdvr687d+5c7t69y9tvv427uy5FPnr0KOvWraNSpUp07txZX/frr78mOTmZXr16Ubp0aQBOnjzJqlWr8PX1pVu3bvq6ixYt4vr167z55pv4+PgA8Ndff7F8+XLKlCnDf/7zH33dJUuWEBcXR9euXXn++ecB+Oeff/jpp5/w8PCgd+/e+rrLli3j4sWLdOjQQT/e7MqVKyxevJhSpUrx7rvv6usuX76c8+fP8/LLLxMQEADA1atXiYiIwMHBgbCwMH3dVatWcfr0aVq1akXt2rUBuHnzJgsWLMDW1pbBgwfr665bt45jx47x0ksv0aBBAwBSUlKYPXs2FhYWDB8+XF938+bNHDp0iMaNG9OkSRP99ztz5kwAhg0bpv8Hs337dvbu3Uv9+vVp3rw5oPshmT59OgCDBg3Czs4OgKioKHbu3EmtWrVo3bq1/nwzZ85Eq9Xy/vvv6+f72L9/P1u3buWFF16gXbt2+rqzZ88mLS2Nd955B1dXVwAOHz7Mhg0bqFKlCh07dtTXnT9/PikpKbz11lv6v8MnTpxgzZo1VKhQgddee01fd+HChdy8eZMePXroJ2o7c+YMK1asoFy5cnTv3l1f94cffiAhIYHXX38dPz8/AGJiYvj555/x8vIiNDRUX3fp0qVcvnyZTp06UblyZQAuXbrEjz/+iJubG3379tXX/eWXX/jnn3945ZVXqF69OgDx8fF89913ODk50a9fP33dlStXcvbsWUJCQggMDATg2rVrfPvtt5QoUYKBAwfq665du5YTJ07QokUL6tWrB+j+fc6dOxdra2vCw8P1dTds2MCRI0do2rQpjRo1AuDu3bt8+eWXAIwYMUJfd+vWrRw4cICgoCBefPFFQJdYZH/3Q4YM0V9Ed+7cSVRUFHXq1DH4j1t23QEDBhSba0TsVfhwBnRpuAQP5zhWH+jKik3Ps2ITlHX7h1fr/cS1ZA9+2nX/GtGh/jLKuF5k3aEOnIvXXSO8XK7QueFiEm+XYsmO+9eIdrWX4+txnk1HX+b0Zd01ws3xKm80jiAl1YHvtt6/RrQOXEVFr9NsP9GKYxd01wjnkjfp8eIC0jJsWbFpsL5us+rr8LY+xvLfXyL6b901wt42hdDms8nSWjBv/f1rRNOqmwnwPcS+vxqz7y/dNcLGKo2+wbprxNx1w9Aq3TUiqMp2avnt5VBMfaJO664RFhot74XovvuvNw4iPVN3jahXMYp6FXdy9J9a7Dh5/xrx39Yz8bbWUtLmfW6nOaIU/HNuP2Wst3Lq8gtsPnr/GtGn5WxsrdNYvP0dku7orhEv+BzmxWob+CuuCuujO+rr9nppPg52KSzd+RbXb+muEVXKnKBlwBr+SajA6oP3rxHdmy7Exf4mv+7uQVyi7hrxvOcZQmqt4PKNcqzYe/8a8XqjH3jOKYGV+15nxSY/VmwCX/cY2tX5mYQkL36Oun+N6NRgKaVLXWbtwU7EJOiuEd6ul+hY/0du3HIjcuf9a0T7ur9Q7rl/2HD4Fc7G6q4RHk7xdG30Hcl3nfhh2/1rRJtaK6ngeZYtx0I4eSkQgFIO13izybfcTS/Bws33rxHNA9ZS2fsEkb+04KUG9SjtbpprxI4dO8grsyZOD15oatSoQYMGDfD19eWnn36iRIkSBXbeiRMnMm7cuAI7vhBCFAUxl3QtOP/WsCaU0f2O4+QA7Zrd3+d27z/ptatB9r08tvduM7IvYVjX/V7dmlWgYkXdtvW9aXLsbA3rPnevbvWK4KvL8bG6V/fe/31y8PGGMr736twrs9AYHtf1XsNGJV/wKKPbfnCmnrYv3t92ufeLWKEsuOYcQULrxpD9cTnfq1u+DDh53K9j8YhpgMp6QjvX+8+t7x3jpfqQee/ADvfeSGl3w/dhZ6v7s2kdyLhX1/5eXXc3w7r29+o2qgVpWt12yXvfkZuzru61m7DnSM4YG9aEsvdidHY0PG6pe99RnepQrapuO/u7d7DP/fusVRXu/T8Mm3ufTUnb3P+e1KgMFZ6/99ncq2tjDQ1q5IxVq+Dvy+bpstMolds/G/OpV68ewcHBtGrVipYtW3Lz5k2DVidfX18GDx7MkCFDGD16NCtXriQ6Olq/PyYmhgoVKnDw4MGHdtXl1uJUrlw5rl69ipubW7Fqhi/IutJVJ1110lVn/r8nT/Id7T8GXQaApUUGGo0iS2uFhcaCnZHg6VZ4rhFx16BZTxu095IAC00m1lZaNn9nSVmvwnWNuHAlnZZvQXqGNdkpmrVlFpu/y8Lbo/BcI2KvQqNuoOHed5+lq2uq796U14jrSTY06gZara6bVqPRosGSHT9aUtrdNNeIGzdu4ObmRlJS0uMnwlaFyK1bt1SpUqXUzJkzVWJiorK2tlY///yzfv+pU6cUoKKiopRSSq1Zs0ZZWFio+Ph4fZ158+YpJycnlZqamufzJiUlKUAlJSWZ7s0IIUQhN2+pUj4v3X/4tVDqx1Xmjip3P67SxVfY41Sq6MRaVOJUquBjNSYPMGuL07Bhw2jfvj2+vr5cuXKFMWPGEB0dzYkTJ3B3d6dfv36sWbOGiIgInJycGDBgAAC7du0CdNl2YGAg3t7eTJ48mbi4OHr27Enfvn2Nmo4gOTkZZ2fnvGWaQghRTHQMg0MnYNjbUDdA1+1UmO+sir2q654p7HFC0Ym1qMQJBRurMXmAWcc4Xbp0iTfffJPr16/j7u5OkyZN2L17t36A5PTp07GwsKBLly6kpaUREhLC7Nmz9a+3tLRk1apV9OvXj6CgIOzt7QkNDWX8+PHmektCCFEkXEnQJU0aDbz+Mni6mTuixyvtXvh/3LMVlViLSpxQeGItdGOczEFanIQQz5pvf4Zxs6BeAPz8hbmjEcK8jMkDZK06IYR4Bq29d/f1yy8+up4QwpAkTkII8YyJvw77juq220jiJIRRJHESQohnzLoduvmbalUFb4/H1xdC3CeJkxBCPGPWbtf92bbZo+sJIXKSxEkIIZ4h1xNh92HddlvpphPCaJI4CSHEM2T9n7oZmF+oBD6lzR2NEEWPJE5CCPEMWbNN9+fL0k0nRL5I4iSEEM+IxGTYdUi3Ld10QuSPJE5CCPGM2LALMrOgagWoUM7c0QhRNEniJIQQz4jsbjppbRIi//K1Vt2mTZvYtGkTCQkJaLVag33ffvutSQITQghhOskpsGO/blumIRAi/4xOnMaNG8f48eOpW7cupUuXRqPRFERcQgghTGhTFGRkQkVfqFze3NEIUXQZnTjNnTuXiIgIevbsWRDxCCGEKAD6SS+lm06IJ2L0GKf09HQaNWpUELEIIYQoALfvwta9um1Z1FeIJ2N04tS3b1+WLFlSELEIIYQoAFt2Q1o6lC8DVZ83dzRCFG156qoLDw/Xb2u1WubPn8/GjRupUaMG1tbWBnWnTZtm2giFEEI8kQcnvZRhqUI8mTwlTocOHTJ4HhgYCMCxY8dMHpAQQgjTuZsKm/fotmV8kxBPLk+J05YtWwo6DiGEEAVg615d8lTWEwIqmzsaIYo+o8c4vf3229y6dStH+e3bt3n77bdNEpQQQgjT0N9NJ910QpiE0YnTokWLuHv3bo7yu3fv8t1335kkKCGEEE8uNV03fxNIN50QppLneZySk5NRSqGU4tatW9jZ2en3ZWVlsWbNGjw8PAokSCGEEMb7cz+k3AGv56BWVXNHI0TxkOfEycXFBY1Gg0ajoXLlnB3lGo2GcePGmTQ4IYQQ+ffgpJcWsjKpECaR58Rpy5YtKKVo0aIFv/zyC66urvp9NjY2+Pr64u3tXSBBCiGEME56Bqzfqdt+WdamE8Jk8pw4NWum+5cXExODj4+PrFEnhBCF2K5DuoV93UtBnermjkaI4sPoteqSkpI4evRojnKNRoOdnR0+Pj7Y2tqaJDghhBD5kz3pZUhTsLQ0byxCFCdGJ06BgYGPbG2ytrbmjTfeYN68eQYDyIUQQjwdmVmw/k/dtnTTCWFaRg8XXL58OZUqVWL+/PlER0cTHR3N/PnzqVKlCkuWLOGbb75h8+bNjBo1qiDiFUII8Ri7o+FmMpRyggY1zR2NEMWL0YnTp59+ysyZM+nTpw8BAQEEBATQp08fpk+fztSpU+nRowdffvkly5cvN+q4kyZNQqPRMHjwYH1ZamoqYWFhuLm54eDgQJcuXYiPjzd43YULF2jXrh0lS5bEw8OD4cOHk5mZaezbEkKIYiP7brqQpmAl3XRCmJTRidPRo0fx9fXNUe7r66sf+xQYGEhsbGyej7lv3z7mzZtHjRo1DMqHDBnC77//zrJly9i2bRtXrlyhc+fO+v1ZWVm0a9eO9PR0du3axaJFi4iIiGD06NHGvi0hhCgWsrJg3Q7d9ssy6aUQJmd04uTv78+kSZNIT0/Xl2VkZDBp0iT8/f0BuHz5Mp6ennk6XkpKCj169GDBggWUKlVKX56UlMQ333zDtGnTaNGiBXXq1GHhwoXs2rWL3bt3A7B+/XpOnDjBDz/8QGBgIG3btuXjjz9m1qxZBvEJIcSzYv8xuHoTnB2hUW1zRyNE8WN04jRr1ixWrVpF2bJlCQ4OJjg4mLJly7Jq1SrmzJkDwPnz53n//ffzdLywsDDatWtHcHCwQfmBAwfIyMgwKPf398fHx4eoKN0aAlFRUQQEBBgkaSEhISQnJ3P8+HFj35oQQhR52XfTtWoE1kbf/iOEeByj/1k1atSImJgYFi9ezJkzZwB47bXX6N69O46OjgD07NkzT8eKjIzk4MGD7Nu3L8e+uLg4bGxscHFxMSj39PQkLi5OX+ffLVvZz7Pr5CYtLY20tDT98+Tk5DzFK4QQhZlWC2uzu+nkbjohCkS+/j/i6OjIe++990QnvnjxIoMGDWLDhg1PfdqCiRMnyvIwQohi59BJiL8GDiWhSR1zRyNE8ZSvxOns2bNs2bKFhIQEtFqtwb68Dsw+cOAACQkJ1K59vxM+KyuL7du389VXX7Fu3TrS09NJTEw0aHWKj4/Hy8sLAC8vL/bu3Wtw3Oy77rLr5GbkyJGEh4frnycnJ1OuXLk8xS2EEIVVdjddyyCwtTFvLEIUV0YnTgsWLKBfv34899xzeHl5GUyGqdFo8pw4tWzZMscM5L1798bf358RI0ZQrlw5rK2t2bRpE126dAHg9OnTXLhwgaCgIACCgoL49NNPSUhIwMPDA4ANGzbg5OREtWrVHnpuW1tbmd1cCFGsKAV/3JuGQLrphCg4RidOn3zyCZ9++ikjRox4ohM7OjrywgsvGJTZ29vj5uamL+/Tpw/h4eG4urri5OTEgAEDCAoKomHDhgC0bt2aatWq0bNnTyZPnkxcXByjRo0iLCxMEiMhxDPl6Bm4FA8l7eCl+uaORojiy+jE6ebNm7z22msFEUsO06dPx8LCgi5dupCWlkZISAizZ8/W77e0tGTVqlX069ePoKAg7O3tCQ0NZfz48U8lPiGEKCyyu+laNAQ7+X+jEAVGo5RSxrygT58+1KtX74kHhxcmycnJODs7k5SUhJOTk7nDEUIIoygFzf4D/1yBWaPhlebmjkiIosWYPMDoFqeKFSvy0UcfsXv3bgICArC2tjbYP3DgQGMPKYQQ4gmcOKdLmmxtoHlDc0cjRPFmdOI0f/58HBwc2LZtG9u2bTPYp9FoJHESQoinLHttupcagH0J88YiRHFndOIUExNTEHEIIYTIB6VgzVbddtumZg1FiGeC0UuuZEtPT+f06dNkZmaaMh4hhBBGOPs3nLsINta6+ZuEEAXL6MTpzp079OnTh5IlS1K9enUuXLgAwIABA5g0aZLJAxRCCPFw2d10TeuCk4N5YxHiWWB04jRy5EgOHz7M1q1bDZZKCQ4OZunSpSYNTgghxKOtyZ708kXzxiHEs8LoMU6//fYbS5cupWHDhgazhlevXp1z586ZNDghhBAPd+4CnDoPVpbQqrG5oxHi2WB0i9PVq1f1y5s86Pbt2waJlBBCiIKV3U3XuDY4O5o3FiGeFUYnTnXr1mX16tX659nJ0tdff61fQ04IIUTBy06c2sradEI8NUZ31U2YMIG2bdty4sQJMjMzmTlzJidOnGDXrl055nUSQghRMC5cgWNnwcICWks3nRBPjdEtTk2aNCE6OprMzEwCAgJYv349Hh4eREVFUadOnYKIUQghxL9kDwpvWBPcXMwaihDPFKNbnACef/55FixYYOpYhBBC5NEf2XfTSTedEE9VnhKn5OTkPB9QFskVQoiCdTkeDp0EjQZCZLZwIZ6qPCVOLi4uj71jTimFRqMhKyvLJIEJIYTI3R87dH/WCwAPV/PGIsSzJk+J05YtWwo6DiGEEHm05t59ODLppRBPX54Sp2bNpBNdCCEKg/hrcOC4bruNJE5CPHX5XuRXCCHE0/fHDlAKalWD0u7mjkaIZ48kTkIIUYSskbvphDArSZyEEKKIuHYT9h7RbbeVbjohzEISJyGEKCLW/wlaLdSoAuW8zB2NEM+mfCVOmZmZbNy4kXnz5nHr1i0Arly5QkpKikmDE0IIcV92N520NglhPkbPHP7PP//Qpk0bLly4QFpaGq1atcLR0ZHPPvuMtLQ05s6dWxBxCiHEM+1mEuw6qNuWxEkI8zG6xWnQoEHUrVuXmzdvUqJECX15p06d2LRpk0mDE0IIobNhF2Rpodrz4FfW3NEI8ewyusVpx44d7Nq1CxsbG4Py8uXLc/nyZZMFJoQQ4r7sSS9l7iYhzMvoFietVpvrsiqXLl3C0dHRJEEJIYS4LykF/jyg25ZpCIQwL6MTp9atWzNjxgz9c41GQ0pKCmPGjOHll182ZWxCCCGATVGQkQmVfHUPIYT5GN1VN3XqVEJCQqhWrRqpqal0796ds2fP8txzz/Hjjz8WRIxCCPFMW5u9Np20NglhdkYnTmXLluXw4cNERkZy5MgRUlJS6NOnDz169DAYLC6EEOLJpdyBbXt123I3nRDml695nKysrPjPf/7D5MmTmT17Nn379s1X0jRnzhxq1KiBk5MTTk5OBAUFsXbtWv3+1NRUwsLCcHNzw8HBgS5duhAfH29wjAsXLtCuXTtKliyJh4cHw4cPJzMzMz9vSwghCp3NuyEtQ3cnnX8Fc0cjhMhTi9PKlSvzfMBXX301z3XLli3LpEmTqFSpEkopFi1aRIcOHTh06BDVq1dnyJAhrF69mmXLluHs7Ez//v3p3LkzO3fuBCArK4t27drh5eXFrl27iI2NpVevXlhbWzNhwoQ8xyGEEIXV2uy16V4Ejca8sQghQKOUUo+rZGFh2DCl0Wj498s09/5F53bHnTFcXV2ZMmUKXbt2xd3dnSVLltC1a1cATp06RdWqVYmKiqJhw4asXbuWV155hStXruDp6QnA3LlzGTFiBFevXs0xZcLDJCcn4+zsTFJSEk5OTk8UvxBCmMqdu1C7M9xNhVVzIaCKuSMSongyJg/IU1edVqvVP9avX09gYCBr164lMTGRxMRE1q5dS+3atfnjjz/yHXRWVhaRkZHcvn2boKAgDhw4QEZGBsHBwfo6/v7++Pj4EBUVBUBUVBQBAQH6pAkgJCSE5ORkjh8//tBzpaWlkZycbPAQQojCZuteXdJU1gteqGzuaIQQkI/B4YMHD2bu3Lk0adJEXxYSEkLJkiV59913OXnypFHHO3r0KEFBQaSmpuLg4MDy5cupVq0a0dHR2NjY4OLiYlDf09OTuLg4AOLi4gySpuz92fseZuLEiYwbN86oOIUQ4mnTd9M1k246IQoLoweHnzt3LkcyA+Ds7Mzff/9tdABVqlQhOjqaPXv20K9fP0JDQzlx4oTRxzHGyJEjSUpK0j8uXrxYoOcTQghjpabr5m8C3fgmIUThYHTiVK9ePcLDww3ubouPj2f48OHUr1/f6ABsbGyoWLEiderUYeLEidSsWZOZM2fi5eVFeno6iYmJBvXj4+Px8vICwMvLK8dddtnPs+vkxtbWVn8nX/ZDCCEKkx374PZdKO0ONf3NHY0QIpvRidO3335LbGwsPj4+VKxYkYoVK+Lj48Ply5f55ptvnjggrVZLWloaderUwdra2mDh4NOnT3PhwgWCgoIACAoK4ujRoyQkJOjrbNiwAScnJ6pVq/bEsQghhLlkr03X9kWwyNfEMUKIgmD0GKeKFSty5MgRNmzYwKlTpwCoWrUqwcHB+jvr8mrkyJG0bdsWHx8fbt26xZIlS9i6dSvr1q3D2dmZPn36EB4ejqurK05OTgwYMICgoCAaNmwI6JZ/qVatGj179mTy5MnExcUxatQowsLCsLW1NfatCSFEoZCeARt36bZltnAhChejEyfQTT3QunVrWrdu/UQnT0hIoFevXsTGxuLs7EyNGjVYt24drVq1AmD69OlYWFjQpUsX0tLSCAkJYfbs2frXW1pasmrVKvr160dQUBD29vaEhoYyfvz4J4pLCCHMaedBSL4N7q5Qp7q5oxFCPChP8zj927Zt2/j888/1d9BVq1aN4cOH07RpU5MH+DTIPE5CiMJk+GT4aS307ACfDDZ3NEIUfyafx+lBP/zwA8HBwZQsWZKBAwcycOBA7OzsaNmyJUuWLMl30EIIISAjE9brFkeQbjohCiGjW5yqVq3Ku+++y5AhQwzKp02bxoIFC4yex6kwkBYnIURh8ecB6DEM3Fxg789gZWnuiIQo/gq0xen8+fO0b98+R/mrr75KTEyMsYcTQgjxgOy76Vo3lqRJiMLI6MSpXLlyBlMEZNu4cSPlypUzSVBCCPEsysqCdX/qtqWbTojCyei76oYOHcrAgQOJjo6mUaNGAOzcuZOIiAhmzpxp8gCFEOJZse8oXLsJzo4QVMvc0QghcmN04tSvXz+8vLyYOnUqP/30E6Ab97R06VI6dOhg8gCFEOJZsebe2nStG4N1viaLEUIUtHz90+zUqROdOnUydSxCCPHM0mrvL+rbVtamE6LQMnqM08WLF7l06ZL++d69exk8eDDz5883aWBCCPEsOXAcEq6Doz00qWPuaIQQD2N04tS9e3e2bNkCQFxcHMHBwezdu5cPP/xQZuwWQoh8ym5tCm4EtjbmjUUI8XBGJ07Hjh2jfv36APz0008EBASwa9cuFi9eTEREhKnjE0KIYk8p6aYToqgwOnHKyMjQL6C7ceNGXn31VQD8/f2JjY01bXRCCPEMOHwKriRASTtoVs/c0QghHsXoxKl69erMnTuXHTt2sGHDBtq0aQPAlStXcHNzM3mAQghR3GW3NrUMAjtb88YihHg0oxOnzz77jHnz5vHSSy/x5ptvUrNmTQBWrlyp78ITQgiRN0rdn4ZAuumEKPyMno7gpZde4tq1ayQnJ1OqVCl9+bvvvkvJkiVNGpwQQhR3x/+CC1d0LU0vNTB3NEKIx8nXPE6WlpYGSRNA+fLlTRGPEEI8U7LXpnupPtiXMG8sQojHy1PiVLt2bTZt2kSpUqWoVasWGo3moXUPHjxosuCEEKI4e/BuOlmbToiiIU+JU4cOHfR30nXs2LEg4xFCiGfGmb/h/EWwsYYWDc0djRAiL/KUOI0ZMybXbSGEEPmX3U33Yj3djOFCiMIv38tI7t+/n5MnTwJQrVo16tSRNQKEEMIY+m46uZtOiCLD6MTp0qVLvPnmm+zcuRMXFxcAEhMTadSoEZGRkZQtW9bUMQohRLHz1wU4HQNWlrplVoQQRYPR8zj17duXjIwMTp48yY0bN7hx4wYnT55Eq9XSt2/fgohRCCGKnezWpsZ1wNnRvLEIIfLO6Banbdu2sWvXLqpUqaIvq1KlCl9++SVNmzY1aXBCCFFcZY9vkm46IYoWo1ucypUrR0ZGRo7yrKwsvL29TRKUEEIUZ/9chhN/gaUFtG5i7miEEMYwOnGaMmUKAwYMYP/+/fqy/fv3M2jQID7//HOTBieEEMVRdjddw0BwdTZrKEIII2mUUsqYF5QqVYo7d+6QmZmJlZWupy97297e8H7aGzdumC7SApScnIyzszNJSUk4OTmZOxwhRDH3aj84fAo+HQL/edXc0QghjMkDjB7jNGPGjPzGJYQQz7xLcbqkSaOBEOmmE6LIMTpxCg0NLYg4hBDimfDHDt2f9QPA3dW8sQghjGf0GCeAc+fOMWrUKN58800SEhIAWLt2LcePHzfqOBMnTqRevXo4Ojri4eFBx44dOX36tEGd1NRUwsLCcHNzw8HBgS5duhAfH29Q58KFC7Rr146SJUvi4eHB8OHDyczMzM9bE0KIApV9N11bWZtOiCLJ6MRp27ZtBAQEsGfPHn799VdSUlIAOHz4sNHLsWzbto2wsDB2797Nhg0byMjIoHXr1ty+fVtfZ8iQIfz+++8sW7aMbdu2ceXKFTp37qzfn5WVRbt27UhPT2fXrl0sWrSIiIgIRo8ebexbE0KIAhV3FQ7c+/9lW5m9RYiiSRmpYcOGaurUqUoppRwcHNS5c+eUUkrt2bNHlSlTxtjDGUhISFCA2rZtm1JKqcTERGVtba2WLVumr3Py5EkFqKioKKWUUmvWrFEWFhYqLi5OX2fOnDnKyclJpaWl5em8SUlJClBJSUlPFL8QQjzKwl+V8nlJqU5h5o5ECPEgY/IAo1ucjh49SqdOnXKUe3h4cO3atSdK4pKSkgBwddV1/B84cICMjAyCg4P1dfz9/fHx8SEqKgqAqKgoAgIC8PT01NcJCQkhOTn5oV2HaWlpJCcnGzyEEKKgrc3uppNJL4UosoxOnFxcXIiNjc1RfujQIcqUKZPvQLRaLYMHD6Zx48a88MILAMTFxWFjY6NfEy+bp6cncXFx+joPJk3Z+7P35WbixIk4OzvrH+XKlct33EIIkRdXb8CeI7ptSZyEKLqMTpy6devGiBEjiIuLQ6PRoNVq2blzJ8OGDaNXr175DiQsLIxjx44RGRmZ72Pk1ciRI0lKStI/Ll68WODnFEI829b/CUpBTX8o62XuaIQQ+WV04jRhwgT8/f0pV64cKSkpVKtWjRdffJFGjRoxatSofAXRv39/Vq1axZYtWyhbtqy+3MvLi/T0dBITEw3qx8fH4+Xlpa/z77vssp9n1/k3W1tbnJycDB5CCFGQ1tybLVxam4Qo2oxOnGxsbFiwYAHnz59n1apV/PDDD5w6dYrvv/8eS0tLo46llKJ///4sX76czZs34+fnZ7C/Tp06WFtbs2nTJn3Z6dOnuXDhAkFBQQAEBQVx9OhR/bQIABs2bMDJyYlq1aoZ+/aEEMLkbiRB1CHdtiROQhRtRk+Ama1cuXJPPDYoLCyMJUuWsGLFChwdHfVjkpydnSlRogTOzs706dOH8PBwXF1dcXJyYsCAAQQFBdGwYUMAWrduTbVq1ejZsyeTJ08mLi6OUaNGERYWhq2t7RPFJ4QQprD+T8jSQrWKUD7/Q0GFEIVAvhMnU5gzZw4AL730kkH5woULeeuttwCYPn06FhYWdOnShbS0NEJCQpg9e7a+rqWlJatWraJfv34EBQVhb29PaGgo48ePf1pvQwghHmntvdnCX5ZJL4Uo8oxe5Lc4kkV+hRAFJekW1OkMGZmwaRFU9DF3REKIfzMmD8jXkitCCCHyZuMuXdJUubwkTUIUB5I4CSFEAVp772466aYTongwOnEqX74848eP58KFCwURjxBCFBu3bsP2fbptuZtOiOLB6MRp8ODB/Prrr1SoUIFWrVoRGRlJWlpaQcQmhBBF2ubdkJYBFcpBFb/H1xdCFH75Spyio6PZu3cvVatWZcCAAZQuXZr+/ftz8ODBgohRCCGKpDUPrE2n0Zg3FiGEaeR7jFPt2rX54osvuHLlCmPGjOHrr7+mXr16BAYG8u233yI36wkhnmV37sLWvbptGd8kRPGR73mcMjIyWL58OQsXLmTDhg00bNiQPn36cOnSJf73v/+xceNGlixZYspYhRCiyPh1A6SmQWkPqF7R3NEIIUzF6MTp4MGDLFy4kB9//BELCwt69erF9OnT8ff319fp1KkT9erVM2mgQghRVCxeCR9O123HJcDSNdCtnXljEkKYhtGJU7169WjVqhVz5syhY8eOWFtb56jj5+dHt27dTBKgEEIUBZlZsDsaIlfD71vulyvgf9OgWX0o7W6u6IQQpmJ04nT+/Hl8fX0fWcfe3p6FCxfmOyghhCgKtFrYd1SXKK3ZBtcTc6+XpYW/L0viJERxYHTi9LikSQghijOlIPqkLllavRXirt3fV8oJXqoPv23S1ctmaSGL+wpRXOQpcSpVqhSaPN5Le+PGjScKSAghChul4PhfsGqLLmG6FHd/n5M9hDSF9s2hUW2wtoKGgbruuSytLmmaEC6tTUIUF3lKnGbMmKHfvn79Op988gkhISEEBQUBEBUVxbp16/joo48KJEghhDCHM3/D75th1VY4f/F+eUk7aNUYXm0BTeuCrY3h67q1041p+vuyrqVJkiYhig+NMnLCpS5dutC8eXP69+9vUP7VV1+xceNGfvvtN1PG91QYsyqyEKJ4+/uyrlXp981wOuZ+ua0NtAyCV5pDiwZQws58MQohTMuYPMDoxMnBwYHo6GgqVjScmOSvv/4iMDCQlJQU4yM2M0mchHi2XYrTtSqt2gJHz9wvt7bStRy1bw7BjcChpNlCFEIUIGPyAKMHh7u5ubFixQqGDh1qUL5ixQrc3NyMPZwQRou9CjGXwK+sdIGI/Iu/Dmu2wsotcPD4/XJLC2hSR9eyFNIEnB3NFqIQohAyOnEaN24cffv2ZevWrTRo0ACAPXv28Mcff7BgwQKTByjEgyJXw8ipoFVgYQETw2ViQZF31xNh7XZdV9yew/fvfNNooEFNeLU5tHkR3FzMGaUQojAzuqsOdInSF198wcmTJwGoWrUqAwcO1CdSRY101RUNsVch6A3D27wtNLBjCZT1Ml9conBLugXr/tSNWdp5UHenW7Y61XXdcC83A8/nzBejEMK8CrSrDqBBgwYsXrw4X8EJkV9/bDdMmkDX8tThfXjnNXjjZSjlbJ7YROGScgc27NSNW9q2FzIy7+8LqAztW8ArL0EZT3NFKIQoqvLV4nTu3DkWLlzI+fPnmTFjBh4eHqxduxYfHx+qV69eEHEWKGlxKvz2HIZeI3SLpj6MrY3u9vDQTrofR/FsuZsKm/foWpY274a09Pv7/CvoWpZeaS4TUQohcirQu+q2bdtG27Ztady4Mdu3b+fkyZNUqFCBSZMmsX//fn7++ecnCt4cJHEq3HYehD4f6n4YK/rq5tPR3ptYcNxAXcIUsRyOn73/mjrVdQlU2xfBJudyiqIIe/DmAFdn2L4PVm6GjbvgTur9ehXK3U+WKpc3W7hCiCKgQBOnoKAgXnvtNcLDw3F0dOTw4cNUqFCBvXv30rlzZy5duvREwZuDJE6F1479uqQpLV13W/j88XAzOefEgkrBwRMQ8atuzbDMLF25eyno3h56tJcxLMVB5GoYOU2XOAPY2UDqAy1LZb10rY6vNIdqz+sGfQshxOMU+DxOR48exc/PzyBx+vvvv/H39yc1NfXxBylkJHEqnLbsgf9+BGkZ0KIhzBmn+6F8nPjrELkKfvgdEq7ryqwsda1PoZ2g7gvyg/pvT2uKB6V0SfCt27pxSCm34dYduH1H92dKLuUpd3T1byTBqfM5j+leCl5tqUuYavrLdyuEMF6BDg53cXEhNjYWPz8/g/JDhw5RpowMHhCmsXEX9BsL6RnQujHMGpP3LjdPNxgUCu/30A0oX/Tb/RXsf9+ia4no1Qk6tpTZn8GwFedhUzxkZt1LbrITmzuGCU7Kbbh9N/eE6N/1HryrzRS+GKVbI04IIZ4Go1uchg0bxp49e1i2bBmVK1fm4MGDxMfH06tXL3r16sWYMWMKKtYCIy1OhcsfO6D/eN2dUG1fhC8/0s3g/CSO/wXf/Qa/bbw/wNzZUXcnXs9Xwcf7icMukmIuQvPQnHcr+vvpWvqyE6S7Jm5I1mh0s3DrH/aGzx3tHygvofszPQOGTzaM1dICdkbKRKhCiCdToF116enphIWFERERQVZWFlZWVmRlZdG9e3ciIiKwtLR8ouDNQRKnwmP1Nhj4sa6Fo31zmP6/J0+aHpSYDEvXwvcr4GKsrkyj0XUFhnbULdhqYWG68xU2ySmw/xjsPQJ7jkD0yfvjhfLC1kaX1NjfS2Ycc0l6sp/n2PdA/RK2+fucI1fD/6bpWq0sLWCCTIAqhDCBAk2csl24cIFjx46RkpJCrVq1qFSpUr6CLQwkcSocVm6GwZ/qfhQ7BsPUD3RjkwpCVhZs3QuLlsO2fffLK5SDXh2gSwg4ORTMuZ+mG0m6JGnvEdh9GE6ee3yipNHA5yPA19swQbIvWTjuUIy9mvPmACGEeBJPJXFKT08nJiaG559/Hiur/DUJbN++nSlTpnDgwAFiY2NZvnw5HTt21O9XSjFmzBgWLFhAYmIijRs3Zs6cOQZJ2o0bNxgwYAC///47FhYWdOnShZkzZ+LgkPdfPUmczO/X9TD0M92P+mtt4LNh8LQaL89f1HXjLftD1y0FUNIOOrfWtUJV9nvUqwuXuKu6lqS9R3RzX539J2ed8mWgfg1oWBPq14SdB6QVRwjxbCvQxOnOnTsMGDCARYsWAXDmzBkqVKjAgAEDKFOmDB988EGej7V27Vp27txJnTp16Ny5c47E6bPPPmPixIksWrQIPz8/PvroI44ePcqJEyews9ON6m3bti2xsbHMmzePjIwMevfuTb169ViyZEme45DEybyWroERn+vGrrzZTvfDbY7uspQ7sHyDrhXqwYSjUS3d3XjBjQquBSw/lNJ1N+65lyTtPQL/XMlZr4qfLlFqUEP3Z27TMkgrjhDiWVagidOgQYPYuXMnM2bMoE2bNhw5coQKFSqwYsUKxo4dy6FDh/IVtEajMUiclFJ4e3szdOhQhg0bBkBSUhKenp5ERETQrVs3Tp48SbVq1di3bx9169YF4I8//uDll1/m0qVLeHvnbcSvJE7ms+R33R1dAP95FT4eZP4xRkrBrkO6u/E27LzfteXtoYuxWzvzLAKrFPx1AfYe1nW77T0CcdcM61hYQPWK9xKlmlAvQDdJpBBCiIcr0OkIfvvtN5YuXUrDhg3RPDBhSvXq1Tl37pzx0T5ETEwMcXFxBAcH68ucnZ1p0KABUVFRdOvWjaioKFxcXPRJE0BwcDAWFhbs2bOHTp06mSweYXqLlsPoL3TbvTvDmP6FYw4ejQYa19Y9LsfD4pXw42q4kgCTv4aZi3RrnYV2ghpVCi6OrCw4FQN7onWtSvuOwvVEwzrWVroYGtTUPepU1w3eFkIIUTCMTpyuXr2Kh4dHjvLbt28bJFJPKi4uDgBPT8NVOD09PfX74uLicsRiZWWFq6urvk5u0tLSSEu7v+hZcnKyqcIWefTtzzBulm77ndfhw/cKR9L0b2U84f/egYGhsGqzbmmXo2fg53W6R62qujmh2jXT3XH2JDIydcfee1iXKO0/Csm3DevY2kDt6vfGJ9XQnV/mohJCiKfH6MSpbt26rF69mgEDBgDok6Wvv/6aoKAg00ZXQCZOnMi4cePMHcYza95SmDBXt/1+d/i/voUzaXqQnQ10baO72y76pK4bb9UWOHRS9/h0Drz5im5pl7yOEUpN0x0rezD3geM550tyKAl1A+6PT6pRpXDc2SaEEM8qoxOnCRMm0LZtW06cOEFmZiYzZ87kxIkT7Nq1i23btpksMC8vLwDi4+MpXbq0vjw+Pp7AwEB9nYSEBIPXZWZmcuPGDf3rczNy5EjCw8P1z5OTkylXrpzJYhcPN2uxrrsLYGBPCO9d+JOmB2k0UKua7vHhe7ouvMUrdWONvvweZi+GkKbwViddohN37f5SJo72cPD4/cHch0/pJnV8kIvT/SSpQQ2oWrFwDUgXQohnndGJU5MmTTh8+DATJ04kICCA9evXU7t2baKioggICDBZYH5+fnh5ebFp0yZ9opScnMyePXvo168foFtwODExkQMHDlCnTh0ANm/ejFarpUGDBg89tq2tLba2tiaLVeTNzEUwLUK3Hf6WblmUoszdVZf89XsT1u+E75brBm2v2aZ7eD2nWzcv+/YLjSbnDN0ebroEqUFN3Z8Vfc0/OF4IIcTDGZU4ZWRk8N///pePPvqIBQsWPPHJU1JS+Ouvv/TPY2JiiI6OxtXVFR8fHwYPHswnn3xCpUqV9NMReHt76++8q1q1Km3atOGdd95h7ty5ZGRk0L9/f7p165bnO+pEwVMKpi7UtciArmsurId5YzIlayvdGKd2zXQTTH73G/y6Iecdb0qBlzs0qX3/rjdf76LV4iaEEM86o6cjcHZ2Jjo6Osciv/mxdetWmjdvnqM8NDSUiIgI/QSY8+fPJzExkSZNmjB79mwqV66sr3vjxg369+9vMAHmF198IRNgFhJKwWcLYM6Puuf/ew/++4Z5Y3oaNuyEvqNylkdOh6DApx6OEEKIRyjQeZxCQ0MJDAxkyJAhTxRkYSKJU8FQCj6ZA18v0z0fEwZvdzVvTE9L7FVo1M1weRNZkFYIIQqnAp3HqVKlSowfP14/47e9veGkMQMHDjT2kKIYUgrGfQULf9U9/3gQ9Opo1pCeqtLuMDE851ImkjQJIUTRZnSL06O66DQaDefPn3/ioJ42aXEyLa0WPpoJP6zUPZ8YDt3bmzcmc5GlTIQQovAr0BanmJiYfAcmij+tVreESuRq3aDnycPh9bbmjsp8SrtLwiSEEMWJ0YmTEA+TlaVbrHfZH7pb6qeOgM6tzR2VEOJxlFJkZmaSlZVl7lCEKBCWlpZYWVmZZIUToxOnByeOfJBGo8HOzo6KFSvSoUMHXF1dnzg4UXRkZsGwz2D5Bt14nun/gw4tzR2VEOJx0tPTiY2N5c6dO+YORYgCVbJkSUqXLo2NzZOtj2X0GKfmzZtz8OBBsrKyqFJFt8LpmTNnsLS0xN/fn9OnT6PRaPjzzz+pVq3aEwX3tMgYpyeTmQVDJsDKzbpZrr8YBe1eMndUQojH0Wq1nD17FktLS9zd3bGxsTHpmqNCFAZKKdLT07l69SpZWVlUqlQJi3/NNFygY5yyW5MWLlyoP3hSUhJ9+/alSZMmvPPOO3Tv3p0hQ4awbt06Yw8vipiMTBjwMazdrpsI8qvR0KapuaMSQuRFeno6Wq2WcuXKUbJkSXOHI0SBKVGiBNbW1vzzzz+kp6djZ5f/1dGNbnEqU6YMGzZsyNGadPz4cVq3bs3ly5c5ePAgrVu35tq1aw85SuEiLU75k5YO/cfrlhuxsYa546Bl0VjnWQgBpKamEhMTg5+f3xP9kAhRFDzq77sxeYDRq2IlJSXlWFgX4OrVqyQnJwPg4uJCenq6sYcWRUhqOrw3Rpc02VrDgk8kaRJCCFH8GZ04dejQgbfffpvly5dz6dIlLl26xPLly+nTp49+Dbm9e/caLIsiipfUNHh3FGzeDbY28M0EeKm+uaMSQoii66233tL/hhaEv//+G41GQ3R0dL6PERERgYuLi8lielLly5dnxowZT/28Ro9xmjdvHkOGDKFbt25kZmbqDmJlRWhoKNOnTwfA39+fr7/+2rSRikLhbqpuDbY/D0AJO/h2AjSqZe6ohBCiaJs5cyZGjpx5ZkRERDB48GASExPNHQqQj8TJwcGBBQsWMH36dP0s4RUqVDBYVDcwMNBkAYrC4/ZdePt/sDsaStpBxCRoUNPcUQkhRNHn7Oxs7hBEHhndVZfNwcGBGjVqUKNGDYOkSRRPKXcgdIQuaXIoCd9PkaRJCGE+L730EgMGDGDw4MGUKlUKT09PFixYwO3bt+nduzeOjo5UrFiRtWvX6l+TlZVFnz598PPzo0SJElSpUoWZM2fq96emplK9enXeffddfdm5c+dwdHTk22+/fWgsiYmJ9O3bF3d3d5ycnGjRogWHDx/W7x87diyBgYHMmzdPfwfj66+/TlJSkr7Ov7vqfv75ZwICAihRogRubm4EBwdz+/ZtQDeNxPjx4ylbtiy2trYEBgbyxx9/GMS0d+9eatWqhZ2dHXXr1uXQoUM54j527Bht27bFwcEBT09PevbsafRNXStWrKB27drY2dlRoUIFxo0bp++NAt0cj19//TWdOnWiZMmSVKpUiZUrVxocY+XKlVSqVAk7OzuaN2/OokWL0Gg0JCYmsnXrVnr37k1SUhIajQaNRsPYsWP1r71z5w5vv/02jo6O+Pj4MH/+fKPizxclVFJSkgJUUlKSuUMplJJuKdUxTCmfl5R6oZ1SB4+bOyIhhCncvXtXnThxQt29ezfHvrS0NJWWlqa0Wq2+LDMzU6WlpamMjAyT1zVWs2bNlKOjo/r444/VmTNn1Mcff6wsLS1V27Zt1fz589WZM2dUv379lJubm7p9+7ZSSqn09HQ1evRotW/fPnX+/Hn1ww8/qJIlS6qlS5fqj3vo0CFlY2OjfvvtN5WZmakaNmyoOnXq9MhYgoODVfv27dW+ffvUmTNn1NChQ5Wbm5u6fv26UkqpMWPGKHt7e9WiRQt16NAhtW3bNlWxYkXVvXt3/TFCQ0NVhw4dlFJKXblyRVlZWalp06apmJgYdeTIETVr1ix169YtpZRS06ZNU05OTurHH39Up06dUv/3f/+nrK2t1ZkzZ5RSSt26dUu5u7ur7t27q2PHjqnff/9dVahQQQHq0KFDSimlbt68qdzd3dXIkSPVyZMn1cGDB1WrVq1U8+bNH/o+Fy5cqJydnfXPt2/frpycnFRERIQ6d+6cWr9+vSpfvrwaO3asvg6gypYtq5YsWaLOnj2rBg4cqBwcHPSfzfnz55W1tbUaNmyYOnXqlPrxxx9VmTJlFKBu3ryp0tLS1IwZM5STk5OKjY1VsbGx+s/B19dXubq6qlmzZqmzZ8+qiRMnKgsLC3Xq1Klc43/U33dj8gBJnJQkTo+SmKzUq+/pkqaA9kodzv3voxCiCHrUD8mkSZPUpEmT9EmHUkrt3LlTTZo0Sa1Zs8ag7tSpU9WkSZNUYmKivmzv3r1q0qRJauXKlQZ1Z86cqSZNmqQSEhL0Zdk/5sZo1qyZatKkif55Zmamsre3Vz179tSXxcbGKkBFRUU99DhhYWGqS5cuBmWTJ09Wzz33nOrfv78qXbq0unbt2kNfv2PHDuXk5KRSU1MNyp9//nk1b948pZQucbK0tFSXLl3S71+7dq2ysLBQsbGxSinDxOnAgQMKUH///Xeu5/T29laffvqpQVm9evXU+++/r5RSat68ecrNzc3ge50zZ45B4vTxxx+r1q1bGxzj4sWLClCnT5/O9bz/TpxatmypJkyYYFDn+++/V6VLl9Y/B9SoUaP0z1NSUhSg1q5dq5RSasSIEeqFF14wOMaHH36oT5xyO282X19f9Z///Ef/XKvVKg8PDzVnzpxc4zdV4iRr1YmHupkE/xkOx85CKSdYPBWqVzR3VEIIoVOjRg39tqWlJW5ubgQEBOjLPD09AQym0Jk1axbffvstFy5c4O7du6Snp+cYlzt06FB+++03vvrqK9auXYubm9tDYzh8+DApKSk56ty9e5dz587pn/v4+FCmTBn986CgILRaLadPn8bLy8vgtTVr1qRly5YEBAQQEhJC69at6dq1K6VKlSI5OZkrV67QuHFjg9c0btxY3z148uRJatSoYTBXUVCQ4Xwxhw8fZsuWLbkOtTl37lye7ow/fPgwO3fu5NNPP9WXZWVlkZqayp07d/STqj74Pdnb2+Pk5KT/Tk6fPk29evUMjlu/ft5v037w2BqNBi8vr1ynTDIlSZxErq4nwn+GwYlz4OYCS6aCfwVzRyWEeFqGDBkCgLW1tb6sQYMG1K1bN8dyFf37989Rt3bt2tSsWTNH3ffeey9H3QeTHWM8eAzQ/XA+WJa9fIxWqwUgMjKSYcOGMXXqVIKCgnB0dGTKlCns2bPH4DgJCQn6pcTOnj1LmzZtHhpDSkoKpUuXZuvWrTn25ffWfUtLSzZs2MCuXbtYv349X375JR9++CF79ux5ZBJnjJSUFNq3b89nn32WY1/p0qXzfIxx48bRuXPnHPseTNpy+56yv5MnVZDHfhhJnEQO125C96FwOgbcS8GSaVC5vLmjEkI8TbkthGppaYmlpWWB1H0adu7cSaNGjXj//ff1ZQ+2CmV7++23CQgIoE+fPrzzzjsEBwdTtWrVXI9Zu3Zt4uLisLKyonz58g8994ULF7hy5Qre3t4A7N69GwsLC/2ar/+m0Who3LgxjRs3ZvTo0fj6+rJ8+XLCw8Px9vZm586dNGvWzOC9ZbfUVK1ale+//57U1FR9ArN79+4ccf/yyy+UL18eK6v8pQK1a9fm9OnTVKyY/66IKlWqsGbNGoOyffv2GTy3sbEhKysr3+cwtXzfVSeKn9irsHobdBmgS5o83CByhiRNQojioVKlSuzfv59169Zx5swZPvrooxw/0rNmzSIqKopFixbRo0cPOnbsSI8ePR66GkZwcDBBQUF07NiR9evX8/fff7Nr1y4+/PBD9u/fr69nZ2dHaGgohw8fZseOHQwcOJDXX389RzcdwJ49e5gwYQL79+/nwoUL/Prrr1y9elWfvA0fPpzPPvuMpUuXcvr0aT744AOio6MZNGgQAN27d0ej0fDOO+9w4sQJ1qxZw+eff25wjrCwMG7cuMGbb77Jvn37OHfuHOvWraN37955TlJGjx7Nd999x7hx4zh+/DgnT54kMjKSUaNG5en1AP/97385deoUI0aM4MyZM/z0009EREQA91sMy5cvT0pKCps2beLatWvcuXMnz8cvCJI4CQAiV0OjN+D9sfD3ZXB2gJ9mQEUfc0cmhBCm8d///pfOnTvzxhtv0KBBA65fv27Q+nTq1CmGDx/O7NmzKVeuHACzZ8/m2rVrfPTRR7keU6PRsGbNGl588UV69+5N5cqV6datG//8849+jBVAxYoV6dy5My+//DKtW7emRo0azJ49O9djOjk5sX37dl5++WUqV67MqFGjmDp1Km3btgVg4MCBhIeHM3ToUAICAvjjjz/0t/SDbrqg33//naNHj1KrVi0+/PDDHF1y2a1WWVlZtG7dmoCAAAYPHoyLi0uO7tWHCQkJYdWqVaxfv5569erRsGFDpk+fjq+vb55eD+Dn58fPP//Mr7/+So0aNZgzZw4ffvghALa2tgA0atSI9957jzfeeAN3d3cmT56c5+MXBKMX+S2OnsVFfjOzIP4aXEmA42dh7Ffw4N8ECwvYFQml3c0XoxCiYMkiv0/H2LFj+e23355ouZNnyaeffsrcuXO5ePGiSY9rqkV+ZYxTMaQUJN+GK/G6xOjyv/68kgBx1+BR4+e0Wl3LkyROQgghCtLs2bOpV68ebm5u7Ny5kylTpuhvOCiMJHEqgtIzdImPPjFK0G1n/xl7VTfT9+NYW0FpD91dc4dOGO6ztIDyZXJ9mRBCCGEyZ8+e5ZNPPuHGjRv4+PgwdOhQRo4cae6wHkq66ijYrrrYqxBzCfzK5q31RilITP5XMvSv7YTrht1qD+PmAt4eDzw8ocwD2+6ldF1yoBvj9L9pkKXVJU0TwqFbuyd660KIQk666sSzRLrqioDI1TBymq7by8ICJoZDx1YQd/WBbrNckqPUtMcf29Zal/xkJ0VlPA3/LO0OJYy4DnZrB83q67rnypeRLjohhBAiN5I4FZDYqzByKmjvtQxptTDic90jL9xd77UOZSdEHoaJkpsL3LtT02RKu0vCJIQQQjyKJE4FJObS/aTp30rY5UyGyjzQleblDrY554gTQogCISM2xLPAVH/Pi03iNGvWLKZMmUJcXBw1a9bkyy+/NGq9G1PzK6vrnnvwzjULC/hjAVT2M31rkRBCGCt7uYo7d+5QokQJM0cjRMHKnjjz38u0GKtYJE5Lly4lPDycuXPn0qBBA2bMmEFISAinT5/Gw8PDLDGVdteNafr3gOsqst6bEKKQsLS0xMXFRb8oasmSJfWzNQtRXCiluHPnDgkJCbi4uDzxEj/F4q66Bg0aUK9ePb766itAt6BjuXLlGDBgAB988MFjX1/Qd9XJgGshRGGllCIuLo7ExERzhyJEgXJxccHLyyvX/xw8U3fVpaenc+DAAYM5HywsLAgODiYqKsqMkenIgGshRGGm0WgoXbo0Hh4eZGRkmDscIQqEtbW1yRaTLvKJ07Vr18jKyjJYEwjA09OTU6dO5fqatLQ00tLu3/OfnJxcoDEKIURhZ2lpabIfFiGKs2dykd+JEyfi7Oysf2Qv5iiEEEII8ShFPnF67rnnsLS0JD4+3qA8Pj4eLy+vXF8zcuRIkpKS9A9TLyQohBBCiOKpyCdONjY21KlTh02bNunLtFotmzZtIigoKNfX2Nra4uTkZPAQQgghhHicIj/GCSA8PJzQ0FDq1q1L/fr1mTFjBrdv36Z37955en32jYUy1kkIIYR49mT//udlooFikTi98cYbXL16ldGjRxMXF0dgYCB//PFHjgHjD3Pr1i0AGeskhBBCPMNu3bqFs7PzI+sUi3mcnpRWq+XKlSs4OjqafPK35ORkypUrx8WLF6VL0ETkMzU9+UxNTz5T05PP1PTkM9VRSnHr1i28vb2xsHj0KKZi0eL0pCwsLChbtmyBnkPGUpmefKamJ5+p6clnanrymZqefKY8tqUpW5EfHC6EEEII8bRI4iSEEEIIkUeSOBUwW1tbxowZg62trblDKTbkMzU9+UxNTz5T05PP1PTkMzWeDA4XQgghhMgjaXESQgghhMgjSZyEEEIIIfJIEichhBBCiDySxKmAzZo1i/Lly2NnZ0eDBg3Yu3evuUMqsiZOnEi9evVwdHTEw8ODjh07cvr0aXOHVWxMmjQJjUbD4MGDzR1KkXf58mX+85//4ObmRokSJQgICGD//v3mDqvIysrK4qOPPsLPz48SJUrw/PPP8/HHH+dpeQyhs337dtq3b4+3tzcajYbffvvNYL9SitGjR1O6dGlKlChBcHAwZ8+eNU+whZwkTgVo6dKlhIeHM2bMGA4ePEjNmjUJCQkhISHB3KEVSdu2bSMsLIzdu3ezYcMGMjIyaN26Nbdv3zZ3aEXevn37mDdvHjVq1DB3KEXezZs3ady4MdbW1qxdu5YTJ04wdepUSpUqZe7QiqzPPvuMOXPm8NVXX3Hy5Ek+++wzJk+ezJdffmnu0IqM27dvU7NmTWbNmpXr/smTJ/PFF18wd+5c9uzZg729PSEhIaSmpj7lSIsAJQpM/fr1VVhYmP55VlaW8vb2VhMnTjRjVMVHQkKCAtS2bdvMHUqRduvWLVWpUiW1YcMG1axZMzVo0CBzh1SkjRgxQjVp0sTcYRQr7dq1U2+//bZBWefOnVWPHj3MFFHRBqjly5frn2u1WuXl5aWmTJmiL0tMTFS2trbqxx9/NEOEhZu0OBWQ9PR0Dhw4QHBwsL7MwsKC4OBgoqKizBhZ8ZGUlASAq6urmSMp2sLCwmjXrp3B31WRfytXrqRu3bq89tpreHh4UKtWLRYsWGDusIq0Ro0asWnTJs6cOQPA4cOH+fPPP2nbtq2ZIyseYmJiiIuLM7gGODs706BBA/m9yoWsVVdArl27RlZWFp6engblnp6enDp1ykxRFR9arZbBgwfTuHFjXnjhBXOHU2RFRkZy8OBB9u3bZ+5Qio3z588zZ84cwsPD+d///se+ffsYOHAgNjY2hIaGmju8IumDDz4gOTkZf39/LC0tycrK4tNPP6VHjx7mDq1YiIuLA8j19yp7n7hPEidRJIWFhXHs2DH+/PNPc4dSZF28eJFBgwaxYcMG7OzszB1OsaHVaqlbty4TJkwAoFatWhw7doy5c+dK4pRPP/30E4sXL2bJkiVUr16d6OhoBg8ejLe3t3ym4qmTrroC8txzz2FpaUl8fLxBeXx8PF5eXmaKqnjo378/q1atYsuWLZQtW9bc4RRZBw4cICEhgdq1a2NlZYWVlRXbtm3jiy++wMrKiqysLHOHWCSVLl2aatWqGZRVrVqVCxcumCmiom/48OF88MEHdOvWjYCAAHr27MmQIUOYOHGiuUMrFrJ/k+T3Km8kcSogNjY21KlTh02bNunLtFotmzZtIigoyIyRFV1KKfr378/y5cvZvHkzfn5+5g6pSGvZsiVHjx4lOjpa/6hbty49evQgOjoaS0tLc4dYJDVu3DjHNBlnzpzB19fXTBEVfXfu3MHCwvDnytLSEq1Wa6aIihc/Pz+8vLwMfq+Sk5PZs2eP/F7lQrrqClB4eDihoaHUrVuX+vXrM2PGDG7fvk3v3r3NHVqRFBYWxpIlS1ixYgWOjo76vndnZ2dKlChh5uiKHkdHxxzjw+zt7XFzc5NxY09gyJAhNGrUiAkTJvD666+zd+9e5s+fz/z5880dWpHVvn17Pv30U3x8fKhevTqHDh1i2rRpvP322+YOrchISUnhr7/+0j+PiYkhOjoaV1dXfHx8GDx4MJ988gmVKlXCz8+Pjz76CG9vbzp27Gi+oAsrc9/WV9x9+eWXysfHR9nY2Kj69eur3bt3mzukIgvI9bFw4UJzh1ZsyHQEpvH777+rF154Qdna2ip/f381f/58c4dUpCUnJ6tBgwYpHx8fZWdnpypUqKA+/PBDlZaWZu7QiowtW7bkev0MDQ1VSummJPjoo4+Up6ensrW1VS1btlSnT582b9CFlEYpmXpVCCGEECIvZIyTEEIIIUQeSeIkhBBCCJFHkjgJIYQQQuSRJE5CCCGEEHkkiZMQQgghRB5J4iSEEEIIkUeSOAkhhBBC5JEkTkIIIYQQeSSJkxCi2HnppZcYPHiwucPIQaPR8Ntvv5k7DCHEE5DESQhhcm+99RYajQaNRoONjQ0VK1Zk/PjxZGZmArB161b9fo1Gg6enJ126dOH8+fMGx9m1axcvv/wypUqVws7OjoCAAKZNm0ZWVtYjz//rr7/y8ccf65+XL1+eGTNmmPx9PszYsWMJDAzMUR4bG0vbtm2fWhxCCNOTxEkIUSDatGlDbGwsZ8+eZejQoYwdO5YpU6YY1Dl9+jRXrlxh2bJlHD9+nPbt2+uTouXLl9OsWTPKli3Lli1bOHXqFIMGDeKTTz6hW7duPGq1KFdXVxwdHU3+ntLT05/o9V5eXtja2pooGiGEWZh5rTwhRDEUGhqqOnToYFDWqlUr1bBhQ6XU/QVHb968qd+/ePFiBahTp06plJQU5ebmpjp37pzj2CtXrlSAioyMfOj5H1ysuFmzZjkWNs22Y8cO1aRJE2VnZ6fKli2rBgwYoFJSUvT7fX191fjx41XPnj2Vo6OjfkHU//u//1OVKlVSJUqUUH5+fmrUqFEqPT1dKaXUwoULH7oQNaCWL1+uP/6RI0dU8+bNlZ2dnXJ1dVXvvPOOunXrVo7PccqUKcrLy0u5urqq999/X38uIcTTJy1OQoinokSJEo9ssSlRogSga9VZv349169fZ9iwYTnqtW/fnsqVK/Pjjz/m6by//vorZcuWZfz48cTGxhIbGwvAuXPnaNOmDV26dOHIkSMsXbqUP//8k/79+xu8/vPPP6dmzZocOnSIjz76CABHR0ciIiI4ceIEM2fOZMGCBUyfPh2AN954g6FDh1K9enX9+d54440ccd2+fZuQkBBKlSrFvn37WLZsGRs3bsxx/i1btnDu3Dm2bNnCokWLiIiIICIiIk/vXQhhelbmDkAIUbwppdi0aRPr1q1jwIABudaJjY3l888/p0yZMlSpUoU1a9YAULVq1Vzr+/v7c+bMmTyd39XVFUtLSxwdHfHy8tKXT5w4kR49eugHkVeqVIkvvviCZs2aMWfOHOzs7ABo0aIFQ4cONTjmqFGj9Nvly5dn2LBhREZG8n//93+UKFECBwcHrKysDM73b0uWLCE1NZXvvvsOe3t7AL766ivat2/PZ599hqenJwClSpXiq6++wtLSEn9/f9q1a8emTZt455138vT+hRCmJYmTEKJArFq1CgcHBzIyMtBqtXTv3p2xY8ca1ClbtixKKe7cuUPNmjX55ZdfsLGx0e9XjxjH9KQOHz7MkSNHWLx4scH5tFotMTEx+qStbt26OV67dOlSvvjiC86dO0dKSgqZmZk4OTkZdf6TJ09Ss2ZNfdIE0LhxY7RaLadPn9YnTtWrV8fS0lJfp3Tp0hw9etSocwkhTEcSJyFEgWjevDlz5szBxsYGb29vrKxyXm527NiBk5MTHh4eBoO5K1euDOiSi0aNGuV43cmTJ6lWrdoTxZeSksJ///tfBg4cmGOfj4+PfvvBxAYgKiqKHj16MG7cOEJCQnB2diYyMpKpU6c+UTwPY21tbfBco9Gg1WoL5FxCiMeTxEkIUSDs7e2pWLHiI+v4+fnh4uKSo7x169a4uroyderUHInTypUrOXv2rMF0A49jY2OTYwqD2rVrc+LEicfG+G+7du3C19eXDz/8UF/2zz//PPZ8/1a1alUiIiK4ffu2PjnbuXMnFhYWVKlSxaiYhBBPjwwOF0IUOvb29sybN48VK1bw7rvvcuTIEf7++2+++eYb3nrrLbp27crrr7+e5+OVL1+e7du3c/nyZa5duwbAiBEj2LVrF/379yc6OpqzZ8+yYsWKHIOz/61SpUpcuHCByMhIzp07xxdffMHy5ctznC8mJobo6GiuXbtGWlpajuP06NEDOzs7QkNDOXbsGFu2bGHAgAH07NlT300nhCh8JHESQhRKXbt2ZcuWLVy4cIGmTZtSpUoVpk+fzocffkhkZCQajSbPxxo/fjx///03zz//PO7u7gDUqFGDbdu2cebMGZo2bUqtWrUYPXo03t7ejzzWq6++ypAhQ+jfvz+BgYHs2rVLf7ddti5dutCmTRuaN2+Ou7t7rncAlixZknXr1nHjxg3q1atH165dadmyJV999VWe35cQ4unTqIIcfSmEEEIIUYxIi5MQQgghRB5J4iSEEEIIkUeSOAkhhBBC5JEkTkIIIYQQeSSJkxBCCCFEHkniJIQQQgiRR5I4CSGEEELkkSROQgghhBB5JImTEEIIIUQeSeIkhBBCCJFHkjgJIYQQQuSRJE5CCCGEEHn0/+GZ73INcMoSAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: PPO learning curve (greedy eval per iteration)\n",
"fig, ax = plt.subplots(figsize=(6, 3))\n",
"ax.plot(ppo_curve, marker=\"o\", ms=3, color=\"#1E40FF\")\n",
"ax.axhline(500, ls=\":\", color=\"#888\", label=\"max episode length\")\n",
"ax.set_xlabel(\"PPO iteration\"); ax.set_ylabel(\"greedy episode length\")\n",
"ax.set_title(\"PPO on CartPole\"); ax.legend(); plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "49267e8a",
"metadata": {},
"source": [
"> **Interpretation.** PPO's curve is smoother and climbs higher than REINFORCE's. The advantage estimate cuts variance; the clip keeps each update inside a trust region so the policy does not lurch. This is the algorithm InstructGPT and ChatGPT use for RLHF, and the next part runs it on a language model instead of a cart.\n",
"\n",
"> **Key takeaways**\n",
"> - The policy gradient is $(a-p)s$ for a logistic action; we verified it against finite differences before trusting autograd.\n",
"> - GAE is a backward recursion with a bias-variance knob $\\lambda$; both limits ($\\lambda=0$ TD, $\\lambda=1$ MC) are exactly checkable.\n",
"> - PPO clips the probability ratio to a trust region. The `min` makes the clip asymmetric across advantage sign; that asymmetry is the whole objective.\n"
]
},
{
"cell_type": "markdown",
"id": "4bd7f3b2",
"metadata": {},
"source": [
"## Part 5 — RLHF on a toy language model\n",
"\n",
"> **Objectives**\n",
"> - Train a Bradley-Terry reward model on preference pairs and verify it scores chosen above rejected.\n",
"> - Run PPO fine-tuning of a tiny LM against that reward, and watch the policy collapse when the KL penalty is off.\n",
"> - Implement DPO and GRPO as one-function variants on the same preferences.\n",
"\n",
"RLHF turned GPT-3 into ChatGPT. The pipeline: start from a pretrained policy, train a *reward model* from human preference comparisons, then fine-tune the policy with PPO against the reward model plus a KL penalty that keeps it near the starting policy. We run the whole thing on a deliberately tiny language model so it trains in seconds and every claim is checkable.\n",
"\n",
"The toy LM is a one-block GPT over a 6-token vocabulary. Token 5 is \".\" (the period). The reward we will optimize is \"more periods\", which is the smallest reproduction of the classic mode-collapse demo: optimize it without a KL leash and the policy degenerates to `.......`.\n"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "48a7c21f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.140982Z",
"iopub.status.busy": "2026-06-11T20:07:45.140888Z",
"iopub.status.idle": "2026-06-11T20:07:45.149616Z",
"shell.execute_reply": "2026-06-11T20:07:45.149289Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TinyGPT: 13536 parameters · forward output (2, 8, 6)\n"
]
}
],
"source": [
"# A genuinely tiny GPT (one transformer block) over a 6-token vocab.\n",
"VOCAB = 6\n",
"PERIOD = 5 # the \".\" token; \"more periods\" is our toy reward target\n",
"SEQ_LEN = 16\n",
"\n",
"class Block(torch.nn.Module):\n",
" def __init__(self, d, h, T):\n",
" super().__init__()\n",
" self.h, self.dk = h, d // h\n",
" self.qkv = torch.nn.Linear(d, 3 * d, bias=False)\n",
" self.out = torch.nn.Linear(d, d, bias=False)\n",
" self.ln1 = torch.nn.LayerNorm(d); self.ln2 = torch.nn.LayerNorm(d)\n",
" self.ffn = torch.nn.Sequential(torch.nn.Linear(d, 4 * d), torch.nn.GELU(),\n",
" torch.nn.Linear(4 * d, d))\n",
" self.register_buffer(\"mask\", torch.triu(torch.ones(T, T), 1).bool().view(1, 1, T, T))\n",
" def forward(self, x):\n",
" B, T, C = x.shape\n",
" q, k, v = self.qkv(self.ln1(x)).split(C, dim=-1)\n",
" q = q.view(B, T, self.h, self.dk).transpose(1, 2) # (B,h,T,dk)\n",
" k = k.view(B, T, self.h, self.dk).transpose(1, 2)\n",
" v = v.view(B, T, self.h, self.dk).transpose(1, 2)\n",
" att = (q @ k.transpose(-2, -1) / math.sqrt(self.dk)).masked_fill(self.mask[:, :, :T, :T], float(\"-inf\"))\n",
" h = (F.softmax(att, dim=-1) @ v).transpose(1, 2).contiguous().view(B, T, C) # (B,T,C)\n",
" x = x + self.out(h)\n",
" return x + self.ffn(self.ln2(x))\n",
"\n",
"class TinyGPT(torch.nn.Module):\n",
" def __init__(self, vocab=VOCAB, d=32, h=4, T=SEQ_LEN):\n",
" super().__init__()\n",
" self.d, self.T = d, T\n",
" self.tok = torch.nn.Embedding(vocab, d); self.pos = torch.nn.Embedding(T, d)\n",
" self.block = Block(d, h, T); self.lnf = torch.nn.LayerNorm(d)\n",
" self.head = torch.nn.Linear(d, vocab, bias=False)\n",
" def trunk(self, idx):\n",
" T = idx.shape[1]; pos = torch.arange(T, device=idx.device)\n",
" return self.lnf(self.block(self.tok(idx) + self.pos(pos))) # (B,T,d)\n",
" def forward(self, idx):\n",
" return self.head(self.trunk(idx)) # (B,T,vocab)\n",
"\n",
"torch.manual_seed(SEED)\n",
"base = TinyGPT()\n",
"nparams = sum(p.numel() for p in base.parameters())\n",
"smoke = base(torch.randint(0, VOCAB, (2, 8)))\n",
"check_shape(smoke, (2, 8, VOCAB)) # randn smoke test before any training\n",
"print(f\"TinyGPT: {nparams} parameters · forward output {tuple(smoke.shape)}\")"
]
},
{
"cell_type": "markdown",
"id": "19827591",
"metadata": {},
"source": [
"The reward model is the trunk of a GPT with the unembedding swapped for a scalar head reading the last position. We train it with the Bradley-Terry loss on preference pairs $(y_w, y_l)$:\n",
"\n",
"$$\\mathcal{L}_{\\text{RM}} = -\\mathbb{E}\\left[\\log\\sigma\\big(r_\\phi(y_w) - r_\\phi(y_l)\\big)\\right]$$\n",
"\n",
"Our synthetic preferences: \"chosen\" sequences contain more periods than \"rejected\" ones. A correct reward model learns to score period-heavy sequences higher.\n"
]
},
{
"cell_type": "markdown",
"id": "11726654",
"metadata": {},
"source": [
"### Exercise 19.7 — The Bradley-Terry reward-model loss\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"Fill in `bradley_terry_loss(r_chosen, r_rejected)`: the negative log-likelihood that the chosen item beats the rejected one under the Bradley-Terry model, $-\\log\\sigma(r_w - r_l)$, averaged over the batch. The checks pin the value at a tie (a margin of 0 gives $\\log 2$) and the monotonicity (a bigger margin gives a smaller loss).\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "56290f86",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.150539Z",
"iopub.status.busy": "2026-06-11T20:07:45.150458Z",
"iopub.status.idle": "2026-06-11T20:07:45.153961Z",
"shell.execute_reply": "2026-06-11T20:07:45.153669Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 19.7 Bradley-Terry tie = log 2: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 19.7 Bradley-Terry monotone in margin: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def bradley_terry_loss(r_chosen, r_rejected):\n",
" \"\"\"Mean -log sigma(r_chosen - r_rejected). Both are (batch,) tensors.\"\"\"\n",
" # TODO 1: margin = r_chosen - r_rejected\n",
" margin = None\n",
" attempted(margin)\n",
" # TODO 2: return the mean of -logsigmoid(margin) (use F.logsigmoid)\n",
" raise NotImplementedError # remove once the TODOs are done\n",
"\n",
"def _bt_tie():\n",
" loss = bradley_terry_loss(torch.tensor([0.0, 1.0]), torch.tensor([0.0, 1.0]))\n",
" assert abs(loss.item() - math.log(2)) < 1e-5, \\\n",
" f\"tie should cost log(2)=0.693, got {loss.item():.4f}\"\n",
"\n",
"def _bt_monotone():\n",
" big = bradley_terry_loss(torch.tensor([3.0]), torch.tensor([0.0]))\n",
" small = bradley_terry_loss(torch.tensor([0.5]), torch.tensor([0.0]))\n",
" assert big.item() < small.item(), \\\n",
" f\"a larger winning margin must cost less: margin 3 -> {big.item():.3f} vs margin 0.5 -> {small.item():.3f}\"\n",
"\n",
"check(\"19.7 Bradley-Terry tie = log 2\", _bt_tie)\n",
"check(\"19.7 Bradley-Terry monotone in margin\", _bt_monotone)"
]
},
{
"cell_type": "markdown",
"id": "02ed125f",
"metadata": {},
"source": [
"Hint 1 (conceptual)$-\\log\\sigma(x)$ is `-F.logsigmoid(x)`. The argument is the reward margin $r_w - r_l$. Average over the batch with `.mean()`.\n",
"\n",
"Hint 2 (the line)`return -F.logsigmoid(r_chosen - r_rejected).mean()`.\n",
"\n",
"Help — \"loss is negative\"You dropped the minus sign. `logsigmoid` is always negative (it is a log of a number in $(0,1)$), so the loss `-logsigmoid(...)` is positive. If yours is negative you returned `logsigmoid` without negating.\n"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "8e34f26a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.154785Z",
"iopub.status.busy": "2026-06-11T20:07:45.154709Z",
"iopub.status.idle": "2026-06-11T20:07:45.158039Z",
"shell.execute_reply": "2026-06-11T20:07:45.157810Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 19.7 Bradley-Terry tie = log 2\n",
"[ ok ] 19.7 Bradley-Terry monotone in margin\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines bradley_terry_loss; the checks below re-verify it.\n",
"def bradley_terry_loss(r_chosen, r_rejected):\n",
" return -F.logsigmoid(r_chosen - r_rejected).mean()\n",
"\n",
"check(\"19.7 Bradley-Terry tie = log 2\", _bt_tie, required=True)\n",
"check(\"19.7 Bradley-Terry monotone in margin\", _bt_monotone, required=True)"
]
},
{
"cell_type": "markdown",
"id": "d7d32d41",
"metadata": {},
"source": [
"Now train the reward model on synthetic preferences. Chosen sequences are mostly periods; rejected are mostly non-periods. We expect the loss to fall and the model to score chosen above rejected.\n",
"\n",
"> **Runtime:** a few seconds on CPU.\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "448b7269",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.158890Z",
"iopub.status.busy": "2026-06-11T20:07:45.158816Z",
"iopub.status.idle": "2026-06-11T20:07:45.234632Z",
"shell.execute_reply": "2026-06-11T20:07:45.234239Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"RM loss: 0.788 -> 0.000\n",
"mean reward chosen 4.61 vs rejected -4.77\n"
]
}
],
"source": [
"class RewardModel(torch.nn.Module):\n",
" def __init__(self, base):\n",
" super().__init__()\n",
" self.trunk = base # shares the tiny-GPT trunk\n",
" self.score = torch.nn.Linear(base.d, 1)\n",
" def forward(self, tokens):\n",
" h = self.trunk.trunk(tokens) # (B,T,d)\n",
" return self.score(h[:, -1, :]).squeeze(-1) # scalar per sequence\n",
"\n",
"torch.manual_seed(SEED)\n",
"rm = RewardModel(TinyGPT())\n",
"opt_rm = torch.optim.Adam(rm.parameters(), lr=1e-2)\n",
"\n",
"def make_pref_batch(n=16, seed=0):\n",
" g = torch.Generator().manual_seed(seed)\n",
" chosen = torch.full((n, 8), PERIOD) # mostly periods\n",
" chosen[:, :2] = torch.randint(0, PERIOD, (n, 2), generator=g)\n",
" rejected = torch.randint(0, PERIOD, (n, 8), generator=g) # no periods\n",
" return chosen, rejected\n",
"\n",
"rm_losses = []\n",
"for step in range(40):\n",
" chosen, rejected = make_pref_batch(seed=step)\n",
" loss = bradley_terry_loss(rm(chosen), rm(rejected))\n",
" opt_rm.zero_grad(); loss.backward(); opt_rm.step()\n",
" rm_losses.append(loss.item())\n",
"\n",
"chosen, rejected = make_pref_batch(seed=999)\n",
"print(f\"RM loss: {rm_losses[0]:.3f} -> {rm_losses[-1]:.3f}\")\n",
"print(f\"mean reward chosen {rm(chosen).mean():.2f} vs rejected {rm(rejected).mean():.2f}\")\n",
"assert rm_losses[-1] < rm_losses[0], \"Bradley-Terry loss should fall as the RM learns the preference\"\n",
"assert rm(chosen).mean() > rm(rejected).mean(), \"the RM must score period-heavy sequences higher\""
]
},
{
"cell_type": "markdown",
"id": "b9ffc117",
"metadata": {},
"source": [
"> **Interpretation.** The reward model learned the preference: period-heavy sequences score higher. Now the dangerous step. We treat the LM as a policy, sample completions, score them with a *handwritten* period-counting reward (cleaner than the learned RM for a controlled demo), and run PPO with a KL penalty back to the frozen starting policy. The KL coefficient is the knob that decides whether the policy stays grounded or collapses.\n",
"\n",
"> **Runtime:** ~5-20 s on CPU.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "7c29921b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.235694Z",
"iopub.status.busy": "2026-06-11T20:07:45.235591Z",
"iopub.status.idle": "2026-06-11T20:07:45.802600Z",
"shell.execute_reply": "2026-06-11T20:07:45.802232Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"KL=0.0 final reward 0.88 diversity 0.69 sample [5, 5, 5, 5, 5, 5, 5, 5]\n",
"KL=0.3 final reward 0.29 diversity 1.00 sample [5, 5, 0, 5, 4, 1, 2, 0]\n"
]
}
],
"source": [
"class ActorCriticLM(torch.nn.Module):\n",
" def __init__(self, base):\n",
" super().__init__()\n",
" self.base = base\n",
" self.value_head = torch.nn.Linear(base.d, 1)\n",
" def forward(self, idx):\n",
" h = self.base.trunk(idx)\n",
" return self.base.head(h), self.value_head(h).squeeze(-1) # logits, values\n",
"\n",
"def period_reward(completion):\n",
" '''Fraction of period tokens in each completion. completion: (B, Tnew).'''\n",
" return (completion == PERIOD).float().mean(dim=-1)\n",
"\n",
"def rlhf_train(kl_coef, steps, seed=SEED, batch=16, new_tokens=8):\n",
" torch.manual_seed(seed)\n",
" shared = TinyGPT()\n",
" actor = ActorCriticLM(shared)\n",
" ref = TinyGPT(); ref.load_state_dict(shared.state_dict()) # frozen reference\n",
" for p in ref.parameters(): p.requires_grad_(False)\n",
" opt = torch.optim.Adam(actor.parameters(), lr=1e-3)\n",
" prompt = torch.zeros(batch, 1, dtype=torch.long) # BOS = token 0\n",
" reward_hist, diversity_hist = [], []\n",
" for it in range(steps):\n",
" full = prompt.clone(); logps, reflps, vals = [], [], []\n",
" with torch.no_grad(): # rollout\n",
" for _ in range(new_tokens):\n",
" logits, v = actor(full)\n",
" dist = Categorical(logits=logits[:, -1])\n",
" nxt = dist.sample()\n",
" logps.append(dist.log_prob(nxt))\n",
" reflps.append(Categorical(logits=ref(full)[:, -1]).log_prob(nxt))\n",
" vals.append(v[:, -1])\n",
" full = torch.cat([full, nxt[:, None]], dim=1)\n",
" _, v_last = actor(full); vals.append(v_last[:, -1])\n",
" comp = full[:, 1:]\n",
" logps = torch.stack(logps, 1); reflps = torch.stack(reflps, 1); vals = torch.stack(vals, 1)\n",
" kl = logps - reflps # per-token KL to reference\n",
" rew = -kl_coef * kl\n",
" rew[:, -1] += period_reward(comp) # scalar reward at the last token\n",
" # GAE over the per-token reward stream\n",
" adv = torch.zeros_like(rew); last = torch.zeros(batch)\n",
" for t in reversed(range(new_tokens)):\n",
" delta = rew[:, t] + vals[:, t + 1] - vals[:, t]\n",
" last = delta + 0.95 * last; adv[:, t] = last\n",
" ret = adv + vals[:, :-1]\n",
" adv = (adv - adv.mean()) / (adv.std() + 1e-8)\n",
" for _ in range(4): # PPO epochs\n",
" logits, vv = actor(full)\n",
" dist = Categorical(logits=logits[:, :-1])\n",
" ratio = (dist.log_prob(full[:, 1:]) - logps).exp()\n",
" s1 = ratio * adv; s2 = torch.clamp(ratio, 0.8, 1.2) * adv\n",
" loss = -torch.min(s1, s2).mean() + 0.5 * F.mse_loss(vv[:, :-1], ret) - 0.01 * dist.entropy().mean()\n",
" opt.zero_grad(); loss.backward()\n",
" torch.nn.utils.clip_grad_norm_(actor.parameters(), 0.5); opt.step()\n",
" with torch.no_grad():\n",
" unique = len({tuple(r.tolist()) for r in comp}) # distinct completions / batch\n",
" reward_hist.append(period_reward(comp).mean().item())\n",
" diversity_hist.append(unique / batch)\n",
" return reward_hist, diversity_hist, comp\n",
"\n",
"rew0, div0, sample0 = rlhf_train(kl_coef=0.0, steps=RLHF_STEPS) # no leash\n",
"rewK, divK, sampleK = rlhf_train(kl_coef=0.3, steps=RLHF_STEPS) # KL leash\n",
"print(f\"KL=0.0 final reward {rew0[-1]:.2f} diversity {div0[-1]:.2f} sample {sample0[0].tolist()}\")\n",
"print(f\"KL=0.3 final reward {rewK[-1]:.2f} diversity {divK[-1]:.2f} sample {sampleK[0].tolist()}\")"
]
},
{
"cell_type": "markdown",
"id": "d6454d58",
"metadata": {},
"source": [
"> **Predict:** which run has higher reward, and which has more diverse samples? AnswerKL=0.0 gets the higher reward (it is free to maximize periods) but its diversity collapses toward a single repeated string. KL=0.3 keeps diversity high but earns less reward. You cannot have both; that tension is the entire operating tradeoff of RLHF.\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "0ee93246",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.803758Z",
"iopub.status.busy": "2026-06-11T20:07:45.803670Z",
"iopub.status.idle": "2026-06-11T20:07:45.922506Z",
"shell.execute_reply": "2026-06-11T20:07:45.922166Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3kAAAFJCAYAAAAmFeRRAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAArTJJREFUeJzs3Xd4U3UbxvFvunfpoFBKKRXZW5AhAspGZIgvoOyykY0ookxFEJAlApWNigIqiMqwWESGiMres1BWN3TP5Lx/HIjWFmhK2tPxfK4rl8lJzjl3Ukny5Ld0iqIoCCGEEEIIIYQoEiy0DiCEEEIIIYQQwnykyBNCCCGEEEKIIkSKPCGEEEIIIYQoQqTIE0IIIYQQQogiRIo8IYQQQgghhChCpMgTQgghhBBCiCJEijwhhBBCCCGEKEKkyBNCCCGEEEKIIkSKPCGEEEIIIYQoQqTIE0I8sXXr1qHT6bh27ZrWUYQQQhRgL7zwAi+88EK+n+fatWvodDrWrVuX5+fOrfx6bUTxIEWeEEIIIYQQBczt27eZPn06x48f1zqKKISstA4ghBBCCCFEXvLz8yM5ORlra2utozxUUFBQptu3b99mxowZlC9fnjp16mgTShRa0pInRAGRmJiodYSHMhgMpKSkaB1DCCGEyBWdToednR2Wlpb5el5TPtttbGywsbHJwzSiOJEiTwgNTJ8+HZ1Ox9mzZ+nZsydubm48//zzAHz55ZfUq1cPe3t73N3dee2117hx44Zx308++QRLS0vu3btn3DZ//nx0Oh3jx483btPr9Tg7OzNx4kTjto8//pjnnnsODw8P7O3tqVevHt9++22WfDqdjpEjR7JhwwaqV6+Ora0tu3btAuDMmTO0aNECe3t7ypYty8yZMzEYDOZ+iYQQQjxCfHw8Y8eOpXz58tja2uLl5UXr1q05evSo8TH79++nW7dulCtXDltbW3x9fRk3bhzJycmZjtW/f3+cnJwIDQ3l5ZdfxsnJCR8fH5YuXQrAqVOnaNGiBY6Ojvj5+fHVV19l2v/BuOx9+/YxdOhQPDw8cHFxoW/fvty9e/exzyU1NZVp06bx9NNPG3O+/fbbpKam5ui1WLFiBRUqVMDe3p4GDRqwf//+LI/575i8jz/+GJ1Ox/Xr17M8dtKkSdjY2GTKfvjwYdq1a4erqysODg40b96cgwcPZtrvUZ/tYWFhBAQEULZsWWxtbfH29qZz586ZxrL/e0ze3r17efbZZwEICAhAp9MZ80+bNg1ra2siIyOzZB8yZAglSpSQH2aFFHlCaKlbt24kJSUxa9YsBg8ezIcffkjfvn2pWLEiCxYsYOzYsQQHB9OsWTNjUde0aVMMBgMHDhwwHmf//v1YWFhk+mA7duwYCQkJNGvWzLht8eLF1K1bl/fff59Zs2ZhZWVFt27d2L59e5Zse/bsYdy4cfTo0YPFixdTvnx5wsLCePHFFzl+/DjvvPMOY8eO5fPPP2fx4sV59yIJIYTIYtiwYSxfvpxXX32VZcuWMWHCBOzt7Tl37pzxMd988w1JSUkMHz6cJUuW0LZtW5YsWULfvn2zHE+v19O+fXt8fX2ZO3cu5cuXZ+TIkaxbt4527dpRv3595syZg7OzM3379iUkJCTLMUaOHMm5c+eYPn06ffv2ZcOGDXTp0gVFUR76PAwGA506deLjjz+mY8eOLFmyhC5durBw4UJ69Ojx2Ndh9erVDB06lNKlSzN37lyaNGlCp06dMv04mp3u3buj0+nYvHlzlvs2b95MmzZtcHNzA9TPw2bNmhEXF8e0adOYNWsW9+7do0WLFvz5559Z9v/vZzvAq6++ytatWwkICGDZsmWMHj2a+Ph4QkNDs81XtWpV3n//fUAt3L744gu++OILmjVrRp8+fcjIyGDTpk2Z9klLS+Pbb7/l1Vdfxc7O7rGvnSjiFCFEvps2bZoCKK+//rpx27Vr1xRLS0vlww8/zPTYU6dOKVZWVsbter1ecXFxUd5++21FURTFYDAoHh4eSrdu3RRLS0slPj5eURRFWbBggWJhYaHcvXvXeKykpKRMx05LS1Nq1KihtGjRItN2QLGwsFDOnDmTafvYsWMVQDl8+LBxW0REhOLq6qoASkhISO5eECGEECZxdXVVRowY8cjH/Pc9X1EUZfbs2YpOp1OuX79u3NavXz8FUGbNmmXcdvfuXcXe3l7R6XTKxo0bjdvPnz+vAMq0adOM29auXasASr169ZS0tDTj9rlz5yqAsm3bNuO25s2bK82bNzfe/uKLLxQLCwtl//79mXIGBgYqgHLw4MGHPr+0tDTFy8tLqVOnjpKammrcvmLFCgXIdJ6QkBAFUNauXWvc1rhxY6VevXqZjvnnn38qgPL5558riqJ+xlasWFFp27atYjAYjI9LSkpS/P39ldatWxu3ZffZrijqawko8+bNe+hzUZSsr81ff/2VJfO/szds2DDTti1btiiA8uuvvz7yPKJ4kJY8ITQ0bNgw4/UtW7ZgMBjo3r07UVFRxkvp0qWpWLEiv/76KwAWFhY899xz7Nu3D4Bz584RHR3NO++8g6IoHDp0CFBb92rUqEGJEiWM57C3tzdev3v3LrGxsTRt2jRT954HmjdvTrVq1TJt27FjB40aNaJBgwbGbSVLlqRXr15P/mIIIYTIsRIlSnD48GFu37790Mf8+z0/MTGRqKgonnvuORRF4dixY1keP2jQoEzHr1y5Mo6OjnTv3t24vXLlypQoUYKrV69m2X/IkCGZJjYZPnw4VlZW7Nix46EZv/nmG6pWrUqVKlUyffa1aNECwPjZl52///6biIgIhg0blmksW//+/XF1dX3ofg/06NGDI0eOcOXKFeO2TZs2YWtrS+fOnQE4fvw4ly5domfPnkRHRxvzJSYm0rJlS/bt25dlyMK/P9tB/TvY2Niwd+/eHHVfzYm+ffty+PDhTNk3bNiAr68vzZs3N8s5ROEmRZ4QGvL39zdev3TpEoqiULFiRUqWLJnpcu7cOSIiIoyPbdq0KUeOHCE5OZn9+/fj7e3NM888Q+3atY1dNg8cOEDTpk0zne+nn36iUaNG2NnZ4e7uTsmSJVm+fDmxsbGPzPbA9evXqVixYpbtlStXzvVrIIQQwnRz587l9OnT+Pr60qBBA6ZPn56l8AoNDaV///64u7vj5OREyZIljQXAf9/37ezsKFmyZKZtrq6ulC1bFp1Ol2V7dsXKfz8fnJyc8Pb2fuQaqpcuXeLMmTNZPvcqVaoEkOmz778ejKf773mtra156qmnHrrfA926dcPCwsLY7VFRFL755hvat2+Pi4uLMR9Av379smRctWoVqampWV7L/35+2traMmfOHHbu3EmpUqVo1qwZc+fOJSws7LEZH6ZHjx7Y2tqyYcMGQP17/vTTT/Tq1SvL30sUT7KEghAa+vevrAaDAZ1Ox86dO7Od/cvJycl4/fnnnyc9PZ1Dhw6xf/9+YzHXtGlT9u/fz/nz54mMjMxU5O3fv59OnTrRrFkzli1bhre3N9bW1qxduzbLIPr/ZhNCCFGwdO/enaZNm7J161aCgoKYN28ec+bMYcuWLbRv3x69Xk/r1q2JiYlh4sSJVKlSBUdHR27dukX//v2ztD49bNbJh21XHjHOzhQGg4GaNWuyYMGCbO/39fU1y3myU6ZMGZo2bcrmzZt59913+eOPPwgNDWXOnDmZ8gHMmzfvocsY/PvzGbL//Bw7diwdO3bk+++/5+eff2bKlCnMnj2bPXv2ULduXZOzu7m58fLLL7NhwwamTp3Kt99+S2pqKr179zb5WKJokiJPiAKiQoUKKIqCv7+/8RfMh2nQoAE2Njbs37+f/fv389ZbbwHQrFkzVq5cSXBwsPH2A9999x12dnb8/PPP2NraGrevXbs2xxn9/PyMv2r+24ULF3J8DCGEEObh7e3NG2+8wRtvvEFERATPPPMMH374Ie3bt+fUqVNcvHiR9evXZ5poZffu3XmW59KlS7z44ovG2wkJCdy5c4eXXnrpoftUqFCBEydO0LJlS5NboPz8/IznfdC9EyA9PZ2QkBBq16792GP06NGDN954gwsXLrBp0yYcHBzo2LFjpnwALi4utGrVyqR8/1WhQgXefPNN3nzzTS5dukSdOnWYP38+X375ZbaPf9zr0bdvXzp37sxff/3Fhg0bqFu3LtWrV3+ijKLokO6aQhQQXbt2xdLSkhkzZmT5hVRRFKKjo4237ezsePbZZ/n6668JDQ3N1JKXnJzMJ598QoUKFfD29jbuY2lpiU6nQ6/XG7ddu3aN77//PscZX3rpJf74449Ms4lFRkYau4sIIYTIe3q9PksXQS8vL8qUKWNcduBBC9y/P08URcnT2ZBXrFhBenq68fby5cvJyMigffv2D92ne/fu3Lp1i5UrV2a5Lzk5+ZHrzNWvX5+SJUsSGBhIWlqacfu6desyLTP0KK+++iqWlpZ8/fXXfPPNN7z88ss4Ojoa769Xrx4VKlTg448/JiEhIcv+2S1j8F9JSUlZljSoUKECzs7Oj1wm4kGOhz2X9u3b4+npyZw5c/jtt9+kFU9kIi15QhQQFSpUYObMmUyaNIlr167RpUsXnJ2dCQkJYevWrQwZMoQJEyYYH9+0aVM++ugjXF1dqVmzJqB+yFeuXJkLFy7Qv3//TMfv0KEDCxYsoF27dvTs2ZOIiAiWLl3K008/zcmTJ3OU8e233+aLL76gXbt2jBkzBkdHR1asWIGfn1+OjyGEEOLJxMfHU7ZsWf73v/9Ru3ZtnJyc+OWXX/jrr7+YP38+AFWqVKFChQpMmDCBW7du4eLiwnfffWe2iT+yk5aWRsuWLenevTsXLlxg2bJlPP/883Tq1Omh+/Tp04fNmzczbNgwfv31V5o0aYJer+f8+fNs3ryZn3/+mfr162e7r7W1NTNnzmTo0KG0aNGCHj16EBISwtq1a3M0Jg/Uz80XX3yRBQsWEB8fn2XZBgsLC1atWkX79u2pXr06AQEB+Pj4cOvWLX799VdcXFz48ccfH3mOixcvGl+XatWqYWVlxdatWwkPD+e111576H4VKlSgRIkSBAYG4uzsjKOjIw0bNjSO+bO2tua1117j008/xdLSktdffz1Hz1kUE1pN6ylEcfZgmuXIyMgs93333XfK888/rzg6OiqOjo5KlSpVlBEjRigXLlzI9Ljt27crgNK+fftM2wcNGqQAyurVq7Mce/Xq1UrFihUVW1tbpUqVKsratWuNWf4NeOjU3CdPnlSaN2+u2NnZKT4+PsoHH3ygrF69WpZQEEKIfJKamqq89dZbSu3atRVnZ2fF0dFRqV27trJs2bJMjzt79qzSqlUrxcnJSfH09FQGDx6snDhxIsu0/P369VMcHR2znKd58+ZK9erVs2z38/NTOnToYLz9YAmF3377TRkyZIji5uamODk5Kb169VKio6OzHPPfywQoiroUwpw5c5Tq1asrtra2ipubm1KvXj1lxowZSmxs7GNfj2XLlin+/v6Kra2tUr9+fWXfvn1ZzpPdEgoPrFy5UgEUZ2dnJTk5OdtzHDt2TOnatavi4eGh2NraKn5+fkr37t2V4OBg42Me9tkeFRWljBgxQqlSpYri6OiouLq6Kg0bNlQ2b9782Ndm27ZtSrVq1RQrK6ts8z9Y8qFNmzaPfZ1E8aJTFDONnBVCCCGEEMXOunXrCAgI4K+//npoq5vIGydOnKBOnTp8/vnn9OnTR+s4ogCRMXlCCCGEEEIUQitXrsTJyYmuXbtqHUUUMDImTwghhBBCiELkxx9/5OzZs6xYsYKRI0dmmixGCJAiTwghhBBCiEJl1KhRhIeH89JLLzFjxgyt44gCSMbkCSGEEEIIIUQRImPyhBBCCCGEEKII0bS75r59+5g3bx5Hjhzhzp07bN26lS5dujxyn7179zJ+/HjOnDmDr68vkydPzrIe2KMYDAZu376Ns7MzOp3uyZ6AEEKIAkNRFOLj4ylTpgwWFsX7N0z5rBNCiKIpp591mhZ5iYmJ1K5dmwEDBuRoVqCQkBA6dOjAsGHD2LBhA8HBwQwaNAhvb2/atm2bo3Pevn0bX1/fJ40uhBCigLpx4wZly5bVOoam5LNOCCGKtsd91hWYMXk6ne6xLXkTJ05k+/btnD592rjttdde4969e+zatStH54mNjaVEiRLcuHEDFxeXJ40thBCigIiLi8PX15d79+7h6uqqdRxNyWedEEIUTTn9rCtUs2seOnSIVq1aZdrWtm1bxo4d+9B9UlNTSU1NNd6Oj48HwMXFRT74hBCiCJLuif+8BvJZJ4QQRdPjPusK1aCFsLAwSpUqlWlbqVKliIuLIzk5Odt9Zs+ejaurq/Ei3VeEEEIIIYQQRVmhKvJyY9KkScTGxhovN27c0DqSEEIIIYQQQuSZQtVds3Tp0oSHh2faFh4ejouLC/b29tnuY2tri62tbX7EE0IIIYQQQgjNFaoir3HjxuzYsSPTtt27d9O4cWOzn0uv15Oenm7244qcs7GxKfbToAshhBBCCGEqTYu8hIQELl++bLwdEhLC8ePHcXd3p1y5ckyaNIlbt27x+eefAzBs2DA+/fRT3n77bQYMGMCePXvYvHkz27dvN1smRVEICwvj3r17ZjumyB0LCwv8/f2xsbHROooQQgghhBCFhqZF3t9//82LL75ovD1+/HgA+vXrx7p167hz5w6hoaHG+/39/dm+fTvjxo1j8eLFlC1bllWrVuV4jbyceFDgeXl54eDgILO0aeTBQr537tyhXLly8ncQQhRa+/btY968eRw5coQ7d+48drkggL179zJ+/HjOnDmDr68vkydPpn///vmSVwghROGnaZH3wgsv8Khl+tatW5ftPseOHcuTPHq93ljgeXh45Mk5RM6VLFmS27dvk5GRgbW1tdZxhBAiVxITE6lduzYDBgyga9euj318SEgIHTp0YNiwYWzYsIHg4GAGDRqEt7e3WX/UfBRFgeSUfDmVEEIUO/Z2kNftF4VqTF5eezAGz8HBQeMkAjB209Tr9VLkCXFfyvnzxAcFYV2uHA71n8Xap4y0dBdw7du3p3379jl+fGBgIP7+/syfPx+AqlWrcuDAARYuXPjQIu+/a8LGxcU9Uea40Ahq9fd6omMIIYTI3rkd4JD9nJFmI0VeNuQLU8EgfwchMovbuZPb70xC+deXeavSpXGoVw+H+vVwqF8fmwoV0MmERYXaoUOHaNWqVaZtbdu2ZezYsQ/dZ/bs2cyYMcNsGQwJ8YAUeUIIUVhJkSeEEAWcoihELVtG1JJPAbB/5hnQ60k+c4aMsDDitm8n7v4EVJaurtjXUws+h/r1sKtaFZ20hBcqYWFhlCpVKtO2UqVKERcXR3JycrZLBk2aNMk4rh3UljxfX99cZ3Aq7caBritzvb8Q5hK3+xfSLl3CrWdPvCZM0DqOEGZhb5f355AiTwghCjBDSgp33n2PuPvLx7j374/XWxPQWVpiSEoi+eRJkv4+QtLff5N84gT62FgS9uwhYc8eAHQODjjUqa0WfvXqY1+7FhYPWVdUFF7mXhPW2sMd31GDzXY8IXIr4Zlq3Bg4iJTvNmDzRgBWMmeCEDkiRZ4QQhRQGZGR3Bg5kpQTJ8HKitLTpuLWrZvxfgsHBxwbNcKxUSMAlPR0Us6eJenvv9XC7+hRDLGxJP5+iMTfD6k7WVtjX7262r2zYSMcmzwn3TsLmNKlSxMeHp5pW3h4OC4uLtm24glRlDk+9xx2tWqRcvIkMevW4fXmm1pHEqJQkE/2IqJ///5ZpuT+9ttvsbOzY/78+dne/yROnjxJ06ZNsbOzw9fXl7lz5z52n9DQUDp06ICDgwNeXl689dZbZGRkmC2TEEVJyvnzhHTvQcqJk1i4ulJu1apMBV52dNbW2NeujcfAgfguX0alQ7/jv20bpaZOweWll7Dy8oL0dJKPHyd61WpuDB7MrTFjMSQl5dOzEjnRuHFjgoODM23bvXs3jRs31iiRENrR6XR4Dh8GwN0NX6GXdYyFyBEp8oqoVatW0atXL5YvX86bZv7VKy4ujjZt2uDn58eRI0eYN28e06dPZ8WKFQ/dR6/X06FDB9LS0vj9999Zv34969atY+rUqWbNJkRREL9nD9d69iLjzh1sypfHf9NGHBs1NPk4OgsL7CpXwr1nT3wWzOfp3/ZSYXcQ3rNn4/pqV3TW1sTv3s313n1I/0/LkTCfhIQEjh8/zvHjxwF1iYTjx48b14GdNGkSffv2NT5+2LBhXL16lbfffpvz58+zbNkyNm/ezLhx47SIL4TmnF54AdsqVTAkJRHz+RdaxxGiUJAi7zEURcGQlKTJ5VFrCD7K3LlzGTVqFBs3biQgIMDMrwhs2LCBtLQ01qxZQ/Xq1XnttdcYPXo0CxYseOg+QUFBnD17li+//JI6derQvn17PvjgA5YuXUpaWprZMwpRGCmKQvSqVdwcMRIlKQnH5xpTftNGbMqXN8vxdTodNr6+lHilC2U+/JBy69Zi6eZGytmzXPtfN5JPnTbLeURmf//9N3Xr1qVu3boAjB8/nrp16xp/5Lpz546x4APw9/dn+/bt7N69m9q1azN//nxWrVqVb2vkCVHQ6HQ6PIeprXkxX36JPj5e40RCFHwyJu8xlORkLjxTT5NzVz56BJ2Ja/ZNnDiRZcuW8dNPP9GyZcsc79e+fXv279//0Pv9/Pw4c+YMoE7v3axZM+M6dqBO7z1nzhzu3r2Lm5tblv0PHTpEzZo1M80Y17ZtW4YPH86ZM2eMX36EKK6UtDTuTJ9B7JYtALj1fJ1Skybl6cyYDvXqUf6bzdwcPpzUS5e53qcPZT76CJd2UkyY0wsvvPDIH+3WrVuX7T7Hjh3Lw1RCFC7ObVpjU6ECaVeucHfDV3gOG6p1JCEKNCnyipCdO3eybds2goODadGihUn7rlq1iuTk5Ife/+/FyMPCwvD39890/4PiLSwsLNsi72FTgj+4T4jiLOPuXW6OGkXy30fAwoJS776Le+9e+XJum7Jl8fv6a26NH0/ivv3cGjuWtDGj8Rg2TNaqFEIUGDoLCzyHDeX2W28Ts24d7n37YGHiD+FCFCdS5D2Gzt6eykePaHZuU9SqVYuoqCimTZtGgwYNcHJyyvG+Pj4+psYTQphB6uXL3Bg2nPSbN7FwcsJn4UKcmj6frxksnZzwXbaMiHnziFn/OZGLPyH1ylW8P5yJhRmn5RdCiCfh0r49kZ9+Svr1UO5u3ITHAPMPSRGiqJAxeY+h0+mwcHDQ5GLqr+g+Pj7s3buXW7du0a5dO+JN6LPevn17nJycHnqpXr268bEPm977wX3Zyc0+QmhNSUsjevUarvfrT9j77xO3Ywfp4RFmO37C/v1ce+110m/exNrXl/Ibv873Au8BnZUVpSZNovT06WBlRdxPP3G9b18yIiM1ySOEEP+ls7LCc8gQAKLXrsGQkqJxIiEKLmnJK2L8/Pz47bffePHFF2nXrh27du3C2dn5sfuZ0l2zcePGvPfee6Snpxu37969m8qVK2fbVfPBPh9++CERERF4eXkZ93FxcaFatWqmPEUh8kXioUOEfTCTtKtXAUg6fJi7X30NgLWvLw7166trzdWrh7Wfn0k/yiiKwt0vNxA+ezYYDDjUr4/Pkk+wesi/n/zk9loPbMr7cXPMWFJOnCSkew98A5djV7my1tGEEALXTp2IXLqUjNt3uPfdd7j3yp+u7UIUNlLkFUG+vr7s3buXF198kbZt27Jr1y4AYmNjjVN4P+Dh4YGvr69J3TV79uzJjBkzGDhwIBMnTuT06dMsXryYhQsXGh+zdetWJk2axPnz5wFo06YN1apVo0+fPsydO5ewsDAmT57MiBEjsJXuYKIASQ8LI3zOHOJ3qv9uLN3d8Rg4gPTwcJL+/pvU8xdIv3GD2Bs3iN26VX1MSU8c6tXHoV49HJ6tj23FiugsLbM9vpKeTtisWdz7eiMArl274j19Grp/TWSkNcdGjSi/8WtuDn+DtGvXuPZ6T3w+/hjnFi9qHU0IUczprK3xHDyYsBnvE71qNW7duhWo908hCgop8oqosmXLZir0vL292bt3b5ZZLAcOHMiqVatMOrarqytBQUGMGDGCevXq4enpydSpUxlyvwsFqAXlhQsXjLctLS356aefGD58OI0bN8bR0ZF+/frx/vvvP9kTFcJMlLQ0Yj7/nMhly1GSksDCArfXX6fkmNFYurgYH6ePjyf5+HGS/vqbpCNHSDl5En1kFPG7dhF//wcVC2dn7J+pq7b21auPfY3q6Gxs0MfGcmvcOBJ/PwQ6HV4T3sR9wIACOcGJrb8/5Tdt5OaYsST98Qc3R4zAa8IE3AcEFMi8Qojiw7VrV6KWB5Jx5w73tm3DrVs3rSMJUeDolNwuxlZIxcXF4erqSmxsLC7/+uIGkJKSQkhICP7+/tjZ2WmUUDwgfw+RXxJ//52wmR8au2ba161L6alTsKta9bH7GlJTSTl5kqQjR0j6+wjJR49iSErK9BidrS32tWuTHh5G+vVQdA4O+Hw8D2cTZ8HVgpKeTtjMD7m3aRNQMFseH3jU+3txI6+FKOpi1q8nfPZHWPv6UmHnDnRW0m4hioecvr/LvwghRLGVfucO4XPmGlvgLD088JowAdfOndBZ5GxeKgtbWxyefRaHZ58FQMnIIOX8BZKP/E3S33+T9PcR9HfvkvTnnwBYeXvju3wZdlWq5M2TMjOdtTWlp0/DtkIFwj/6iNgtW0gPDS0wYwiFEMVTie7difpsBek3bhC3fTuunTtrHUmIAkWKPCFEsaOkpRG9fj1Ry5ajJCerXTN79aLkqJGZumbmhs7KCvsa1bGvUR33fv1QFIW0q1fVYi82lhKvdMGqZEkzPZP8odPpcO/bB5vyftwaN56kv//m2v0JWWwrVNA6nhCiGLKwt8c9oD+R8xcQFfgZLi+//NCx0EIUR7KEghCiWEk4eJCrnbsQOX8BSnIy9s88g/+W7yj93rtPXOBlR6fTYVuhAm49uuM5ZHChK/D+zalZM8pv/BrrsmVJv3GDa6+9TsKBg1rHEkIUU26v98TC1ZW0kBDig4Ly5ZxKejrFbKSTKKSkJU8IUSyk37lD+EdziP/5ZwAsPT3xmvAmrp07y0QiJrCtWJHymzdxc9Roko8c4cbQoZSaNAm3Xj2LzOuo1+tZt24dwcHBREREYDAYMt2/Z88ejZIJIf7N0skR9759iFryKVHLA3Fu2zbHXe1zI+XsWUKHDsW2wtP4rVubZ+cRwhykJU8IUaQZ0tKI+mwFV17qoBZ4Fha49e1DhZ07KNGlS5EpTPKTlbs75dauwbVLF9DrCZ85kyvt2hH56VLSrl/XOt4TGzNmDGPGjEGv11OjRg1q166d6SKEKDjce/fGwtGR1IsXSfj11zw7T8qFi4QOGIg+MoqkP/4g9fLlPDuXEOYgLXlCiCIr4cBBwmfOJO3aNQDs69VTZ82Uhb2fmIWNDd6zZ2FbsSKRn35K+vVQoj79lKhPP8W+dm1cOnXEpX17rNzdtY5qso0bN7J582ZeeuklraMIIR7D0tUVt969if7sM6KWLcepRQuz/3iXevUqoQMGoL93z7gtLiiIkk8/bdbzCGFO0pInhChyFIOBsA9ncWPQINKuXcPS05Myc+fg9+UXUuCZkU6nw2PgACod2E+ZuXNwfP55sLAg+cQJwj+YyaVmzbkxbDhxO3ZgSE7WOm6O2djY8LR8eROi0HDv1xedvT0pZ86QeOCAWY+ddu0aof36o4+OxrZaVbzefhuA+KDdZj2PEOYmRZ4QokhR9HruTJ7C3S++AJ3O2DXTtVMn6ZqZRywcHXHt1Ilyq1ZS8be9lJr0DnbVq0NGBgl793Jr/Jtcer4ptye9S+KhQyh6vdaRH+nNN99k8eLFMrmCEIWElbs7bq+9BqDOmmymf7tpN29yvX8AGZGR2FaqRLnVq3F9pQtYWpJ6/jxpoaFmOY8QeUG6awohigwlPZ3bE98hbscOsLSkzOxZuHbqpHWsYsWqZEnc+/XDvV8/Uq9cIfbHH4n74UfSb98mdutWYrduxcrLC5eXX8a1U0dsK1cuEMV3165dM93es2cPO3fupHr16lhbW2e6b8uWLfkZTQiRA+4B/bm7YQPJx46RdPhPHBs1fKLjpd++TWi//mSEhWHz1FOUW7vGuDaoQ4NnSTr0B/G7d+MxcKA54gthdtKSJ4QoEgxpadwcO04t8Kyt8VmwQAo8jdlWqIDX2LFU+GU3fhu+pESPHli4upIREUHMmjWEdHmFkE6diVq5kvQ7dzTN6urqmunyyiuv0Lx5czw9PbPcJ4QoeKy9vCjxv/8BEBUY+ETHSg+P4Hr/ANJv3cLarxzl1q7FysPDeL9LmzaAOi5PiIJKpxSz/ihxcXG4uroSGxuLy3/WxEpJSSEkJAR/f3/s7Ow0Spg7/fv35969e3z//ffGbd9++y29e/fmww8/5NSpU1nufxInT55kxIgR/PXXX5QsWZJRo0bx9v1+6tmJjo6mV69enDx5kujoaLy8vOjcuTOzZs3K8nd4oDD/PUT+MiQnc3PUaBIPHEBnY0PZJZ/g1Ly51rFENgxpaSTu20fsDz+S8OuvKOnp6h06HQ7PPot7//44t3gxV8d+1Pt7cSOvhSiO0m/f5nLbdpCejt9XX+HwTF2Tj5ERFcX1vv1Iu3oV67Jl8fvic6y9vTM/JjKSS82ag6Lw9N5fsS5d2lxPQYjHyun7u7TkFVGrVq2iV69eLF++nDfffNOsx46Li6NNmzb4+flx5MgR5s2bx/Tp01mxYsVD97GwsKBz58788MMPXLx4kXXr1vHLL78wbNgws2YTxY8+IZEbQ4aqBZ69Pb6fBUqBV4BZ2Njg3KoVZT9ZTMUD+yn9wfs4PPssKApJf/5JWqj2SzCEhIRw6dKlLNsvXbrEtfsztQohCh7rMmUo0aUzAFGBy03eP+PuXUIDBpB29SpW3t6UW7cuS4EHard0+2eeAWQCFlFwyZi8x1AUSE7R5tz2dpCboSpz585l2rRpbNy4kVdeecXsuTZs2EBaWhpr1qzBxsaG6tWrc/z4cRYsWMCQIUOy3cfNzY3hw4cbb/v5+fHGG28wb948s+cTxYc+Lo4bg4eQfOIEFk5O+K74DIf7H7yi4LN0dcWtWzfcunUj/dYtYrfvwKUALFvQv39/BgwYQMWKFTNtP3z4MKtWrWLv3r3aBBNCPJbHkCHc27KVxH37ST51GvuaNXK0n/7ePUIHDCT10iWsvLzwW7cWm7I+D328S5vWJB85QnxQEO59+5grvhBmI0XeYySnQFWNvnOc2wEO9qbtM3HiRJYtW8ZPP/1Ey5Ytc7xf+/bt2b9//0Pv9/Pz48yZMwAcOnSIZs2aYWNjY7y/bdu2zJkzh7t37+J2f2Dyo9y+fZstW7bQXFpcRC5l3L1L6MCBpJ49h6WrK76rVuX4w1wUPNY+PngOGax1DACOHTtGkyZNsmxv1KgRI0eO1CCRECKnbHx9cX25A7HbfiDqs0B8P/30sfvo4+MJHTSY1HPnsPTwoNy6tdj4+T1yH+fWrQmf/RFJR46QERWFlaenuZ6CEGYhRV4RsnPnTrZt20ZwcDAtWrQwad9Vq1aR/Ih1rP49u1xYWBj+/v6Z7i9VqpTxvkcVea+//jrbtm0jOTmZjh07smrVKpNyCgGQHhFB6IABpF2+on4gr1mDXeVKWscSRYROpyM+Pj7L9tjYWPQFfPkHIQR4DB2qjvv9JZiUCxceuT6qPiGRG4OHkHL6NJYlSlBu7Rpsn3rqseewLlMGu5o1STl1ivhfgnF7rYc5n4IQT0yKvMewt1Nb1LQ6tylq1apFVFQU06ZNo0GDBjg5OeV4Xx+fh3dJMKeFCxcybdo0Ll68yKRJkxg/fjzLli3Ll3OLoiH99m2uBwSQfj0Uq1KlKLd2LbZP+T9+RyFyqFmzZsyePZuvv/4aS0tLAPR6PbNnz+b555/XOJ0Q4nFsn3oK53Ztid+5i+jPPsNnwYJsH2dITubmsGEkHz+OhYsL5dasxq5Szn8wdG7TWi3ygoKkyBMFjhR5j6HTmd5lUis+Pj58++23vPjii7Rr146dO3fi7Oyco31N6a5ZunRpwsPDM93/4Hbpx8wwVbp0aUqXLk2VKlVwd3enadOmTJkyBe9sBjYL8V9poaFc79+fjNt3sPbxodz6ddiULat1LFHEfPTRRzRv3pzKlSvTtGlTAPbv309cXBx79uzROJ0QIic8hw0jfucu4nbuwnPkyCytc4bUVG6OGEHS339j4eREudWrsKtWzaRzuLRpQ+T8BST++Sf6e/ewLFHCjM9AiCcjs2sWMX5+fvz222+EhYXRrl27bLscZWfVqlUcP378oZcdO/5pzmzcuDH79u0j/cHU58Du3bupXLlyjsbjPWAwGABITU3N8T6i+Eq9coXrvXqTcfsONuXL47fhSynwRJ6oXr06J0+epHv37kRERBAfH0/fvn05f/48NWrIuE8hCgO7ypVxatkSFIXozzLP/m1IS+PmqFEk/n4InYMDvitWYF+zpsnnsPHzw7ZyZcjIIH7Pr+aKLoRZSEteEeTr68vevXt58cUXadu2Lbt27QLU8STHjx/P9FgPDw98fX1N6q7Zs2dPZsyYwcCBA5k4cSKnT59m8eLFLFy40PiYrVu3MmnSJM6fPw/Ajh07CA8P59lnn8XJyYkzZ87w1ltv0aRJE8qXL//Ez1kUbSnnzhE6cBD6mBhsK1Wi3JrVMshd5JnQ0FB8fX2ZNWtWtveVK1dOg1RCCFN5DhtGQnAwsT/9hOfIEdj4+qKkp3Nr3HgS9+1HZ2dHuc8Cc7We3gPObVqTeuEC8UFBlOhq/hnNhcgtackrosqWLcvevXuJioqibdu2xMXFsXfvXurWrZvpMmPGDJOP7erqSlBQECEhIdSrV48333yTqVOnZlo+ITY2lgsXLhhv29vbs3LlSp5//nmqVq3KuHHj6NSpEz/99JNZnq8oupJPnOB6v/7oY2Kwq1GDcuvXSYEn8pS/vz+RkZFZtkdHR2eZdEoIUXDZ16yBY9OmoNcTvWIlSkYGtya8RUJwMDobG3yXLVXX6XwCLm3aAJB48CD6hARzxBbCLHSKoihah8hPj1olPiUlhZCQEPz9/bGzM3HWE2F28vcQSX/9xY2hwzAkJWH/zDP4fhaIZQ7HmYri51Hv76awsLAgPDyckiVLZtp+/fp1qlWrRmJi4pNGzXPmei2EKOySjh7les9eYG2NU9OmJOzZg87amrJLP8WpWbMnPr6iKFx9qQNpISGU+fhjXF/uYIbUQjxcTt/fpbumEKJASjhwkJsjR6KkpODQqBG+y5Zi4eCgdSxRhI0fPx5Ql1CYMmUKDv/6/02v13P48GHq1KmjUTohRG44PPMMDg0bknT4MAl79oCVFT6LF5mlwAP1/cK5TRuiP/uM+KAgKfJEgSFFnhCiwInfs4dbY8aipKfj2LwZZRcvxkJac0UeO3bsGKD+Mn/q1ClsbGyM99nY2FC7dm0mTJigVTwhRC55Dh9G6OHDYGmJz8cf42ziWsKP49ymNdGffUbC/v0YkpOxsC8k07KLIk2KPCFEgRK3eze3xo2HjAyc27TB5+N56P71ZVuIvPLrr+rseAEBASxevFi6OQpRRDg2aoTPgvlYenji2LCB2Y9vV60a1j4+pN+6RcL+/cZxekJoSSZeEUIUGMmnTnN7wluQkYFLx474LJgvBZ7Id2vXrpUCT4gixuWll/KkwIP7XTZbtwYgPmh3npxDCFNJS142HqzfJrRVzOYEKvbSw8O5OWIESmoqjs2bUeaj2egsLbWOJYqpv//+m82bNxMaGkpaWlqm+7Zs2aJRKiFEQeXcpg0x69aRsHcvhrQ0LOQHSqExKfL+xcbGBgsLC27fvk3JkiWxsbFBp9NpHatYUhSFyMhIdDod1tbWWscRecyQnMzNN0aQERGBzdMV8Jk/Xwo8oZmNGzfSt29f2rZtS1BQEG3atOHixYuEh4fzyiuyDpYQIiv7OrWx8vIiIyKCxN9/x/mFF7SOJIo5KfL+xcLCAn9/f+7cucPt27e1jlPs6XQ6ypYti6V82S/SFIOB25PeJeXMGSzd3PBdvhxLJyetY4libNasWSxcuJARI0bg7OzM4sWL8ff3Z+jQoXh7e2sdTwhRAOksLHBu1Yq7X31FfNBuKfKE5qTI+w8bGxvKlStHRkYGer1e6zjFmrW1tRR4xUDU0mXE79oF1taUXfIJNr6+WkcSxdyVK1fo0EGdBt3GxobExER0Oh3jxo2jRYsWzJgxQ+OEQoiCyLlNG+5+9RUJwcEo6dPRSU8koaEcFXkP1g7KiQULFpgUYOnSpcybN4+wsDBq167NkiVLaNDg4QNjFy1axPLlywkNDcXT05P//e9/zJ4926yLZT/oIijdBIXIW7HbtxO1dCkA3tOn4VC/vsaJhAA3Nzfi4+MB8PHx4fTp09SsWZN79+6RlJSkcTohREHlUL8elm5u6O/eJemvv3B87jmtI4liLEdF3oO1gx44evQoGRkZVK5cGYCLFy9iaWlJvXr1TDr5pk2bGD9+PIGBgTRs2JBFixbRtm1bLly4gJeXV5bHf/XVV7zzzjusWbOG5557josXL9K/f390Op3JxaUQQlvJJ09y5933AHAPCKDEq69qnEgIVbNmzdi9ezc1a9akW7dujBkzhj179rB7925atmypdTwhRAGls7LCuVVL7n3zLXFBQVLkCU3pFBOnMFywYAF79+5l/fr1uLm5AXD37l0CAgJo2rQpb775Zo6P1bBhQ5599lk+/fRTQJ3V0tfXl1GjRvHOO+9kefzIkSM5d+4cwcHBxm1vvvkmhw8f5sCBA9meIzU1ldTUVOPtuLg4fH19iY2NlSmyhdBIelgY17p1JyMyEqfmzSm7bKlMtCKeWFxcHK6urk/8/h4TE0NKSgplypTBYDAwd+5cfv/9dypWrMjkyZONn30FmbleCyGEaRL27+fG4CFYenpS8be98tkmzC6n7+8mr5M3f/58Zs+enelDzs3NjZkzZzJ//vwcHyctLY0jR47QqlWrf8JYWNCqVSsOHTqU7T7PPfccR44c4c8//wTg6tWr7Nixg5deeumh55k9ezaurq7Gi6+M9xFCU4akJHUmzchIbCtWpMz8j+VDUBQo7u7ulClTBlA/l9555x1++OEH5s+fXygKPCGEdhwbNsTC2Rl9VBTJx49rHUcUYyZPvBIXF0dkZGSW7ZGRkcYxDDkRFRWFXq+nVKlSmbaXKlWK8+fPZ7tPz549iYqK4vnnn0dRFDIyMhg2bBjvvvvuQ88zadKkTGMKH7TkCSHyn3EmzbNnsXRzo6zMpCkKKL1ez9atWzl37hwA1apVo3PnzlhZyXxlQoiH09nY4NziRWK3/UB8UBAOJg5lEsJcTG7Je+WVVwgICGDLli3cvHmTmzdv8t133zFw4EC6du2aFxmN9u7dy6xZs1i2bBlHjx5ly5YtbN++nQ8++OCh+9ja2uLi4pLpIoTQRtSnnxL/88/qTJqfLsGmrI/WkYTI4syZM1SqVIl+/fqxdetWtm7dSr9+/ahYsSKnT5/WOp4QooBzbtMGgLjduzFxVJQQZmPyT5KBgYFMmDCBnj17kp6erh7EyoqBAwcyb968HB/H09MTS0tLwsPDM20PDw+ndOnS2e4zZcoU+vTpw6BBgwCoWbMmiYmJDBkyhPfeew8LC5NrViFEPon9aTtRy5YD4D1jhvy6KQqsQYMGUb16df7+++9MY8/79+/PkCFD+P333zVOKIQoyBybNEHn4EDG7TuknD6Nfc2aWkcSxZBJVZFer+fvv//mww8/JDo6mmPHjnHs2DFiYmJYtmwZjo6OOT6WjY0N9erVyzSJisFgIDg4mMaNG2e7T1JSUpZC7sE6avJLiRAFV/KJE9y5363afeAASnR9ReNEQjzc8ePHsx17/uGHH2aZbVoIIf7Lws4Op+bNAIgPCtI4jSiuTCryLC0tadOmDffu3cPR0ZFatWpRq1Ytk4q7fxs/fjwrV65k/fr1nDt3juHDh5OYmEhAQAAAffv2ZdKkScbHd+zYkeXLl7Nx40ZCQkLYvXs3U6ZMoWPHjrJothAFVPqdO9wYORIlLQ2nF1/Ey4R1N4XQQqVKlbL0MgGIiIjg6aef1iCREKKwcXnQZTMoSBoihCZM7q5Zo0YNrl69ir+//xOfvEePHkRGRjJ16lTCwsKoU6cOu3btMk7GEhoamqnlbvLkyeh0OiZPnsytW7coWbIkHTt25MMPP3ziLEII8zMkJXHjjRHoI6OwrVSJMvPmyUyaokCKi4szXp89ezajR49m+vTpNGrUCIA//viD999/nzlz5mgVUQhRiDg1a4bO1pb066GkXryI3f21pYXILyavk7dr1y4mTZrEBx98QL169bK04hX0iU1k7SAh8odiMHBrzFjid+/G0t2d8ps3y0QrIk89yfu7hYUFOp3OePvBR+ODbf++rdfrzZQ478hnnRDauzFiJAnBwXi+8QYlR4/SOo4oInL6/m5yS96DNek6deqU5QOxsHz4CSHyXuQnnxC/ezc6mUlTFAK//vqr1hGEEEWMS5vWJAQHE787SIo8ke9MLvLkg1AI8TixP/5IdOBnAJT+4H0cnnlG40RCPFrz5s3z9PhLly5l3rx5hIWFUbt2bZYsWUKDBg0e+vhFixaxfPlyQkND8fT05H//+x+zZ8/Gzs4uT3MKIczH6cUXwdqa1EuXSb0agu1TTz7USYicMrnIy+sPQiFE4ZZ8/Dh33psMgMfgQZTo0kXbQEJobNOmTYwfP57AwEAaNmzIokWLaNu2LRcuXMDLyyvL47/66iveeecd1qxZw3PPPcfFixfp378/Op2OBQsWaPAMhBC5YenigmOjRiTu3098UBC2w4ZqHUkUI7leWC4pKYnz589z8uTJTBchRPGVfvs2N0aOUmfSbNmSkuPGaR1JCM0tWLCAwYMHExAQQLVq1QgMDMTBwYE1a9Zk+/jff/+dJk2a0LNnT8qXL0+bNm14/fXX+fPPP/M5uRDiSTm3aQ3IUgoi/5lc5EVGRvLyyy/j7OxM9erVqVu3bqaLEKL4UfR60m/dUmfSjIrCtnJlfObOQWeR69+RhCgS0tLSOHLkCK1atTJus7CwoFWrVhw6dCjbfZ577jmOHDliLOquXr3Kjh07jGPis5OamkpcXFymixBCe84tW4KFBSlnz5J286bWcUQxYnJ3zbFjx3Lv3j0OHz7MCy+8wNatWwkPD2fmzJnMnz8/LzIKITRmSEkh/c4d0m/fJv32bTLu3CH91m3j7fTwcMjIAMDSwwPfZUuxyOX6mUJoJSkpCQcHB7MeMyoqCr1eb1wa6IFSpUpx/vz5bPfp2bMnUVFRPP/88yiKQkZGBsOGDePdd9996Hlmz57NjBkzzJpdCPHkrNzdcXj2WZIOHyY+aDceAwK0jiSKCZOLvD179rBt2zbq16+PhYUFfn5+tG7dGhcXF2bPnk2HDh3yIqcQIg/pY2NJv3VLLeT+XbzdL+z00dGPP4iVFbb+/njP/ABrH5lJUxQ+np6etGjRgk6dOtGpUydKly6tSY69e/cya9Ysli1bRsOGDbl8+TJjxozhgw8+YMqUKdnuM2nSJMaPH2+8HRcXh6+vb35FFkI8gnOb1veLvCAp8kS+MbnIS0xMNA4Ud3NzIzIykkqVKlGzZk2OHj1q9oBCiLwVtWIlkTmYzEHn4IB1GW+sy5RRL973/+tTBmtvb6y8vGShc1GonT9/nm3btrF582ZGjx5N7dq1jQVfzZo1c3VMT09PLC0tCQ8Pz7Q9PDz8oUXklClT6NOnD4MGDQKgZs2aJCYmMmTIEN577z0ssukGbWtri62tba4yCiHylnOr1oR/MJPk48dJDw/H+j8t+0LkBZOLvMqVK3PhwgXKly9P7dq1+eyzzyhfvjyBgYF4e3vnRUYhRB7JuHuXqMBAACzd3bH28blfwHlnKuCsy5TBwtU109qYQhQ15cqVY9SoUYwaNYrY2Fh27NjBtm3bmDdvHu7u7saCr3nz5ljm8AcNGxsb6tWrR3BwMF3uzzRrMBgIDg5m5MiR2e6TlJSUpZB7cL4Hi7ILIQoP61Je2NetS/KxY8Tv/gX33r20jiSKAZOLvDFjxnDnzh0Apk2bRrt27diwYQM2NjasW7fO3PmEEHkoZv16lKQkbKtWxX/Ld1LECXGfq6srr7/+Oq+//jrp6ens3buXH374gYCAAOLj41myZAm9euXsi9r48ePp168f9evXp0GDBixatIjExEQCAtRuW3379sXHx4fZs2cD0LFjRxYsWEDdunWN3TWnTJlCx44dc1xcCiEKFuc2bdQiLyhIijyRL0wu8nr37m28Xq9ePa5fv8758+cpV64cnp6eZg0nhMg7+thY7n65AQDPN4ZLgSfEQ1hbW9O6dWtat27NkiVLOHbsGBn3JxrKiR49ehAZGcnUqVMJCwujTp067Nq1yzgZS2hoaKaWu8mTJ6PT6Zg8eTK3bt2iZMmSdOzYkQ8//NDsz00IkT+cW7cmYs4ckv7+m4yYGKzc3bWOJIo4nWJi34+rV6/y1FNP5VWePBcXF4erqyuxsbG4uLhoHUcIzUR+upSoTz/FtmJF/Ld9L8sdiEJP3t//Ia+FEAVPyKv/I+XMGUq/PwO37t21jiMKqZy+v5v8re7pp5+mXLly9OnTh9WrV3P58uUnCiqEyH/6hARiPv8cuN+KJwWeEEIIkaec27QBID5ot8ZJRHFg8je7GzduMHv2bOzt7Zk7dy6VKlWibNmy9OrVi1WrVuVFRiGEmd39cgOGuDhsKlQwfugIIYQQIu84t2kNQOIff6CPjdU4jSjqTC7yfHx86NWrFytWrODChQtcuHCBVq1asXnzZoYOHZoXGYUQZqRPSCRm7VoAPIcNlWUPhBBCiHxg6++PbcWKkJFB/K+/ah1HFHEmF3lJSUkEBQXx7rvv8txzz1GrVi1OnDjByJEj2bJlS15kFEKY0b2NX6OPjcXGzw+X9u21jiNEgZOcnExSUpLx9vXr11m0aBFBQUEaphJCFAXSZVPkF5Nn1yxRogRubm706tWLd955h6ZNm+Lm5pYX2YQQZmZITiZ6jdqK5zFsGDork98ChCjyOnfuTNeuXRk2bBj37t2jYcOGWFtbExUVxYIFCxg+fLjWEYUQhZRzmzZELV1K4oED6BMSsXRy1DqSKKJMbsl76aWX0Ov1bNy4kY0bN/LNN99w8eLFvMgmhDCzu5s2oY+JwbpsWVxf7qB1HCEKpKNHj9K0aVMAvv32W0qVKsX169f5/PPP+eSTTzROJ4QozGwrVcTarxxKWhqJ+/dpHUcUYSYXed9//z1RUVHs2rWLxo0bExQURNOmTY1j9YQQBZMhJYXo1asB8Bg6BJ21tcaJhCiYkpKScHZ2BiAoKIiuXbtiYWFBo0aNuH79usbphBCFmU6nw+V+l8046QIu8lCu502vWbMmTZo0oXHjxjz77LNERESwadMmc2YTQpjRvW+/Qx8ZhVUZb0p07qx1HCEKrKeffprvv/+eGzdu8PPPP9Pm/heyiIgIWXNOCPHEnNu0BSBhz69kREVpnEYUVSYXeQsWLKBTp054eHjQsGFDvv76aypVqsR3331HZGRkXmQUQjwhQ1oa0StXAuA5eDA6GxuNEwlRcE2dOpUJEyZQvnx5GjZsSOPGjQG1Va9u3boapxNCFHZ2NapjV6sWSmoqMevWaR1HFFE6RVEUU3Z49tlnad68OS+88AJNmzbF1dU1r7LliZyuEi9EUXJ34ybCpk/HqlQpKuwOwkKKPFEEmfP9PSwsjDt37lC7dm0sLNTfQ//8809cXFyoUqWKOeLmKfmsE6Jgi//1V24OfwMLBwcqBP+ClUxiKHIop+/vJk+t99dffz1RMCFE/lLS0oha8RkAHoMGSYEnRA6ULl2a0qVLZ9rWoEEDjdIIIYoapxdewLZqVVLPnePuF19QcvRorSOJIiZXY/L2799P7969ady4Mbdu3QLgiy++4MCBA2YNJ4R4crE//EDG7TtYenpSotv/tI4jRIGXmJjIlClTeO6553j66ad56qmnMl2EEOJJ6XQ6PIcNAyDmiy/Rx8drnEgUNSa35H333Xf06dOHXr16cezYMVJTUwGIjY1l1qxZ7Nixw+whhRC5o2RkEPXZCgA8Bg7Ews5O40RCFHyDBg3it99+o0+fPnh7e6PT6bSOJIQogpxbt8Lm6QqkXb7C3Q0bjEWfEOZgcpE3c+ZMAgMD6du3Lxs3bjRub9KkCTNnzjRrOCHEk4n96SfSb9zA0s0Ntx7dtY4jRKGwc+dOtm/fTpMmTbSOIoQownQWFngOHcbtt94iZt163Pv0wcJRFkcX5mFyd80LFy7QrFmzLNtdXV25d++eOTIJIcxA0euJDlTH4rkPCMDCwUHjREIUDm5ubri7u2sdQwhRDLi0b4e1Xzn09+5xd6MsRSbMx+Qir3Tp0ly+fDnL9gMHDshYBSEKkLidu0i7dg1LV1fcXu+pdRwhCo0PPviAqVOnkpSUpHUUIUQRp7OywnPIUACi167FkJKicSJRVJjcXXPw4MGMGTOGNWvWoNPpuH37NocOHWLChAlMmTIlLzIKIUykGAxELV8OgHv/flg6SfcPIXJq/vz5XLlyhVKlSlG+fHmsra0z3X/06FGNkgkhiiLXTh2JXPopGbfvcO+bb3Hv01vrSKIIMLnIe+eddzAYDLRs2ZKkpCSaNWuGra0tEyZMYNSoUXmRUQhhovigINKuXMHC2Rm33vJhIYQpunTponUEIUQxorO2xnPwYMJmvE/06tWU6NFdljsST8ykxdD1ej0HDx6kVq1aODg4cPnyZRISEqhWrRpOTk55mdNsZIFYUdQpBgMhr3Ql9cIFPEeMoOSokVpHEiJfyPv7P+S1EKJwMaSmcqV1GzIiIig9Y4ZMliYeKqfv7yaNybO0tKRNmzbcvXsXGxsbqlWrRoMGDQpNgSdEcZCwZw+pFy5g4eiIe98+WscRotA6cuQIX375JV9++SXHjh3TOo4QogizsLXFY+AAAKJXrEBJT9c4kSjsTJ54pUaNGly9ejUvsgghnpCiKEQuWwaAW+/eWLq6apxIiMInIiKCFi1a8OyzzzJ69GhGjx5NvXr1aNmyJZGRkVrHE0IUUSW6d8fS3Z30W7eI3b5d6ziikDO5yJs5cyYTJkzgp59+4s6dO8TFxWW6CCG0k/Dbb6SePYfOwQH3/v20jiNEoTRq1Cji4+M5c+YMMTExxMTEcPr0aeLi4hg9erTW8YQQRZSFvT3uAf0BiA78DEWv1zaQKNRMnnjlpZdeAqBTp07odDrjdkVR0Ol06OV/SCE0oSgKUcvUGTXdXn8NKzc3jRMJUTjt2rWLX375hapVqxq3VatWjaVLl9KmTRsNkwkhijq313sSvWo1adeuEf/zz7jc/94thKlMLvJ+/fXXvMghhHhCiQcOknLyJDo7OzwCArSOI0ShZTAYsiybAGBtbY3BYNAgkRCiuLB0UsfTRy35lKjlgTi3a4fOwuSOd0KYXuQ1b948L3IIIZ6A2op3fyxejx5YeXpqnEiIwqtFixaMGTOGr7/+mjJlygBw69Ytxo0bR8uWLTVOJ4Qo6tz79CFm7TpSL10iYc8enFu10jqSMJPUqyFYeXpgmQ+zHstPA0IUAUmHD5N87Bg6Gxvc78/OJYTInU8//ZS4uDjKly9PhQoVqFChAv7+/sTFxbFkyRKt4wkhijhLFxfcevUCIGrZckxY7UwUYKlXQ7jety+hAQPQ37uX5+czuSVPCFHwRC1VW/FKdOuGtZeXxmmEKNx8fX05evQov/zyC+fPnwegatWqtJJf04UQ+cS9fz9iPv+clLNnSdy/H6dmzbSOJJ5A2vXrhPbvjz4qCquSJfPlnFLkCVHIJf31F0l//YXO2hqPwYO0jiNEkaDT6WjdujWtW7fWOooQohiycnPD7bXXiFm7lqhly3Fs2jTThIei8Ei7eYvr/QPIiIjAtuLTlFuzGssSJfL8vFLkCVHIRS1XZ9R07doV69KlNU4jROH0ySefMGTIEOzs7Pjkk08e+VhZRkEIkR88BgRwd8MGko8fJ+nwYRwbNdI6kjBR+p07hPbvT8adO9j4+1Nu7dp8m/1cijwhCrGkY8dI/P0QWFnhMXiw1nGEKLQWLlxIr169sLOzY+HChQ99nE6nkyJPCJEvrEqWpES3btzdsEFtzZMir1BJj4ggtH8A6TdvYl2uHOXWrc3XifFyVOTVrVs3x03ER48efaJAQoicM7bide6ETVkfjdMIUXiFhIRke10IIbTkMWggdzdvJunPP0k6cgSHevW0jiRyICM6mtCAAaRdv461jw9+69ZiXapUvmbI0eyaXbp0oXPnznTu3Jm2bdty5coVbG1teeGFF3jhhRews7PjypUrtG3bNq/zCiHuSz51isR9+8HSEs+hQ7WOI0SR8f7775OUlJRle3JyMu+//74GiYQQxZW1tzclunQBIGp5oLZhRI5k3L2rFnhXrmBVujTl1q/D+v5yPPkpR0XetGnTjJfIyEhGjx7NoUOHWLBgAQsWLOD3339n7NixhIeHmxxg6dKllC9fHjs7Oxo2bMiff/75yMffu3ePESNG4O3tja2tLZUqVWLHjh0mn1eIwiw9PJzbb08EwPXll7EpV07jREIUHTNmzCAhISHL9qSkJGbMmKFBIiFEceYxZDBYWpJ44ADJp05pHUc8gj4ujhsDB5F68SJWJUvit24tNmXLapLF5HXyvvnmG/r27Ztle+/evfnuu+9MOtamTZsYP34806ZN4+jRo9SuXZu2bdsSERGR7ePT0tJo3bo1165d49tvv+XChQusXLkSHx/ppiaKj7SbN7neqzdpISFYeXtTcuwYrSMJUaQoipLtEIUTJ07g7u6uQSIhRHFm4+uL68svA9KaV5DpExIIHTSYlLNnsfTwoNy6tdiUL69ZHpMnXrG3t+fgwYNUrFgx0/aDBw9iZ2dn0rEWLFjA4MGDCQgIACAwMJDt27ezZs0a3nnnnSyPX7NmDTExMfz+++9YW1sDUP4xL15qaiqpqanG23FxcSZlFKIgSQ0JITRgABlhYViXK4ff2jVYe3trHUuIIsHNzQ2dTodOp6NSpUqZCj29Xk9CQgLDhg3TMKEQorjyGDqE2B9+IGHPHlLOn8euShWtI4l/MSQmcmPIUFJOnsSyRAnKrVmDbYUKmmYyucgbO3Ysw4cP5+jRozRo0ACAw4cPs2bNGqZMmZLj46SlpXHkyBEmTZpk3GZhYUGrVq04dOhQtvv88MMPNG7cmBEjRrBt2zZKlixJz549mThxIpaWltnuM3v2bOleI4qElIsXCR0wEH1UFDYVKlBuzRqsS8nC50KYy6JFi1AUhQEDBjBjxgxcXV2N99nY2FC+fHkaN26sYUIhRHFl+9RTuLRvR9yOnUR99hllHzELsMhfhuRkbgx/g+SjR7FwcaHcmtXYVa6kdSzTi7x33nmHp556isWLF/Pll18CULVqVdauXUv37t1zfJyoqCj0ej2l/jPTTKlSpTh//ny2+1y9epU9e/bQq1cvduzYweXLl3njjTdIT09n2rRp2e4zadIkxo8fb7wdFxeHr69vjnMKURAknz7DjYED0cfGYlulCuXWrMZKuo0JYVb9+vUDwN/fnyZNmmBlJasMCSEKDo+hw4jbsZP4XT+TOvKK5i1FAgypqdwcMZKkP//EwtGRcqtWYletmtaxgFyuk9e9e3eTCjpzMRgMeHl5sWLFCiwtLalXrx63bt1i3rx5Dy3ybG1tsbW1zeekQphP0tFj3BgyBENCAna1alFu5Qos/9XCIIQwr+bNm3PlyhXWrl3LlStXWLx4MV5eXuzcuZNy5cpRvXp1rSMKIYohu8qVcGrVkoRfgolesYIyc+ZoHalYU9LSuDV6DIm//47OwQHflSuwr1VL61hGJk+88sCRI0f48ssv+fLLLzl27JjJ+3t6emJpaZllRs7w8HBKly6d7T7e3t5UqlQpU9fMqlWrEhYWRlpamskZhCjoEv/4g9BBgzAkJGBfvx7l1qyWAk+IPPbbb79Rs2ZNDh8+zJYtW4wzbZ44ceKhPygKIUR+8Bw2HIDYn7aTFhqqcZriS0lP5+b48ST89hs6Ozt8ly/H4ZlntI6ViclFXkREBC1atODZZ59l9OjRjB49mnr16tGyZUsiIyNzfBwbGxvq1atHcHCwcZvBYCA4OPihYx6aNGnC5cuXMRgMxm0XL17E29sbGxsbU5+KEAVawr593Bg6DCUpCccmTSi3ciWWTk5axxKiyHvnnXeYOXMmu3fvzvTZ0qJFC/744w8Nkwkhijv7GtVxbNYU9HqiV67UOk6xpGRkcOvtt0n4JRidjQ1ll36KY8MGWsfKwuQib9SoUcTHx3PmzBliYmKIiYnh9OnTxMXFMXr0aJOONX78eFauXMn69es5d+4cw4cPJzEx0TjbZt++fTNNzDJ8+HBiYmIYM2YMFy9eZPv27cyaNYsRI0aY+jSEKNDigoK4MWIkSmoqTi1aUHb5Mizs7bWOJUSxcOrUKV555ZUs2728vIiKitIgkRBC/ONBa96977eRfvu2xmmKF0Wv5/a77xK/cxdYW1N2ySc4NWmidaxsmTwmb9euXfzyyy9UrVrVuK1atWosXbqUNm3amHSsHj16EBkZydSpUwkLC6NOnTrs2rXLOBlLaGgoFhb/1KG+vr78/PPPjBs3jlq1auHj48OYMWOYOHGiqU9DiAIr9sefuP3OO6DX4/JSe8rMmYPu/pIhQoi8V6JECe7cuYO/v3+m7ceOHZN1WYUQmnN4pi4OjRqR9McfRK9aTempOZ/dXuSeYjBwZ9o04n74EaysKLtwAU7Nm2sd66FMLvIMBoNxjbp/s7a2ztSNMqdGjhzJyJEjs71v7969WbY1btxYusuIIuvuN98QNnUaKAqur7yC98wP0D1keRAhRN547bXXmDhxIt988w06nQ6DwcDBgweZMGECffv21TqeEELgOWwYoX/8wb1vv8Vj2FCsvWRJpbykKAphH3xA7LffgYUFPh/Pw7lVK61jPZLJ3TVbtGjBmDFjuP2v5uFbt24xbtw4WrZsadZwQhQnMZ9/QdiUqaAouPV8He8PZ0qBJ4QGZs2aRZUqVfD19SUhIYFq1arRrFkznnvuOSZPnqx1PCGEwKFhA+yfeQYlLY2Y1Wu0jlPkRcyZy72vN4JOR5k5H+HSrp3WkR5LpyiKYsoON27coFOnTpw5c8a43tyNGzeoUaMGP/zwA2XLls2ToOYSFxeHq6srsbGxuLi4aB1HCACiVqwkcsECANwHDMDrrQnodDqNUwlRuJj7/T00NJTTp0+TkJBA3bp1qVixohlS5g/5rBOi6EvYv58bg4eoLUvzP8alfXutIxVJadeucaWd+tp6fziTEq++qmmenL6/m9xd09fXl6NHj/LLL78YFy2vWrUqrQp4k6UQBZGiKER+8gnRywMB8BwxAs+RI6TAE6IAKFeuHOXKldM6hhBCZMvx+edx/d+rxH77HbcmvIXO2rrAdyEsjJJPnQLAvm5dzQs8U+RqMXSdTkfr1q1p3bq1ufMIUWwoikLER3OIWb8eAK8Jb+IxaJDGqYQonsaPH5/jxy643+ouhBBa0ul0eM+YgZKWRtwPP3Jz3HjKLvkE5xde0DpakZJy5iwAdtWra5zENDkq8j755BOGDBmCnZ0dn3zyySMfa+oyCkIUR4rBQNj773Nv4yYASk2ejHvvXhqnEqL4OnbsWI4eJ63sQoiCRGdpSZlZs1DS0onftYtbo8dQdvmyAjutf2GUcvZ+kfevlQUKgxyNyfP39+fvv//Gw8Mjy5TSmQ6m03H16lWzBjQ3GacgtKZkZHDnvcnEbtsGOh3eMz8oVM3/QhRUBfn9fenSpcybN4+wsDBq167NkiVLaNDg4Yvn3rt3j/fee48tW7YQExODn58fixYt4qWXXsrR+QryayGEMD8lPZ2b48apC3Tb2eG74jMcH/EeI3JGURQuNmiIIT4e/++3YlelitaRzDsmLyQkJNvrQgjTRS5apBZ4lpaUmTMH15c7aB1JCPEQN27cADBONJYbmzZtYvz48QQGBtKwYUMWLVpE27ZtuXDhAl7ZTHuelpZG69at8fLy4ttvv8XHx4fr169TokSJXGcQQhRtOmtrfBYs4OaoUST+to8bw4ZTbtVKHJ55RutohVr6zZsY4uPRWVtjW6GC1nFMYtISCunp6VSoUIFz587lVR4hirSMu3eJ+XIDgBR4QhRQGRkZTJkyBVdXV8qXL0/58uVxdXVl8uTJpKenm3y8BQsWMHjwYAICAqhWrRqBgYE4ODiwZk32056vWbOGmJgYvv/+e5o0aUL58uVp3rw5tWvXftKnJoQowixsbCj7ySc4PvccSlISNwYPIfnkSa1jFWoPxuPZVq6MLpt1wgsyk4o8a2trUlJS8iqLEEXe3Q1foaSkYFetGi4dctbtSgiRv0aNGsWKFSuYO3cux44d49ixY8ydO5fVq1ebPO48LS2NI0eOZJqB2sLCglatWnHo0KFs9/nhhx9o3LgxI0aMoFSpUtSoUYNZs2ah1+sfep7U1FTi4uIyXYQQxY+FrS1ll36KQ4MGGBITCR00mOQzZ7SOVWgV1vF4kIvF0EeMGMGcOXPIyMjIizxCFFmGpCTufvklAB6DBsoEDkIUUF999RXr1q1j6NCh1KpVi1q1ajF06FBWr17NV199ZdKxoqKi0Ov1lCpVKtP2UqVKERYWlu0+V69e5dtvv0Wv17Njxw6mTJnC/PnzmTlz5kPPM3v2bFxdXY2XJ+leKoQo3Czs7fFdvgz7Z57BEBfHjQEDSblwUetYhVLK/d6LdtWraZzEdCYvofDXX38RHBxMUFAQNWvWxNHRMdP9W7ZsMVs4IYqSe99tQX/vHta+vji3aaN1HCHEQ9ja2lK+fPks2/39/bGxscnz8xsMBry8vFixYgWWlpbUq1ePW7duMW/ePKZNm5btPpMmTcq0DERcXJwUekIUYxaOjviu+IzQAQNJOXmS0IAA/L74vNCNK9OSoiik3G8FtatWDIq8EiVK8KrMBCiESZSMDGLWrgXAY0AAOqtcLVEphMgHI0eO5IMPPmDt2rXY2toCanfIDz/8kJEjR5p0LE9PTywtLQkPD8+0PTw8nNKlS2e7j7e3N9bW1lhaWhq3Va1albCwMNLS0rItNG1tbY1ZhRACwNLJiXIrV3A9IIDUs+cI7a8WejbZ/IglssqIiEAfEwOWlthWqqR1HJOZ/E1z7f0vqkKInIvbuYv027exdHfH9ZVXtI4jhHiEY8eOERwcTNmyZY2TnZw4cYK0tDRatmxJ165djY99XO8VGxsb6tWrR3BwMF26dAHUlrrg4OCHFoxNmjThq6++wmAwYGGhjqq4ePEi3t7e+dKSKIQoOixdXSm3ejWh/fqTevEi1/sH4PflF9iULat1tALPOOlKhQpY2NlpnMZ0uWpOyMjIYO/evVy5coWePXvi7OzM7du3cXFxwcnJydwZhSjUFEUhevVqANz79C6UbxRCFCfZ9Vh5kq6P48ePp1+/ftSvX58GDRqwaNEiEhMTCQgIAKBv3774+Pgwe/ZsAIYPH86nn37KmDFjGDVqFJcuXWLWrFkmT/oihBAAVm5ulFu7hut9+pJ29Sqh/frj98XnWJcpo3W0Aq0wT7oCuSjyrl+/Trt27QgNDSU1NZXWrVvj7OzMnDlzSE1NJTAwMC9yClFoJR44SOr58+gcHHB7/XWt4wghHsPcPVZ69OhBZGQkU6dOJSwsjDp16rBr1y7jZCyhoaHGFjtQC8qff/6ZcePGUatWLXx8fBgzZgwTJ040ay4hRPFh5eFBubVrud63D+nXQ9UWvS++wLpU1rU6haowT7oCuSjyxowZQ/369Tlx4gQeHh7G7a+88gqDBw82azghioLoVasAcOv2PyxlMWMhiqWRI0c+tHvm3r17s2xr3Lgxf/zxRx6nEkIUJ9alvPBbt47rffqSHhqqTsby+XqsPD21jlYgGVvyCuGkK5CLJRT279/P5MmTs4wLKF++PLdu3TJbMCGKguRTp0g6fBisrHDv31/rOEKIHIiOjmbEiBFUq1YNT09P3N3dM12EEKKwsvb2pty6dVh5e6tdNwMGkHH3rtaxCpyMmBgy7twBwLZKMemuaTAYsl2Q9ebNmzg7O5sllBBFRfQqdSyea4cOWHt7a5xGCJETffr04fLlywwcOJBSpUrJmpZCiCLFpqwPfuvWcr1PX1IvXSJ0wED81q6R3kb/knJW7app4+eHpZPjYx5dMJlc5LVp04ZFixaxYsUKAHQ6HQkJCUybNo2XXnrJ7AGFKKzSrl0jPigIAPeBAzROI4TIqf3793PgwAHjzJpCCFHU2Pj5UW7dWq737UfquXOEDhpMubVrsJQGGwBSzt3vqllIx+NBLrprzp8/n4MHD1KtWjVSUlLo2bOnsavmnDlz8iKjEIVS9Jq1oCg4NW+OXSFcX0WI4qpKlSokJydrHUMIIfKU7VNPUW7NaixLlCDl9GlujhqNoihaxyoQCvt4PMhFkVe2bFlOnDjBu+++y7hx46hbty4fffQRx44dw8tLZugRAiAjMpLY778HwGPwIG3DCCFMsmzZMt577z1+++03oqOjiYuLy3QRQoiiwq5SJcqtXYPOzo6kP/4g8eDvWkcqEIpCkZerdfKsrKzo3bu3ubMIUWTEfPElSloa9nXqYF+vntZxhBAmKFGiBHFxcbRo0SLTdkVR0Ol02Y5LF0KIwsqualXcenQnZv3nRC1fjmOT54r1WGR9fDzp10MBsC2ka+RBLou8CxcusGTJEs7dXz+iatWqjBw5kipVqpg1nBCFkT4hgbtffw2Ax6CBxfqNUojCqFevXlhbW/PVV1/JxCtCiGLBfcBA7n71NclHjpD01184NmigdSTNPFgfz6qMN1ZubhqnyT2Ti7zvvvuO1157jfr169O4cWMA/vjjD2rWrMnGjRt59dVXzR5SiMLk3uZvMMTHY/PUUzj9pyVACFHwnT59mmPHjlG5cmWtowghRL6wLuWF6/9e5d7XG9XWvGJc5KU+WAS9EHfVhFyMyXv77beZNGkShw4dYsGCBSxYsIDff/+dd999l7fffjsvMgpRaChpacSsXw+Ax8AB6CxM/icmhNBY/fr1uXHjhtYxhBAiX3kOGgRWViQd+oOkY8e0jqOZojAeD3JR5N25c4e+fftm2d67d2/u3F80UIjiKvbHn8gID8fKywuXjh21jiOEyIVRo0YxZswY1q1bx5EjRzh58mSmixBCFEXWPj64du4EQFRgoMZptFNUijyTu2u+8MIL7N+/n6effjrT9gMHDtC0aVOzBROisFEMBqLXrAHAvV9fLGxsNE4khMiNHj16ADBgwD/rW+p0Opl4RQhR5HkOGULs1u9J/G0fyWfOYF+9utaR8pUhOZnUK1cBsKtazIq8Tp06MXHiRI4cOUKjRo0AdUzeN998w4wZM/jhhx8yPVaI4iJh717SrlzBwsmJEve/JAohCp+QkBCtIwghhCZs/Pxw6dCBuB9/JDowkLJLlmgdKV+lXrwIBgOWnp5YeZXUOs4TMbnIe+ONNwB1HaFly5Zlex8gv3aKYid65SoA3F5/DUsnJ43TCCFyy8/PT+sIQgihGc+hQ4j76Sfid/9CysWL2FWqpHWkfPNPV82qhX5mZZPH5BkMhhxdpMATxUnS0aMkHzuGztoatz59tI4jhHhCV65cYdSoUbRq1YpWrVoxevRorly5onUsIYTIc7ZPP41zmzYARH+2QuM0+auojMeDXBR5QoisHrTiuXbpgrWXl8ZphBBP4ueff6ZatWr8+eef1KpVi1q1anH48GGqV6/O7t27tY4nhBB5znPYUADidu4ktRh1YU85U3SKvFwthi6E+EfqpUsk/Por6HS4DwjQOo4Q4gm98847jBs3jo8++ijL9okTJ9K6dWuNkgkhRP6wq1oVpxdfJOHXX4lesZIys2dpHSnPKWlppFy6BBSNIk9a8oR4QtFr1gLg3KoVtv7+GqcRQjypc+fOMXDgwCzbBwwYwNn7XXmEEKKo8xw+DIDYH34g7eZNjdPkvdQrVyA9HQsXF6x9fLSO88SkyBPiCaSHhRH7008AeAzK+qVQCFH4lCxZkuPHj2fZfvz4cbykO7YQopiwr1ULxyZNQK83Dkspyv49Hq+wT7oC0l1TiCcSs249pKfj8Oyz2NeurXUcIYQZDB48mCFDhnD16lWee+45AA4ePMicOXMYP368xumEECL/eA4fRuLBg8Ru2YLn8GFYly6tdaQ8U5TG40EuizyDwcDly5eJiIjAYDBkuq9Zs2ZmCSZEQaePjeXe5s0AeAwepHEaIYS5TJkyBWdnZ+bPn8+kSZMAKFOmDNOnT2f06NEapxNCiPzjUL8+DvXrk/T330SvXkPp997VOlKeMbbkVa2qcRLzMLnI++OPP+jZsyfXr19HUZRM98naeKI4ufv1RgxJSdhWroxj06ZaxxFCmIlOp2PcuHGMGzeO+Ph4AJydnTVOJYQQ2vB8YzihAwZyb/NmPIcOwcrTU+tIZqfo9aRcuACAXfWi0ZJn8pi8YcOGUb9+fU6fPk1MTAx37941XmJiYvIioxAFjiElhZgvvgDUsXhFoe+2EEIVEhLCpfszrDk7OxsLvEuXLnHt2jUNkwkhRP5zaNwYu9q1UFJTiV67Vus4eSLt2jWU5GR0Dg7Y+PlpHccsTC7yLl26xKxZs6hatSolSpTA1dU100WI4iD2+23oo6OxKuONS7t2WscRQphR//79+f3337NsP3z4MP3798//QEIIoSGdTofn8OGA2osp4+5djROZn7GrZpUq6CwtNU5jHiYXeQ0bNuTy5ct5kUWIQkHR64leuwYAj/4B6KytNU4khDCnY8eO0aRJkyzbGzVqlO2sm0IIUdQ5NW+ObbWqKElJxHz+udZxzK6oTboCuRiTN2rUKN58803CwsKoWbMm1v/5glurVi2zhROiIIrfvZv066FYurpS4n+vah1HCGFmOp3OOBbv32JjY2XcuRCiWNLpdHgOG8at0WO4+8WXeAQEYOnionUssylqk65ALoq8V19Vv9QOGDDAuE2n06Eoiky8Ioo8RVGMa8W49eqFhYODxomEEObWrFkzZs+ezddff43l/W47er2e2bNn8/zzz2ucTgghtOHcqhW2FZ8m9dJl7m7YYOzCWdgpikLKuXNA0Zl0BXJR5IWEhORFDiEKhaTDh0k5cwadnR1ufXprHUcIkQfmzJlDs2bNqFy5Mk3vz5y7f/9+4uLi2LNnj8bphBBCGzoLCzyGDuP2hAnErFuPe9++WDg6ah3riaXfvIkhPh6dtTW2FSpoHcdsTB6T5+fn98iLEEXZg1a8Eq++ipWbm8ZphBB5oVq1apw8eZLu3bsTERFBfHw8ffv25fz589SoUUPreEIIoRmX9u2w8fNDHxvL3Y2btI5jFg/G49lWrlyk5lnI1WLoAGfPniU0NJS0tLRM2zt16mTysZYuXcq8efMICwujdu3aLFmyhAYNGjx2v40bN/L666/TuXNnvv/+e5PPK4QpUs6dI/HgQbC0xD2gv9ZxhBB5qEyZMsyaNUvrGEIIUaDoLC3xGDKEO++9R/Tatbj16omFnZ3WsZ5IURyPB7ko8q5evcorr7zCqVOnjGPxAOM6YaaOydu0aRPjx48nMDCQhg0bsmjRItq2bcuFCxfw8vJ66H7Xrl1jwoQJxq40QuQlQ1ISYTPeB8ClXTtsypbVOJEQQgghRP5z7dSRqKVLSb99m3vffIt7IR++UhTH40EuumuOGTMGf39/IiIicHBw4MyZM+zbt4/69euzd+9ekwMsWLCAwYMHExAQQLVq1QgMDMTBwYE1a9Y8dB+9Xk+vXr2YMWMGTz311COPn5qaSlxcXKaLEKZQ0tK4OWYsycePY+HigufIEVpHEkIIIYTQhM7aGo8hgwGIXrUKw3969RUmiqKQcuYMULSWT4BcFHmHDh3i/fffx9PTEwsLCywsLHj++eeZPXs2o0ePNulYaWlpHDlyhFatWv0TyMKCVq1acejQoYfu9/777+Pl5cXAgQMfe47Zs2dnWqzd19fXpIyieFP0em6/8w6J+/ejs7fH97NAbP39tY4lhBBCCKEZ165dsfLyIiM8nNit32sdJ9cyIiLQx8SApSW2lSppHcesTC7y9Ho9zs7OAHh6enL79m1AnZDlwoULJh0rKioKvV5PqVKlMm0vVaoUYWFh2e5z4MABVq9ezcqVK3N0jkmTJhEbG2u83Lhxw6SMovhSFIWwDz4gbsdOsLam7JIlONStq3UsIYQQQghNWdjY4DFIbWyJXrECJT1d40S5Y5x0pUKFQj+28L9MLvJq1KjBiRMnAGjYsCFz587l4MGDvP/++4/tOvmk4uPj6dOnDytXrsTT0zNH+9ja2uLi4pLpIkRORC5ezL2Nm0Cnw2fuHJyeb6J1JGGCr36ERt3hpGm/PRUbG36Ami9Ds17w+niYMAcWroPNO+HgUbh2C1ILbw+cJ9KiRQvu3buXZXtcXBwtWrTI/0BCCFEAlejWDUsPD9Jv3SL2p+1ax8mVojrpCuRi4pXJkyeTmJgIqN0mX375ZZo2bYqHhwebNpk2laqnpyeWlpaEh4dn2h4eHk7p0qWzPP7KlStcu3aNjh07GrcZDAb1iVhZceHCBSoUofUthHai164jOvAzAEpPn45L+/YaJxKmCI+GD5ZBUgp8thGWTtM6UcESnwgfrYS4RPVy/fbDH+vlAT5e4FNKvZQpBWX/dd3VKf9y55e9e/dmmTkaICUlhf3792uQSAghCh4Le3s8AvoT8fF8oj/7DNdOHdFZWmodyyRFddIVyEWR17ZtW+P1p59+mvPnzxMTE4Obm5txhs2csrGxoV69egQHB9OlSxdALdqCg4MZOXJklsdXqVKFU6dOZdo2efJk4uPjWbx4sYy3E2Zx77stRMyZA0DJ8eNx69Fd40TCVAvWqgUewO6DEJcALkWwGMmtz79XX5MKvjD7TbgVDrci4FbYv66HQ0oqRESrl2Pnsj+WsyNUKg8fvQmVCvlw1ZMnTxqvnz17NtOwAb1ez65du/Dx8dEimhBCFEglXnud6JWrSLt2jbhdu3Dt0EHrSCYxtuQVsUlX4AnWybt8+TJXrlyhWbNmuLu7G5dSMNX48ePp168f9evXp0GDBixatIjExEQCAgIA6Nu3Lz4+PsyePRs7O7ssC9GWKFECQBaoFWYR/8sv3JkyBQD3AQPwGDxI40TCVOevql0OAdxdISYWdu6DHi9pm6ugSEyGlZvV6yP7QMPa2T9OUdTX7nY43AxXi77bEer12/dvx8SqrYJHzsAbM+Cnz8DONv+ei7nVqVMHnU6HTqfLtlumvb09S5Ys0SCZEEIUTJZOjrj160vUJ0uIDvwMl/bt0VmYPBpMExkxMWTcuQOAbRXprkl0dDTdu3fn119/RafTcenSJZ566ikGDhyIm5sb8+fPN+l4PXr0IDIykqlTpxIWFkadOnXYtWuXcTKW0NBQLArJ/yyicEv84w9ujRsPBgOu/3sVr7cmmNw6LbQ3KxAMBnipOdSoCHNXwZbdUuQ98MU2uBsHfmWg0yOGl+l04FFCvdSsnP1jkpLVsXt934ZL19XXemohXmEkJCQERVF46qmn+PPPPylZsqTxPhsbG7y8vLAsZF2RhBAir7n37k3MmrWkXrpEfHAwLq1bax0pR1LOql1UbMqXx9LJUeM05mdy9TRu3Disra0JDQ3FwcHBuL1Hjx7s2rUrVyFGjhzJ9evXSU1N5fDhwzRs2NB43969e1m3bt1D9123bh3ff/99rs4rxAPJp05x840RKOnpOLdujff06VLgFUK//Qm//QXWVvDOYOhyf3WWP46rLU/FXXLKv1rxeoPVE9YrDvZQ7WmY+7Z6e/W3cODIkx1TS35+fpQvXx6DwUD9+vXx8/MzXry9vaXAE0KIbFi6uODWuxcA0csDc927L7+lnHvQVbPoteJBLoq8oKAg5syZQ9myZTNtr1ixItevXzdbMCHyS+qVK9wYPARDUhIOjRtRZv7H6Kxy3ZNZaESvh1nqXDn07QJ+PurkII3qqNu+/0WrZAXHVz9B1F0oWxpeMeMPrS0aQe9O6vUJcyA23nzH1sLs2bNZs2ZNlu1r1qxhzv3xukIIIf7h3q8fOgcHUs6eJXHfPq3j5EhRHo8HuSjyEhMTM7XgPRATE4OtbSEejCGKpfRbtwgdMBD9vXvY1apF2SWfYmFjo3UskQvf7FLH47k6w+g+/2zver+Y2bJbHWdWXKWkQeBG9frIXmprpzm9NwzK+8CdSJiy2LzHzm+fffYZVapUybK9evXqBAYGapBICCEKNis3N9xeew2AqPuzkxd0UuT9R9OmTfn888+Nt3U6HQaDgblz5/Liiy+aNZwQeSkjKorQAQPJCA/H5ukK+H4WWCT7ZBcHickwf616fXQfKPGv5TBfag62NnD5Opy+qE2+gmDTdnWWzDJe8Grbxz/eVA72sOhdsLSAbcHwwx7znyO/hIWF4e3tnWV7yZIluXN/kL4QQojM3Pv3AwsLko8dI+3GDa3jPJI+Pp7066EA2BbBNfIgF0Xe3LlzWbFiBe3btyctLY23336bGjVqsG/fPunGIgoNfXw8oYOHkHb9OtZlylBu9Wqs3Ny0jiVyacUmtYApVwb6dM58n7MjtLm/jv2W3fmfrSBITYPlX6vXh78ONtZ5c5661dSxfgDvLYSwyLw5T17z9fXl4MGDWbYfPHiQMmXKaJBICCEKPmsvLxyefRaA+KCC/YH7YH08qzLeRfb7n8lFXo0aNbh48SLPP/88nTt3JjExka5du3Ls2DFZiFwUCobkZG4MH07quXNYenhQbs1qrO/P5ioKn/Ao+GyTev2dwWqr3X89GH/2wx7I0OdftoLi211qN8pSntA9j2cZHdUHalVW1+GbMFed6bSwGTx4MGPHjmXt2rVcv36d69evs2bNGsaNG8fgwYO1jieEEAWWcxv1Azc+KEjjJI+W+mAR9CLaVRNyuU6eq6sr7733nrmzCJHnlPR0bo0dR/LfR7BwcqLcqpXYlC+vdSzxBD5eo84a+Ux1tWtmdpo9qy4FEHUX9v2lThRSXKRnwLKv1OvDeoBdHg85tbZSu22+NAT2/w3rv4eArnl7TnN76623iI6O5o033iAtLQ0AOzs7Jk6cyKRJkzROJ4QQBZdzq9aEfzCT5BMnSA8Lw7p0aa0jZauoj8eDXBZ5KSkpnDx5koiICAz/+Zm2U6dOZgkmBIBiMJhtUU3FYOD2u++R8Ntv6Ozs8P0sELsi2g+7uDh3RZ1wBWDycHVtt+xYW0HHF2HdVti6u3gVeVuC1AXMS7rB6y/nzzkrlIN3h8LUT2D2Z/B8Pajolz/nNgedTsecOXOYMmUK586dw97enooVK8rkYkII8RjWpbywr1uX5GPHiN/9C+59emsdKVtS5GVj165d9O3bl6ioqCz36XQ69Ppi2BdK5InoNWuJ+PhjLEuUwNrbG+syZe5fvLEyXi+DZYkSj13TTlEUwj+cRdyPP4KVFWUXL8KhXr18eiYir8wKVGfMfPkFqFf90Y/t2kYt8oIOQkISOGWdJLjIydDDpxvU60NeA3u7/Dt33y4QfEhdt3Dsh7B1ad6NBcwrTk5OPHt/fIkQQoiccW7TRi3ygoIKZJFnSE4m9cpVQIq8TEaNGkW3bt2YOnUqpWQck8gjhsREogIDwWBAHxODPiaGlDNnsn2szt5eLfj+Uwg+uG7l5UXU8kDubtgAOh1lZs/GqflD+vWJQmPvn7Dvb7VwmJiDYVK1KkMFX7hyA3bug27t8j6j1n4IhtDb4O4KvTvm77l1Opj3NrQeAKcvwSdfwIQB+ZshtxITE/noo48IDg7OtsfK1atXNUomhBAFn3Pr1kTMmUPSkSNkREdj5eGhdaRMUi9eBIMBS09PrL28tI6TZ0wu8sLDwxk/frwUeCJP3du2DUNcHNblylF28SLS79wh/dZt9b+3bxsv+qgolORk0q5cIe3KlewPZmmprpQNlJr8Hq4d86nPmsgzGXr4cLl6vd8r6qyaj6PTwStt4OPVahfGol7k6fWw5Ev1+uDu6hIH+a2UJ8weD2/MgKUb4MWGj29xLQgGDRrEb7/9Rp8+ffD29n5sTwEhhBD/sCnrg1316qScOUP8L8G49eiudaRM/umqWbSH7Jhc5P3vf/9j7969MpOmyDOKwcDd9epajO59+mBXtepDx84ZUlPJeFD4PSgEb/+rGAwLg/R0AEqOGY17r1759jxE3vlmJ1y8pi58PsqEniBdWqlF3qHj6myT3iXzKqH2ftoLV2+oawb27aJdjg4vwCu/q2Mhx82CnavAUYOC0xQ7d+5k+/btNGnSxGzHXLp0KfPmzSMsLIzatWuzZMkSGjRo8Nj9Nm7cyOuvv07nzp35/vvvzZZHCCHyknObNmqRFxRUgIu8ottVE3JR5H366ad069aN/fv3U7NmTaytMw+yGD16tNnCieIpYd8+0q5f55xrEz4L6cHbEeoCztmxsLXFpnz5h86Qqej1ZERFoyQnySyaRURiMsxfo14f01ct9HLKtzQ0rAWHT8L3v6hrxhVFBgMs+UK9PuBV7ccfzhgNh0/A9dswcxnMflPbPI/j5uaGu7u72Y63adMmxo8fT2BgIA0bNmTRokW0bduWCxcu4PWIrkLXrl1jwoQJNG3a1GxZhBAiPzi3aU3kwoUkHj6MPjYWS1dXrSMZpZwpHkWeydMWfv311wQFBfHdd9+xZMkSFi5caLwsWrQoDyKK4iZm/Xr0WLDQcy5bf7XmzY/UyTVyQ2dpiXUpLynwipDAryHyLvhls/B5TrzSRv3vlqDc/39V0O3cD5eug4sj9C8Ayxe4OsH8d9TrX/2kTshSkH3wwQdMnTqVpKQksxxvwYIFDB48mICAAKpVq0ZgYCAODg6sWbPmofvo9Xp69erFjBkzeOqppx57jtTUVOLi4jJdhBBCK7b+/thWrAgZGcTv+VXrOEZKWhoply4BUuRl8d577zFjxgxiY2O5du0aISEhxosMRhdPKuXCRZIO/cEf9i0JTVJ/Sf/9GGzeqXEwUSCERcKKzer1d4bkbrbGl5qDrbXa3fPMZbPGKxAMBlii9nYm4FW1wCoInqsLg7qp19+eB9H3NI3zSPPnz+fnn3+mVKlS1KxZk2eeeSbTxRRpaWkcOXKEVq1aGbdZWFjQqlUrDh16eLX7/vvv4+XlxcCBA3N0ntmzZ+Pq6mq8+Pr6mpRTCCHMzbmN+qtqQVoYPfXKFUhPx8LFBWsfH63j5CmTu2umpaXRo0cPLMy0dpkQ/xbz+XoU4Lsy4yEJ/MtCyE2YuRxebARe5utBJQqhj9dASirUrwHtm+XuGK5O0PI52PGb2ppXo6J5M2rtl9/h3FV13NuAV7VOk9lbg9TF6C9eg0nz4bP3H762oZa6dOlitmNFRUWh1+uzTFZWqlQpzp8/n+0+Bw4cYPXq1Rw/fjzH55k0aRLjx4833o6Li5NCTwihKec2bYhaupTEgwfRJyRi6eSodaRM4/GK+qRaJhd5/fr1Y9OmTbz77rt5kUcUYxnR0cT9+BOnbJ7lbFJ5bK1h4wIY+J46Bfv0T2DZdO3y3Y2FX/+EahWgUnnQ8neO1DQ4eQHiEtSFpm1ttMuSX85chm9/Vq8/auHznOjaWi3yfgiGd4eBlaV5MmpNUWDx/Va8/q+ok64UJHY2sOg96Dwcfj6gLmTfvb3WqbKaNm2aZueOj4+nT58+rFy5Ek9PzxzvZ2trm6vF2vV6Pen3J6cS+c/a2hpLyyLyBiTEf9hWqoiNnx9p16+T8NteXDt00DpSsRmPB7ko8vR6PXPnzuXnn3+mVq1aWSZeWbBggdnCieLl7qZNKGlpbCk3FtLhf+2gdEmY8xZ0Ggbbf4OgA9Dm+fzPlpIGvd9Si01QJ/t4tqZ6aVALalYCa5P/NeVcQhIcPQN/noI/T8Dx82qhB+qU9Cs+AE+3vDu/1hRFXTJBUaBTC6j7hO/NzRuAm4s6tu/gEfV2UbDnD/X/UQc7GFSwJjMzqv40vDkAPloB05dAozpQzlvrVHnH09MTS0tLwsPDM20PDw+ndOnSWR5/5coVrl27RseO/yxs+GCdPisrKy5cuGCW2a0VRSEsLIx79+498bHEkylRogSlS5cu8q0KovjR6XQ4t2lD9MqVxAftLhhFXjGZWRNyUeSdOnWKunXrAnD69OlM98kblMgtQ1oad7/6mhCrivyR/gwWFjCkh3pfjYrq9eVfw+TF6pdCl3weZ/TB0n++PAPExqvd4n75Xb1tbwd1q6oFX4Na8Ew1dVtuxcTCX6fUGQn/OgVnLoE+83rMeJSAtHQ4ckZtGVk9C6o8fn6GQunXw3DwqDoG761BT348G2vo2AI+/x627M6fIk+vh5Bb6oLsefFWqSj/zKjZu7O6AHpBNaS7OvnKX6dg/GzYtFBdzlJL7u7uXLx4EU9PT9zc3B75eRYTE5Pj49rY2FCvXj2Cg4ON3UANBgPBwcGMHDkyy+OrVKnCqVOnMm2bPHky8fHxLF682GxdMB8UeF5eXjg4OMjntwYURSEpKYmIiAgAvL2L8K8doth6UOQl7NuHITkZC3vt1tBR9HpSLlwAiv4aeZCLIu/XXwvODDmi6IjbsQN9VBTflZoCQPumUP5f42HH9oOd++DaLZizEj4cl3/ZtgXDlz+o15fPgCbPwNnL8OdJ9fLXKbgbp04Q8/sx9XFWllCzMjS439L3bM1HT/V/O+Kfgu7wSbh8Petjypa+f7za6n+f8lXXQRvwrvq6dB0JS6ZAy8bmfw20lKGHWYHq9f5dzdfq07W1WuT9fEBdliEv125TFBg9U127rmVjmPe2WqSb0/6/4dg5tevukALaiveApSUsmATtBqr/z6/YrP1yFgsXLsTZ2dl43ZxFz/jx4+nXrx/169enQYMGLFq0iMTERAICAgDo27cvPj4+zJ49Gzs7O2rUqJFp/xIlSgBk2Z5ber3eWOB5eHiY5Zgid+zvf+GNiIjAy8tLum6KIseuRnWsy5Qh/fZtEg4cwKV1a82ypF27hpKcjM7BARs/P81y5Jc87GAmRM4oikLM+s8JtyzDr5atQIFh//nCZ2errq31+ni14OrUAhrWzvtsl0PhnY/V6yN7wwv3W3xqV1Evg7ursxleDv2n6PvzpLrQ9rGz6uWzTWrLTWV/tdhrWAv8feHUBbWg++sk3AzPeu6KfupzfNAlNLu1AiuUg++XwrDp8Mdxdfzie8PUWQy1+GFerzd/i8zmHepyACVc1L+BudSp+s/EPrv2wattzXfs//ruZ7XAA7UFq+1AWDgJmtY3z/H/PRavdycoWQgmKCrnDdNHwVtz1XUPmz2rduXUSr9+/YzX+/fvb9Zj9+jRg8jISKZOnUpYWBh16tRh165dxslYQkND83Uyswdj8BwcNF5AUQD//B3S09OlyBNFjk6nw7l1a2LWryc+aLemRZ6xq2aVKuiKwb81naIU1ZWishcXF4erqyuxsbG4uBSwWQmKqcQ//yS0bz8C3SbzvV0vnqsLXz9kaOfEj2HjdrUVa+cqdSKHvJKcAp3fgAsh0LgubJiXswJGUdSi7a+T/xRxV248eh9LC6hR6Z8isH5N07rbpaXD1MXw9Xb19msd4IMxuVtiIDcuXYdpn6gtmfWqwyutoUNzcHvCLoMJSdC8N0Tdhekj1SUBzGnx57BgrTp5zYaPzXvsB26EqS1WCUlqAXb4hPp6gfojwVsDn3zinN+PqT+A2FrD/q+gVM7n69CUosDQqWpraqXy8ONnT/Zv2lzv75aWlty5cyfLQuXR0dF4eXmh1+tzHzKfPOq1SElJISQkBH9/f+zsnqBfuTAL+XuIoi7p6FGu9+yFhZMTFX8/iIWNNrPFhX80h5h163Dr3ZvSk9/TJIM55PSzTlryhOZi1n9OnK4Euxy6geHR3bbeHaa2hFy9oY4/eitnS0jlypTFaoFX0g0+mZzzFiqdDnxLq5eu9xfejoxRu6X9dUpt6bt+G6pX/Kc75zPVn6y7oI212tJZsby63MTG7WoXzsDpT15oPUpCkloorflW7VYJ8Pdp9TJ9CbzYELq0Vrso5ubLe+BGtcDzLwu9Opk3O8ArrdQi7+BRdQ2+0iXNe3y9Xh1zlpCkLvvw/mhIz1D/Rl9sg5Wb1QJtyWS1VTa3HrTi9ehQeAo8UP+tzH5THVd68RrMWwVT3tA6ldq7IDupqanYaPTlRAghCiv7OnWwKlmSjMhIkg4dwql5c01yGFvyqhb98XggRZ7QWFpoKAl79vCj4zBSDDZUe/rRXdhcndQWqmHTIPBrePkFqPrkE81lsXmnOr27hQV8MuXJ1+cr6a4uwv1SHr6v6XQw8H/qWMbRM9Xum11GqBOyPP0EBUR2FAV+/FUtVsKj1G2tn1O7U/51Crb+ok4WE3RQvbg4wksvqEVVg1o5W37iTqRaBEHuFz5/nHJl1OLr79PwfTAMe828x1/5jVrUO9qr3TMtLdXLzLHQ/Fm1q+KZS9BhKEwbCa+9ZHo32z9Pqn9rayvtx7XlhkcJmPuWOrZ01TfQopE67lULn3zyCaB2L1q1ahVOTv/M8KTX69m3bx9VqlTRJpwQQhRSOgsLnFu34u5XXxMXFKRJkacoCinnzgFgV73oz6wJICuaC03FfPElKYotP7iqExAMe+3xX3LbN4O2TdWWo7fnqa0l5nTuCkxepF4fHwDP1TXv8fNay8aw5VMoW0ptzXvlDXVSDnO5GAKvvwmjPlALvHJlYM0sWPWhOs5tcHfYsQJ2r4ERPcGnFMQlqq2LPcZBk9fVyXMuhjz6PB+vVhc+b1AL2ubhshld7w8P2LrbvMc9e1l9DqAWcOXKZL6/dRP4ebVa0CSnqGM/h0+He3GmneeT+zNqdmuX/bjNwqBlY+j5snr90HHtcixcuJCFCxeiKAqBgYHG2wsXLiQwMJCkpCQCAwO1Cyjo379/lsXqv/32W+zs7Jg/f3629z+JkydP0rRpU+zs7PD19WXu3LmP3Sc0NJQOHTrg4OCAl5cXb731FhkZGWbLJERh5NxG7dqUELwHRYN/D+k3b2KIj0dnbY2tGZahKQykyBOa0cfHE/vddwQ5dCXW4IyvN3R4IWf7fjBabR06eQHWbjFfpoQkeGOGugZd8wZqkVIYVfaHbcvV8XFxidBvojqT5JNISFJb7toPhkPH1HFk4wNg99rsZ/Ss5A9vD4YDX8HmRfB6B/VvdjsCln0FrQeox1qx+Z/WwAdOX4LvgtTr7w3L20lkXn5RbSU8f1UtzMwhJQ3GzlK7ZrZp8vAFv0t5wpfzYNJQdUbWnfug3SC1ZS4njpxRC3grS3ijl3mya2XyG7B+DkwYoF2GkJAQQkJCaN68OSdOnDDeDgkJ4cKFC/z88880bNhQu4Aii1WrVtGrVy+WL1/Om2++adZjx8XF0aZNG/z8/Dhy5Ajz5s1j+vTprFix4qH76PV6OnToQFpaGr///jvr169n3bp1TJ061azZhChsHOrXx7JECfT37pH011/5fv4Hi6DbVq6MzjqfJizQmBR5QjP3vvuO9KRUtpYYDMDgbuqX1Zwo5amOzwP4eA2E3nnyPIoCk+ar4/28S8Kid3PWrbCg8nSDrxaok6DoDeoYw6mf/DN2LqcURV1G4sW+avfJDL3asha8Hsb0ffxYOwsLdZbQjybAX1tg+XR1f2srtaj6cDk07A69JsC3u/4pJh8sfF4nj7vOuzqrXQRBXTPPHD5erY7n9HRTx5w9qki1sFBbsLcuVbva3omE18ar/1+nP+bHzgeteF3bqGNACzNH+39mr9Xar7/+ipubm/G2Xq/n+PHj3L17V8NUeUdRFAxJSZpcnmTut7lz5zJq1Cg2btxoXI7CnDZs2EBaWhpr1qyhevXqvPbaa4wePZoFCx4yMxgQFBTE2bNn+fLLL6lTpw7t27fngw8+YOnSpaSlpZk9oxCFhc7KCqdWLQGICwrK9/MXp0XQH5AxeUITil7P3S++ZL9dW+4opXF3fXhrx8P0eAm+/wX+OAHvLoAv5j5Zi88X2+CHPWqh+enUgr2YdE7Z2ahjwSr6wdxVsH4rXLupPr+cLCh/MUQtDv84od72KwMzRqsTquQ2z4OxiffiYPtedfzeX6fgwBH1MmmBOluorbXaEpgfuraBXfvVYnbSkCdbBuL3Y+rYMoA5E9RCLydqVYYdK9VZSr/ZpU4sdOAIfPJe1q6eoLZi7z2szsw6spC34hU0Y8eOpWbNmgwcOBC9Xk+zZs04dOgQDg4O/PTTT7zwwgtaRzQrJTmZC8/U0+TclY8eQZeLpRwmTpzIsmXL+Omnn2jZsmWO92vfvj379+9/6P1+fn6cOXMGgEOHDtGsWbNMk+20bduWOXPmcPfu3Uw/BDxw6NAhatasaVwe48E+w4cP58yZM9StW8j6/wthRi5t2hD77XfE//ILpadMQZePv6Qbx+MVg0XQH5AiT2giPjiYtFu3+KbUMgD6vwL2Js4cbWGhtg61HaB2WdsSlPu1zk5egA/UKLwzRJ2Mo6jQ6WBEL3XZiXGz4be/4JWRsOZD8PPJfp/4RFi0HtZ+p7YC2tmqxxjSw3zLVpRwUWfM7NVJbYndFgxbg/5ZbiLg1fxrnXqxoZonIlot0nK7fl1cArz5kdoK+XoHaPWcafs72sPHE9VJWd5doK6z2H6wOlHLK/9ZWuiT+zNqdm758L+jyJ1vvvmG3r3VRRl//PFHrl27xvnz5/niiy947733OHjwoMYJi7edO3eybds2goODadGihUn7rlq1iuTk5Ifeb/2vblxhYWH4+/tnuv9B8RYWFpZtkRcWFpapwPvvPkIUZ46NGmHh7Iw+Mork48dxeCZ/ZtlSFIWU+z/eSEueEHksZv3nHLN5jisWlbC3g75dcncc/7Iwtr86kcf7y9RxdDltOXkgNl6d8CItXR0/Nahb7rIUdO2bQdnSMOg9uHxdXQPws/czLyqvKGrr6KzP1IIH1K6VU0bkbcFVzhtG9VZbpE5fVKfT75TzH+efmI21OlPrlz+oPxbktsib9ok65rBcGfU1y62OLaBuNXVc31+n1P/+9pc6s6yzI5y5DLt/Vwt4cy4QL1TR0dGULq3+D79jxw66detGpUqVGDBgAIsXL9Y4nfnp7O2pfPSIZuc2Va1atYiKimLatGk0aNAg0yyoj+PjI7+ICKEVnY0NTi++QNwPPxL/c1C+FXkZERHoY2LA0hLbSpXy5ZwFQSEecSQKq+RTp0k+coTNzmpfvNdeerK13AZ3h2pPq93/Znxq2r6KAhPmwM0w8PWGj9/J20k+tFazEvywXO0aeDdOHQe3ead634UQ6DFWLSgiotXxYes+ghUf5F+Lmk4HNSurLbLW+fwT1IM1DXfth6SH/9D/UNv3qmP6LCzULrJPsu4hqAX5xoXq5DaWFursn+0Hw9GzaldOgI4vPtn6eiJ7pUqV4uzZs+j1enbt2kXr1mozalJSEpZP0pe3gNLpdFg4OGhy0eXiDdfHx4e9e/dy69Yt2rVrR3x8fI73bd++PU5OTg+9VK9e3fjY0qVLEx4enmn/B7cf/AjwX7nZR4jixOX+LJvxu3c/0ZhcUxgnXalQAQs7E7uNFWJS5Il8F/P551yyqsZxm0ZYWjx5y5m1FcydoH65/mGPulh6Tq36Rl3HzcYalk9T1+Er6kp5qrNddmiuTuzx1ly12Gs/CA6fVLtmThgIP6/J/di7wuiZamphm5QCuw6Ytm94FLy7UL3+Rk/zdfe1slQnt9m8WC36btyB/41SZ+EEtfVTmF9AQADdu3enRo0a6HQ6WrVqBcDhw4dlnbwCws/Pj99++42wsDCTCr1Vq1Zx/Pjxh1527NhhfGzjxo3Zt28f6enpxm27d++mcuXK2XbVfLDPqVOniIiIyLSPi4sL1YpRNzEhHsbx+efROTiQfvs2KafP5Ms5i9si6A9IkSfyVXp4BHE7d/KN0yDg/+3deVhUVR8H8O9lgBnZFZRFEFBLTYXUktTXTEVALcUNU98EIzHD1wortEywzV2zUrJc0FbLXMoFQRJzwXwT1NTE5UXUZFGQfWfu+8eVUZJB0Flg+H6eZx65M+ee+5vDHQ+/ueeeIw3Jc9bAl5vdO91JFt9ZId1Tdj9/nAYW3p4Je16oVEdz0UIhTb7y6mRp+9Bx6d473/5AfLSUPGjq3rumQhDu3Pe2rQETf4milCjn5gPdHrnTppr0RDdgz5fSbKNVSum5YQOkZSpI8yIjI7F27VqEhITg8OHDkMvlAACZTIbZs2frOTqq5uLigoSEBGRlZcHX1xf5+dIik3l5efckb1evSjf7tm3bFh07dlT7cHV1VdU/ceJEmJqaIjg4GGfOnMHmzZuxcuVKhIWFqcps27atRuLv4+ODxx57DC+88AJOnjyJvXv3Yu7cuQgNDVWdR0TNmZFCAYunnwYAFOhols3mtgh6Nd6TRzp167tvcV10xKEW0gwp08Zrru6wIGmo3ZXr0kyS77+qvmx2LhA6X1oOYMQg4N8jNBdHU2FkJA0F7OQuDTN8YWTjmcJeX/y9gRXRwKEkIDMbsLe9/z5f7ZDul5ObAivelq4Ka4OVBfDJXGm5h1+PShMEkfaMHTv2nucCAwP1EAnVxdnZGQkJCRg4cCB8fX3h6OiIhISEe2axDA4Oxtq1axtUt7W1NWJjYxEaGopevXrBzs4O8+bNQ0jInQ9fXl4eUlJSVNsymQw7d+7E9OnT0adPH5ibmyMwMBDvvffew71RIgNi5TMEBTExKIiNReuw1x9o2HZDNMflEwBAEHU1ILaRyM/Ph7W1NfLy8mBlZaXvcJoVZWkpLj4zEB8rX8Uu8+fxjBewcaFmj3HouDT0UBCAH1cCT3avJQ4lEDRb+sO8vQvwy+eARcNn8CYDNWoGkHQGeGc6EBJQd9lLV4BhIUBpGRA5Q5oRlPTnYf5//+STTxASEgKFQoFPPvmkzrIzZ858mDB1oq62KC0tRWpqKtzd3aFoRvenNFb8fVBzU1VYhAt9+0IsL4f7jh1QdNLeZCiVOTm40LcfAODRP/6AzMJca8fSlfr2dbySRzqT9/PPuJkvQ6z9aADA9Oc1f4x/9QLG+UnrjIUvlYa4yf8x7HDVt1KCp5BLC3MzwaO7jR4iJXnbYutO8ioqpSUpSsuk8y5wlO5iJM1bsWIFJk2aBIVCgRUrVqgtJwhCk0jyiIgaK5mFOcz/9S8U/vorCmJjtZrklZ6VhmqaurkZRILXELwnjwAAyuJipM+LwPm+/ZCzaZPGZzwSRRE5mzZhh9m/UQFTPN655tT9mjR3OtC6pXSVZdU3NV87kgws3yD9/MFrQOf22omBmq5nn5Em8zl7CTj3P/XlPvsaOHlOGka5NFwa/kpNV2pqKmxtbVU/q3v87391nBRERFQvlj7STfDavi+v9K/qoZrNa9IVgEkeASg5cwapo8cg94cfUJWTg8yPFuDqtGmovHlTY8coOnwEuZfSsdNiIgDg5QnaW6rAxgqYf/uL9tXfSksDANI9VjPfl4ZrjvOTHkT/1NJauu8NkJYtqE3yXcsYfPg64NhaN7EREREZAsuBAwFjY5RduICy1FStHae53o8HcLhmsyYqlcjZEI2sjz9GeYWArfZv4ISdH8amfYgnftuP/430h9PCBbDo3/+hj5WzcSNizMahULBCexdp0XFtGjYAGNJXWjA6fIk0Bf3M94Ebt6SJRuqalIVo1BBg7yFpYfi3XgLuXhqtuEQaplmllCbtGTFIf3GS5tw9Y+L9LF++XIuREBEZPpm1Ncy9vFB0+DAKYuMgn6ad2cSY5FGzU5GVhfTZc1B05AiOyZ/GGtcP8Hd5ayAHOGG5GmNb/YzJV+bi6tQQtAqcjNazZsHI9MHm1C+7dAm5B49ia5v5AKT7nLS9nrAgAO+/BiSeAJL/AkbPAP48Ly1QHRUpLSFApM6gpwBrSyDjpnQO/avXndc+/BxIvQY42ElDfskwJCcn19hOSkpCZWUlOnWS1lY5f/48ZDIZevXqVdvuRETUQJY+PreTvFjYaSHJq/j7b1SkXQEAyJvZGnkAh2s2SwX79yN1pD8u/Z6GSNsozGu1Bn+Xt4a9nTSFPABsqRiBWY/E4Ypxe+Rs3ITLAeNRdunSAx0vZ9NXSGgxDDdljmjdChjlo8E3UwfH1sCcadLPf56X/l34BtChnW6OT02X3FS6Nw+oOWRz/+/A1z9LPy+bLSWCZBj279+vejz33HMYMGAArl27hqSkJCQlJeHq1asYOHAghg8fru9QiYgMgqX3YMDICKVnzqD82t8arbsiKwtXXgwGACi6dYNxy5Yarb8pYJLXjChLS5Hx/gf43/TXsKliAkLa7MJR02dgLJPWq/t1I7DyHWD9R0Ara+BCoT1mOv6MPXZBKD13DqljxuLW95sbNClL5a1buLXjZ2wxlz5owWN1u8j2xGeB3h7Sz/8ewaF1VH/VC6Pv+Q0oKQVy8qRFzwHgxTE1r+6RYVm2bBkWLFiAlnf9UdCyZUt88MEHWLZsmR4jIyIyHMa2tjC7PTqiIE7NTfAPoDI7G1emvIjytDSYtG0L509WaqzupoRJXjNRev48Lo8LQMyWvzGt9S/4yvI/KIccfXsAMeuAt1++s5TA4D7Sc08/AZRWyrDSJBwLOnyDvDIFMiIj8ffMmai8datex839cQuOiV5IM3kEFmYiJj2nxTdZCyMjYO0HwJr37kzGQlQfT3QDXByBohLp/rw5y4AbOUBHVyB8qr6jI23Kz8/HjRs37nn+xo0bKCgo0ENERESGydJHGt6lqVk2K2/dkhK8S5dg7OCAdhujYeLkpJG6mxomeQZOFEXkfPMNEgNew+ybryGiVRTSjdvBwQ5YNQ/4dhnwiOu9+9nbAhsXScsRmBgDvxX3xIz2+3DCrB8K4vYh1X8Uio7+XvexKypw65tv8IOFdBVv0nMCrCy08S7rZm0J+PUHjLV8HyAZFkEARt0evvzBaiDmoHQOffy2tMYiGa5Ro0ZhypQp2Lp1K65du4Zr167hp59+QnBwMEaPHq3v8IiIDEb1UgolycmoyMx6qLqq8vNxNfgllJ0/D+PWreEavQGmzs6aCLNJYpJnwCpzcnBp2qtYtiIHU2224XfFIBjLRLz8PPDrJuDZgXUvY2BkBEwNALavBjq4AFnF5phj8yU2Os9HaWY2rkyZgqzlKyBWVNS6f/7eWJzKccQZ0ydgaiIieKyW3iiRloy+ff/ojdsXrl8PArprb81WaiQ+//xzDB06FBMnToSrqytcXV0xceJE+Pn5YfXq1foOj4jIYJjY26OFp7RwcsG+Bx+yWVVYiCsvTUXp2bOQ2dqiXfQGmLq5aSjKpqlRJHmrVq2Cm5sbFAoFvLy8cOzYMbVlv/zyS/Tv3x8tW7ZEy5Yt4e3tXWf55qrw8GF8478Ik/56E19b/gcVghz9eoqIWSdgzjRplsn66vYIsHMNMGE4IIoCvqsKwFuP7MF1Ixdkf/EFLk+chPK0tBr7iKKInI0b8ePtq3ijhgiwt9PkOyTSPndnoMftCbl6dZXWdyTDZ2ZmhtWrVyM7OxvJyclITk5GTk4OVq9eDXNzc32H16wFBQXB39+/xnNbtmyBQqHAsmXLan39YZw6dQr9+/eHQqGAi4sLFi9eXGf57Oxs+Pn5wcnJCXK5HC4uLpgxYwby8/M1FhORobkzZPPBkjxlURGuhkxD6alTkNnYoN369ZB36KDJEJskvSd5mzdvRlhYGCIiIpCUlARPT0/4+voiK6v2S7YJCQmYMGEC9u/fj8TERLi4uMDHxwd//63ZWXmaKrG8HEmRX+DFN8oxT7YIGcYusLepwOoI4JulQq1DM+vDrIU0M2VUpDT88WyhM2Y478K+VhNQ8uefSB01Grnbt6smZSlJPoHzfxUhUeENQRARMl5z75FIlyL/AwQMBT6bxyG/zY25uTk8PDzg4eHB5K6RWrt2LSZNmoSoqCjMmjVLo3Xn5+fDx8cHrq6uOH78OJYsWYLIyEh88cUXavcxMjLCyJEj8fPPP+P8+fOIjo7Gvn378PLLL2s0NiJDYukrJXnF//0vKnNyGrSvsqQEV6e/gpKkJBhZWcFl3VooOnHIDdAI1slbvnw5pk6diilTpgCQhsns2rUL69evx+zZs+8p/80339TYXrt2LX766SfEx8dj8uTJOom5sco7dxnLXj+Gb0sCUaGQw1iowktjlZg5xaRBV+7qMmwA8HgX4PWPgKMnjbFUPg/J7YdgeuqrUM6eg6KDh+AQGYGcjRuxxfxFAIBPPwEduWwBNVGPd5EeRIZKFKUZZPWhhaLu2wbqsnjxYkREROD777/HqFGjNBsYpL83ysvLsX79epiamqJr1644ceIEli9fjpCQ2tf0atmyJaZPn67adnV1xSuvvIIlS5ZoPD4iQ2Hq7Az5Y11QdvYvFMTHo+W4cfXaT1lWhmuhM1B87BiMzM3Rbu2XaNG1q5ajbTr0muSVl5fj+PHjmDNnjuo5IyMjeHt7IzExsV51FBcXo6KiAq1atar19bKyMpSVlam2DXHIhFIp4udlR/DRL27IlAUAAvCUezY+jLRFx3aav/Tg1EaasGX1d8CKDUB8SR+cbR+PN69Nx2O7dqE4OQkZWSJ+tZOGtXCIGxFR41VSCnQZpp9j/7VbGinSUOHh4Vi9ejV27tyJwYMH13u/oUOH4uDBg2pfd3V1xZkzZwAAiYmJePrpp2FqemfdH19fXyxatAi3bt2qscSGOtevX8fWrVsxYMCAesdI1BxZ+fjgxtm/UBAbV68kTywvx98zX0XRkSMQzMzg8uUXaOHhoYNImw69Jnk3b95EVVUV7O3tazxvb2+Pc+fO1auO8PBwODk5wdvbu9bXFyxYgPnz5z90rI2RUikibtNZrPpOhpPl/QAZ0FqWg3n/McJzI2wf+NvR+pDJgP/8G+jXE3j1Q+DKdUu8YfsVXsAmBFxfgm2Wr6NSMEVvD6DnY9qLg4iImpc9e/Zgx44diI+Px6BBDVv8dO3atSgpKVH7uomJiernjIwMuLu713i9+u+VjIyMOpO8CRMmYMeOHSgpKcFzzz2HtWvXNihOoubG0scHNz5eiaKjR1GVnw+ZlZXasmJFBa6FhaHwwAEICgVcoqJg1rOnDqNtGvQ+XPNhLFy4EN9//z0SEhKgUChqLTNnzhyEhYWptvPz8+Hi4qKrELWiqlKJX9b8idXbzJFSJV2WNhYr8ELXM3hjsQcszHV3q2XPx4DdXwDzVgJb4wRsRCCSHxmAiwWtAQAvP6+zUIiI6AG0UEhX1PR17Iby8PDAzZs3ERERgd69e8PCov5r87Rt27bhB3wAK1asQEREBM6fP6/6O4QzsxKpJ2/fHqYdO6D84iUU7t8P65Ejay0nVlbi77feQuG+eAimpnBe9RnMvXrrONqmQa9Jnp2dHWQyGTIzM2s8n5mZCQcHhzr3Xbp0KRYuXIh9+/bBo47Ls3K5HHK5YSxqVVFWiR9XnsKavba4rJSmm5WLJfB3/RMz3nJDu66P6yUuS3NgxdvA008Ccz8GThW6AQLQyR0Y9JReQiIionoShAcbMqkvbdu2xZYtWzBw4ED4+flhz549sLS0rNe+DRmu6eDgUOvfJ9Wv1cXBwQEODg7o3LkzWrVqhf79++Pdd9+Fo6NjveIkao6sfHxw82IU8mPjak3yxKoqXH/7bRTsiQFMTOD86Sew6NdPD5E2DXpN8kxNTdGrVy/Ex8erpjxWKpWIj4/HjBkz1O63ePFifPjhh9i7dy+eeOIJHUWrPyWF5fh6ySms+80J6ZAuR5uJhQh45Cymz+kIh/aN4xuMUUOkaeZnfgicPAeETXnwG+qJiIjUcXV1xYEDB1SJXkxMTL0SvYYM1+zTpw/eeecdVFRUqJ6Pi4tDp06d6nU/XjWlUgkANeYHIKJ7Wfr44ObqKBQdOgRlURGM7prVWFQqkR4RgfyffwGMjeG8YjkseK9rnfQ+XDMsLAyBgYF44okn0Lt3b3z88ccoKipSzbY5efJktG3bFgsWLAAALFq0CPPmzcO3334LNzc3ZGRkAAAsLCwaNGTjYSx57RgOn7NEL7d8PNXXAk8Nc4Olnean1y7MKcaGhacRfcwdNwUpmbUSc/Fvj/MImd0ZLZ0aR3J3t3ZOwLbPgLwCwEb9cGoiIqKH4uLigoSEBAwcOBC+vr6IiYkBAOTl5eHEiRM1ytra2sLFxaVBwzUnTpyI+fPnIzg4GOHh4Th9+jRWrlyJFStWqMps27YNc+bMUc0jsHv3bmRmZuLJJ5+EhYUFzpw5gzfffBP9+vWDWzNfmJnofuSdOsGkXTtUXLmCwt9+g9XQoQCktZcz3n8feVt+AoyM0HbpEliqmYuD7tB7kjd+/HjcuHED8+bNQ0ZGBh5//HHExMSobm6+cuUKjIzu3GMWFRWF8vJyjB07tkY9ERERiIyM1EnMh89ZIrmsC5JTgLUpgNH6SnQyuYCezrfg9aQcfZ91Ret2Ng9c/63r+fhi4Tl8c+oR5Am9AQGwFW8g8MlUBM/pBotWjS+5u5sgMMEjIiLtc3Z2rpHoOTo6IiEhAT169KhRLjg4uMGTn1hbWyM2NhahoaHo1asX7OzsMG/evBrLJ+Tl5SElJUW13aJFC3z55Zd4/fXXUVZWBhcXF4wePbrWJaGIqCZBEGDlMwTZa9chPzYWVkOHQhRFZC1ciNzvvgcEAU4LF8DKz0/foTYJgli9enUzkZ+fD2tra+Tl5cGqjpl76vLXkWs4tDcDx04b4UROW2TB/p4ybkZp6GGfBa8exug3rC3adW1z33ozU28hasEFbD7/GIoF6aqkI64j+Onr+PebHmhhYXqfGoiImi9N/P9uKOpqi9LSUqSmpsLd3V3tpGWkO/x9EN1RcuoULgeMh2BmhkePHMbNVauQ/aX0BY3jhx/AZswYPUeof/Xt6/R+Ja8p6tLXGV36OmPq7e3LJzNwOOY6jp1QIjmzDdLEdrisdMXldFdsSwewG3BEOh63vY4nu4vo52OPR72cYWQk3bB25UwWPlt8GdvTuqPs9pU7VyENL/tlY9yrHjCRO+ntvRIRERER6YKie3cYOzqiMj0d1155BUVHpHWzHSLmMcFrICZ5GuDm6QA3TwdMur2dmXoLR3al4fc/ynH8mi0uVrohXXBEerYj9iQASABaitnwtL4KM9MqxN7wQKXQBhCAR2WX8Ip/AUa87AGZsav+3hQRERERkQ4JggDLId64tekrVYJn//YctJwwQc+RNT1M8rTA3r0lRs1oiVG3t/OyinB092UcTSzE8cvWOFvmjluCLRLybaUCAuBpeg6hE6owZPJjqit8RERERETNiZWvL25t+goA0OaNWWg1ebKeI2qamOTpgHUbc/gGdYVvkLRdUliO43tTcOS3PGRkCxjlb4n+YzvrNUYiIiIiIn1r0bMnWoeFwdjWFjZjRus7nCaLSZ4etLAwxb/GdMK/OLSYiIj0pJnNu9Zo8fdAVJMgCLALmXr/glQno/sXISIiIkNRvbB3cXGxniMh4M7v4e6F2ImIHhav5BERETUjMpkMNjY2yMrKAgCYmZlBEHgvuK6Jooji4mJkZWXBxsYGMplM3yERkQFhkkdERNTMODg4AIAq0SP9sbGxUf0+iIg0hUkeERFRMyMIAhwdHdGmTRtUVFToO5xmy8TEhFfwiEgrmOQRERE1UzKZjEkGEZEB4sQrREREREREBoRJHhERERERkQFhkkdERERERGRAmt09edWLjubn5+s5EiIi0qTq/9e5uDT7OiIiQ1Xfvq7ZJXkFBQUAABcXFz1HQkRE2lBQUABra2t9h6FX7OuIiAzb/fo6QWxmX3kqlUpcv34dlpaWD7z4a35+PlxcXHD16lVYWVlpOELtYMy6wZh1gzHrRlOLWRRFFBQUwMnJCUZGzftuBPZ1jFmbGLNuMGbdaGox17eva3ZX8oyMjODs7KyRuqysrJrEyXA3xqwbjFk3GLNuNKWYm/sVvGrs6xizLjBm3WDMutGUYq5PX9e8v+okIiIiIiIyMEzyiIiIiIiIDAiTvAcgl8sREREBuVyu71DqjTHrBmPWDcasG00xZtKcpvj7Z8y6wZh1gzHrRlOMuT6a3cQrREREREREhoxX8oiIiIiIiAwIkzwiIiIiIiIDwiSPiIiIiIjIgDDJIyIiIiIiMiBM8tRYtWoV3NzcoFAo4OXlhWPHjtVZ/scff0Tnzp2hUCjQvXt37N69W0eRAgsWLMCTTz4JS0tLtGnTBv7+/khJSalzn+joaAiCUOOhUCh0FLEkMjLynhg6d+5c5z76bGcAcHNzuydmQRAQGhpaa3l9tPNvv/2G5557Dk5OThAEAdu3b6/xuiiKmDdvHhwdHdGiRQt4e3vjwoUL9623oZ8JTcVcUVGB8PBwdO/eHebm5nBycsLkyZNx/fr1Out8kPNLUzEDQFBQ0D3H9/Pzu2+9+mpnALWe24IgYMmSJWrr1HY7k3axr9Mu9nPawX5Own6u4TE3p36OSV4tNm/ejLCwMERERCApKQmenp7w9fVFVlZWreWPHDmCCRMmIDg4GMnJyfD394e/vz9Onz6tk3gPHDiA0NBQHD16FHFxcaioqICPjw+Kiorq3M/Kygrp6emqR1pamk7ivVvXrl1rxHDo0CG1ZfXdzgDw3//+t0a8cXFxAIBx48ap3UfX7VxUVARPT0+sWrWq1tcXL16MTz75BJ9//jl+//13mJubw9fXF6WlpWrrbOhnQpMxFxcXIykpCe+++y6SkpKwdetWpKSkYMSIEfettyHnlyZjrubn51fj+N99912ddeqznQHUiDU9PR3r16+HIAgYM2ZMnfVqs51Je9jX6Qb7Oc1jP3cH+7mGxdys+jmR7tG7d28xNDRUtV1VVSU6OTmJCxYsqLV8QECAOHz48BrPeXl5idOmTdNqnOpkZWWJAMQDBw6oLbNhwwbR2tpad0HVIiIiQvT09Kx3+cbWzqIoiq+++qrYoUMHUalU1vq6vtsZgLht2zbVtlKpFB0cHMQlS5aonsvNzRXlcrn43Xffqa2noZ8JTcZcm2PHjokAxLS0NLVlGnp+PYzaYg4MDBRHjhzZoHoaWzuPHDlSHDRoUJ1ldNnOpFns67SP/Zz2sZ/z1GxwarCf89RcYDrCK3n/UF5ejuPHj8Pb21v1nJGREby9vZGYmFjrPomJiTXKA4Cvr6/a8tqWl5cHAGjVqlWd5QoLC+Hq6goXFxeMHDkSZ86c0UV4NVy4cAFOTk5o3749Jk2ahCtXrqgt29jauby8HF9//TVefPFFCIKgtlxjaOdqqampyMjIqNGO1tbW8PLyUtuOD/KZ0La8vDwIggAbG5s6yzXk/NKGhIQEtGnTBp06dcL06dORnZ2ttmxja+fMzEzs2rULwcHB9y2r73amhmNfpzvs53SL/Rz7ufoy9H6OSd4/3Lx5E1VVVbC3t6/xvL29PTIyMmrdJyMjo0HltUmpVOK1115Dv3790K1bN7XlOnXqhPXr12PHjh34+uuvoVQq0bdvX1y7dk1nsXp5eSE6OhoxMTGIiopCamoq+vfvj4KCglrLN6Z2BoDt27cjNzcXQUFBass0hna+W3VbNaQdH+QzoU2lpaUIDw/HhAkTYGVlpbZcQ88vTfPz88OmTZsQHx+PRYsW4cCBAxg6dCiqqqpqLd/Y2nnjxo2wtLTE6NGj6yyn73amB8O+Tjf/B7Of0z32c+zn6svQ+zljfQdAmhUaGorTp0/fd6xwnz590KdPH9V237590aVLF6xZswbvv/++tsMEAAwdOlT1s4eHB7y8vODq6ooffvihXt+q6Nu6deswdOhQODk5qS3TGNrZkFRUVCAgIACiKCIqKqrOsvo+v55//nnVz927d4eHhwc6dOiAhIQEDB48WOvHf1jr16/HpEmT7juBgr7bmZqnptLXNfXPB/s53WM/pzuG3s/xSt4/2NnZQSaTITMzs8bzmZmZcHBwqHUfBweHBpXXlhkzZmDnzp3Yv38/nJ2dG7SviYkJevTogYsXL2opuvuzsbHBo48+qjaGxtLOAJCWloZ9+/bhpZdeatB++m7n6rZqSDs+yGdCG6o7vrS0NMTFxdX57WZt7nd+aVv79u1hZ2en9viNpZ0B4ODBg0hJSWnw+Q3ov52pftjX6ef8ZD+nfezn2M/VR3Po55jk/YOpqSl69eqF+Ph41XNKpRLx8fE1vqm6W58+fWqUB4C4uDi15TVNFEXMmDED27Ztw6+//gp3d/cG11FVVYU///wTjo6OWoiwfgoLC3Hp0iW1Mei7ne+2YcMGtGnTBsOHD2/QfvpuZ3d3dzg4ONRox/z8fPz+++9q2/FBPhOaVt3xXbhwAfv27YOtrW2D67jf+aVt165dQ3Z2ttrjN4Z2rrZu3Tr06tULnp6eDd5X3+1M9cO+Tj/nJ/s57WM/x36uPppFP6ffeV8ap++//16Uy+VidHS0ePbsWTEkJES0sbERMzIyRFEUxRdeeEGcPXu2qvzhw4dFY2NjcenSpeJff/0lRkREiCYmJuKff/6pk3inT58uWltbiwkJCWJ6errqUVxcrCrzz5jnz58v7t27V7x06ZJ4/Phx8fnnnxcVCoV45swZncQsiqI4a9YsMSEhQUxNTRUPHz4sent7i3Z2dmJWVlatMeu7natVVVWJ7dq1E8PDw+95rTG0c0FBgZicnCwmJyeLAMTly5eLycnJqhm6Fi5cKNrY2Ig7duwQT506JY4cOVJ0d3cXS0pKVHUMGjRI/PTTT1Xb9/tMaDPm8vJyccSIEaKzs7N44sSJGud4WVmZ2pjvd35pM+aCggLxjTfeEBMTE8XU1FRx3759Ys+ePcVHHnlELC0tVRuzPtu5Wl5enmhmZiZGRUXVWoeu25m0h32d9rGf0w72cxL2cw2LuVpz6eeY5Knx6aefiu3atRNNTU3F3r17i0ePHlW9NmDAADEwMLBG+R9++EF89NFHRVNTU7Fr167irl27dBYrgFofGzZsUBvza6+9pnp/9vb24rBhw8SkpCSdxSyKojh+/HjR0dFRNDU1Fdu2bSuOHz9evHjxotqYRVG/7Vxt7969IgAxJSXlntcaQzvv37+/1vOhOi6lUim+++67or29vSiXy8XBgwff815cXV3FiIiIGs/V9ZnQZsypqalqz/H9+/erjfl+55c2Yy4uLhZ9fHzE1q1biyYmJqKrq6s4derUezqxxtTO1dasWSO2aNFCzM3NrbUOXbczaRf7Ou1iP6cd7Ock7OcaFnO15tLPCaIoig96FZCIiIiIiIgaF96TR0REREREZECY5BERERERERkQJnlEREREREQGhEkeERERERGRAWGSR0REREREZECY5BERERERERkQJnlEREREREQGhEkeERERERGRAWGSR0REREREZECY5BFpSVBQEARBgCAIMDExgbu7O9566y2UlpbWKCcIArZv315rHQkJCRAEAbm5ufe85ubmho8//rjGdvXxqh/Ozs4PFHtkZCQef/zxB9qXiIiaD/Z1RI2Tsb4DIDJkfn5+2LBhAyoqKnD8+HEEBgZCEAQsWrRIK8d77733MHXqVNW2TCbTynGIiIiqsa8janx4JY9Ii+RyORwcHODi4gJ/f394e3sjLi5Oa8eztLSEg4OD6tG6dWu1ZRMSEtC7d2+Ym5vDxsYG/fr1Q1paGqKjozF//nycPHlS9S1pdHQ0ACA3NxcvvfQSWrduDSsrKwwaNAgnT55U1Vn9reiaNWvg4uICMzMzBAQEIC8vT2vvmYiI9It9Hfs6anyY5BHpyOnTp3HkyBGYmprqOxRUVlbC398fAwYMwKlTp5CYmIiQkBAIgoDx48dj1qxZ6Nq1K9LT05Geno7x48cDAMaNG4esrCzs2bMHx48fR8+ePTF48GDk5OSo6r548SJ++OEH/PLLL4iJiUFycjJeeeUVfb1VIiLSIfZ1RI0Dh2sSadHOnTthYWGByspKlJWVwcjICJ999lmD66ntfoPi4uJ7ngsPD8fcuXNV2x999BFmzpx5T7n8/Hzk5eXh2WefRYcOHQAAXbp0Ub1uYWEBY2NjODg4qJ47dOgQjh07hqysLMjlcgDA0qVLsX37dmzZsgUhISEAgNLSUmzatAlt27YFAHz66acYPnw4li1bVqM+IiIyDOzr2NdR48Mkj0iLBg4ciKioKBQVFWHFihUwNjbGmDFjGlzPwYMHYWlpWeO5Z5555p5yb775JoKCglTbdnZ2tdbXqlUrBAUFwdfXF0OGDIG3tzcCAgLg6OioNoaTJ0+isLAQtra2NZ4vKSnBpUuXVNvt2rVTdXoA0KdPHyiVSqSkpLDjIyIyQOzr2NdR48Mkj0iLzM3N0bFjRwDA+vXr4enpiXXr1iE4OLhB9bi7u8PGxqbGc8bG93587ezsVMe7nw0bNmDmzJmIiYnB5s2bMXfuXMTFxeGpp56qtXxhYSEcHR2RkJBwz2v/jI2IiJoP9nVEjQ/vySPSESMjI7z99tuYO3cuSkpK9B0OAKBHjx6YM2cOjhw5gm7duuHbb78FAJiamqKqqqpG2Z49eyIjIwPGxsbo2LFjjcfd36JeuXIF169fV20fPXoURkZG6NSpk27eFBER6Q37OvZ11DgwySPSoXHjxkEmk2HVqlU1nk9NTcWJEydqPIqKirQWR2pqKubMmYPExESkpaUhNjYWFy5cUN2r4Obmporp5s2bKCsrg7e3N/r06QN/f3/Exsbi8uXLOHLkCN555x388ccfqroVCgUCAwNx8uRJHDx4EDNnzkRAQACHrxARNRPs64j0j8M1iXTI2NgYM2bMwOLFizF9+nSYm5sDAMLCwu4pe/DgQa3FYWZmhnPnzmHjxo3Izs6Go6MjQkNDMW3aNADAmDFjsHXrVgwcOBC5ubnYsGEDgoKCsHv3brzzzjuYMmUKbty4AQcHBzz99NOwt7dX1d2xY0eMHj0aw4YNQ05ODp599lmsXr1aa++FiIgaF/Z1RPoniKIo6jsIIjIMkZGR2L59O06cOKHvUIiIiLSCfR01BRyuSUREREREZECY5BERERERERkQDtckIiIiIiIyILySR0REREREZECY5BERERERERkQJnlEREREREQGhEkeERERERGRAWGSR0REREREZECY5BERERERERkQJnlEREREREQGhEkeERERERGRAfk/IOrPsNILWYQAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"KL=0 collapsed sample (token 5 is the period): [5, 5, 5, 5, 5, 5, 5, 5]\n"
]
}
],
"source": [
"# viz: reward and diversity under two KL settings\n",
"fig, ax = plt.subplots(1, 2, figsize=(9, 3.4))\n",
"ax[0].plot(rew0, color=\"#d62728\", label=\"KL=0.0\"); ax[0].plot(rewK, color=\"#1E40FF\", label=\"KL=0.3\")\n",
"ax[0].set_xlabel(\"RLHF step\"); ax[0].set_ylabel(\"mean period reward\"); ax[0].set_title(\"reward\"); ax[0].legend()\n",
"ax[1].plot(div0, color=\"#d62728\", label=\"KL=0.0\"); ax[1].plot(divK, color=\"#1E40FF\", label=\"KL=0.3\")\n",
"ax[1].set_xlabel(\"RLHF step\"); ax[1].set_ylabel(\"distinct completions / batch\"); ax[1].set_title(\"sample diversity\"); ax[1].legend()\n",
"plt.tight_layout(); plt.show()\n",
"collapsed = sample0[0].tolist()\n",
"print(f\"KL=0 collapsed sample (token {PERIOD} is the period): {collapsed}\")\n",
"assert div0[-1] <= divK[-1] + 1e-9, \"the unleashed run should be no more diverse than the KL-leashed run\"\n",
"assert rew0[-1] >= rewK[-1] - 1e-9, \"the unleashed run should earn at least as much proxy reward\""
]
},
{
"cell_type": "markdown",
"id": "d15894dc",
"metadata": {},
"source": [
"> **Interpretation.** With no KL penalty the reward saturates and the samples collapse onto a near-constant high-period string: this is *mode collapse*, the toy version of the failure that bites production RLHF. The KL term pulls the policy back toward the reference and preserves diversity, at the cost of reward. This is why the KL coefficient $\\beta$ is the single most discussed knob in RLHF: it bounds how hard the optimizer can exploit the reward.\n",
"\n",
"> **Caveat:** the KL penalty bounds exploitation; it does not fix a wrong reward. If the reward model is systematically biased, a strong KL just makes the bias milder. Part 6 makes that failure concrete with an exactly-measurable example.\n",
"\n",
"DPO (Rafailov et al., 2023) folds the reward model out entirely. The KL-regularized objective has a closed-form optimal policy $\\pi^*(y\\mid x)\\propto\\pi_\\text{ref}(y\\mid x)\\exp(r(x,y)/\\beta)$; substituting the implied reward into the Bradley-Terry likelihood gives a loss on preference triples alone:\n",
"\n",
"$$\\mathcal{L}_{\\text{DPO}} = -\\mathbb{E}\\left[\\log\\sigma\\!\\left(\\beta\\log\\frac{\\pi_\\theta(y_w)}{\\pi_\\text{ref}(y_w)} - \\beta\\log\\frac{\\pi_\\theta(y_l)}{\\pi_\\text{ref}(y_l)}\\right)\\right]$$\n",
"\n",
"No reward model, no rollout, no separate KL term. The check that pins it: when the policy equals the reference, both log-ratios are zero, the margin is zero, and the loss is exactly $\\log 2$.\n"
]
},
{
"cell_type": "markdown",
"id": "9ce7ba03",
"metadata": {},
"source": [
"### Exercise 19.8 — The DPO loss\n",
"`Difficulty 3/5 · ~12 min`\n",
"\n",
"Fill in `dpo_loss`. Inputs are summed log-probabilities of chosen and rejected completions under the policy and the (frozen) reference. Compute the two log-ratios, form the $\\beta$-scaled margin, return $-\\log\\sigma(\\text{margin})$ averaged. The checks pin the policy-equals-reference identity ($\\log 2$) and the direction (pushing chosen up relative to rejected lowers the loss).\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "cc89c5d4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.923995Z",
"iopub.status.busy": "2026-06-11T20:07:45.923891Z",
"iopub.status.idle": "2026-06-11T20:07:45.928397Z",
"shell.execute_reply": "2026-06-11T20:07:45.927847Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] 19.8 DPO = log 2 at policy=ref: not attempted yet — fill in the TODO above, then re-run.\n",
"[ -- ] 19.8 DPO direction: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def dpo_loss(logp_chosen, logp_rejected, ref_chosen, ref_rejected, beta=0.1):\n",
" \"\"\"All four are (batch,) summed log-probs. Returns the mean DPO loss.\"\"\"\n",
" # TODO 1: log_ratio_chosen = logp_chosen - ref_chosen\n",
" # TODO 2: log_ratio_rejected = logp_rejected - ref_rejected\n",
" log_ratio_chosen = None\n",
" log_ratio_rejected = None\n",
" attempted(log_ratio_chosen, log_ratio_rejected)\n",
" # TODO 3: margin = beta * (log_ratio_chosen - log_ratio_rejected)\n",
" # TODO 4: return mean of -logsigmoid(margin)\n",
" raise NotImplementedError # remove once the TODOs are done\n",
"\n",
"def _dpo_at_ref():\n",
" # policy == reference -> every log-ratio is 0 -> margin 0 -> loss log(2)\n",
" z = torch.tensor([-2.0, -1.0])\n",
" loss = dpo_loss(z, z, z, z, beta=0.1)\n",
" assert abs(loss.item() - math.log(2)) < 1e-5, \\\n",
" f\"at policy=reference the DPO loss is log(2)=0.693, got {loss.item():.4f}\"\n",
"\n",
"def _dpo_direction():\n",
" ref = torch.tensor([-2.0])\n",
" # policy raises chosen, lowers rejected relative to ref -> positive margin -> loss < log 2\n",
" better = dpo_loss(torch.tensor([-1.0]), torch.tensor([-3.0]), ref, ref, beta=0.5)\n",
" assert better.item() < math.log(2), \\\n",
" f\"raising chosen and lowering rejected must drop the loss below log(2); got {better.item():.4f}\"\n",
"\n",
"check(\"19.8 DPO = log 2 at policy=ref\", _dpo_at_ref)\n",
"check(\"19.8 DPO direction\", _dpo_direction)"
]
},
{
"cell_type": "markdown",
"id": "1fae094f",
"metadata": {},
"source": [
"Hint 1 (conceptual)The DPO \"implicit reward\" of a completion is $\\beta(\\log\\pi_\\theta - \\log\\pi_\\text{ref})$. The loss is the Bradley-Terry loss on those implicit rewards: $-\\log\\sigma(\\text{reward}_w - \\text{reward}_l)$.\n",
"\n",
"Hint 2 (pseudocode)\n",
"\n",
"```python\n",
"lr_w = logp_chosen - ref_chosen\n",
"lr_l = logp_rejected - ref_rejected\n",
"margin = beta * (lr_w - lr_l)\n",
"return -F.logsigmoid(margin).mean()\n",
"```\n",
"\n",
"\n",
"Help — \"loss at policy=ref is 0, not 0.69\"You returned the margin or forgot the `-logsigmoid`. At zero margin, $\\sigma(0)=0.5$ and $-\\log 0.5 = \\log 2 \\approx 0.693$. If you got 0 you likely returned `margin.mean()` directly.\n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "3fc0d6c6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.929400Z",
"iopub.status.busy": "2026-06-11T20:07:45.929312Z",
"iopub.status.idle": "2026-06-11T20:07:45.932558Z",
"shell.execute_reply": "2026-06-11T20:07:45.932102Z"
},
"collapsed": true,
"jupyter": {
"source_hidden": true
},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] 19.8 DPO = log 2 at policy=ref\n",
"[ ok ] 19.8 DPO direction\n",
"[ ok ] DPO reduces to log(2) exactly when the policy has not moved off the reference\n"
]
}
],
"source": [
"#@title Solution { display-mode: \"form\" }\n",
"# solution: redefines dpo_loss; the checks below re-verify it.\n",
"def dpo_loss(logp_chosen, logp_rejected, ref_chosen, ref_rejected, beta=0.1):\n",
" log_ratio_chosen = logp_chosen - ref_chosen\n",
" log_ratio_rejected = logp_rejected - ref_rejected\n",
" margin = beta * (log_ratio_chosen - log_ratio_rejected)\n",
" return -F.logsigmoid(margin).mean()\n",
"\n",
"check(\"19.8 DPO = log 2 at policy=ref\", _dpo_at_ref, required=True)\n",
"check(\"19.8 DPO direction\", _dpo_direction, required=True)\n",
"print(\"[ ok ] DPO reduces to log(2) exactly when the policy has not moved off the reference\")"
]
},
{
"cell_type": "markdown",
"id": "0696e450",
"metadata": {},
"source": [
"GRPO (Shao et al., 2024), the DeepSeek variant, removes PPO's value network. For each prompt it samples a group of $G$ completions and uses the group's mean reward as the baseline, normalizing by the group's standard deviation:\n",
"\n",
"$$\\hat A_i = \\frac{r_i - \\text{mean}(r_{1:G})}{\\text{std}(r_{1:G})}$$\n",
"\n",
"That is a one-function change with no critic to train. We verify the group-relative advantages have mean zero per group.\n"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "92b2243f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.933462Z",
"iopub.status.busy": "2026-06-11T20:07:45.933378Z",
"iopub.status.idle": "2026-06-11T20:07:45.936261Z",
"shell.execute_reply": "2026-06-11T20:07:45.935986Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"group advantage row means (want ~0): [0.0, 0.0]\n",
"[ ok ] GRPO replaces the critic with a per-prompt group baseline\n"
]
}
],
"source": [
"def grpo_advantages(rewards):\n",
" '''rewards: (n_prompts, group_size). Group-normalized advantages, same shape.'''\n",
" mean = rewards.mean(dim=1, keepdim=True)\n",
" std = rewards.std(dim=1, keepdim=True) + 1e-8\n",
" return (rewards - mean) / std\n",
"\n",
"g_rew = torch.tensor([[1.0, 2.0, 3.0, 4.0], [10.0, 11.0, 12.0, 13.0]])\n",
"g_adv = grpo_advantages(g_rew)\n",
"check_shape(g_adv, (2, 4))\n",
"print(\"group advantage row means (want ~0):\", g_adv.mean(dim=1).tolist())\n",
"assert g_adv.mean(dim=1).abs().max() < 1e-5, \"each group's advantages must center at zero (the baseline is the group mean)\"\n",
"print(\"[ ok ] GRPO replaces the critic with a per-prompt group baseline\")"
]
},
{
"cell_type": "markdown",
"id": "5025f0c2",
"metadata": {},
"source": [
"> **Interpretation.** PPO, DPO, and GRPO all train on the same preference signal. PPO routes it through a learned reward model and a value network; DPO folds the reward model into a closed-form loss; GRPO keeps PPO's clip but swaps the value network for a group baseline. The differences are smaller than the marketing implies: each is one term of the same family swapped out.\n",
"\n",
"> **Key takeaways**\n",
"> - The reward model is a Bradley-Terry classifier on preference pairs; it learns to score chosen above rejected.\n",
"> - PPO against a reward with no KL leash collapses the policy onto a high-reward degenerate string. The KL term bounds exploitation.\n",
"> - DPO removes the reward model (loss is $\\log 2$ at policy=reference); GRPO removes the critic (group-mean baseline). Same data, different bookkeeping.\n"
]
},
{
"cell_type": "markdown",
"id": "f46c1eb3",
"metadata": {},
"source": [
"## Part 6 — Reward hacking: when the proxy and the truth diverge\n",
"\n",
"> **Objectives**\n",
"> - Build an MDP with two rewards: the *true* objective and a *proxy* that adds a tempting bonus.\n",
"> - Solve both exactly with value iteration; show the proxy-optimal policy never reaches the true goal.\n",
"> - Measure the *true* return of the proxy-optimal policy: exactly zero.\n",
"\n",
"Every algorithm in this chapter optimizes a scalar reward. Reward hacking is the gap between the reward you wrote and the outcome you wanted. We make it exact. A 1x7 corridor. The *true* reward pays $+1$ for reaching the goal at the far right and nothing else. The *proxy* reward adds a $+0.3$ bonus every time the agent steps onto a \"shiny\" cell near the start. A real designer might add that bonus as a well-meaning shaping term. We will solve the proxy MDP optimally and then audit the result under the *true* reward.\n"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "e8c5c0d5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.937065Z",
"iopub.status.busy": "2026-06-11T20:07:45.936998Z",
"iopub.status.idle": "2026-06-11T20:07:45.940308Z",
"shell.execute_reply": "2026-06-11T20:07:45.939936Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"corridor built · true goal at cell 6 · shiny bonus cell at 2\n"
]
}
],
"source": [
"CORRIDOR = 7\n",
"GOAL_C = CORRIDOR - 1 # true goal at the far right\n",
"SHINY = 2 # the \"shiny\" cell that pays a proxy bonus\n",
"BONUS = 0.3 # proxy-only bonus for stepping onto SHINY\n",
"\n",
"def build_corridor(use_proxy):\n",
" Sc, Ac = CORRIDOR, 2 # actions: 0=left, 1=right\n",
" Pc = np.zeros((Sc, Ac, Sc)); Rc = np.zeros((Sc, Ac, Sc))\n",
" for s in range(Sc):\n",
" for a in range(Ac):\n",
" ns = s if s == GOAL_C else (max(0, s - 1) if a == 0 else min(Sc - 1, s + 1))\n",
" Pc[s, a, ns] = 1.0\n",
" if s != GOAL_C:\n",
" if ns == GOAL_C: Rc[s, a, ns] += 1.0 # true reward: reach the goal\n",
" if use_proxy and ns == SHINY: Rc[s, a, ns] += BONUS # proxy-only bonus\n",
" return Pc, Rc\n",
"\n",
"P_true, R_true = build_corridor(use_proxy=False)\n",
"P_proxy, R_proxy = build_corridor(use_proxy=True)\n",
"print(\"corridor built · true goal at cell\", GOAL_C, \"· shiny bonus cell at\", SHINY)"
]
},
{
"cell_type": "markdown",
"id": "5a4ad7cd",
"metadata": {},
"source": [
"Solve both MDPs to optimality with the value iteration from Part 1. The true-optimal policy should march right toward the goal. The proxy-optimal policy is the interesting one.\n"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "cf32c5d9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.941056Z",
"iopub.status.busy": "2026-06-11T20:07:45.940989Z",
"iopub.status.idle": "2026-06-11T20:07:45.945676Z",
"shell.execute_reply": "2026-06-11T20:07:45.945175Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"true-optimal policy: ['right', 'right', 'right', 'right', 'right', 'right']\n",
"proxy-optimal policy: ['right', 'right', 'left', 'left', 'left', 'left']\n"
]
}
],
"source": [
"V_true_opt, pi_true_opt = value_iteration(P_true, R_true, gamma=0.95)\n",
"V_proxy_opt, pi_proxy_opt = value_iteration(P_proxy, R_proxy, gamma=0.95)\n",
"ACT = {0: \"left\", 1: \"right\"}\n",
"print(\"true-optimal policy: \", [ACT[a] for a in pi_true_opt[:GOAL_C]])\n",
"print(\"proxy-optimal policy:\", [ACT[a] for a in pi_proxy_opt[:GOAL_C]])"
]
},
{
"cell_type": "markdown",
"id": "f6548987",
"metadata": {},
"source": [
"> **Stop and think:** the proxy-optimal policy near the shiny cell points back toward it, not toward the goal. From the start, following the proxy policy, will the agent ever reach the goal? Trace it on paper before running the next cell.\n"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "06251139",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.946528Z",
"iopub.status.busy": "2026-06-11T20:07:45.946455Z",
"iopub.status.idle": "2026-06-11T20:07:45.948903Z",
"shell.execute_reply": "2026-06-11T20:07:45.948500Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"true-optimal reaches the goal: True\n",
"proxy-optimal reaches the goal: False\n"
]
}
],
"source": [
"def reaches_goal(pi, start=0, max_steps=50):\n",
" s = start\n",
" for _ in range(max_steps):\n",
" s = max(0, s - 1) if pi[s] == 0 else min(CORRIDOR - 1, s + 1)\n",
" if s == GOAL_C:\n",
" return True\n",
" return False\n",
"\n",
"print(\"true-optimal reaches the goal: \", reaches_goal(pi_true_opt))\n",
"print(\"proxy-optimal reaches the goal:\", reaches_goal(pi_proxy_opt))"
]
},
{
"cell_type": "markdown",
"id": "c0ac99a4",
"metadata": {},
"source": [
"Now the measurement that names the failure. Evaluate each policy under the *true* reward (exact policy evaluation from Part 2). The proxy-optimal policy maximized the proxy. What did it earn on the objective we actually cared about?\n"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "e3974348",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.949693Z",
"iopub.status.busy": "2026-06-11T20:07:45.949623Z",
"iopub.status.idle": "2026-06-11T20:07:45.952315Z",
"shell.execute_reply": "2026-06-11T20:07:45.951919Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"true return from the start:\n",
" true-optimal policy : 0.774\n",
" proxy-optimal policy: 0.000\n",
"\n",
"the proxy-optimal policy got the maximum proxy reward and ZERO true reward.\n"
]
}
],
"source": [
"def corridor_policy_eval(pi, P, R, gamma=0.95):\n",
" idx = np.arange(CORRIDOR)\n",
" P_pi = P[idx, pi]\n",
" r_pi = np.einsum(\"st,st->s\", P_pi, R[idx, pi])\n",
" return np.linalg.solve(np.eye(CORRIDOR) - gamma * P_pi, r_pi)\n",
"\n",
"true_value_true_pol = corridor_policy_eval(pi_true_opt, P_true, R_true)[0]\n",
"true_value_proxy_pol = corridor_policy_eval(pi_proxy_opt, P_true, R_true)[0]\n",
"print(f\"true return from the start:\")\n",
"print(f\" true-optimal policy : {true_value_true_pol:.3f}\")\n",
"print(f\" proxy-optimal policy: {true_value_proxy_pol:.3f}\")\n",
"assert true_value_proxy_pol < 1e-9, \\\n",
" \"the proxy-optimal policy earns ZERO true reward: it never reaches the goal\"\n",
"assert true_value_true_pol > 0.5, \"the true-optimal policy earns substantial true reward\"\n",
"print(\"\\nthe proxy-optimal policy got the maximum proxy reward and ZERO true reward.\")"
]
},
{
"cell_type": "markdown",
"id": "dcd38a8a",
"metadata": {},
"source": [
"> **Interpretation.** The proxy-optimal policy is not buggy. It optimally maximizes the reward it was given. The reward was wrong. A $+0.3$ shaping bonus, added with good intentions, turned a goal-reaching task into a stay-near-the-shiny-cell task, and the optimizer did exactly what was asked. True return: $0.000$. This is reward hacking with no neural network, no approximation error, no noise. It is a property of the objective, and it is why the reward signal, not the algorithm, is the bottleneck of every method in this chapter.\n"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "b983dd30",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:45.953015Z",
"iopub.status.busy": "2026-06-11T20:07:45.952949Z",
"iopub.status.idle": "2026-06-11T20:07:46.131794Z",
"shell.execute_reply": "2026-06-11T20:07:46.131359Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAD5CAYAAADbedYGAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAATx1JREFUeJzt3XdYFNf6B/Dv0pbepIkgYEexghUVCxENUVFjTyhqNPZecy3ExBJ7RLHkXjAary2KxtiNldgVo7EjtqgoFhBQVPb8/uC3c12Wqugy+P08zz4Pe/bMmXfOzgzvzp45qxBCCBARERERyYyergMgIiIiInobTGSJiIiISJaYyBIRERGRLDGRJSIiIiJZYiJLRERERLLERJaIiIiIZImJLBERERHJEhNZIiIiIpIlJrJEREREJEtMZKlIKBQKTJky5aNa95QpU6BQKDTK3N3dERoa+sFjuX37NoyNjREbG/tB1qfL97u4WLJkCcqWLYuMjIwiazM0NBTm5uZF1l5xpz6GkpKSSsR65Cg6OhoKhQInT57UdSiS/fv3Q6FQYP/+/boOpVA+huO3WbNmaNasma7D0MBElgps27ZtH33yUlx9++23qF+/Pnx9fXUdykcjNDQUL1++xNKlS3UdSrE3bdo0xMTE6DoMoneWnp6OKVOmyC7JLsmYyFKBbdu2DeHh4Tm+9vz5c/zrX//6wBEVP5cvX8by5cs/6DofPnyIFStW4Ouvv/6g6/3YGRsbIyQkBHPnzoUQQtfhFGtMZKmkSE9PR3h4OBPZYoSJbBFQqVR48eKFrsPQKWNjYxgYGOg6DJ1TKpUwNDT8oOtctWoVDAwM0LZt23zrpqenf4CI3t3r16/x8uXLHF9LS0v7wNHkrkuXLrh58yb27dun61CIiD5KTGT/n3oM1aVLl9ClSxdYWlqiVKlSGDp0qFaSqlAoMGjQIPzyyy+oVq0alEolduzYAQA4c+YM2rRpA0tLS5ibm6Nly5Y4evSotOwff/wBPT09TJo0SaPN1atXQ6FQIDIyEgDg5+eHmjVr5hhr5cqVERAQkO82PXjwAL1794ajoyOMjY1Rs2ZNrFixQqPOjRs3oFAoMHv2bMybNw9ubm4wMTGBn58fzp8/L9ULDQ3FokWLpO1XP97skzeHHaj788qVK/jiiy9gZWUFe3t7TJw4EUII3L59G+3bt4elpSWcnJwwZ84cjbhevnyJSZMmwdvbG1ZWVjAzM0OTJk3eOmFQj7lau3YtJkyYACcnJ5iZmaFdu3a4ffu2Vv3169fD29sbJiYmsLOzwxdffIF//vkn3/XkNEb26dOnGD58ONzd3aFUKuHi4oLg4GAkJSUhNTUVZmZmGDp0qFZbd+7cgb6+PqZPn57nOmNiYlC/fn2tsVnNmjWDl5cXTp06haZNm8LU1BQTJkwAAGRkZGDy5MmoUKEClEolXF1dMWbMGK3xnhkZGRg+fDjs7e1hYWGBdu3a4c6dO/n2A1Dw9/DNfXD+/PkoX748lEolLly4IO1HFy5cQI8ePWBjY4PGjRsDyEp2p06dKtV3d3fHhAkTNLZhxIgRKFWqlMYV08GDB0OhUODHH3+UyhITEzWOPwBYuHAhqlWrBlNTU9jY2MDHxwerV6/WiN3b2xu2trbYvHlzgfrk2LFj+PTTT2FjYwMzMzPUqFEDCxYs0Kr3zz//ICgoCObm5rC3t8eoUaOQmZmpUWf27Nlo1KgRSpUqBRMTE3h7e2PDhg1abe3evRuNGzeGtbU1zM3NUblyZWk/UCvo/lCQtrJTKBRIS0vDihUrpPNGTsdIaGgorK2tYWVlhbCwsBw/dK1atUo6Lm1tbdGtW7ccj9/cJCUl5Xt+L8h+BWQd65999hkOHz6MevXqwdjYGOXKlcPPP/+sUS+nsfTA/8am3rhxQyo7efIkAgICYGdnBxMTE3h4eKBXr14F3r433bx5EwMGDEDlypVhYmKCUqVKoXPnzhrre1N6ejr69euHUqVKwdLSEsHBwXjy5IlGnc2bNyMwMBDOzs5QKpUoX748pk6dqrVvqs89Fy5cQPPmzWFqaooyZcrghx9+0FrvnTt3EBQUBDMzMzg4OGD48OG5jjsvyHlZPU61IMfQmjVr4O3tDQsLC1haWqJ69eo5Ho9qN27cgL29PQAgPDxc2p+zD7kryLpVKhXmz5+PatWqwdjYGI6OjujXr59Wn+dm/fr1qFq1KoyNjeHl5YVNmzYhNDQU7u7uGvXS0tIwcuRIuLq6QqlUonLlypg9e7bWt0hRUVFo0aIFHBwcoFQqUbVqVY3zYbEmSAghxOTJkwUAUb16ddG2bVsREREhvvjiCwFAfPnllxp1AQhPT09hb28vwsPDxaJFi8SZM2fE+fPnhZmZmShdurSYOnWqmDFjhvDw8BBKpVIcPXpUWn7gwIHCwMBAnDp1SgghxN27d4Wtra3w9/cXKpVKCCHE8uXLBQBx7tw5jXUfP35cABA///xzntuTnp4uPD09haGhoRg+fLj48ccfRZMmTQQAMX/+fKleQkKCtN3u7u5i5syZIjw8XNja2gp7e3tx//59IYQQf/75p/jkk08EALFy5Urp8WafTJ48Was/a9WqJbp37y4WL14sAgMDBQAxd+5cUblyZdG/f3+xePFi4evrKwCIAwcOSMs/fPhQlC5dWowYMUJERkaKH374QVSuXFkYGhqKM2fOaL0fb647J/v27ZO2s0aNGmLu3Lli3LhxwtjYWFSqVEmkp6dLdaOiogQAUbduXTFv3jwxbtw4YWJiItzd3cWTJ0+0tvFNbm5uIiQkRHr+7Nkz4eXlJfT19cVXX30lIiMjxdSpU0XdunWl7ejZs6dwdHQUr1+/1mjrhx9+EAqFQty8eTPX7Xr58qUwMTERI0aM0HrNz89PODk5CXt7ezF48GCxdOlSERMTIzIzM0WrVq2EqampGDZsmFi6dKkYNGiQMDAwEO3bt9doQ30M9OjRQ0RERIiOHTuKGjVqFKjPC/oeqvfBqlWrinLlyokZM2aIefPmiZs3b0p9XLVqVdG+fXuxePFisWjRIiGEECEhIQKA+Pzzz8WiRYtEcHCwACCCgoKktjdu3Kh1HNWsWVPo6emJzz//XCpbv369ACDOnz8vhBBi2bJlUttLly4VCxYsEL179xZDhgzR2k5/f3/h7e2dZ18IIcSuXbuEkZGRcHNzE5MnTxaRkZFiyJAhwt/fX6oTEhIijI2NRbVq1USvXr1EZGSk6NSpkwAgFi9erNGei4uLGDBggIiIiBBz584V9erVEwDE1q1bpTrnz58XRkZGwsfHRyxYsEAsWbJEjBo1SjRt2lSqU9D9oSBt5WTlypVCqVSKJk2aSOeNP//8Uwjxv2Oodu3aomPHjmLx4sWiT58+AoAYM2aMRjvfffedUCgUomvXrmLx4sUiPDxc2NnZaR2XOSnM+b0g+5UQWcd65cqVhaOjo5gwYYKIiIgQderUEQqFQtqP3lx3durzTEJCghBCiMTERGFjYyMqVaokZs2aJZYvXy6++eYb4enpmee25Wb9+vWiZs2aYtKkSWLZsmViwoQJwsbGRri5uYm0tDStOKpXry6aNGkifvzxRzFw4EChp6cnmjZtKv1PEkKIoKAg0aVLFzFr1iwRGRkpOnfuLACIUaNGaazbz89PODs7C1dXVzF06FCxePFi0aJFCwFAbNu2TaqXnp4uKlWqJIyNjcWYMWPE/Pnzhbe3t3SO2bdvn1ac+Z2XC3oM7dq1SwAQLVu2FIsWLRKLFi0SgwYNEp07d861T1NTU0VkZKQAIDp06CDtz2fPni3UuoUQok+fPsLAwEB89dVXYsmSJWLs2LHCzMxM1K1bV7x8+TLP93br1q1CoVBI/8smTpwobGxshJeXl3Bzc5PqqVQq0aJFC6FQKESfPn1ERESEaNu2rQAghg0bptFm3bp1RWhoqJg3b55YuHChaNWqlQAgIiIitN5bPz+/POP70JjI/j/1yaZdu3Ya5QMGDBAApB1ViKzESU9PT/z9998adYOCgoSRkZGIj4+Xyu7evSssLCw0TvZpaWmiQoUKolq1auLFixciMDBQWFpaaiQsT58+FcbGxmLs2LEa6xgyZIgwMzMTqampeW7P/PnzBQCxatUqqezly5eiYcOGwtzcXKSkpAgh/pdEmJiYiDt37kh1jx07JgCI4cOHS2UDBw7M8YSs7pOcEtm+fftKZa9fvxYuLi5CoVCIGTNmSOVPnjwRJiYmGgng69evRUZGhsY6njx5IhwdHUWvXr3yXHdO1IlsmTJlpG0XQoh169YJAGLBggVSHzk4OAgvLy/x/Plzqd7WrVsFADFp0iStbXxT9kR20qRJAoDYuHGjVkzqfxA7d+4UAMT27ds1Xq9Ro0a+J4xr164JAGLhwoVar/n5+QkAYsmSJRrlK1euFHp6euLQoUMa5UuWLBEARGxsrBBCiLi4OAFADBgwQKNejx49CtTnBX0P1fugpaWlePDggUZ9dR93795do1wdW58+fTTKR40aJQCIP/74QwghxIMHDzT+iTx9+lTo6emJzp07C0dHR2m5IUOGCFtbW+k9ad++vahWrVqe26fWt29fYWJikmed169fCw8PD+Hm5qaVdL2ZKKiTqG+//VajTu3atbWS5Tc/fAmRte96eXmJFi1aSGXz5s0TAMTDhw9zja2g+0NB2sqNmZmZxnGhpn5/sx/THTp0EKVKlZKe37hxQ+jr64vvv/9eo965c+eEgYGBVnlu68nv/F7Q/UqIrGMdgDh48KBU9uDBA6FUKsXIkSO11p1d9kR206ZNAoA4ceJEnttSUNn3DyGEOHLkiNaFEHUc3t7eGgnUDz/8IACIzZs359lmv379hKmpqXjx4oVUpj73vLmejIwM4eTkJDp16iSVqf9PrVu3TipT/398M5EtzHm5oMfQ0KFDhaWlpdYFhPw8fPgw1/NfQdd96NAhAUD88ssvGvV27NiRY3l21atXFy4uLuLZs2dS2f79+wUAjUQ2JiZGABDfffedxvKff/65UCgU4tq1a1JZTu9tQECAKFeunEZZcUxkObQgm4EDB2o8Hzx4MICsG53e5Ofnh6pVq0rPMzMzsWvXLgQFBaFcuXJSeenSpdGjRw8cPnwYKSkpAABTU1NER0fj4sWLaNq0KX7//XfMmzcPZcuWlZazsrJC+/bt8d///lf6CiAzMxNr166VvobJy7Zt2+Dk5ITu3btLZYaGhhgyZAhSU1Nx4MABjfpBQUEoU6aM9LxevXqoX7++1nYXVp8+faS/9fX14ePjAyEEevfuLZVbW1ujcuXKuH79ukZdIyMjAFlfwTx+/BivX7+Gj48PTp8+/dbxBAcHw8LCQnr++eefo3Tp0tJ2njx5Eg8ePMCAAQNgbGws1QsMDESVKlXw+++/F2p9v/76K2rWrIkOHTpovab+utHf3x/Ozs745ZdfpNfOnz+Pv/76C1988UWe7T969AgAYGNjk+PrSqUSYWFhGmXr16+Hp6cnqlSpgqSkJOnRokULAJC++lf3yZAhQzSWHzZsWJ4xqRX2PezUqZP0tV122W9kU8c2YsQIjfKRI0cCgPQ+2dvbo0qVKjh48CAAIDY2Fvr6+hg9ejQSExNx9epVAMChQ4fQuHFj6T2xtrbGnTt3cOLEiXy308bGBs+fP89z/PGZM2eQkJCAYcOGwdraWuO1nL52zr69TZo00Tg+AMDExET6+8mTJ0hOTkaTJk00+la9rs2bN0OlUuUYW0H3h4K09bZy2t5Hjx5J58yNGzdCpVKhS5cuGjE6OTmhYsWKBR5ylN/5vaD7lVrVqlXRpEkT6bm9vb3Wuayg1P27detWvHr1qtDLZ/fm/vHq1Ss8evQIFSpUgLW1dY7HX9++fTXG9/fv3x8GBgYa/wPebPPZs2dISkpCkyZNkJ6ejkuXLmm0Z25urnH+MjIyQr169TT6Ztu2bShdujQ+//xzqczU1BR9+/bVaOttzsv5HUPW1tZIS0vD7t27tZZ9V/mte/369bCyssInn3yisT97e3vD3Nw8z/357t27OHfuHIKDgzWGk/n5+aF69eoadbdt2wZ9fX2tc/jIkSMhhMD27dulsjff2+TkZCQlJcHPzw/Xr19HcnJy4TrgA2Mim03FihU1npcvXx56enpa44o8PDw0nj98+BDp6emoXLmyVpuenp5QqVQaY7l8fX3Rv39/HD9+HAEBATmOgwoODsatW7dw6NAhAMCePXuQmJiIL7/8EkBWYnv//n2Nh/oGmZs3b6JixYrQ09N8iz09PaXX89puAKhUqVKu46kK6s3kHMhK0I2NjWFnZ6dVnn1s0IoVK1CjRg0YGxujVKlSsLe3x++///5OB1X27VQoFKhQoYK0nep+yel9rFKlila/5Sc+Ph5eXl551tHT00PPnj0RExMjJUO//PILjI2N0blz5wKtR/1hJ7syZcpIyaTa1atX8ffff8Pe3l7jUalSJQBZY6uBrL7Q09ND+fLlNZbPqW9yU5j3MPsxlddr6tgqVKigUe7k5ARra2uN96lJkybSMXTo0CH4+PjAx8cHtra2OHToEFJSUnD27FmNhGTs2LEwNzdHvXr1ULFiRQwcODDXOXrVfZ9TQqoWHx8PAPnuC0DWjZPZE3obGxut42Pr1q1o0KABjI2NYWtrC3t7e0RGRmr0bdeuXeHr64s+ffrA0dER3bp1w7p16zQS0YLuDwVp621lP0+oP5ipt/nq1asQQqBixYpacV68eFGKMT/5nd8Ls1/lFLc69oKOc3yTn58fOnXqhPDwcNjZ2aF9+/aIiop663mKnz9/jkmTJkljI+3s7GBvb4+nT5/mePxl7xtzc3OULl1a43/A33//jQ4dOsDKygqWlpawt7eXktXsbbq4uGgdE9n75ubNm6hQoYJWveznmMKelwtyDA0YMACVKlVCmzZt4OLigl69ekn3uryLgqz76tWrSE5OhoODg9b+nJqamuf+rN7W7PtoTmU3b96Es7OzxsUbIOc8IDY2Fv7+/jAzM4O1tTXs7e2l8e/FPZHlbeb5yO2f05ufXt5GRkaGNH1HfHw80tPTYWpqqlEnICAAjo6OWLVqFZo2bYpVq1bByckJ/v7+ALImwc/+D37fvn3FarJifX39ApUBmsnYqlWrEBoaiqCgIIwePRoODg7SjU/qpKAkCQ4OxqxZsxATE4Pu3btj9erV+Oyzz2BlZZXncqVKlQKAXP9x5rSfqlQqVK9eHXPnzs1xGVdX10JGn7PCvod5HVO5vZZX8qjWuHFjLF++HNevX8ehQ4fQpEkTKBQKNG7cGIcOHYKzszNUKpVGIuvp6YnLly9j69at2LFjB3799VcsXrwYkyZN0pqC7smTJzA1NX3nc4JabsfHmw4dOoR27dqhadOmWLx4MUqXLg1DQ0NERUVp3JBmYmKCgwcPYt++ffj999+xY8cOrF27Fi1atMCuXbugr69f4P2hIG0V9TarzwkqlQoKhQLbt2/Pse7bTkKf2/5TkP0KKNi5LLe2st/8o1AosGHDBhw9ehS//fYbdu7ciV69emHOnDk4evRoobdx8ODBiIqKwrBhw9CwYUNYWVlBoVCgW7dub/Xh4+nTp/Dz84OlpSW+/fZblC9fHsbGxjh9+jTGjh2r1WZB+uZ9Kci+6ODggLi4OOzcuRPbt2/H9u3bERUVheDgYK2boot63SqVCg4ODhrfwr0pt2+m3pf4+Hi0bNkSVapUwdy5c+Hq6gojIyNs27YN8+bNK/JvYIoaE9lsrl69qpEcXrt2DSqVSutOwOzs7e1hamqKy5cva7126dIl6OnpaSQIkydPxsWLFzF79myMHTsW48aN07iLGsg6IHr06IHo6GjMnDkTMTEx+Oqrr6QDxcnJSetrEfVMB25ubvjrr7+gUqk0rsqqv/5xc3PT2u7srly5orHdBT25F4UNGzagXLly2Lhxo8Z6J0+e/E7tZt9OIQSuXbuGGjVqAPhfv1y+fFn6alXt8uXLWv2Wn/Lly2vM/pAbLy8v1K5dG7/88gtcXFxw69YtLFy4MN/lypYtCxMTEyQkJBQqprNnz6Jly5Z5vqdubm5QqVSIj4/XuBKS0z6ek/f1Hr4Z29WrV6WrC0DW7ANPnz7VeJ/UCeru3btx4sQJjBs3DgDQtGlTREZGwtnZGWZmZvD29tZYh5mZGbp27YquXbvi5cuX6NixI77//nuMHz9e4+vNhIQEjRhyor6qff78eemD6Lv49ddfYWxsjJ07d0KpVErlUVFRWnX19PTQsmVLtGzZEnPnzsW0adPwzTffYN++ffD39y/w/lCQtnLzrueO8uXLQwgBDw8P6Urx28jv/F6Y/aqg1FeXnz59qjGsJLdvdxo0aIAGDRrg+++/x+rVq9GzZ0+sWbNGY5hWQWzYsAEhISEaM8K8ePECT58+zbH+1atX0bx5c+l5amoq7t27h08//RRA1swvjx49wsaNG9G0aVOpXmHOPdm5ubnh/PnzEEJo7CPZzzFFfV5WMzIyQtu2bdG2bVuoVCoMGDAAS5cuxcSJE3O84gkUzf/B8uXLY8+ePfD19S30B2D1tl67dk3rtexlbm5u2LNnD549e6ZxVTZ7HvDbb78hIyMDW7Zs0fiWQS7TCnJoQTbqKabU1MlEmzZt8lxOX18frVq1wubNmzW+iklMTMTq1avRuHFjWFpaAsiagmf27NkYNmwYRo4cidGjRyMiIkJr3CoAfPnll3jy5An69euH1NRUjTFHxsbG8Pf313ioT5qffvop7t+/j7Vr10r1X79+jYULF8Lc3Bx+fn4a64mJidGYxuT48eM4duyYxnarx+XmdiIsSupk/c1P78eOHcORI0feqd2ff/4Zz549k55v2LAB9+7dk7bTx8cHDg4OWLJkicZXetu3b8fFixcRGBhYqPV16tQJZ8+exaZNm7Rey35l4ssvv8SuXbswf/58lCpVKt99Dsga9+zj41Oon5fs0qUL/vnnnxx/uOH58+fSPK3q9Wf/gDV//vwCred9vYcApH+u2WNRX1V8833y8PBAmTJlMG/ePLx69Ur69bMmTZogPj4eGzZsQIMGDTTmQVaPPVYzMjJC1apVIYTQGr94+vRpNGrUKM9469SpAw8PD8yfP1/r+HmbK1T6+vpQKBQaV/Vu3Lih9aMDjx8/1lq2Vq1aACDt3wXdHwrSVm7MzMze6bzRsWNH6OvrIzw8XKu/hBBa71du8ju/F2a/Kij1hxj1OG0A0nRkb3ry5InWthW0f3Oir6+v1d7ChQu1rgSrLVu2TGPfjoyMxOvXr6W+yel4fvnyJRYvXlzo2NQ+/fRT3L17V2PauPT0dCxbtkyjXlGflwHtY1xPT0+6oJFXf6u/OX2X/blLly7IzMzE1KlTtV57/fp1nm07OzvDy8sLP//8M1JTU6XyAwcO4Ny5cxp1P/30U2RmZiIiIkKjfN68eVAoFHm+t8nJyTl+MC6OeEU2m4SEBLRr1w6tW7fGkSNHsGrVKvTo0SPXOV3f9N1330nzLA4YMAAGBgZYunQpMjIypPnzXrx4gZCQEFSsWBHff/89gKz56H777TeEhYXh3LlzGjdy1a5dG15eXtINGXXq1CnQdvTt2xdLly5FaGgoTp06BXd3d2zYsAGxsbGYP3++1piZChUqoHHjxujfvz8yMjKkZGrMmDFSHfUVqyFDhiAgIAD6+vro1q1bgeIprM8++wwbN25Ehw4dEBgYiISEBCxZsgRVq1bVOHgLy9bWFo0bN0ZYWBgSExMxf/58VKhQAV999RWArMRw5syZCAsLg5+fH7p3747ExEQsWLAA7u7uGD58eKHWN3r0aGzYsAGdO3dGr1694O3tjcePH2PLli1YsmSJxn7Vo0cPjBkzBps2bUL//v0L/MMK7du3xzfffIOUlBTpw1JevvzyS6xbtw5ff/019u3bB19fX2RmZuLSpUtYt24ddu7cCR8fH9SqVQvdu3fH4sWLkZycjEaNGmHv3r05XgnIyft6D4Gsbx5CQkKwbNky6SvP48ePY8WKFQgKCtK4sgRkJa1r1qxB9erVpQ97derUgZmZGa5cuYIePXpo1G/VqhWcnJzg6+sLR0dHXLx4EREREQgMDNQ4dk6dOoXHjx+jffv2ecarp6eHyMhItG3bFrVq1UJYWBhKly6NS5cu4e+//8bOnTsLtf2BgYGYO3cuWrdujR49euDBgwdYtGgRKlSogL/++kuq9+233+LgwYMIDAyEm5sbHjx4gMWLF8PFxUWaj7eg+0NB2sqNt7c39uzZg7lz58LZ2RkeHh6oX79+gbe3fPny+O677zB+/HjcuHEDQUFBsLCwQEJCAjZt2oS+ffti1KhR+baT3/m9sPtVQbRq1Qply5ZF7969MXr0aOjr6+M///kP7O3tcevWLaneihUrsHjxYnTo0AHly5fHs2fPsHz5clhaWkoJNpA1T+qKFSuQkJCQ5zeFn332GVauXAkrKytUrVoVR44cwZ49e6ThSNm9fPkSLVu2RJcuXXD58mUsXrwYjRs3Rrt27QAAjRo1go2NDUJCQjBkyBAoFAqsXLnynYYKfPXVV4iIiEBwcDBOnTqF0qVLY+XKlVrD7Ir6vAxk3Yj8+PFjtGjRAi4uLrh58yYWLlyIWrVq5fkNi4mJCapWrYq1a9eiUqVKsLW1hZeXV4HGv6v5+fmhX79+mD59OuLi4tCqVSsYGhri6tWrWL9+PRYsWKBxA1x206ZNQ/v27eHr64uwsDA8efIEERER8PLy0ji3tm3bFs2bN8c333yDGzduoGbNmti1axc2b96MYcOGSR+yWrVqJV2dVl80W758ORwcHHDv3r0Cb5fOfLD5EYo59RQpFy5cEJ9//rmwsLAQNjY2YtCgQRrTfQiRNd3TwIEDc2zn9OnTIiAgQJibmwtTU1PRvHlzac5EIYQYPny40NfXF8eOHdNY7uTJk8LAwED0799fq031NCjTpk0r1DYlJiaKsLAwYWdnJ4yMjET16tVFVFSURh311EezZs0Sc+bMEa6urtKcj29OOSZE1hRCgwcPFvb29kKhUGhMKYNcpt/KPlVPSEiIMDMz04rVz89PY7ojlUolpk2bJtzc3IRSqRS1a9cWW7duFSEhIRrTi+S07pyop9/673//K8aPHy8cHByEiYmJCAwMzHGe1rVr14ratWsLpVIpbG1tRc+ePTWmJ3tzG9+UffotIYR49OiRGDRokChTpowwMjISLi4uIiQkRCQlJWmt99NPPxUANPaZ/CQmJgoDAwONeX2F0O7TN718+VLMnDlTVKtWTSiVSmFjYyO8vb1FeHi4SE5Oluo9f/5cDBkyRJQqVUqYmZmJtm3bitu3bxeozwv6Hr65D2aX234khBCvXr0S4eHhwsPDQxgaGgpXV1cxfvx4jWmA1BYtWiQAaB1f/v7+AoDYu3evRvnSpUtF06ZNRalSpYRSqRTly5cXo0eP1ugbIYQYO3asKFu2rMYUWnk5fPiw+OSTT4SFhYUwMzMTNWrU0Jg6LbfjI6d97d///reoWLGiUCqVokqVKiIqKkqr3t69e0X79u2Fs7OzMDIyEs7OzqJ79+7iypUrGm0VZH8oaFs5uXTpkmjatKkwMTERAKRjJLf3N/vUVGq//vqraNy4sTAzMxNmZmaiSpUqYuDAgeLy5ct5rr8w5/eC7ldubm4iMDBQa105TU906tQpUb9+fWFkZCTKli0r5s6dq7WNp0+fFt27dxdly5YVSqVSODg4iM8++0ycPHlSo61OnToJExOTfOfOffLkiXT+Nzc3FwEBAeLSpUta5yh1HAcOHBB9+/YVNjY2wtzcXPTs2VM8evRIo83Y2FjRoEEDYWJiIpydncWYMWOk6QPfnPM1t3NPTufvmzdvinbt2glTU1NhZ2cnhg4dKk1D9WabQhTsvFzQY2jDhg2iVatWwsHBQXpf+vXrJ+7du5dnvwqRNa+6t7e3MDIy0jgXFub4FSJrvmpvb29hYmIiLCwsRPXq1cWYMWPE3bt3841hzZo1okqVKkKpVAovLy+xZcsW0alTJ1GlShWNes+ePRPDhw8Xzs7OwtDQUFSsWFHMmjVL65y1ZcsWUaNGDWFsbCzNKf+f//xH6zgsjtNvMZH9f3n9w9S1+fPn5zsx/tvKK4koSdSJ7Pr163UdSp6CgoJE+fLlC71cr169ROPGjd9DRJSbFy9eCCcnJ40fGCF63xwcHLR+gIBIiKwfe3nzB1Y+FhwjW8wJIfDvf/8bfn5+OU71QiXHvXv38Pvvv0vTqxXG5MmTceLEiVyniKKiFxUVBUNDQ605I4nel7///hvPnz/H2LFjdR0K6dCrV6/w+vVrjbL9+/fj7NmzxWrWog+FY2SLqbS0NGzZsgX79u3DuXPnCvxb7iQ/CQkJiI2NxU8//QRDQ0P069ev0G2ULVtW6zfj6f36+uuvmcTSB1WtWjXpRyLo4/XPP//A398fX3zxBZydnXHp0iUsWbIETk5OH+U5iYlsMfXw4UP06NED1tbWmDBhgjTgnkqeAwcOICwsDGXLlsWKFSvg5OSk65CIiKiYsrGxgbe3N3766Sc8fPgQZmZmCAwMxIwZM3K9ma8kUwjxAWYnJiIiIiIqYhwjS0RERESyJIuhBSqVCnfv3oWFhcUH/XUpIiIiIvqwhBB49uwZnJ2dNX6dNCeySGTv3r1bZL//TkRERETF3+3bt+Hi4pJnHVkksupf0rl9+3aBfrmIiIiIiOQpJSUFrq6uWr9CmhNZJLLq4QSWlpZMZImIiIg+AgUZTsqbvYiIiIhIlpjIEhEREZEsMZElIiIiIlmSxRhZItKdzMxMvHr1StdhEOmEoaEh9PX1dR0GEeWCiSwR5UgIgfv37+Pp06e6DoVIp6ytreHk5MR5zImKISayRJQjdRLr4OAAU1NT/hOnj44QAunp6Xjw4AEAoHTp0jqOiIiyYyJLRFoyMzOlJLZUqVK6DodIZ0xMTAAADx48gIODA4cZEBUzhb7Z6+DBg2jbti2cnZ2hUCgQExOT7zL79+9HnTp1oFQqUaFCBURHR79FqET0oajHxJqamuo4EiLdUx8HHCtOVPwUOpFNS0tDzZo1sWjRogLVT0hIQGBgIJo3b464uDgMGzYMffr0wc6dOwsdLBF9WBxOQMTjgKg4K/TQgjZt2qBNmzYFrr9kyRJ4eHhgzpw5AABPT08cPnwY8+bNQ0BAQGFXT0REREQE4APMI3vkyBH4+/trlAUEBODIkSO5LpORkYGUlBSNBxERERHRm977zV7379+Ho6OjRpmjoyNSUlLw/PlzaSD9m6ZPn47w8PD3HRoRFdLFKp4fdH2ely5+0PV9SNHR0Rg2bBinN3tH+/fvR/PmzfHkyRNYW1sXWV2i4sCtua4jyHJzn64jyF2x/GWv8ePHIzk5WXrcvn1b1yEREVEx1KhRI9y7dw9WVlaFrhsdHc2Elkjm3vsVWScnJyQmJmqUJSYmwtLSMsersQCgVCqhVCrfd2hERHj58iWMjIx0HcZH723fByMjIzg5ORV5XSKSh/d+RbZhw4bYu3evRtnu3bvRsGHD971qIvrINGvWDIMGDcKgQYNgZWUFOzs7TJw4EUIIqY67uzumTp2K4OBgWFpaom/fvgCAX3/9FdWqVYNSqYS7u7t0gyoAfPvtt3B2dsajR4+kMvVsLCqVCr169cJnn32mEcurV6/g4OCAf//733nGHBMTg4oVK8LY2BgBAQFa30BFRkaifPnyMDIyQuXKlbFy5UqN1xUKBX766Sd06NABpqamqFixIrZs2SK9ntNVx5iYGI078c+ePYvmzZvDwsIClpaW8Pb2xsmTJ3ONWaFQIDIyEm3atIGJiQnKlSuHDRs2aNS5ffs2unTpAmtra9ja2qJ9+/a4ceOG9HpoaCiCgoLw/fffw9nZGZUrV9Zaz5UrV6BQKHDp0iWN8nnz5qF8+fIAsoYLKBQKaYjGzZs30bZtW9jY2MDMzAzVqlXDtm3btOru378fYWFhSE5OhkKhgEKhwJQpU3LdZiIqngqdyKampiIuLg5xcXEAsqbXiouLw61btwBkDQsIDg6W6n/99de4fv06xowZg0uXLmHx4sVYt24dhg8fXjRbQET0hhUrVsDAwADHjx/HggULMHfuXPz0008adWbPno2aNWvizJkzmDhxIk6dOoUuXbqgW7duOHfuHKZMmYKJEydKc15/8803cHd3R58+fQAAixYtwp9//okVK1ZAT08Pffr0wY4dO3Dv3j1pHVu3bkV6ejq6du2aa6zp6en4/vvv8fPPPyM2NhZPnz5Ft27dpNc3bdqEoUOHYuTIkTh//jz69euHsLAw7NunOWAtPDwcXbp0wV9//YVPP/0UPXv2xOPHjwvcZz179oSLiwtOnDiBU6dOYdy4cTA0NMxzmYkTJ6JTp044e/YsevbsiW7duuHixawxza9evUJAQAAsLCxw6NAhxMbGwtzcHK1bt8bLly+lNvbu3YvLly9j9+7d2Lp1q9Y6KlWqBB8fH/zyyy8a5b/88gt69OiRY1wDBw5ERkYGDh48iHPnzmHmzJkwNzfXqteoUSPMnz8flpaWuHfvHu7du4dRo0bl21dEVLwUemjByZMn0bz5/0YfjxgxAgAQEhKC6Oho3Lt3T0pqAcDDwwO///47hg8fjgULFsDFxQU//fQTp94iovfC1dUV8+bNg0KhQOXKlXHu3DnMmzcPX331lVSnRYsWGDlypPS8Z8+eaNmyJSZOnAggK4G6cOECZs2ahdDQUOjr62PVqlWoVasWxo0bhx9//BE//fQTypYtCyArKVJfLR0zZgwAICoqCp07d84xiVJ79eoVIiIiUL9+fQBZSbinpyeOHz+OevXqYfbs2QgNDcWAAQMAZJ1vjx49itmzZ2uch0NDQ9G9e3cAwLRp0/Djjz/i+PHjaN26dYH67NatWxg9ejSqVKkCAKhYsWK+y3Tu3FlK7KdOnYrdu3dj4cKFWLx4MdauXQuVSoWffvpJuvIbFRUFa2tr7N+/H61atQIAmJmZ4aeffspzSEHPnj0RERGBqVOnAsi6Snvq1CmsWrUq123p1KkTqlevDgAoV65cjvWMjIxgZWUFhULB4QZEMlboK7LNmjWDEELrob5yER0djf3792stc+bMGWRkZCA+Ph6hoaFFEDoRkbYGDRpofG3esGFDXL16FZmZmVKZj4+PxjIXL16Er6+vRpmvr6/GcuXKlcPs2bMxc+ZMtGvXTuuKYJ8+fRAVFQUg6z6A7du3o1evXnnGamBggLp160rPq1SpAmtra+nKZm5xqV9Xq1GjhvS3mZkZLC0t8eDBgzzX/aYRI0agT58+8Pf3x4wZMxAfH5/vMtmHhzVs2FCK6+zZs7h27RosLCxgbm4Oc3Nz2Nra4sWLFxptV69ePd9xsd26dcONGzdw9OhRAFlXY+vUqSMl3dkNGTIE3333HXx9fTF58mT89ddf+W4LEclXsZy1gIjofTIzM3ur5Q4ePAh9fX3cuHEDr1+/1ngtODgY169fx5EjR7Bq1Sp4eHigSZMmRRFuvrIPA1AoFFCpVAAAPT09jTHCgPZPrU6ZMgV///03AgMD8ccff6Bq1arYtGnTW8eTmpoKb29vaRia+nHlyhWNDwAFeR+cnJzQokULrF69GgCwevVq9OzZM9f6ffr0wfXr1/Hll1/i3Llz8PHxwcKFC996W4ioeGMiS0QlyrFjxzSeHz16FBUrVoS+vn6uy3h6eiI2NlajLDY2FpUqVZKWW7t2LTZu3Ij9+/fj1q1b0lfdaqVKlUJQUBCioqIQHR2NsLCwfGN9/fq1xk1Vly9fxtOnT+Hp6ZlnXFWrVs23bTV7e3s8e/YMaWlpUpn6Hoc3VapUCcOHD8euXbvQsWNH6epybtRXSN98ro67Tp06uHr1KhwcHFChQgWNR0GmycquZ8+eWLt2LY4cOYLr169rjCPOiaurK77++mts3LgRI0eOxPLly3OsZ2RkpHGlnojkh4ksEZUot27dwogRI3D58mX897//xcKFCzF06NA8lxk5ciT27t2LqVOn4sqVK1ixYgUiIiKkm3/u3LmD/v37Y+bMmWjcuDGioqIwbdo0rWSuT58+WLFiBS5evIiQkJB8YzU0NMTgwYNx7NgxnDp1CqGhoWjQoAHq1asHABg9ejSio6MRGRmJq1evYu7cudi4cWOhbkqqX78+TE1NMWHCBMTHx2P16tXSUDAAeP78OQYNGoT9+/fj5s2biI2NxYkTJ6SkNDfr16/Hf/7zH1y5cgWTJ0/G8ePHMWjQIABZiaednR3at2+PQ4cOISEhAfv378eQIUNw586dAseu1rFjRzx79gz9+/dH8+bN4ezsnGvdYcOGYefOnUhISMDp06exb9++XLfF3d0dqamp2Lt3L5KSkpCenl7o2IhIt977PLJEVHLI4Ze2goOD8fz5c9SrVw/6+voYOnSoNMVWburUqYN169Zh0qRJmDp1KkqXLo1vv/0WoaGhEEIgNDQU9erVkxK1gIAA9O/fH1988QXi4uKkG7r8/f1RunRpVKtWLc9kS83U1BRjx45Fjx498M8//6BJkyYa03UFBQVhwYIFmD17NoYOHQoPDw9ERUWhWbNmBe4PW1tbrFq1CqNHj8by5cvRsmVLTJkyReoTfX19PHr0CMHBwUhMTISdnR06duyY768rhoeHY82aNRgwYABKly6N//73v9KVYlNTUxw8eBBjx46VktAyZcqgZcuWsLS0LHDsahYWFmjbti3WrVuH//znP3nWzczMxMCBA3Hnzh1YWlqidevWmDdvXo51GzVqhK+//hpdu3bFo0ePMHnyZE7BRSQzCpF98FQxlJKSAisrKyQnJ7/VSZCICufFixdISEiAh4cHjI2NdR1OgTVr1gy1atXC/PnzdbL+1NRUlClTBlFRUejYsaNOYvgQFAoFNm3ahKCgIF2H8kHI9Xgg+ftYf6K2MHkfr8gSEb0jlUqFpKQkzJkzB9bW1mjXrp2uQyIi+igwkSUieke3bt2Ch4cHXFxcEB0dDQMDnlqJiD4Enm2JqMTIPof1h+Lu7q41xVVJ9jFtKxEVb5y1gIiIiIhkiYksEREREckSE1kiIiIikiUmskREREQkS0xkiYiIiEiWmMgSERERkSxx+i0iKrAP/SszH/rXZD5G0dHRGDZsGJ4+farrUIpMaGgonj59ipiYGF2HQkTvGa/IElGJ0axZMwwbNkzXYXx0pkyZglq1auk6DCL6CDGRJaKPihACr1+//mDre/ny5QdbV16KSxxEREWJiSwRlQihoaE4cOAAFixYAIVCAYVCgRs3bmD//v1QKBTYvn07vL29oVQqcfjwYYSGhiIoKEijjWHDhqFZs2bSc5VKhenTp8PDwwMmJiaoWbMmNmzYkGcc7u7umDp1KoKDg2FpaYm+ffsCAA4fPowmTZrAxMQErq6uGDJkCNLS0gAAERER8PLyktqIiYmBQqHAkiVLpDJ/f3/861//AgDEx8ejffv2cHR0hLm5OerWrYs9e/YUKI7o6GiULVsWpqam6NChAx49epRv344dOxaVKlWCqakpypUrh4kTJ+LVq1dSe+Hh4Th79qzU79HR0Tm28/r1awwZMgTW1tYoVaoUxo4di5CQEI33ISMjA0OGDIGDgwOMjY3RuHFjnDhxQno9MzMTvXv3lt6TypUrY8GCBfluAxGVTExkiahEWLBgARo2bIivvvoK9+7dw7179+Dq6iq9Pm7cOMyYMQMXL15EjRo1CtTm9OnT8fPPP2PJkiX4+++/MXz4cHzxxRc4cOBAnsvNnj0bNWvWxJkzZzBx4kTEx8ejdevW6NSpE/766y+sXbsWhw8fxqBBgwAAfn5+uHDhAh4+fAgAOHDgAOzs7KSf3H316hWOHDkiJdmpqan49NNPsXfvXpw5cwatW7dG27ZtcevWrTzjOHbsGHr37o1BgwYhLi4OzZs3x3fffZdvP1hYWCA6OhoXLlzAggULsHz5csybNw8A0LVrV4wcORLVqlWT+r1r1645tjNz5kz88ssviIqKQmxsLFJSUrTGsY4ZMwa//vorVqxYgdOnT6NChQoICAjA48ePAWR9uHBxccH69etx4cIFTJo0CRMmTMC6devy3Q4iKnl4sxcRlQhWVlYwMjKCqakpnJyctF7/9ttv8cknnxS4vYyMDEybNg179uxBw4YNAQDlypXD4cOHsXTpUvj5+eW6bIsWLTBy5EjpeZ8+fdCzZ09p/G7FihXx448/ws/PD5GRkfDy8oKtrS0OHDiAzz//HPv378fIkSOlK43Hjx/Hq1ev0KhRIwBAzZo1UbNmTan9qVOnYtOmTdiyZYuUHOcUx8SJE9G6dWuMGTMGAFCpUiX8+eef2LFjR559ob4SDGRd6R01ahTWrFmDMWPGwMTEBObm5jAwMMix39+0cOFCjB8/Hh06dACQdSV627Zt0utpaWmIjIxEdHQ02rRpAwBYvnw5du/ejX//+98YPXo0DA0NER4eLi3j4eGBI0eOYN26dejSpUue6yeikodXZInoo+Dj41Oo+teuXUN6ejo++eQTmJubS4+ff/4Z8fHxhVrX2bNnER0drdFOQEAAVCoVEhISoFAo0LRpU+zfvx9Pnz7FhQsXMGDAAGRkZODSpUs4cOAA6tatC1NTUwBZV2RHjRoFT09PWFtbw9zcHBcvXtS6Ips9josXL6J+/foaZeokPS9r166Fr68vnJycYG5ujn/9619a68pPcnIyEhMTUa9ePalMX18f3t7e0vP4+Hi8evUKvr6+UpmhoSHq1auHixcvSmWLFi2Ct7c37O3tYW5ujmXLlhU6HiIqGXhFlog+CmZmZhrP9fT0IITQKFOP+wSykkUA+P3331GmTBmNekqlslDrSk1NRb9+/TBkyBCtumXLlgWQNePCsmXLcOjQIdSuXRuWlpZScnvgwAGNK8CjRo3C7t27MXv2bFSoUAEmJib4/PPPtW7oyh7H2zhy5Ah69uyJ8PBwBAQEwMrKCmvWrMGcOXPeue23sWbNGowaNQpz5sxBw4YNYWFhgVmzZuHYsWM6iYeIdIuJLBGVGEZGRsjMzCxQXXt7e5w/f16jLC4uDoaGhgCAqlWrQqlU4tatW3kOIyiIOnXq4MKFC6hQoUKudfz8/DBs2DCsX79eGgvbrFkz7NmzB7GxsRpDBGJjYxEaGip9RZ+amoobN27kG4enp6dWwnf06NE8l/nzzz/h5uaGb775Riq7efOmRp2C9LuVlRUcHR1x4sQJNG3aFEDWjVunT5+Wpu4qX748jIyMEBsbCzc3NwBZHy5OnDghDcuIjY1Fo0aNMGDAAKnt/K6QE1HJxaEFRFRiuLu749ixY7hx4waSkpKgUqlyrduiRQucPHkSP//8M65evYrJkydrJLYWFhYYNWoUhg8fjhUrViA+Ph6nT5/GwoULsWLFikLFNXbsWPz555/STVZXr17F5s2bNcaz1qhRAzY2Nli9erVGIhsTE4OMjAyNr9srVqyIjRs3Ii4uDmfPnkWPHj3y3Fa1IUOGYMeOHZg9ezauXr2KiIiIfMfHVqxYEbdu3cKaNWsQHx+PH3/8EZs2bdKo4+7ujoSEBMTFxSEpKQkZGRk5tjV48GBMnz4dmzdvxuXLlzF06FA8efIECoUCQNYV5P79+2P06NHYsWMHLly4gK+++grp6eno3bu3FM/Jkyexc+dOXLlyBRMnTtSY1YCIPi68IktEBVbcf2lr1KhRCAkJQdWqVfH8+XMkJCTkWjcgIAATJ07EmDFj8OLFC/Tq1QvBwcE4d+6cVGfq1Kmwt7fH9OnTcf36dVhbW6NOnTqYMGFCoeKqUaMGDhw4gG+++QZNmjSBEALly5fXuLtfoVCgSZMm+P3339G4cWNpOUtLS1SuXFljmMDcuXPRq1cvNGrUCHZ2dhg7dixSUlLyjaNBgwZYvnw5Jk+ejEmTJklTek2dOjXXZdq1a4fhw4dj0KBByMjIQGBgICZOnIgpU6ZIdTp16oSNGzeiefPmePr0KaKiohAaGqrV1tixY3H//n0EBwdDX18fffv2RUBAAPT19aU6M2bMgEqlwpdffolnz57Bx8cHO3fuhI2NDQCgX79+OHPmDLp27QqFQoHu3btjwIAB2L59e77bT0Qlj0JkHyRWDKWkpMDKygrJycmwtLTUdThEJd6LFy+QkJAADw8PGBsb6zocKqFUKhU8PT3RpUuXPJNpXePxQLryoX8WPDcf+iJGYfI+XpElIqIP4ubNm9i1axf8/PyQkZGBiIgIJCQkoEePHroOjYhkimNkiYjog9DT00N0dDTq1q0LX19fnDt3Dnv27IGnp6euQyMimeIVWSIi+iBcXV0RGxur6zCIqAThFVkiIiIikiUmskSUq4JM6URU0vE4ICq+OLSAiLQYGRlBT08Pd+/ehb29PYyMjKS5Pok+FkIIvHz5Eg8fPoSenh6MjIx0HRIRZcNEloi06OnpwcPDA/fu3cPdu3d1HQ6RTpmamqJs2bLQ0+OXmETFDRNZIsqRkZERypYti9evXxf4Z1+JShp9fX0YGBjwGwmiYoqJLBHlSqFQwNDQEIaGhroOhYiISAu/JyEiIiIiWWIiS0RERESyxESWiIiIiGSJiSwRERERyRITWSIiIiKSJSayRERERCRLTGSJiIiISJaYyBIRERGRLDGRJSIiIiJZYiJLRERERLLERJaIiIiIZImJLBERERHJEhNZIiIiIpIlJrJEREREJEtMZImIiIhIlpjIEhEREZEsMZElIiIiIlliIktEREREssREloiIiIhkiYksEREREckSE1kiIiIikiUmskREREQkS0xkiYiIiEiWmMgSERERkSwxkSUiIiIiWWIiS0RERESyxESWiIiIiGSJiSwRERERyRITWSIiIiKSJSayRERERCRLTGSJiIiISJaYyBIRERGRLDGRJSIiIiJZYiJLRERERLLERJaIiIiIZImJLBERERHJEhNZIiIiIpIlJrJEREREJEtMZImIiIhIlpjIEhEREZEsMZElIiIiIlliIktEREREssREloiIiIhkiYksEREREckSE1kiIiIikiUmskREREQkS0xkiYiIiEiWmMgSERERkSwxkSUiIiIiWWIiS0RERESy9FaJ7KJFi+Du7g5jY2PUr18fx48fz7Xuxo0b4ePjA2tra5iZmaFWrVpYuXLlWwdMRERERAS8RSK7du1ajBgxApMnT8bp06dRs2ZNBAQE4MGDBznWt7W1xTfffIMjR47gr7/+QlhYGMLCwrBz5853Dp6IiIiIPl4KIYQozAL169dH3bp1ERERAQBQqVRwdXXF4MGDMW7cuAK1UadOHQQGBmLq1Kk5vp6RkYGMjAzpeUpKClxdXZGcnAxLS8vChEtEREQkS27NdR1Blpv7Puz6UlJSYGVlVaC8z6AwDb98+RKnTp3C+PHjpTI9PT34+/vjyJEj+S4vhMAff/yBy5cvY+bMmbnWmz59OsLDwwsTGtFH62IVT12HIPG8dFHXIRAR0UekUEMLkpKSkJmZCUdHR41yR0dH3L9/P9flkpOTYW5uDiMjIwQGBmLhwoX45JNPcq0/fvx4JCcnS4/bt28XJkwiIiIi+ggU6ors27KwsEBcXBxSU1Oxd+9ejBgxAuXKlUOzZs1yrK9UKqFUKj9EaEREREQkU4VKZO3s7KCvr4/ExESN8sTERDg5OeW6nJ6eHipUqAAAqFWrFi5evIjp06fnmsgSEREREeWnUEMLjIyM4O3tjb1790plKpUKe/fuRcOGDQvcjkql0riZi4iIiIiosAo9tGDEiBEICQmBj48P6tWrh/nz5yMtLQ1hYWEAgODgYJQpUwbTp08HkHXjlo+PD8qXL4+MjAxs27YNK1euRGRkZNFuCRERERF9VAqdyHbt2hUPHz7EpEmTcP/+fdSqVQs7duyQbgC7desW9PT+d6E3LS0NAwYMwJ07d2BiYoIqVapg1apV6Nq1a9FtBRERERF9dAo9j6wuFGY+MaKPDaffIiIqmTiPbP5531v9RC0RERERka4xkSUiIiIiWWIiS0RERESyxESWiIiIiGSJiSwRERERyRITWSIiIiKSJSayRERERCRLTGSJiIiISJaYyBIRERGRLDGRJSIiIiJZYiJLRERERLLERJaIiIiIZImJLBERERHJEhNZIiIiIpIlJrJEREREJEtMZImIiIhIlpjIEhEREZEsMZElIiIiIlliIktEREREssREloiIiIhkiYksEREREckSE1kiIiIikiUmskREREQkS0xkiYiIiEiWmMgSERERkSwxkSUiIiIiWWIiS0RERESyxESWiIiIiGSJiSwRERERyRITWSIiIiKSpbdKZBctWgR3d3cYGxujfv36OH78eJ71169fjypVqsDY2BjVq1fHtm3b3ipYIiIiIiK1Qieya9euxYgRIzB58mScPn0aNWvWREBAAB48eJBj/T///BPdu3dH7969cebMGQQFBSEoKAjnz59/5+CJiIiI6OOlEEKIwixQv3591K1bFxEREQAAlUoFV1dXDB48GOPGjdOq37VrV6SlpWHr1q1SWYMGDVCrVi0sWbIkx3VkZGQgIyNDep6SkgJXV1ckJyfD0tKyMOESlXgXq3jqOgSJ56WLug6BiKjEcGuu6wiy3Nz3YdeXkpICKyurAuV9BoVp+OXLlzh16hTGjx8vlenp6cHf3x9HjhzJcZkjR45gxIgRGmUBAQGIiYnJdT3Tp09HeHh4YUIj+mgxeSQiKpk+dAIpR4UaWpCUlITMzEw4OjpqlDs6OuL+/fs5LnP//v1C1QeA8ePHIzk5WXrcvn27MGESERER0UegUFdkPxSlUgmlUqnrMIiIiIioGCvUFVk7Ozvo6+sjMTFRozwxMRFOTk45LuPk5FSo+kREREREBVGoRNbIyAje3t7Yu3evVKZSqbB37140bNgwx2UaNmyoUR8Adu/enWt9IiIiIqKCKPTQghEjRiAkJAQ+Pj6oV68e5s+fj7S0NISFhQEAgoODUaZMGUyfPh0AMHToUPj5+WHOnDkIDAzEmjVrcPLkSSxbtqxot4SIiIiIPiqFTmS7du2Khw8fYtKkSbh//z5q1aqFHTt2SDd03bp1C3p6/7vQ26hRI6xevRr/+te/MGHCBFSsWBExMTHw8vIquq0gIiIioo9OoeeR1YXCzCdGRERERPJVmLzvrX6iloiIiIhI15jIEhEREZEsMZElIiIiIlliIktEREREssREloiIiIhkiYnsR+b+1O9wZ/hwvLh8RdehlBjP/tiHGz16ImXHDgiVStfh5CszNQ1Jy5YjoWMnvE5K0nU4JYIQAs/++AM3v/gSyb9t1XU4JUbG9QT8M2YM7o4br+tQSozM1DQkLV2Wdfw/eqTrcIjeGRPZj4xZY1+8+OscEtq3x51hw/HiChPad2XsWQV6pqb4Z9hwJLRvX2wTWnUCG+/vj4cLF8KkVi3o29joOixZUyewCZ064c6AgYBCAZMa1XUdluypE9jrn32G9GPHYd7MT9chyZ46gY1v2RJJEREwqV0b+tbWug6L6J1xHlkZyUxJQWZKyju3I16/xrPdu/Hkl9V4ff8+zJo2hV2/vjD19i6CKOVFvHqFV4mJRdLWi7//xuOfV+L5qVMwcneHbWgIrLt0gUJPt58XM1NSkLR0GZ6uXw9Vejos27SBTfduMHBweOe2Ffr6MCxdugiilBchBJI3xeBRVBReXr0K4xrVYRscApNaNYukfQMHB+gZGRVJW3Ly/Nx5JC1dgtQ/9kHf1hY23bvDsk1rKIqgL/TNzT/KxC0zORlJy5ZnHf/Pn8OyTWvYdOPxT8VbYfI+JrIycqWRLzIfP35v7Vc+Gwc9pfK9tV8c3f/2WzxZ/d/31r7dkMGwHzDgvbVfENc7dkTGhYvvrX2XRRGwaNnyvbVfHD1Zsxb3p0x5b+1btm2LMrN+eG/tF0dCCFyq5gW8p28zFMbGqHzyBBQGhf5BS1m7HtQBGZcuvbf2XRYvgkWLFu+tffo4MZEtoTKuJ+D1g3e/eiiEQMbFi0iO2YyMK1dg5OEO27Aw2HTpUgRRyktmcjJeXCyaJO91UhJStv6O1EOHoGdqAsvAQDiOHQc9E+Miaf9tZVy/jgc//IDUQ4ehZ2oKyzZtYN6yRZF8aFHo68OkTh0o9PWLIFL5UL16hQc/zELKb78hMyUFZo0awbLtZzD8/5/qflfKSpVgYGtbJG3JydMtW/B4+U/IuHoVRh4esApqD+MaNaBQKN65bYNSpaCsWLEIopSXjPh4PJg163/H/6dtYN6Cxz8Vb0xkKVfPz51D4syZeH7yFIy9vGA3aCDM/fyK5B/FxyozORkP5s7D040boW9ujlK9e8Gme3fomZnpOjQNr/75J2uIwcaN0Le0RKk+fWAbFsr3/h2oXrzA03Xr8Wj5crx+/BhW7drBYeQIGNjZ6To02RJCIP3IETyMWITnp0/DuHp1OI4dA1MfH12HJmsv7/yDR8veOP6/+gq2oSE8/qlYYiJLubozfDhe3fkHdgMHMIEtIslbf8eDH36AbfCXxTKBze7VP/8gadlypOzYgXJbtsDQ8d3Hyn3spIQ2Ogp2ffvBpltXXYcke28mtHqmpij703Jdh1QiqBPalJ07Ue63LTAsgrGyREWNiSwRERERyVJh8j5Ov0VEREREssREloiIiIhkiYksEREREckSE1kiIiIikiUmskREREQkS0xkiYiIiEiWmMgSERERkSwxkSUiIiIiWWIiS0RERESyxESWiIiIiGSJiSwRERERyRITWSIiIiKSJQNdB1AQQggAQEpKio4jISIiIqL3SZ3vqfO/vMgikX327BkAwNXVVceREBEREdGH8OzZM1hZWeVZRyEKku7qmEqlwt27d2FhYQGFQqHrcD6olJQUuLq64vbt27C0tNR1OCUC+/T9YL8WPfZp0WOfvh/s16L3MfepEALPnj2Ds7Mz9PTyHgUriyuyenp6cHFx0XUYOmVpafnR7cjvG/v0/WC/Fj32adFjn74f7Nei97H2aX5XYtV4sxcRERERyRITWSIiIiKSJSayxZxSqcTkyZOhVCp1HUqJwT59P9ivRY99WvTYp+8H+7XosU8LRhY3exERERERZccrskREREQkS0xkiYiIiEiWmMgSERERkSwxkSUiIiIiWWIiS0RERESyxES2GFu0aBHc3d1hbGyM+vXr4/jx47oOSdYOHjyItm3bwtnZGQqFAjExMboOSfamT5+OunXrwsLCAg4ODggKCsLly5d1HZbsRUZGokaNGtIv+jRs2BDbt2/XdVglyowZM6BQKDBs2DBdhyJbU6ZMgUKh0HhUqVJF12HJ3j///IMvvvgCpUqVgomJCapXr46TJ0/qOqxii4lsMbV27VqMGDECkydPxunTp1GzZk0EBATgwYMHug5NttLS0lCzZk0sWrRI16GUGAcOHMDAgQNx9OhR7N69G69evUKrVq2Qlpam69BkzcXFBTNmzMCpU6dw8uRJtGjRAu3bt8fff/+t69BKhBMnTmDp0qWoUaOGrkORvWrVquHevXvS4/Dhw7oOSdaePHkCX19fGBoaYvv27bhw4QLmzJkDGxsbXYdWbHEe2WKqfv36qFu3LiIiIgAAKpUKrq6uGDx4MMaNG6fj6ORPoVBg06ZNCAoK0nUoJcrDhw/h4OCAAwcOoGnTproOp0SxtbXFrFmz0Lt3b12HImupqamoU6cOFi9ejO+++w61atXC/PnzdR2WLE2ZMgUxMTGIi4vTdSglxrhx4xAbG4tDhw7pOhTZ4BXZYujly5c4deoU/P39pTI9PT34+/vjyJEjOoyMKG/JyckAspIuKhqZmZlYs2YN0tLS0LBhQ12HI3sDBw5EYGCgxvmV3t7Vq1fh7OyMcuXKoWfPnrh165auQ5K1LVu2wMfHB507d4aDgwNq166N5cuX6zqsYo2JbDGUlJSEzMxMODo6apQ7Ojri/v37OoqKKG8qlQrDhg2Dr68vvLy8dB2O7J07dw7m5uZQKpX4+uuvsWnTJlStWlXXYcnamjVrcPr0aUyfPl3XoZQI9evXR3R0NHbs2IHIyEgkJCSgSZMmePbsma5Dk63r168jMjISFStWxM6dO9G/f38MGTIEK1as0HVoxZaBrgMgopJh4MCBOH/+PMfIFZHKlSsjLi4OycnJ2LBhA0JCQnDgwAEms2/p9u3bGDp0KHbv3g1jY2Ndh1MitGnTRvq7Ro0aqF+/Ptzc3LBu3ToOgXlLKpUKPj4+mDZtGgCgdu3aOH/+PJYsWYKQkBAdR1c88YpsMWRnZwd9fX0kJiZqlCcmJsLJyUlHURHlbtCgQdi6dSv27dsHFxcXXYdTIhgZGaFChQrw9vbG9OnTUbNmTSxYsEDXYcnWqVOn8ODBA9SpUwcGBgYwMDDAgQMH8OOPP8LAwACZmZm6DlH2rK2tUalSJVy7dk3XochW6dKltT6senp6cshGHpjIFkNGRkbw9vbG3r17pTKVSoW9e/dyjBwVK0IIDBo0CJs2bcIff/wBDw8PXYdUYqlUKmRkZOg6DNlq2bIlzp07h7i4OOnh4+ODnj17Ii4uDvr6+roOUfZSU1MRHx+P0qVL6zoU2fL19dWawvDKlStwc3PTUUTFH4cWFFMjRoxASEgIfHx8UK9ePcyfPx9paWkICwvTdWiylZqaqnGlICEhAXFxcbC1tUXZsmV1GJl8DRw4EKtXr8bmzZthYWEhjeG2srKCiYmJjqOTr/Hjx6NNmzYoW7Ysnj17htWrV2P//v3YuXOnrkOTLQsLC62x22ZmZihVqhTHdL+lUaNGoW3btnBzc8Pdu3cxefJk6Ovro3v37roOTbaGDx+ORo0aYdq0aejSpQuOHz+OZcuWYdmyZboOrfgSVGwtXLhQlC1bVhgZGYl69eqJo0eP6jokWdu3b58AoPUICQnRdWiylVN/AhBRUVG6Dk3WevXqJdzc3ISRkZGwt7cXLVu2FLt27dJ1WCWOn5+fGDp0qK7DkK2uXbuK0qVLCyMjI1GmTBnRtWtXce3aNV2HJXu//fab8PLyEkqlUlSpUkUsW7ZM1yEVa5xHloiIiIhkiWNkiYiIiEiWmMgSERERkSwxkSUiIiIiWWIiS0RERESyxESWiIiIiGSJiSwRERERyRITWSIiIiKSJSayRERERCRLTGSJiIiISJaYyBIRERGRLDGRJSIiIiJZ+j9gcSGSMiRj6AAAAABJRU5ErkJggg==",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# viz: proxy reward landscape vs true value, side by side\n",
"fig, ax = plt.subplots(figsize=(7, 2.6))\n",
"cells = np.arange(CORRIDOR)\n",
"ax.bar(cells - 0.2, [BONUS if c == SHINY else 0 for c in cells], width=0.4,\n",
" color=\"#d62728\", label=\"proxy bonus per visit\")\n",
"ax.bar(cells + 0.2, [1.0 if c == GOAL_C else 0 for c in cells], width=0.4,\n",
" color=\"#1E40FF\", label=\"true reward at goal\")\n",
"for s in range(GOAL_C):\n",
" ax.annotate(\"\", xy=(s + (0.35 if pi_proxy_opt[s] == 1 else -0.35), -0.12),\n",
" xytext=(s, -0.12), arrowprops=dict(arrowstyle=\"->\", color=\"#d62728\"))\n",
"ax.set_xticks(cells); ax.set_ylim(-0.25, 1.1); ax.set_yticks([0, 0.3, 1.0])\n",
"ax.set_title(\"proxy-optimal policy (red arrows) chases the bonus, abandons the goal\")\n",
"ax.legend(loc=\"upper center\"); plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "markdown",
"id": "eb1abbf8",
"metadata": {},
"source": [
"> **Key takeaways**\n",
"> - Reward hacking is the optimizer exploiting the gap between the proxy reward and the true objective. It needs no approximation error to appear.\n",
"> - A well-meaning shaping bonus can make the proxy-optimal policy earn zero true reward, exactly and provably.\n",
"> - The mitigation is never \"optimize harder\". It is a better reward (verifiable rewards where possible), a KL leash that bounds drift, and an audit of the policy under a held-out true objective.\n"
]
},
{
"cell_type": "markdown",
"id": "e1b65c96",
"metadata": {},
"source": [
"## Safety lens\n",
"\n",
"Reward hacking is the safety lens of this chapter, and Part 6 already made it exact. Three habits carry from the toy code to a real RLHF run.\n",
"\n",
"**Never trust a single reward curve.** In Part 5 the KL=0 reward climbed beautifully while the policy degenerated to `.......`. A rising training reward is consistent with both success and reward hacking. Train against a *held-out* reward model that was not used to optimize, and report both; a growing gap between training reward and held-out reward is the cleanest single signal that you are exploiting the optimizer's reward, not the objective.\n",
"\n",
"**Prefer verifiable rewards.** Part 6's failure came from a learned/shaped proxy. Where the desired behavior can be checked programmatically (a unit test passes, a proof checks, an answer matches ground truth), use the checker, not a model. The reward becomes a measurement instead of an approximation, and the reward-hacking surface shrinks (it does not vanish: an agent with file access can still edit the test).\n",
"\n",
"**Audit the policy under the true objective.** We evaluated the proxy-optimal policy under the true reward and found zero. The production analog is to run the RLHF-fine-tuned model on adversarial probes designed to elicit the failure mode (sycophancy probes, length-confound regressions, calibration checks) rather than on the distribution you optimized. The full alignment-failure taxonomy and red-team discipline live in Ch 24; this chapter's job was to show you the mechanism in code you can run.\n"
]
},
{
"cell_type": "markdown",
"id": "1fc73c19",
"metadata": {},
"source": [
"## Test yourself\n",
"\n",
"Three parts: concept self-checks with folded answers, two auto-checked problems, and a capstone with a rubric and a folded reference. Try before you peek. Every answer is in this notebook; if unsure, re-run that section.\n"
]
},
{
"cell_type": "markdown",
"id": "868b1e36",
"metadata": {},
"source": [
"### Part A — Concepts\n",
"\n",
"1. Value iteration and policy iteration both solve the gridworld. In Part 2 their values agreed to $10^{-15}$. What does that agreement prove? AnswerThat the Bellman optimality equation has a unique fixed point and both methods found it. Two independent algorithms agreeing to machine precision is strong evidence both are correct; it is the cheapest correctness oracle in the notebook.\n",
"2. The broken value iteration in Part 2 returned an all-zero $V$ with no error. What single check caught it, and why did a shape check not? AnswerThe Bellman residual $\\max_s|\\max_a Q(s,a) - V(s)|$. The broken output had the right shape (16,) full of zeros, so a shape check passed; only a property check that the output satisfies the equation it claims to solve detected the silent wrong answer.\n",
"3. Q-learning's update uses $\\max_{a'} Q(s',a')$ regardless of the action actually taken next. What property of the algorithm does that give it? AnswerOff-policy: it learns the value of the greedy policy while behaving with a different ($\\epsilon$-greedy) policy. SARSA, which uses the actually-taken next action, is the on-policy cousin.\n",
"4. In Part 3, the greedy-from-the-start agent recovered fewer optimal actions than the $\\epsilon$-greedy agent, even though the world is fully observable. Why? AnswerObservability is about the state, not the value estimates. A greedy agent commits to early-and-possibly-wrong $Q$ values and never tries the actions that would correct them. Exploration is coverage of the action space under uncertainty.\n",
"5. The from-scratch policy gradient in Part 4 was $(a - p)\\,s$. How did we know it was right before training anything? AnswerWe compared it to a central finite-difference estimate of $\\log\\pi(a)$ and it matched to $10^{-6}$. Verifying the gradient primitive against a numerical derivative is the move that lets you trust autograd downstream.\n",
"6. GAE at $\\lambda=0$ and $\\lambda=1$ reduce to what two estimators? Which is biased, which is high-variance? Answer$\\lambda=0$ is one-step TD (biased, low variance); $\\lambda=1$ is the Monte-Carlo return (unbiased, high variance). $\\lambda$ interpolates the bias-variance tradeoff; 0.95 is the usual default.\n",
"7. In the PPO surrogate, with a *negative* advantage and a ratio of 2.0, the objective was $-2.0$, not the clipped $-1.2$. Why does the clip not protect that case? AnswerThe `min` takes the more pessimistic term. With $A<0$, the unclipped $r\\cdot A = -2.0$ is more negative than the clipped $-1.2$, so PPO keeps it. The clip only bounds the optimistic direction; a bad action made more likely still gets penalized.\n",
"8. Why does RLHF add a KL penalty to the reward instead of just optimizing the reward model's score? AnswerTo bound how far the policy drifts from the reference (SFT) model. Without it, the optimizer exploits the reward model's failure modes and the policy collapses (Part 5's KL=0 run). The KL is a regularizer, not a fix for a wrong reward.\n",
"9. What does DPO remove from the RLHF pipeline, and what is the value of its loss when the policy equals the reference? AnswerIt removes the explicit reward model and the PPO rollout: the reward is folded into a closed-form loss on preference triples. At policy=reference every log-ratio is zero, the margin is zero, and the loss is exactly $\\log 2 \\approx 0.693$ (Part 5's check).\n",
"10. In Part 6 the proxy-optimal policy earned zero true reward. Was the policy wrong, the algorithm wrong, or the reward wrong? AnswerThe reward. The policy optimally maximized the proxy it was given; the algorithm found the true optimum of the proxy MDP. The $+0.3$ shaping bonus redefined the task. Optimizing harder makes this worse, not better.\n"
]
},
{
"cell_type": "markdown",
"id": "93c85a39",
"metadata": {},
"source": [
"### Part B1 — On-policy SARSA\n",
"`Difficulty 3/5 · ~12 min`\n",
"\n",
"You built Q-learning (off-policy). SARSA is the on-policy cousin: its target uses the value of the *actually taken* next action $a'$ instead of $\\max_{a'}$. Fill in `sarsa_target(Q, r, s_next, a_next, done, gamma)`. The check pins it against a hand-computed value and confirms it differs from the Q-learning target when the greedy next action is not the taken one.\n"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "dc382014",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:46.132697Z",
"iopub.status.busy": "2026-06-11T20:07:46.132600Z",
"iopub.status.idle": "2026-06-11T20:07:46.136598Z",
"shell.execute_reply": "2026-06-11T20:07:46.136165Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] B1 SARSA target: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def sarsa_target(Q, r, s_next, a_next, done, gamma):\n",
" \"\"\"SARSA bootstrap target: r + gamma * Q[s_next, a_next], or just r if done.\"\"\"\n",
" # TODO: terminal -> r ; else r + gamma * Q[s_next, a_next] (the TAKEN next action)\n",
" result = None\n",
" attempted(result)\n",
" return result\n",
"\n",
"def _sarsa():\n",
" Q = np.array([[0.0, 0.0], [1.0, 0.5]])\n",
" # taken next action is a_next=1 with Q[1,1]=0.5: target = 0 + 0.9*0.5 = 0.45\n",
" got = sarsa_target(Q, 0.0, 1, 1, False, 0.9)\n",
" assert abs(got - 0.45) < 1e-9, f\"got {got}, expected 0.45 = 0 + 0.9*Q[1,1]=0.9*0.5\"\n",
" # the Q-learning target would have used max(Q[1])=1.0 -> 0.9, so they differ here\n",
" assert abs(got - 0.9) > 1e-6, \"SARSA must use the taken action, not the max\"\n",
" # terminal: target is r only\n",
" assert abs(sarsa_target(Q, 1.0, 1, 0, True, 0.9) - 1.0) < 1e-9, \"terminal target is r\"\n",
"\n",
"check(\"B1 SARSA target\", _sarsa)"
]
},
{
"cell_type": "markdown",
"id": "bcab2d25",
"metadata": {},
"source": [
"Hint 1Identical structure to the Q-learning target, but index `Q[s_next, a_next]` instead of taking `Q[s_next].max()`. Gate the bootstrap on `not done`.\n",
"Solution\n",
"\n",
"```python\n",
"def sarsa_target(Q, r, s_next, a_next, done, gamma):\n",
" if done:\n",
" return r\n",
" return r + gamma * Q[s_next, a_next]\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 46,
"id": "0a582563",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:46.137427Z",
"iopub.status.busy": "2026-06-11T20:07:46.137348Z",
"iopub.status.idle": "2026-06-11T20:07:46.140550Z",
"shell.execute_reply": "2026-06-11T20:07:46.140241Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] B1 SARSA target\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def sarsa_target(Q, r, s_next, a_next, done, gamma):\n",
" if done:\n",
" return r\n",
" return r + gamma * Q[s_next, a_next]\n",
"\n",
"check(\"B1 SARSA target\", _sarsa, required=True)"
]
},
{
"cell_type": "markdown",
"id": "953042ae",
"metadata": {},
"source": [
"### Part B2 — The per-token KL penalty\n",
"`Difficulty 2/5 · ~10 min`\n",
"\n",
"RLHF shapes a per-token reward as `-kl_coef * (logp_policy - logp_ref)`. Fill in `kl_shaped_reward(logp_policy, logp_ref, scalar_reward, kl_coef)`: subtract the scaled KL at every token and add the scalar reward at the *last* token only. The check pins the shape and the placement.\n"
]
},
{
"cell_type": "code",
"execution_count": 47,
"id": "f86cf271",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:46.141300Z",
"iopub.status.busy": "2026-06-11T20:07:46.141222Z",
"iopub.status.idle": "2026-06-11T20:07:46.145206Z",
"shell.execute_reply": "2026-06-11T20:07:46.144930Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ -- ] B2 KL-shaped reward: not attempted yet — fill in the TODO above, then re-run.\n"
]
},
{
"data": {
"text/plain": [
"False"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def kl_shaped_reward(logp_policy, logp_ref, scalar_reward, kl_coef):\n",
" \"\"\"logp_*: (B, T). scalar_reward: (B,). Returns per-token reward (B, T).\"\"\"\n",
" # TODO 1: per-token reward = -kl_coef * (logp_policy - logp_ref)\n",
" rew = None\n",
" attempted(rew)\n",
" # TODO 2: add scalar_reward to the LAST token position only (rew[:, -1] += scalar_reward)\n",
" raise NotImplementedError # remove once the TODOs are done\n",
"\n",
"def _kl_shaped():\n",
" lp = torch.zeros(2, 4); ref = torch.zeros(2, 4) # zero KL everywhere\n",
" sc = torch.tensor([1.0, 2.0])\n",
" out = kl_shaped_reward(lp, ref, sc, kl_coef=0.1)\n",
" assert out.shape == (2, 4), f\"shape {tuple(out.shape)}, expected (2,4)\"\n",
" assert torch.allclose(out[:, :-1], torch.zeros(2, 3)), \"non-final tokens carry only the KL term (here 0)\"\n",
" assert torch.allclose(out[:, -1], sc), \"the scalar reward lands on the last token only\"\n",
" lp2 = torch.full((1, 3), -1.0); ref2 = torch.zeros(1, 3) # KL term is -0.1*(-1-0)=+0.1\n",
" out2 = kl_shaped_reward(lp2, ref2, torch.tensor([0.0]), kl_coef=0.1)\n",
" assert abs(out2[0, 0].item() - 0.1) < 1e-6, \"per-token KL reward = -kl_coef*(logp_policy - logp_ref)\"\n",
"\n",
"check(\"B2 KL-shaped reward\", _kl_shaped)"
]
},
{
"cell_type": "markdown",
"id": "8c162523",
"metadata": {},
"source": [
"Hint 1The KL term is dense (every token); the scalar reward is sparse (last token). Build the dense part first, then add to `rew[:, -1]`.\n",
"Solution\n",
"\n",
"```python\n",
"def kl_shaped_reward(logp_policy, logp_ref, scalar_reward, kl_coef):\n",
" rew = -kl_coef * (logp_policy - logp_ref)\n",
" rew = rew.clone()\n",
" rew[:, -1] += scalar_reward\n",
" return rew\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "849eba95",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-11T20:07:46.145857Z",
"iopub.status.busy": "2026-06-11T20:07:46.145790Z",
"iopub.status.idle": "2026-06-11T20:07:46.148743Z",
"shell.execute_reply": "2026-06-11T20:07:46.148409Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ ok ] B2 KL-shaped reward\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def kl_shaped_reward(logp_policy, logp_ref, scalar_reward, kl_coef):\n",
" rew = -kl_coef * (logp_policy - logp_ref)\n",
" rew = rew.clone()\n",
" rew[:, -1] += scalar_reward\n",
" return rew\n",
"\n",
"check(\"B2 KL-shaped reward\", _kl_shaped, required=True)"
]
},
{
"cell_type": "markdown",
"id": "64f139e0",
"metadata": {},
"source": [
"### Part C — Capstone: a reward-hacking audit\n",
"\n",
"Open project. Extend Part 6 into a small reward-hacking study and report what you find.\n",
"\n",
"**Deliverables**\n",
"1. A corridor (or grid) MDP with a *true* reward and a *proxy* reward that adds a shaping term you choose. Solve both with value iteration.\n",
"2. A sweep over the bonus magnitude: for each value, record (a) the proxy return of the proxy-optimal policy and (b) its *true* return. Plot both against the bonus.\n",
"3. Identify the bonus threshold at which the proxy-optimal policy stops reaching the goal. Report it.\n",
"4. One paragraph: relate your threshold to the RLHF KL coefficient. Both are knobs that trade reward against fidelity to a true objective.\n",
"\n",
"**Self-assessment (pass / partial / fail)**\n",
"- (a) Both MDPs solve to a Bellman residual near zero.\n",
"- (b) The true-return curve drops to zero past some bonus threshold, and you found that threshold.\n",
"- (c) Your proxy return keeps rising past the threshold while the true return is zero (the Goodhart divergence is visible in the plot).\n",
"- (d) You evaluate the proxy-optimal policy under the *true* reward, not the proxy.\n",
"- (e) The notebook runs top-to-bottom and your value iteration reuses the Part 1 function.\n",
"\n",
"My solution (reference, runs in ~1 s)\n",
"\n",
"```python\n",
"bonuses = np.linspace(0.0, 0.6, 25)\n",
"proxy_returns, true_returns = [], []\n",
"for b in bonuses:\n",
" Pp = np.zeros((CORRIDOR, 2, CORRIDOR)); Rp = np.zeros((CORRIDOR, 2, CORRIDOR))\n",
" for s in range(CORRIDOR):\n",
" for a in range(2):\n",
" ns = s if s == GOAL_C else (max(0, s-1) if a == 0 else min(CORRIDOR-1, s+1))\n",
" Pp[s, a, ns] = 1.0\n",
" if s != GOAL_C:\n",
" if ns == GOAL_C: Rp[s, a, ns] += 1.0\n",
" if ns == SHINY: Rp[s, a, ns] += b\n",
" Vp, pip = value_iteration(Pp, Rp, gamma=0.95)\n",
" proxy_returns.append(Vp[0])\n",
" true_returns.append(corridor_policy_eval(pip, P_true, R_true)[0])\n",
"true_returns = np.array(true_returns)\n",
"# first bonus where the true return collapses to ~0:\n",
"hacked = bonuses[np.argmax(true_returns < 1e-6)]\n",
"print(f\"true return collapses to zero at bonus ~{hacked:.3f}\")\n",
"plt.plot(bonuses, proxy_returns, label=\"proxy return (what we optimized)\")\n",
"plt.plot(bonuses, true_returns, label=\"true return (what we wanted)\")\n",
"plt.axvline(hacked, ls=\":\", color=\"grey\"); plt.xlabel(\"shaping bonus\"); plt.legend(); plt.show()\n",
"```\n",
"\n",
"The proxy return rises monotonically with the bonus while the true return falls off a cliff at the threshold: textbook Goodhart. The threshold is the bonus at which oscillating on the shiny cell out-values the discounted goal reward; it scales with $\\gamma$ and the distance to the goal, exactly like the RLHF KL coefficient scales the price of drifting from the reference.\n"
]
},
{
"cell_type": "markdown",
"id": "32a9cd7c",
"metadata": {},
"source": [
"## Reflection\n",
"\n",
"Write ~150 words, for yourself, on the dumbest bug you hit in this notebook and how you found it. A strong candidate is the early-break sentinel in Part 2: it returns a perfectly-shaped, all-zero value function with no error, and the only thing that catches it is asserting the Bellman residual rather than the shape. Did you trust an output because it had the right shape and a plausible magnitude? What property could you have checked instead? RL is the tier where correct code fails silently and buggy code sometimes succeeds, so the discipline is to assert a *property the answer must satisfy* (a residual, a known limit, a recovered ground truth), not just that something ran. Nobody grades this. Writing it is the point; the habit it builds is what separates an RL practitioner who can debug a flat reward curve from one who guesses.\n"
]
},
{
"cell_type": "markdown",
"id": "fdd141b8",
"metadata": {},
"source": [
"## Going further\n",
"- Sutton and Barto, *Reinforcement Learning* 2e, chapters 3-6 and 13 — the canonical treatment of MDPs, TD learning, and policy gradients at proper depth.\n",
"- Karpathy, *Pong from Pixels* — policy gradients from scratch in 130 lines of NumPy; the `(a - p)\\,s` gradient from Part 4 applied for six million frames.\n",
"- Schulman et al., *Proximal Policy Optimization Algorithms* (2017) and *High-Dimensional Continuous Control Using GAE* (2015) — the two papers behind Part 4.\n",
"- Rafailov et al., *Direct Preference Optimization* (2023) — the derivation that folds the reward model out of RLHF; Part 5's DPO loss is its equation 7.\n",
"- Shao et al., *DeepSeekMath* (2024) — GRPO, the group-baseline variant from Part 5.\n",
"- Lilian Weng, *Reward Hacking in Reinforcement Learning* (2024) — the taxonomy and survey for Part 6's failure mode.\n",
"- ARENA chapter 2 (intro-to-RL through RLHF) — the hands-on PyTorch curriculum this notebook compresses; do all six parts for the full stack.\n",
"\n",
"## What this enables\n",
"- **Ch 20 — Agents.** Tool-using LLM agents are RL by other means: the same MDP, reward, and policy vocabulary, plus the reward-tampering failure mode you saw in Part 6 when an agent can edit its own verifier.\n",
"- **Ch 22 — Mech Interp.** How RLHF changes a model's internals (which features get suppressed or redirected) is among the most active interpretability questions; it builds directly on the RLHF pipeline here.\n",
"- **Ch 23 — Eval Science.** You cannot evaluate an RL-trained model with the techniques used on a pretrained one; Goodhart applies to the eval itself. The held-out-reward discipline from the Safety lens is the entry point.\n",
"- **Ch 24 — Safety and Red Team.** This chapter is the algorithmic foundation; Ch 24 is the adversarial application and the full alignment-failure taxonomy that Part 6 previewed.\n",
"\n",
"The gap this notebook leaves: every environment here had a *known* true reward, which is why we could measure the hacking exactly. At frontier scale nobody can write down the true reward for \"be helpful and honest.\" That is the open problem, and it is why the reward signal, not the algorithm, is the research frontier.\n"
]
},
{
"cell_type": "markdown",
"id": "0363136f",
"metadata": {},
"source": [
"---\n",
"*Built top-to-bottom. If every check above printed `[ ok ]`, you reproduced the chapter: exact dynamic programming, learned control, policy gradients, RLHF, and a reward-hacking failure you can measure. Total running time and verification stamp written by CI.*\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "obvix-nb",
"language": "python",
"name": "obvix-nb"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"obvix": {
"title": "Ch 19 — Reinforcement Learning and RLHF"
}
},
"nbformat": 4,
"nbformat_minor": 5
}