From f2d7b6b31be53f748cb38e470d5d5be1a12dfff3 Mon Sep 17 00:00:00 2001
From: Hendrik Borras <hendrikborras@web.de>
Date: Mon, 1 Feb 2021 01:35:16 +0100
Subject: [PATCH] Added suggestions form testing (#270)
---
.../1-train-mlp-with-brevitas.ipynb | 266 +++++++++++-------
.../2-export-to-finn-and-verify.ipynb | 29 +-
.../3-build-accelerator-with-finn.ipynb | 8 +-
3 files changed, 190 insertions(+), 113 deletions(-)
diff --git a/notebooks/end2end_example/cybersecurity/1-train-mlp-with-brevitas.ipynb b/notebooks/end2end_example/cybersecurity/1-train-mlp-with-brevitas.ipynb
index 91a776f84..650125ee1 100644
--- a/notebooks/end2end_example/cybersecurity/1-train-mlp-with-brevitas.ipynb
+++ b/notebooks/end2end_example/cybersecurity/1-train-mlp-with-brevitas.ipynb
@@ -70,14 +70,14 @@
"output_type": "stream",
"text": [
"Requirement already satisfied: pandas in /workspace/.local/lib/python3.6/site-packages (1.1.5)\n",
+ "Requirement already satisfied: numpy>=1.15.4 in /opt/conda/lib/python3.6/site-packages (from pandas) (1.19.5)\n",
"Requirement already satisfied: pytz>=2017.2 in /opt/conda/lib/python3.6/site-packages (from pandas) (2019.1)\n",
- "Requirement already satisfied: numpy>=1.15.4 in /opt/conda/lib/python3.6/site-packages (from pandas) (1.19.4)\n",
"Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from pandas) (2.8.1)\n",
"Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.6/site-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)\n",
- "Requirement already satisfied: scikit-learn in /workspace/.local/lib/python3.6/site-packages (0.23.2)\n",
- "Requirement already satisfied: scipy>=0.19.1 in /opt/conda/lib/python3.6/site-packages (from scikit-learn) (1.5.2)\n",
+ "Requirement already satisfied: scikit-learn in /workspace/.local/lib/python3.6/site-packages (0.24.1)\n",
"Requirement already satisfied: joblib>=0.11 in /workspace/.local/lib/python3.6/site-packages (from scikit-learn) (1.0.0)\n",
- "Requirement already satisfied: numpy>=1.13.3 in /opt/conda/lib/python3.6/site-packages (from scikit-learn) (1.19.4)\n",
+ "Requirement already satisfied: scipy>=0.19.1 in /opt/conda/lib/python3.6/site-packages (from scikit-learn) (1.5.2)\n",
+ "Requirement already satisfied: numpy>=1.13.3 in /opt/conda/lib/python3.6/site-packages (from scikit-learn) (1.19.5)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /workspace/.local/lib/python3.6/site-packages (from scikit-learn) (2.1.0)\n",
"Requirement already satisfied: tqdm in /opt/conda/lib/python3.6/site-packages (4.31.1)\n"
]
@@ -106,75 +106,6 @@
"**This is important -- always import onnx before torch**. This is a workaround for a [known bug](https://github.com/onnx/onnx/issues/2394)."
]
},
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Define the Quantized MLP Model <a id='define_quantized_mlp'></a>\n",
- "\n",
- "We'll now define an MLP model that will be trained to perform inference with quantized weights and activations.\n",
- "For this, we'll use the quantization-aware training (QAT) capabilities offered by[Brevitas](https://github.com/Xilinx/brevitas).\n",
- "\n",
- "Our MLP will have four fully-connected (FC) layers in total: three hidden layers with 64 neurons, and a final output layer with a single output, all using 2-bit weights. We'll use 2-bit quantized ReLU activation functions, and apply batch normalization between each FC layer and its activation.\n",
- "\n",
- "In case you'd like to experiment with different quantization settings or topology parameters, we'll define all these topology settings as variables."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "input_size = 593 \n",
- "hidden1 = 64 \n",
- "hidden2 = 64\n",
- "hidden3 = 64\n",
- "weight_bit_width = 2\n",
- "act_bit_width = 2\n",
- "num_classes = 1 "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we can define our MLP using the layer primitives provided by Brevitas:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "from brevitas.nn import QuantLinear, QuantReLU\n",
- "import torch.nn as nn\n",
- "\n",
- "model = nn.Sequential(\n",
- " QuantLinear(input_size, hidden1, bias=True, weight_bit_width=weight_bit_width),\n",
- " nn.BatchNorm1d(hidden1),\n",
- " nn.Dropout(0.5),\n",
- " QuantReLU(bit_width=act_bit_width),\n",
- " QuantLinear(hidden1, hidden2, bias=True, weight_bit_width=weight_bit_width),\n",
- " nn.BatchNorm1d(hidden2),\n",
- " nn.Dropout(0.5),\n",
- " QuantReLU(bit_width=act_bit_width),\n",
- " QuantLinear(hidden2, hidden3, bias=True, weight_bit_width=weight_bit_width),\n",
- " nn.BatchNorm1d(hidden3),\n",
- " nn.Dropout(0.5),\n",
- " QuantReLU(bit_width=act_bit_width),\n",
- " QuantLinear(hidden3, num_classes, bias=True, weight_bit_width=weight_bit_width)\n",
- ")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Note that the MLP's output is not yet quantized. Even though we want the final output of our MLP to be a binary (0/1) value indicating the classification, we've only defined a single-neuron FC layer as the output. While training the network we'll pass that output through a sigmoid function as part of the loss criterion, which [gives better numerical stability](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html). Later on, after we're done training the network, we'll add a quantization node at the end before we export it to FINN."
- ]
- },
{
"cell_type": "markdown",
"metadata": {},
@@ -206,7 +137,7 @@
},
{
"cell_type": "code",
- "execution_count": 5,
+ "execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
@@ -216,7 +147,7 @@
},
{
"cell_type": "code",
- "execution_count": 6,
+ "execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
@@ -237,7 +168,7 @@
},
{
"cell_type": "code",
- "execution_count": 7,
+ "execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
@@ -248,6 +179,75 @@
"test_quantized_loader = DataLoader(test_quantized_dataset, batch_size=batch_size, shuffle=True) "
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Define the Quantized MLP Model <a id='define_quantized_mlp'></a>\n",
+ "\n",
+ "We'll now define an MLP model that will be trained to perform inference with quantized weights and activations.\n",
+ "For this, we'll use the quantization-aware training (QAT) capabilities offered by[Brevitas](https://github.com/Xilinx/brevitas).\n",
+ "\n",
+ "Our MLP will have four fully-connected (FC) layers in total: three hidden layers with 64 neurons, and a final output layer with a single output, all using 2-bit weights. We'll use 2-bit quantized ReLU activation functions, and apply batch normalization between each FC layer and its activation.\n",
+ "\n",
+ "In case you'd like to experiment with different quantization settings or topology parameters, we'll define all these topology settings as variables."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "input_size = 593 \n",
+ "hidden1 = 64 \n",
+ "hidden2 = 64\n",
+ "hidden3 = 64\n",
+ "weight_bit_width = 2\n",
+ "act_bit_width = 2\n",
+ "num_classes = 1 "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we can define our MLP using the layer primitives provided by Brevitas:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from brevitas.nn import QuantLinear, QuantReLU\n",
+ "import torch.nn as nn\n",
+ "\n",
+ "model = nn.Sequential(\n",
+ " QuantLinear(input_size, hidden1, bias=True, weight_bit_width=weight_bit_width),\n",
+ " nn.BatchNorm1d(hidden1),\n",
+ " nn.Dropout(0.5),\n",
+ " QuantReLU(bit_width=act_bit_width),\n",
+ " QuantLinear(hidden1, hidden2, bias=True, weight_bit_width=weight_bit_width),\n",
+ " nn.BatchNorm1d(hidden2),\n",
+ " nn.Dropout(0.5),\n",
+ " QuantReLU(bit_width=act_bit_width),\n",
+ " QuantLinear(hidden2, hidden3, bias=True, weight_bit_width=weight_bit_width),\n",
+ " nn.BatchNorm1d(hidden3),\n",
+ " nn.Dropout(0.5),\n",
+ " QuantReLU(bit_width=act_bit_width),\n",
+ " QuantLinear(hidden3, num_classes, bias=True, weight_bit_width=weight_bit_width)\n",
+ ")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that the MLP's output is not yet quantized. Even though we want the final output of our MLP to be a binary (0/1) value indicating the classification, we've only defined a single-neuron FC layer as the output. While training the network we'll pass that output through a sigmoid function as part of the loss criterion, which [gives better numerical stability](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html). Later on, after we're done training the network, we'll add a quantization node at the end before we export it to FINN."
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
@@ -335,7 +335,7 @@
"metadata": {},
"outputs": [],
"source": [
- "num_epochs = 5\n",
+ "num_epochs = 15\n",
"lr = 0.001 \n",
"\n",
"def display_loss_plot(losses, title=\"Training loss\", xlabel=\"Iterations\", ylabel=\"Loss\"):\n",
@@ -360,16 +360,28 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 12,
"metadata": {
"scrolled": true
},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Training loss = 0.130159 test accuracy = 0.797989: 100%|██████████| 15/15 [02:42<00:00, 10.80s/it]\n"
+ ]
+ }
+ ],
"source": [
"import numpy as np\n",
"from sklearn.metrics import accuracy_score\n",
"from tqdm import tqdm, trange\n",
"\n",
+ "# Setting seeds for reproducibility\n",
+ "torch.manual_seed(0)\n",
+ "np.random.seed(0)\n",
+ "\n",
"running_loss = []\n",
"running_test_acc = []\n",
"t = trange(num_epochs, desc=\"Training loss\", leave=True)\n",
@@ -385,12 +397,26 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 13,
"metadata": {
"scrolled": true
},
- "outputs": [],
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAknElEQVR4nO3de5hddX3v8fdnz33PTCbZkwshyUwCDWhEBE+4WLy1XgpqE2yf06LW6ql9kB4Rq7YV2x5t6bEPR1urPeJRTkV7WixSqTYqiuJdKiXhLlEgQK4kEHJP5j7zPX+sNZOdYc9kD5mVPbP35/U8+9lr/dZae3+TZ5LPrPVb6/dTRGBmZjZertIFmJnZzOSAMDOzkhwQZmZWkgPCzMxKckCYmVlJDggzMyvJAWE2AUnflPS26d53ijW8UtL26f5cs3LUV7oAs+kk6XDRah7oB4bT9XdGxI3lflZEXJLFvmazhQPCqkpEtI0uS9oM/H5E3D5+P0n1ETF0Mmszm218iclqwuilGkkfkLQL+LykeZK+Lmm3pH3p8tKiY34g6ffT5bdL+omkv0n3fULSJc9x3xWSfiTpkKTbJV0n6Z/L/HM8P/2u/ZIekrSmaNvrJG1MP3eHpD9K2+enf7b9kvZK+rEk/9u34/IPidWSU4AC0A1cTvLz//l0vQvoBT41yfEXAA8D84GPAp+TpOew7xeBu4BO4C+At5ZTvKQG4GvAt4GFwLuBGyWdme7yOZLLaO3AWcD30vb3A9uBBcAi4E8Bj7Fjx+WAsFoyAnw4Ivojojci9kTELRHRExGHgI8Ar5jk+C0R8X8jYhj4R2AxyX+4Ze8rqQs4D/hQRAxExE+AdWXWfyHQBlybHvs94OvAm9Ltg8AqSXMiYl9E3FPUvhjojojBiPhxeBA2K4MDwmrJ7ojoG12RlJf0WUlbJB0EfgTMlVQ3wfG7RhcioiddbJvivqcCe4vaALaVWf+pwLaIGClq2wIsSZd/E3gdsEXSDyW9JG3/GLAJ+LakxyVdXeb3WY1zQFgtGf9b8/uBM4ELImIO8PK0faLLRtNhJ1CQlC9qW1bmsU8Cy8b1H3QBOwAiYn1ErCW5/PRV4Oa0/VBEvD8iTgPWAO+T9KoT+2NYLXBAWC1rJ+l32C+pAHw46y+MiC3ABuAvJDWmv+X/epmH/yfQA/yJpAZJr0yPvSn9rLdI6oiIQeAgySU1JL1B0i+lfSAHSG77HSn5DWZFHBBWyz4BtADPAHcC3zpJ3/sW4CXAHuB/Al8ieV5jUhExQBIIl5DU/GngdyPiF+kubwU2p5fLrki/B2AlcDtwGPgp8OmI+P60/Wmsasl9VWaVJelLwC8iIvMzGLOp8BmE2Ukm6TxJp0vKSboYWEvSZ2A2o/hJarOT7xTg30ieg9gO/EFE3FvZksyezZeYzMysJF9iMjOzkqrmEtP8+fNj+fLllS7DzGxWufvuu5+JiAWltlVNQCxfvpwNGzZUugwzs1lF0paJtvkSk5mZleSAMDOzkhwQZmZWkgPCzMxKckCYmVlJDggzMyvJAWFmZiXVfEAc6B3kE7c/wgPb91e6FDOzGaVqHpR7riT4xO2P0tJQx9lL51a6HDOzGaPmzyDmNDcwL9/Alr09x9/ZzKyG1HxAAHR1trJ1jwPCzKyYAwLoLuTZ6jMIM7NjOCCArkKeHft7GRz2PO5mZqMcEEBXZ57hkeDJ/b2VLsXMbMZwQJBcYgLY4n4IM7MxDgigu7MVwP0QZmZFHBDAwvYmGutzDggzsyIOCCCXE12FPFv2HKl0KWZmM4YDItVdyLsPwsysiAMi1dWZZ9veHiKi0qWYmc0IDohUVyHPkYFh9hwZqHQpZmYzggMi1d3pW13NzIo5IFJdhdFbXd1RbWYGDogxS+e1IMHWPX6a2swMMg4ISRdLeljSJklXT7Lfb0oKSauL2j6YHvewpF/Lsk6A5oY6TpnTzBafQZiZARlOGCSpDrgOeA2wHVgvaV1EbBy3XzvwHuA/i9pWAZcBLwBOBW6XdEZEDGdVLyQd1R7228wskeUZxPnApoh4PCIGgJuAtSX2+yvgfwF9RW1rgZsioj8ingA2pZ+Xqe7OvCcOMjNLZRkQS4BtRevb07Yxkl4MLIuIb0z12PT4yyVtkLRh9+7dJ1xwVyHP7kP99A5keqJiZjYrVKyTWlIO+Djw/uf6GRFxfUSsjojVCxYsOOGaujxon5nZmCwDYgewrGh9ado2qh04C/iBpM3AhcC6tKP6eMdm4uiw3+6oNjPLMiDWAyslrZDUSNLpvG50Y0QciIj5EbE8IpYDdwJrImJDut9lkpokrQBWAndlWCtw9GE5n0GYmWV4F1NEDEm6ErgNqANuiIiHJF0DbIiIdZMc+5Ckm4GNwBDwrqzvYALoaGmgvbneAWFmRoYBARARtwK3jmv70AT7vnLc+keAj2RWXAmSkjuZfKurmZmfpB6vu9DqMwgzMxwQz9LVmWf7vh6GRzzst5nVNgfEOF2FPIPDwc4DHpPJzGqbA2Kc0VtdPeSGmdU6B8Q4XaPzQrgfwsxqnANinMUdLTTUyXcymVnNc0CMU5cTS+cl81ObmdUyB0QJXYW854Uws5rngChh9GG5CN/qama1ywFRQlchz6G+Ifb3DFa6FDOzinFAlNBV8KB9ZmYOiBK603khfKurmdUyB0QJY2cQnhfCzGqYA6KElsY6FrY3+VkIM6tpDogJdBXy7oMws5rmgJhAV6cDwsxqmwNiAt2FVnYd7KNvMPOJ7MzMZiQHxAS6O/NEwPZ9Posws9rkgJjAMj8LYWY1zgExge7RYb99J5OZ1SgHxAQ6WxtpbaxzQJhZzXJATEASXZ2tvsRkZjXLATGJrkKLA8LMapYDYhLd6RnEyIiH/Taz2pNpQEi6WNLDkjZJurrE9iskPSjpPkk/kbQqbV8uqTdtv0/SZ7KscyJdhTwDQyM8daivEl9vZlZR9Vl9sKQ64DrgNcB2YL2kdRGxsWi3L0bEZ9L91wAfBy5Otz0WEedkVV85iu9kWtzRUslSzMxOuizPIM4HNkXE4xExANwErC3eISIOFq22AjPqWo7nhTCzWpZlQCwBthWtb0/bjiHpXZIeAz4KXFW0aYWkeyX9UNLLSn2BpMslbZC0Yffu3dNZOwCnzm2hLie2+lZXM6tBFe+kjojrIuJ04APAn6fNO4GuiDgXeB/wRUlzShx7fUSsjojVCxYsmPbaGupyLJnb4omDzKwmZRkQO4BlRetL07aJ3ARcChAR/RGxJ12+G3gMOCObMifnYb/NrFZlGRDrgZWSVkhqBC4D1hXvIGll0errgUfT9gVpJzeSTgNWAo9nWOuEujrznlnOzGpSZncxRcSQpCuB24A64IaIeEjSNcCGiFgHXCnp1cAgsA94W3r4y4FrJA0CI8AVEbE3q1on013Is69nkIN9g8xpbqhECWZmFZFZQABExK3ArePaPlS0/J4JjrsFuCXL2so1eqvr1j09nLWko8LVmJmdPBXvpJ7pPOy3mdUqB8RxdHe2Ah7228xqjwPiONqa6ulsbWTrXndUm1ltcUCUoasz7zMIM6s5Dogy+FkIM6tFDogydBfyPLm/l4GhkUqXYmZ20jggytDV2cpIwI79vZUuxczspHFAlOHosN/uqDaz2uGAKMPosN/b3A9hZjXEAVGGhe1NNDfkfCeTmdUUB0QZJNFVyHvYbzOrKQ6IMnUVWj1xkJnVFAdEmUafhYiYUbOimpllxgFRpu7OPL2Dw+w+3F/pUszMTgoHRJm6iob9NjOrBQ6IMnUXRp+FcECYWW1wQJRpybwWJM8LYWa1wwFRpqb6Ok7taHFAmFnNcEBMQVch7+E2zKxmOCCmoLvTw36bWe1wQEzBskKeZw4PcKR/qNKlmJllzgExBaOjuvoswsxqgQNiCroLrYBvdTWz2uCAmIKxh+X2uqPazKpfpgEh6WJJD0vaJOnqEtuvkPSgpPsk/UTSqqJtH0yPe1jSr2VZZ7k6WhroaGnwJSYzqwmZBYSkOuA64BJgFfCm4gBIfTEiXhgR5wAfBT6eHrsKuAx4AXAx8On08yquuzPvS0xmVhOyPIM4H9gUEY9HxABwE7C2eIeIOFi02gqMDpW6FrgpIvoj4glgU/p5FTc6qquZWbXLMiCWANuK1renbceQ9C5Jj5GcQVw1xWMvl7RB0obdu3dPW+GT6e7Ms2NfL0PDIyfl+8zMKqXindQRcV1EnA58APjzKR57fUSsjojVCxYsyKbAcboKeYZGgp0H+k7K95mZVUqWAbEDWFa0vjRtm8hNwKXP8diTpsu3uppZjcgyINYDKyWtkNRI0um8rngHSSuLVl8PPJourwMuk9QkaQWwErgrw1rLNvqw3Bbf6mpmVa4+qw+OiCFJVwK3AXXADRHxkKRrgA0RsQ64UtKrgUFgH/C29NiHJN0MbASGgHdFxHBWtU7FKXOaaazLeeIgM6t6mQUEQETcCtw6ru1DRcvvmeTYjwAfya665yaXE0sLHvbbzKpfWZeYJLVKyqXLZ0haI6kh29Jmru6Cn4Uws+pXbh/Ej4BmSUuAbwNvBb6QVVEzXXdnK1v39hARx9/ZzGyWKjcgFBE9wG8An46I/0rylHNNWlbIc7h/iL1HBipdiplZZsoOCEkvAd4CfCNtmxFDX1RCd8HDfptZ9Ss3IP4Q+CDwlfQOo9OA72dW1QzneSHMrBaUdRdTRPwQ+CFA2ln9TERcNflR1WtZegbhjmozq2bl3sX0RUlzJLUCPwM2SvrjbEubuZob6lg0p8kBYWZVrdxLTKvSkVcvBb4JrCC5k6lmdRda2eZLTGZWxcoNiIb0uYdLgXURMcjRoblrUldn3sNtmFlVKzcgPgtsJpmz4UeSuoGDkx5R5boLeZ462E/f4IwYAcTMbNqVFRAR8fcRsSQiXheJLcCvZFzbjNblO5nMrMqV20ndIenjo5PzSPpbkrOJmtU1+iyEO6rNrEqVe4npBuAQ8Fvp6yDw+ayKmg26O9N5IXwGYWZVqtzRXE+PiN8sWv9LSfdlUM+sMS/fQHtTPVv3uKPazKpTuWcQvZJeOroi6SKgN5uSZgdJLCvkfQZhZlWr3DOIK4D/J6kjXR+b3KeWdXfmefipQ5Uuw8wsE+XexXR/RLwIOBs4OyLOBX4108pmga7OPNv39jI8UtOPhJhZlZrSnNQRcTB9ohrgfRnUM6t0F1oZGB5h18G+SpdiZjbtphQQ42jaqpilusYG7XNHtZlVnxMJiJq/rjI67LfHZDKzajRpJ7WkQ5QOAgEtmVQ0iyzuaKY+J4/qamZVadKAiIj2k1XIbFRfl2PpvBbf6mpmVelELjEZyeRBHm7DzKqRA+IEdXfmPWCfmVWlTANC0sWSHpa0SdLVJba/T9JGSQ9I+m46jPjotmFJ96WvdVnWeSK6C60c6B3kQM9gpUsxM5tWmQWEpDrgOuASYBXwJkmrxu12L7A6Is4Gvgx8tGhbb0Sck77WZFXniRod9tuTB5lZtcnyDOJ8YFNEPB4RA8BNwNriHSLi+xExen3mTmBphvVk4uizEL7MZGbVJcuAWAJsK1rfnrZN5B0k812Pak7nnrhT0qWlDpB0+egcFbt37z7hgp+LsXkh3A9hZlWm3MH6MiXpd4DVwCuKmrsjYoek04DvSXowIh4rPi4irgeuB1i9enVFHtxrbapnfluT72Qys6qT5RnEDmBZ0frStO0Ykl4N/BmwJiL6R9sjYkf6/jjwA+DcDGs9Id2defdBmFnVyTIg1gMrJa2Q1AhcBhxzN5Kkc4HPkoTD00Xt8yQ1pcvzgYuAjRnWekK6/CyEmVWhzAIiIoaAK4HbgJ8DN0fEQ5KukTR6V9LHgDbgX8fdzvp8YIOk+4HvA9dGxIwOiJ0H++gfGq50KWZm0ybTPoiIuBW4dVzbh4qWXz3Bcf8BvDDL2qZTd2eeCNi+r5fTF7RVuhwzs2nhJ6mnweiorr7MZGbVxAExDZZ5Xggzq0IOiGmwoK2JfGMdW/f2VroUM7Np44CYBpKSO5l8q6uZVREHxDTpKuQ93IaZVRUHxDRJziB6GBmp+ZlYzaxKOCCmSXdnnv6hEXYf7j/+zmZms4ADYpp0dbYCHtXVzKqHA2KadPlWVzOrMg6IabJkbgs5edhvM6seDohp0lif49S5LQ4IM6saDohp1N3pW13NrHo4IKbR6K2uZmbVwAExjboKrew9MsChvsFKl2JmdsIcENNobFRXn0WYWRVwQEyj0VtdPey3mVUDB8Q06krPILb4DMLMqoADYhrNaW5gXr7BdzKZWVVwQEyzrs5WtvkMwsyqgANimnUX8mzxvBBmVgUcENOsq5Dnyf19DA6PVLoUM7MT4oCYZl2deYZHgif3e/pRM5vdHBDTrHtsVFf3Q5jZ7OaAmGbdo/NCuKPazGa5TANC0sWSHpa0SdLVJba/T9JGSQ9I+q6k7qJtb5P0aPp6W5Z1TqeF7U001ufY6nkhzGyWyywgJNUB1wGXAKuAN0laNW63e4HVEXE28GXgo+mxBeDDwAXA+cCHJc3LqtbplMvJg/aZWVXI8gzifGBTRDweEQPATcDa4h0i4vsRMfo/6Z3A0nT514DvRMTeiNgHfAe4OMNap1V3wcN+m9nsl2VALAG2Fa1vT9sm8g7gm8/x2BmlqzM5g4iISpdiZvaczYhOakm/A6wGPjbF4y6XtEHSht27d2dT3HPQVcjTMzDMM4cHKl2KmdlzlmVA7ACWFa0vTduOIenVwJ8BayKifyrHRsT1EbE6IlYvWLBg2go/UR7228yqQZYBsR5YKWmFpEbgMmBd8Q6SzgU+SxIOTxdtug14raR5aef0a9O2WaGrkNzqutVDbpjZLFaf1QdHxJCkK0n+Y68DboiIhyRdA2yIiHUkl5TagH+VBLA1ItZExF5Jf0USMgDXRMTerGqdbkvntSD5YTkzm90yCwiAiLgVuHVc24eKll89ybE3ADdkV112mhvqOGVOsycOMrNZbUZ0UlcjPwthZrOdAyIj3Z15D7dhZrOaAyIj3Z2t7D7UT8/AUKVLMTN7ThwQGXn+4nYA/vCm+zjQO1jhaszMps4BkZFfOXMh/+MNq/jeL55mzad+wkNPHqh0SWZmU+KAyIgk3vHSFXzpnRfSPzjCGz/9H3xp/VYPv2Fms4YDImP/pbvAN656KResKPCBWx7kj7/8AL0Dw5Uuy8zsuBwQJ0FnWxNf+G/n855XreSWe7bzxk/fweO7D1e6LDOzSTkgTpK6nHjva87g828/j6cO9rHmU3dw64M7K12WmdmEHBAn2SvPXMjXr3oZv7Swjf9+4z1c87WNDAyNVLosM7NncUBUwJK5Ldz8zpfw9l9ezg13PMFl1/+UnQd6K12WmdkxHBAV0lif4y/WvIBPvflcHt51iNf//U/48aMzZ04LMzMHRIW94exTWffulzK/rZHfveEuPnn7o4yM+FZYM6s8B8QMcPqCNr76rot44zlL+LvbH+HtX1jP3iOejc7MKssBMUPkG+v52996EX/9xhdy52N7eP3f/5h7tu6rdFlmVsMcEDOIJN58QRe3/MEvU18nfuszP+Xzdzzhp6/NrCIcEDPQC5d28PUrX8Yrz1zAX35tI1d+8V4O9XnAPzM7uRwQM1RHvoHr37qaqy95Ht96aBdrP3UHv9h1sNJlmVkNcUDMYLmcuOIVp3Pj71/Aof4hLr3uDm65e3ulyzKzGuGAmAUuPK2Tb1z1Us5ZNpf3/+v9XPFPd3Pzhm1sfuaI+yfMLDP1lS7AyrOwvZl/fscFfPK7j/JPd27hWw/tAmBBexPnLZ/H6u4C568o8LxT2qmvc+6b2YlTtfwGunr16tiwYUOlyzgpRkaCx3Yf5q7Ne9mweR93PbGXHfuToTramuo5t2su5y0vcN7yAucsm0tLY12FKzazmUrS3RGxuuQ2B0R1eHJ/L+vTwFi/eS8PP3WICGioE2ct6eD85QVWLy+wunse81obK12umc0QDogadKBnkLu37mX95n2sf2IvD2w/wMBwMmrsyoVtnLeiwHnL53He8gJL5rYgqcIVm1klOCCMvsFhHth+gPWb97J+817u3ryPQ/1DACzuaOa85QXOXtrBGYvaOWNRO4vmNDk0zGrAZAGRaSe1pIuBTwJ1wD9ExLXjtr8c+ARwNnBZRHy5aNsw8GC6ujUi1mRZa7Vrbqjj/BVJRzbA8Ejw8K5DbNiyl7ueSF7r7n9ybP/25nrOXNTOykXtnLGobWx5flujg8OsRmR2BiGpDngEeA2wHVgPvCkiNhbtsxyYA/wRsG5cQByOiLZyv89nECdu75EBHnnqUNHrMI88dYj9PUef4p6Xbxg7yzhjUdvYsvs1zGanSp1BnA9siojH0yJuAtYCYwEREZvTbZ5SbQYotDZy4WmdXHha51hbRLD7cD+PPnWYh3cd4tGnk+D46r07xi5RAcxva+LMU9pYufBoeKxc1E5HS0Ml/ihmNg2yDIglwLai9e3ABVM4vlnSBmAIuDYivjp+B0mXA5cDdHV1PfdKbUKSWNjezML2Zi76pflj7RHBroN9yVnGrvSM4+nD3LxhGz0Dw2P7nTKnmaXzWlg8t4VTO5o5paOZxR0tnDo3ee9sbSSX8yUrs5loJj8o1x0ROySdBnxP0oMR8VjxDhFxPXA9JJeYKlFkrZLE4o4WFne08IozFoy1j4wEO/b3jp1pPPLUIZ7c38sD2/dz20N9z5p/u7Eux6KOpiQ0OppZPLeFxWmILO5o5tS5LczLN7jfw6wCsgyIHcCyovWlaVtZImJH+v64pB8A5wKPTXqQVVwuJ5YV8iwr5PnV5y06ZltEsPfIADsP9PHk/l52HuhLX73s3N/Hhi37eOrBnQwOH5v1TfW5o6Ext5nFHc0smtNMS0MdzWOvXPJef3S5afS9PkdjXc4hYzZFWQbEemClpBUkwXAZ8OZyDpQ0D+iJiH5J84GLgI9mVqmdFJLobGuis62Js5Z0lNxnZCR45kg/O/cnwfHk/j52HTwaKHc+toenDvUzPMVpWXOCpqLwGA2O8eEyr7WB7s5Wugv55L0zT2vTTD7RNstOZj/5ETEk6UrgNpLbXG+IiIckXQNsiIh1ks4DvgLMA35d0l9GxAuA5wOfTTuvcyR9EBsn+CqrIrnc0T6PFy2bW3Kf4ZHkTKRvcDh9jdA3VLQ82j40Qv/guPah4n1G6E+P23M4+bw9mweeNd3rgvYmlnfm6Sq0srwzT/f89L2z1Z3wVtX8oJzZOAf7Btm6p4cte3rYvOcIW/YcYfOeHrbsOcJTB/uP2XdevoGuzqOBUfxeaPUzIzbzVexBObPZaE5zA2ct6Sh5Gax3YJite48Njq17erh7yz6+dv+TFF/5amuqp7szz/LOVhbOaaKjpYG5LQ3MzTfSkW84Zn1Oc71H4bUZxwFhNgUtjXWceUo7Z57S/qxt/UPDbN/Xy5Y9R9hSdAaycedBfvRI/zHPjZTS3lyfhEa+gbktSYjMbWko3ZYGTL6hnuZGd8JbNhwQZtOkqb6O0xe0cfqC0gMADA2PcLBviP09A+zvHeRAzyAHegfH1vcXrR/oHeTJA70c6Blkf+/gcTvlJY7e1VWfo7kx6XRvaUw74evrxtqaG3Jj+7Y0Hu2sbynqvK+vEw11Oepyoj4n6uty1OdEXU401Im6XC5tV7pPcszoPvW55Fib3RwQZidJfV2OQmsjhSkOSxIRHO4fSsNjcOx9f+8AvQPD9A8lne69A0knfO9A0mnfPzhMb9oZv79nkN7BYfoHR9K25DXFm8GmROJoqORyNNbnaGmsI99YR76xPn2vo6WxntbGugm35RvqyDcdbW9pqKO1KVluqveZU5YcEGYznCTamxtob25g6bzp+9yIYGB4JLmbqyhM+gaHGRoJhkeCoZERhoZHl4Oh4ZGxbYPDIyXbk/VgeGSEwdG24aB/KAmxIwND9Awky/t7BukpWu8ZHJ7SLcw5QWtTPe1N9bQ119PWVE97cwNtzWlb2t7e3DBun+TV1pTsm2+o8xP9JTggzGqUJJrq62iqr4MZcrtuRNA/NDIWFj39SXj0DAzTOzjEkf40SAaGOJKGyuH+oeTVN8Sh/uQS3bZ9PRzuS9qLh36ZiJTcVDAaIq1N9QgYiaSmkYCR9D1ZP9oWY9uCkZHJ95egtbGe1qbkLKitqT5dr6ctbRtrn6gtPf5k3NTggDCzGUPS2IOM03WyNDQ8wpH+YQ71Dx4Nkr4hDqXLh/sHOdw3xMG+o0FzZGBorJ6cIJe+H7suVLQtWS/annv2/sMjwZGBIY70J2F3qG+IXQf6OJKG3JGB8s+gmupzY6HxomVz+d9vOnea/saOckCYWVWrr8vRkc/RkZ8ZZ0mTGT2DGg2QJDTS8Ehfh/uHi5aT91PntmRSjwPCzGyGKD6D6ix7Npzs+MkcMzMryQFhZmYlOSDMzKwkB4SZmZXkgDAzs5IcEGZmVpIDwszMSnJAmJlZSVUzo5yk3cCWE/iI+cAz01RO1mZTrTC76p1NtcLsqnc21Qqzq94TqbU7IhaU2lA1AXGiJG2YaNq9mWY21Qqzq97ZVCvMrnpnU60wu+rNqlZfYjIzs5IcEGZmVpID4qjrK13AFMymWmF21TubaoXZVe9sqhVmV72Z1Oo+CDMzK8lnEGZmVpIDwszMSqr5gJB0saSHJW2SdHWl65mMpGWSvi9po6SHJL2n0jUdj6Q6SfdK+nqlazkeSXMlfVnSLyT9XNJLKl3TRCS9N/0Z+Jmkf5HUXOmaikm6QdLTkn5W1FaQ9B1Jj6bv0zWr6AmZoNaPpT8HD0j6iqS5FSzxGKXqLdr2fkkhaf50fFdNB4SkOuA64BJgFfAmSasqW9WkhoD3R8Qq4ELgXTO8XoD3AD+vdBFl+iTwrYh4HvAiZmjdkpYAVwGrI+IsoA64rLJVPcsXgIvHtV0NfDciVgLfTddngi/w7Fq/A5wVEWcDjwAfPNlFTeILPLteJC0DXgtsna4vqumAAM4HNkXE4xExANwErK1wTROKiJ0RcU+6fIjkP7Alla1qYpKWAq8H/qHStRyPpA7g5cDnACJiICL2V7SoydUDLZLqgTzwZIXrOUZE/AjYO655LfCP6fI/ApeezJomUqrWiPh2RAylq3cCS096YROY4O8W4O+APwGm7c6jWg+IJcC2ovXtzOD/cItJWg6cC/xnhUuZzCdIfmBHKlxHOVYAu4HPp5fE/kFSa6WLKiUidgB/Q/Kb4k7gQER8u7JVlWVRROxMl3cBiypZzBT8HvDNShcxGUlrgR0Rcf90fm6tB8SsJKkNuAX4w4g4WOl6SpH0BuDpiLi70rWUqR54MfB/IuJc4Agz5xLIMdJr92tJQu1UoFXS71S2qqmJ5P76GX+PvaQ/I7m0e2Ola5mIpDzwp8CHpvuzaz0gdgDLitaXpm0zlqQGknC4MSL+rdL1TOIiYI2kzSSX7n5V0j9XtqRJbQe2R8ToGdmXSQJjJno18ERE7I6IQeDfgF+ucE3leErSYoD0/ekK1zMpSW8H3gC8JWb2A2Onk/yycH/6720pcI+kU070g2s9INYDKyWtkNRI0tG3rsI1TUiSSK6R/zwiPl7peiYTER+MiKURsZzk7/V7ETFjf8uNiF3ANklnpk2vAjZWsKTJbAUulJRPfyZexQztUB9nHfC2dPltwL9XsJZJSbqY5PLomojoqXQ9k4mIByNiYUQsT/+9bQdenP5Mn5CaDoi0E+pK4DaSf2A3R8RDla1qUhcBbyX5bfy+9PW6ShdVRd4N3CjpAeAc4K8rW05p6VnOl4F7gAdJ/h3PqGEhJP0L8FPgTEnbJb0DuBZ4jaRHSc6Crq1kjaMmqPVTQDvwnfTf2WcqWmSRCerN5rtm9pmTmZlVSk2fQZiZ2cQcEGZmVpIDwszMSnJAmJlZSQ4IMzMryQFhlpJ0OH1fLunN0/zZfzpu/T+m8/PNsuCAMHu25cCUAiIdNG8yxwRERMyGJ5+txjkgzJ7tWuBl6QNS703ntPiYpPXp/ADvBJD0Skk/lrSO9KlrSV+VdHc6V8Pladu1JCOv3ifpxrRt9GxF6Wf/TNKDkn676LN/UDQ/xY3pU9NIulbJnCAPSPqbk/63YzXjeL/1mNWiq4E/iog3AKT/0R+IiPMkNQF3SBodPfXFJPMGPJGu/15E7JXUAqyXdEtEXC3pyog4p8R3/QbJU9svAuanx/wo3XYu8AKSobzvAC6S9HPgjcDzIiJm0kQ2Vn18BmF2fK8FflfSfSTDq3cCK9NtdxWFA8BVku4nmUNgWdF+E3kp8C8RMRwRTwE/BM4r+uztETEC3Edy6esA0Ad8TtJvADN6nCCb3RwQZscn4N0RcU76WlE0/8KRsZ2kV5KMMfSSiHgRcC9wIlOB9hctDwP16fhh55OMxfQG4Fsn8Plmk3JAmD3bIZKB2kbdBvxBOtQ6ks6YYDKhDmBfRPRIeh7JtLCjBkePH+fHwG+n/RwLSGa1u2uiwtK5QDoi4lbgvSSXpswy4T4Is2d7ABhOLxV9gWSu6uUkY+yLZOa5S0sc9y3girSf4GGSy0yjrgcekHRPRLylqP0rwEuA+0km0PmTiNiVBkwp7cC/S2omObN533P6E5qVwaO5mplZSb7EZGZmJTkgzMysJAeEmZmV5IAwM7OSHBBmZlaSA8LMzEpyQJiZWUn/H/4OY3cvfA9TAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
"source": [
+ "%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"\n",
"loss_per_epoch = [np.mean(loss_per_epoch) for loss_per_epoch in running_loss]\n",
@@ -399,13 +425,57 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 14,
"metadata": {},
- "outputs": [],
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA360lEQVR4nO3deXxV1bn/8c83CSHMYQgyQ0AQQWUQUYvziHWsbS10cGpFr1PV1qptb+u1t712uNfhV4dqHTooaLVaKip11lKthDCDyBwCAQJkYMr8/P7YO3gMGU7IOTkZnvfrdV6cs/ba6zwnxvNkr7XXWjIznHPOuWglJToA55xzrYsnDuecc43iicM551yjeOJwzjnXKJ44nHPONYonDuecc43iicO1S5Jek3RFrOs61x7I53G41kLSnoiXnYFSoDJ8fa2ZPdP8UTnX/njicK2SpA3Ad8zszVqOpZhZRfNH1br4z8kdKu+qcq2epNMk5Uq6Q9JW4ClJPSW9IilfUkH4fFDEOe9K+k74/EpJ/5T0m7DueknnHWLdTEnvS9ot6U1JD0n6cx1xNxRjL0lPSdoSHn854tjFkhZJKpa0VtLUsHyDpLMi6t1d/f6ShkkySd+WlAO8HZb/RdJWSUVh7GMjzu8k6X8lbQyP/zMsmyPpphqfZ4mkLzXyP59rhTxxuLaiH9ALGArMIPjdfip8PQTYD/y2nvOPB1YBfYBfAU9I0iHUfRb4GOgN3A18q573bCjGPxF0yY0F+gL3AUiaDPwRuB1IB04BNtTzPjWdChwJnBu+fg0YGb5HNhDZ5fcb4FjgCwQ/3x8AVcAfgG9WV5I0DhgIzGlEHK61MjN/+KPVPQi+KM8Kn58GlAFp9dQfDxREvH6XoKsL4EpgTcSxzoAB/RpTl+DLvwLoHHH8z8Cfo/xMB2IE+hN8Qfespd7vgPsa+rmEr++ufn9gWBjr8HpiSA/r9CBIbPuBcbXUSwMKgJHh698ADyf698IfzfPwKw7XVuSbWUn1C0mdJf0u7GIpBt4H0iUl13H+1uonZrYvfNq1kXUHALsiygA21RVwAzEODtsqqOXUwcDautqNwoGYJCVLujfs7irmsyuXPuEjrbb3Cn/WzwHflJQETCe4QnLtgCcO11bUvMvje8ARwPFm1p2gOwegru6nWMgDeknqHFE2uJ769cW4KWwrvZbzNgEj6mhzL8FVULV+tdSJ/Fl9HbgYOIvgKmNYRAw7gJJ63usPwDeAM4F9ZvZhHfVcG+OJw7VV3Qi6WQol9QJ+Gu83NLONQBZwt6RUSScCFx5KjGaWRzD28HA4iN5BUnVieQK4StKZkpIkDZQ0Ojy2CJgW1p8EfKWBsLsR3Na8kyDh/CIihirgSeD/JA0Ir05OlNQxPP4hQXfa/+JXG+2KJw7XVt0PdCL4q/kj4PVmet9vACcSfBH/N0F3Tmkdde+n/hi/BZQDnwDbgVsAzOxj4CqCwfIi4D2CAXaA/yS4QigA/otgsL4+fwQ2ApuBFWEckb4PLAXmA7uAX/L5740/AkcTjOW4dsLncTgXR5KeAz4xs7hf8SSCpMuBGWZ2UqJjcc3HrziciyFJx0kaEXYhTSUYP3g5wWHFRTiWcz3wWKJjcc3LE4dzsdWP4PbdPcCDwH+Y2cKERhQHks4F8oFtNNwd5toY76pyzjnXKH7F4ZxzrlFSEh1Ac+jTp48NGzYs0WE451yrsmDBgh1mllGzvF0kjmHDhpGVlZXoMJxzrlWRtLG2cu+qcs451yieOJxzzjWKJw7nnHON4onDOedco3jicM451yhxTRySpkpaJWmNpDtrOT5E0juSFobbTn4x4thd4XmrwlmqUbXpnHMuvuKWOMLNaB4CzgPGANMljalR7cfA82Y2AZgGPByeOyZ8PRaYSrC0dHKUbTrnnIujeM7jmEywxeY6AEmzCBZ8WxFRx4Du4fMewJbw+cXALDMrBdZLWhO2RxRtOudcq1BSXsnCnEIW5xaSkiT6dO1I766p9O7SkT5dU+nZJZUOyS1vRCGeiWMgn982Mxc4vkadu4F/SLoJ6EKwC1n1uZH7AuSGZUTRJgCSZgAzAIYMGdL46J1zLsaKS8pZsKGAjzfs4uP1u1iSW0h5Zf3rBfbo1IHeXVPp0yVMKl1T6RUmlt5hWfXzHp06kJQUz00uA4meOT4deNrM/jfcLe1Pko6KRcNm9hjhcs+TJk3ylRydc80uf3cp88Mk8fH6XazcWowZdEgWRw/swbdPGs7kzJ4cO6QXADv3lrJzbxk795SyY08ZO/eUBWV7ytixp5Q12/fw7/VlFOwro7b1aZOTRM/OYSIJk8kd541mYHqnmH6ueCaOzXx+v+VBYVmkbxOMYWBmH0pKA/o0cG5DbTrnXLMzM3IL9vPx+l0HksW6HXsB6NQhmYlD07nlzFEcl9mTCYN70ik1+aA2enTuwPCDVoY6WEVlFQX7yj+XVHbt/SzRBEmnlMUFhTH+lIF4Jo75wEhJmQRf7tOAr9eok0Ow0f3Tko4E0gjW+J8NPCvp/4ABwEjgY0BRtOmcc3FXVWWsyd9z4Gpi/oZd5BWVANA9LYXJmb342nGDmZzZi6MG9ojpWEVKchIZ3TqS0a1jzNps1PvHq2Ezq5B0IzAXSAaeNLPlku4BssxsNvA94HFJtxIMlF9pwQYhyyU9TzDoXQHcYGaVALW1Ga/P4JxzkVZv2817n+YfSBQF+8oB6NutI5Mzex14jOrbrVnGGhKlXWzkNGnSJPPVcZ1zh6Kqynhn1XaenLeeeWt2AjC0d2cmD+vFcZm9OD6zF0N6dUZqe4lC0gIzm1SzPNGD48451yLtLa3gxexcnpq3gfU79tKvexo/mHoEl04YRL8eaYkOL6E8cTjnXITcgn388cONzPw4h90lFYwbnM6D0ydw3lH9WuScikTwxOGca/fMjAUbC3hy3npeX7YVSZx3VD+uPimTiUN6Jjq8FscTh3Ou3SqrqOK1ZXk88c/1LMktontaCtecMpzLTxwW87kPbYknDudcu7NrbxkzP87hjx9uYFtxKcMzuvCzS47iyxMH0jnVvxYb4j8h51y78em23Tw1bz1/zd5MaUUVJ4/sw71fPoZTR2a06dtnY80Th3OuTauqMt77NJ8n563ng9U76JiSxKUTB3HVlGGMOqxbosNrlTxxOOfapH1lFbyYvZmn5q1nXf5eDuvekdvPPYLpk4fQq0tqosNr1TxxOOdaHTNjT2kF+btLg8eeUnaE/1aXLdhYQHFJBccM6sED08Zz3lH9SU3x22ljwROHc67FKCmvPJAI8neXsiMiERxIEGFZSXnVQedX72nRp1sqZ4zuy7dOHMrEIT3b5KzuRPLE4ZxLmHlrdvDwu2vIKyohf3cpu0sqaq3Xq0sqGV2DRf2GDe1CRrdgP4qMbh3J6Jp2YMG/9Gbaj6K988ThnGt2+8oq+OVrn/CHDzcyqGcnxg1K55SRHcNE0PFAIqjeEc9nbLcsnjica8Cqrbu54dlsunRMoV/3jvTrnka/Hp3o16Mjh3VPo3+PTvTrnlbr/gruYAs2FvC95xexYec+rp6SyQ+mHkFaB//ZtSaeOJyrh5lxzyvL2VZcwvjB6azN38u/1uxkd+nBXSrd01Lo36MTh/VICxJMmFD69ehIv+6d6NcjjZ6dO7Tb/vbSikruf3M1v3tvLf17dGLmNSdw4ojeiQ7LHQJPHM7V491V+cxbs5OfXDCGq0/KPFC+t7SCrcUlbC0KH8UlbCsuIa8o+PeTvGLy95QetL1nakpSkEy6p3FYjzSOGtCdq6Zktvm7fZZvKeK25xazattuph03mB9fMIauHf3rp7Xy/3LO1aGisopfvLqSYb07880Thn7uWJeOKYzI6MqIjK51nl9eWUX+7tIgqRR9llSqE86S3EL+vngLsxdv4cHpE+ptq7WqqKzikXfX8sBbq+nZJZUnr5zEGaMPS3RYronimjgkTQUeINit7/dmdm+N4/cBp4cvOwN9zSxd0unAfRFVRwPTzOxlSU8DpwJF4bErzWxR/D6Fa6+ey9rE6u17ePSbEw/piqBDchID0jsxoJ7F8t5YsY0fvLCYCx78Jz+9cAxfO25wm+nKWrN9D997fhGLc4u4aNwA/uuisfT0iXdtQtx2AJSUDHwKnA3kEuxBPt3MVtRR/yZggpldXaO8F7AGGGRm+8LE8YqZvRBtLL4DoGusPaUVnPbrd8js04Xnrz0xrl/m24pLuO35Rcxbs5PzjurH/1x6NOmdW+8XbFWV8dS/NvCr1z+hc2oy/33J0Zx/TP9Eh+UOQV07AMazY3UysMbM1plZGTALuLie+tOBmbWUfwV4zcz2xSFG52r16Ltr2bGnjB+dPybuVwCHdU/jT1cfz13njeaNFds474EP+Gjdzri+Z7xs2rWP6Y9/xM9eWcFJh/dh7q2neNJog+KZOAYCmyJe54ZlB5E0FMgE3q7l8DQOTig/l7RE0n2SOtbR5gxJWZKy8vPzGx+9a7fyivbz+AfruGjcAMYPTm+W90xKEteeOoK/Xv8F0jokM/3xj/jN3FWUVx48O7olMjNmfpzD1PvfZ/mWYn71lWP4/RWT6NutfW+x2la1lFs5pgEvmFllZKGk/sDRwNyI4rsIxjyOA3oBd9TWoJk9ZmaTzGxSRkZGfKJ2bdKv567CgNvPPaLZ3/uYQem8ctNJfPXYQfz2nTV89dEPydnZsi+2txWXcPXT87nrr0s5ZlA6r99yMpdNajtjNe5g8Uwcm4HBEa8HhWW1qe2qAuAy4CUzK68uMLM8C5QCTxF0iTkXE8s2F/HSws1cNWUYg3t1TkgMXTqm8KuvjOO3X5/A2vw9fPHBD3hpYW5CYqmPmTF78RbOue99Ply3k7svHMMz3zmeQT0T83NzzSeeiWM+MFJSpqRUguQwu2YlSaOBnsCHtbRx0LhHeBWCgj9nLgGWxTZs116ZGT+fs5L0Th24/rTDEx0OFxwzgNdvOYUx/btz63OLuWXWQopLyhs+sRns2lvGjc8u5OaZCxme0YVXbz6ZK6dk+jpR7UTcbsc1swpJNxJ0MyUDT5rZckn3AFlmVp1EpgGzrMbtXZKGEVyxvFej6WckZQACFgHXxeszuPbl7U+28+G6nfzXRWPp0alDosMBYGB6J2bOOIGH3lnDA2+tZkFOAfd/bQLHDu2ZsJjeXLGNO/+6lKL9Zfxg6hHMOHk4Kb6WVLsSt9txWxK/Hdc1pLyyiqn3v48ZzL31lBa5qN6CjQV8d9ZC8opKuOXMkVx/+uEkN+Nf+MUl5dzz9xW8sCCXI/t35/8uG8eR/bs32/u75lfX7bg+c9w5YNb8TazN38tj3zq2RSYNgGOH9uTV757Mf768jP9941M+WL2D+6aNZ2A9EwyboryyipV5xWRtKGBBTgEfrd1Jwb4ybjz9cG4+c2SbXybF1c0Th2v3dpeUc/8bnzI5sxdnj2nZy2F0T+vAA9MmcNoRGfz4pWWcd//7/M+lx8RkrkThvjIW5hSStXEXCzYWsHhTEfvLgxsdB6Z3YsrhfbhqyjAmDElcN5lrGTxxuHbvkXfXsnNvGU+df2SruYX0SxMGMXFIT26etYgbns3mvU8H8dMLx9IlyoUDzYz1O/ayYGPBgcfq7XsASE4SYwd0Z9rkwRw7tCfHDu1J/x7xuapxrZMnDpcQZtYivqQ3F+7niX+u55LxAzhmUHqiw2mUob278MJ1J/LAm6t56N01zN9QwIPTJnD0oB4H1S0pr2Tp5iIWbCwga0MB2TkF7NpbBkCPTh2YOCSdSyYMZOKQnowb3IPOqf7V4Ormvx0uIW5/YQl5Rft59JvH0i0tcXcw/aZ6st/U0QmLoSk6JCfx/XOP4KSRfbj1uUVc+sg8vn/OEXxpwkCyc4IriayNBSzbXER5ZXAjzPA+XThzdN8DVxMjMrr6bbSuUfyuKtfsCvaWcdzP36Siypg0tCd/uHpy1F0ssbQ0t4gLf/tP/uO0EdzRShNHpMJ9Zdz54lJeX771QFlqShLjBvXg2KG9OHZoTyYOSad311pX6XHuIH5XlWsx/rFiKxVVxk1nHM7D767l6qfn89RVxzVr94iZ8d9zVtC7SyrXnzai2d43ntI7p/LINycyZ2keW4tKmDi0J0cN6OF3P7mY88Thmt0rS/IY2rszt509ipGHdeOWWQv5zh+yePLK45pt7+k3V27n3+t38bOLxya0qyzWJHHBMQMSHYZr4/xPEdesdu0t419rd3L+0f2RxEXjBvCbr47jw3U7ueaPWZSUVzbcSBOVV1bxP6+uZERGF6ZNHhL393OurfHE4ZrV3OVbqayyz807uHTiIH556TF8sHoH1z+TTWlFfJPHzI9zWLdjL3edd2SLneznXEvm/9e4ZjVnSR6ZfbowpsZSFZcdN5hffOlo3v5kOzc+uzBu+1AUl5Rz/5urOXF4b848sm9c3sO5ts4Th2s2O/eU8q+1Ow50U9X09eOHcM/FY3ljxTZunhmf5PHwO2sp2FfGj1rRZD/nWhpPHK7ZvL58K1VGvctjXH7iMP7zgjG8tmwrtz63iIoYJo9Nu/bx5Lz1fGnCQI4aePAkOedcdPyuKtds5izJY3hGF0b361ZvvW+flElFZRX/89ondEhO4jdfHReTVWB/849VCPj+Oc2/s59zbYknDtcs8neX8tG6ndx4+uFRdRFde+oIKqqMX89dRXKS+NWXj2nS7OZFmwr526It3Hj64QyI02qyzrUXnjhcs/ismyr6OQY3nH44ZRVVPPDWajoki59fcvQhJQ8z4xdzVtKnayrXtZHJfs4lUlwTh6SpwAMEOwD+3szurXH8PuD08GVnoK+ZpYfHKoGl4bEcM7soLM8EZgG9gQXAt8ysLJ6fwzXdnCVbOLxvV0Yd1rVR591y1kgqqqp46J21JCeJn118VKMHtecu38bHG3bx8y8dRdcELG3iXFsTt/+LJCUDDwFnA7nAfEmzzWxFdR0zuzWi/k3AhIgm9pvZ+Fqa/iVwn5nNkvQo8G3gkTh8BBcj23eX8O/1u7j5jJGN/tKXxPfPOYKKSuN3768jJSmJn144Jup2yiqquPe1lYzs25WvTRp8KOE752qI511Vk4E1ZrYuvCKYBVxcT/3pwMz6GlTwbXEG8EJY9AfgkqaH6uLp9WVbsQbupqqPJO48bzRXT8nk6X9t4BevriTaxTmf+fdGNuzcxw+/eKTvi+1cjMTzun0gsCnidS5wfG0VJQ0FMoG3I4rTJGUBFcC9ZvYyQfdUoZlVRLQ5sI42ZwAzAIYM8WUlEumVJXmMOqwrow6r/26q+kjiPy84koqqKh7/YD0dkpO4/dwj6r3yKNpfzgNvrWbK4b057YiMQ35v59zntZQO32nAC2YWudbEUDPbLGk48LakpUBRtA2a2WPAYxAsqx7TaF3UthWXMH/DLm45c1ST25LE3ReOpbzSePjdtaQkJ3Hb2XW3+9A7ayjaX84Pv+iT/ZyLpXgmjs1AZKfyoLCsNtOAGyILzGxz+O86Se8SjH+8CKRLSgmvOupr07UAry3Na1I3VU1JSeLnlxxFRWUVD761mg5J4qYzRx5Ub9OufTw9bwNfnjiIsQN8sp9zsRTPTt/5wEhJmZJSCZLD7JqVJI0GegIfRpT1lNQxfN4HmAKssKBj+x3gK2HVK4C/xfEzuCaaszSP0f26cXjfxt1NVZ+kJHHvl4/h0gkD+d83PuWRd9ceVOeXr39CUpJP9nMuHuKWOMIrghuBucBK4HkzWy7pHkkXRVSdBsyyz492HglkSVpMkCjujbgb6w7gNklrCMY8nojXZ3BNs7WohPkbCrggRlcbkZKTxK+/Oo4Lxw3gl69/wu8/WHfgWHZOAa8syWPGycPp1yMt5u/tXHsX1zEOM3sVeLVG2U9qvL67lvP+BRxdR5vrCO7Yci3cnKV5AHzx6NgnDgiSx32XjaOisor/nrOSlCRxxReG8fM5K+nTtSPXnuqT/ZyLh5YyOO7aoDlLtjCmf3eGZ8Sum6qmlOQkHpw+geufyebuv69gQU4hCzYW8D+XHp2Qfcydaw/8xnYXF5sL95OdUxizQfH6dEhO4rdfn8AZo/vy98VbOOKwblzmk/2cixv/k8zFxWthN9X5ceqmqqljSjIPf2MiD761mi8e3T8mq+k652rnicPFxStL8jhqYHeG9enSbO+Z1iGZH0wd3Wzv51x75V1VLuY27drHok2FnH909CvhOudaD08cLuZeW9a83VTOueblicPF3JwleRwzqAdDendOdCjOuTjwxOFiatOufSzOLfKrDefaME8cLqbiPenPOZd4njhcTM1Zkse4wekM7uXdVM61VZ44XMxs3LmXpZuLuMCvNpxr0zxxuJip7qY67+h+CY7EORdPdU4AlHRpFOeXhAsZOsecJXlMGJLOoJ7eTeVcW1bfzPHHCfa6qG/thlOosfqta5/W79jL8i3F/Pj8IxMdinMuzupLHK+Z2dX1nSzpzzGOx7VSr/rdVM61G3WOcZjZNxs6OZo6rn14ZUkek4b2ZEB6p0SH4pyLs6gHxyUdLunPkl6UdGKU50yVtErSGkl31nL8PkmLwsenkgrD8vGSPpS0XNISSV+LOOdpSesjzhsf7Wdw8bE2fw8r84qbZQl151zi1Tc4nmZmJRFFPwN+ED7/OzC+voYlJQMPAWcDucB8SbMjtoDFzG6NqH8TMCF8uQ+43MxWSxoALJA018wKw+O3m9kLUXw+1wxeXZKHBOcd5YnDufagviuOv0u6POJ1OTAMGApURtH2ZGCNma0zszJgFnBxPfWnAzMBzOxTM1sdPt8CbAcyonhPlwBzluZx3NBevr+3c+1EfYljKtBd0uuSTgG+D5wLfAn4RhRtDwQ2RbzODcsOImkokAm8XcuxyUAqsDai+OdhF9Z9kjrW0eYMSVmSsvLz86MI1x2KNdt388nW3d5N5Vw7Ut/geKWZ/Rb4GnAR8ADwlJl9z8w+iXEc04AXzOxzVzKS+gN/Aq4ys6qw+C5gNHAc0Au4o474HzOzSWY2KSPDL1bi5ZUD3VQ+6c+59qK+MY7jgduBMuAXwH6Cv/Q3Az+LGG+oy2YgcuPnQWFZbaYBN9R4/+7AHOBHZvZRdbmZ5YVPSyU9RXAl5BJkzpI8Jg/rRd/u3k3lXHtRX1fV74CbgbuB35nZWjObBswGnoui7fnASEmZklIJksPsmpUkjQZ6Ah9GlKUCLwF/rDkIHl6FIEnAJcCyKGJxcfDptt2s3r6HC7ybyrl2pb4JgBUEg+FdCK46ADCz94D3GmrYzCok3QjMBZKBJ81suaR7gCwzq04i04BZZmYRp19GMCu9t6Qrw7IrzWwR8IykDIIZ7YuA6xqKxcXHK0vySBKc691UzrUr+vz3dcQBaRRwLUHSeNjMNtVasRWYNGmSZWVlJTqMNsXMOOv/3qNvtzRmzjgh0eE45+JA0gIzm1SzvM4rDjP7FPheXKNyrdaqbbtZm7+Xq6ZkJjoU51wzq3OMQ9IrDZ0cTR3XNs0Ju6mmejeVc+1OfWMcJ0k6aDA7goAxMY7HtQJmxpwleZw4ojd9utY6jcY514bVlzjqm+VdrazhKq6tWZm3m3U79vKdk4cnOhTnXALUN8bR4J1Trn2as3QLyUni3LGHJToU51wC+NaxrlGqu6m+MKI3vb2byrl2yROHa5TlW4rZsHMf5/uGTc61Ww0mDkkXSvIE44BgJdygm8rvpnKuvYomIXwNWC3pV+HyIK6dqu6mmnJ4H3p2SU10OM65BGkwcYTbw04gWNb86XBnvhmSusU9OteiLNtcTM6ufb42lXPtXFRdUGZWDLxAsBlTf4I9ObLDXftcO/HK0i10SBbnjvFuKufas2jGOC6S9BLwLtABmGxm5wHj8CVJ2o3qbqqTDu9Dj84dEh2Ocy6B6psAWO3LwH1m9n5koZntk/Tt+ITlWpoluUXkFuznlrNGJToU51yCRZM47gaqN09CUifgMDPbYGZvxSsw17LMWZpHh2Rx9hif9OdcexfNGMdfgKqI15VhmWsnqrupThmZQY9O3k3lXHsXTeJIMbPIjZzKAL8Xsx1ZuKmQzYX7Od/vpnLOEV3iyJd0UfULSRcDO6JpXNJUSaskrZF0Zy3H75O0KHx8Kqkw4tgVklaHjysiyo+VtDRs88FwC1kXR3OW5JGanMRZ3k3lnCO6MY7rCLZr/S3BUuqbgMsbOklSMvAQcDaQC8yXNNvMVlTXMbNbI+rfRDBfBEm9gJ8CkwADFoTnFgCPANcA/wZeBaYCr0XxOdwhqKoyXl2axymjMuie5t1UzrnoJgCuNbMTCPbeONLMvmBma6JoezKwxszWhd1bs6h/qfbpwMzw+bnAG2a2K0wWbwBTJfUHupvZR+Ee5X8ELokiFneIFm4qIK+oxCf9OecOiOaKA0nnA2OBtOqeITO7p4HTBhJcnVTLBY6vo/2hQCbwdj3nDgwfubWUuzh5c+V2UpLEmUf2TXQozrkWIpoJgI8SrFd1E0FX1VeBoTGOYxrwgplVxqrBcFmULElZ+fn5sWq23VmwsYCxA3vQzbupnHOhaAbHv2BmlwMFZvZfwIlANLPANgODI14PCstqM43PuqnqO3dz+LzBNs3sMTObZGaTMjIyogjX1VReWcWS3EImDklPdCjOuRYkmsRREv67T9IAoJxgvaqGzAdGSsqUlEqQHA7awzxccbcn8GFE8VzgHEk9JfUEzgHmmlkeUCzphPBuqsuBv0URizsEn+TtpqS8iolDeiY6FOdcCxLNGMffJaUDvwayCe5yeryhk8ysQtKNBEkgGXjSzJZLugfIMrPqJDINmBUOdlefu0vSzwiSD8A9ZrYrfH498DTQieBuKr+jKk6ycwoAmDjUE4dz7jP1Jo5wA6e3zKwQeFHSK0CamRVF07iZvUpwy2xk2U9qvL67jnOfBJ6spTwLOCqa93dNk51TwGHdOzKgR1qiQ3HOtSD1dlWZWRXBXIzq16XRJg3X+i3YWMDEIT3xOZbOuUjRjHG8JenLPkO7fdm+u4Tcgv0+vuGcO0g0ieNagkUNSyUVS9otqTjOcbkEy95YCMDEoekJjcM51/I0ODhuZr5FbDu0MKeADsli7IAeiQ7FOdfCNJg4JJ1SW3nNjZ1c25KdU8DYAT1I65Cc6FCccy1MNLfj3h7xPI1gDaoFwBlxicglXFlFFUtyi/jG8bFeIMA51xZE01V1YeRrSYOB++MVkEu8lXnFlFZU+fiGc65W0QyO15QLHBnrQFzLcWDin99R5ZyrRTRjHP+PYLY4BIlmPMEMctdGZecU0r9HGgPSOyU6FOdcCxTNGEdWxPMKYKaZzYtTPK4FyA4n/jnnXG2iSRwvACXVS55LSpbU2cz2xTc0lwjbi0vYXLifq6YMS3QozrkWKqqZ4wQLClbrBLwZn3BcovnChs65hkSTONLMbE/1i/B55/iF5BIpO6eQ1OQkxg7onuhQnHMtVDSJY6+kidUvJB0L7I9fSC6RsjcWcNTA7nRM8Yl/zrnaRTPGcQvwF0lbCLaO7UewlaxrY8oqqliyuYjLT/CJf865ukUzAXB+uEvfEWHRKjMrj29YLhGWbymirKLKxzecc/VqsKtK0g1AFzNbZmbLgK6Sro+mcUlTJa2StEbSnXXUuUzSCknLJT0blp0uaVHEo0TSJeGxpyWtjzg2PtoP6+qXnVMI+MQ/51z9oumqusbMIjdzKpB0DfBwfSdJSibYBOpsgtnm8yXNNrMVEXVGAncBU8J2+4bv8Q7BREMk9QLWAP+IaP52M3shithdI2TnFDCgRxr9fMc/51w9ohkcT47cxClMCKlRnDcZWGNm68ysDJgFXFyjzjXAQ2ZWAGBm22tp5yvAaz5vJP4WbixggndTOecaEE3ieB14TtKZks4EZoZlDRkIbIp4nRuWRRoFjJI0T9JHkqbW0s608D0j/VzSEkn3SeoYRSyuAVuLSthSVOLdVM65BkWTOO4A3gb+I3y8xeeXWm+KFGAkcBowHXhcUnr1QUn9gaOBuRHn3AWMBo4DeoXxHUTSDElZkrLy8/NjFG7b9dnChumJDcQ51+I1mDjMrMrMHjWzr5jZV4AVwP+Lou3NwOCI14PCski5wGwzKzez9cCnBImk2mXAS5F3cZlZngVKgacIusRqi/sxM5tkZpMyMjKiCLd9y95YQGpKku/455xrUFTLqkuaIOlXkjYA9wCfRHHafGCkpExJqQRdTrNr1HmZ4GoDSX0Iuq7WRRyfTo1uqvAqhHDc5RJgWTSfwdUvO6eAYwb2IDXlUFbad861J3XeVSVpFMEX93RgB/AcIDM7PZqGzaxC0o0E3UzJwJNmtlzSPUCWmc0Oj50jaQVQSXC31M7w/YcRXLG8V6PpZyRlEExGXARcF+VndXUorahk2eZirvSFDZ1zUajvdtxPgA+AC8xsDYCkWxvTuJm9Crxao+wnEc8NuC181Dx3AwcPpmNmvmVtjC3fUkxZZZWPbzjnolJfv8SlQB7wjqTHwzuqVE9910plb/Qd/5xz0aszcZjZy2Y2jeAOpncI1qzqK+kRSec0U3yuGSzMKWRgeif6dveJf865hkVzV9VeM3vWzC4kuDNqIXXcAutap+ycAl+fyjkXtUbdQmNmBeFtrmfGKyDXvLYU7ievqMTHN5xzUfN7L9u5zyb++RWHcy46njjaueyNhXRMSeLI/r7jn3MuOp442rnsnAKOGeQT/5xz0fNvi3aspLyS5VuKvJvKOdconjjaseVbiiivNL+jyjnXKJ442rHsjYWAD4w75xrHE0c7lp1TwOBencjo5luaOOei54mjnTKzYOKfX2045xrJE0c7taWohG3FpZ44nHON5omjnfKFDZ1zh8oTRzuVnVNAWockRvfvluhQnHOtjCeOdio7p5BjBqXTIdl/BZxzjePfGu1QSXklK3zin3PuEMU1cUiaKmmVpDWS7qyjzmWSVkhaLunZiPJKSYvCx+yI8kxJ/w7bfC7cz9w1wrLN4cQ/XxHXOXcI4pY4JCUDDwHnAWOA6ZLG1KgzErgLmGJmYwk2i6q238zGh4+LIsp/CdxnZocDBcC34/UZ2qoDK+L6jHHn3CGI5xXHZGCNma0zszJgFnBxjTrXAA+ZWQGAmW2vr0FJAs4AXgiL/gBcEsug4620opJfvf7JgS/vRFiwsYAhvTrTp6tP/HPONV48E8dAYFPE69ywLNIoYJSkeZI+kjQ14liapKyw/JKwrDdQaGYV9bQJgKQZ4flZ+fn5Tf4wsVBWUcUNz2Tz8Ltr+dFLyzCzZo8hmPhX6N1UzrlDlujB8RRgJHAaMB14XFJ6eGyomU0Cvg7cL2lEYxoOdyqcZGaTMjIyYhjyoSmrqOKGZ7N5c+V2zhzdl5V5xby/ekezx5FbsJ/83aUc691UzrlDFM/EsRkYHPF6UFgWKReYbWblZrYe+JQgkWBmm8N/1wHvAhOAnUC6pJR62mxxyiqquPHZbN5YsY17Lh7LI988lsO6d+TRd9c2eyzVXWQT/I4q59whimfimA+MDO+CSgWmAbNr1HmZ4GoDSX0Iuq7WSeopqWNE+RRghQV9O+8AXwnPvwL4Wxw/Q5OVV1Zx08xs/rFiG/910VguP3EYqSlJfPukTD5ct5PFmwqbNZ6FOYV0Tk1mdD+f+OecOzRxSxzhOMSNwFxgJfC8mS2XdI+k6ruk5gI7Ja0gSAi3m9lO4EggS9LisPxeM1sRnnMHcJukNQRjHk/E6zM0VXllFTfPXMjc5dv46YVjuOILww4cmz55CN3SUnj0vea96qje8S/FJ/455w5RSsNVDp2ZvQq8WqPsJxHPDbgtfETW+RdwdB1triO4Y6tFK6+s4ruzFvLasq385IIxXDUl83PHu6V14FsnDOWR99ayfsdeMvt0iXtMwcS/YmacMjzu7+Wca7v8z844qKis4pZZi3h16VZ+fP6RXH1SZq31rpwyjA7JSTz2/rpmiWtJbhEVVeYzxp1zTeKJI8YqKqv47nOLmLM0jx+ffyTfObnuv+77dkvjyxMH8WJ2Ltt3l8Q9ts8GxtPj/l7OubbLE0cMVVRWcevzi5mzJI8ffnF0vUmj2oxThlNeWcVT8zbEPb7sjQUM692Z3j7xzznXBJ44YqSisorbnl/M3xdv4a7zRjPjlOimnWT26cJ5R/Xjzx9tZHdJedzi+2zin3dTOeeaxhNHDFRWGd/7y2JmL97CHVNHc+2pjZqryHWnjmB3SQUzP86JU4TBxL8de0qZ4BP/nHNN5ImjiSqrjO//ZTF/W7SF2889gv84rXFJA+CYQel8YURvnvjnekorKuMQZcTChj6+4ZxrIk8cTVBZZdz+l8W8tHAzt597BDecfvght3XtqSPYVlzK3xZuiWGEn8neWEDn1GSOOMwn/jnnmsYTxyGqrDJ+8MIS/rpwM987e1STkgbAKSP7MKZ/dx59fy1VVbFf/DA7p5Bxg9J94p9zrsn8W+QQVFUZd7y4hBezc7n1rFHcdObIJrcpiWtPHc66/L28uXJbDKL8zL6yClbkFfvChs65mPDE0UhVVcadf13CCwtyueWskXz3rKYnjWrnH92fwb068eh7a2O65PqS3CIqq4yJQ9Nj1qZzrv3yxNEIVVXGXX9dyvNZudx85khuOWtUTNtPSU7impOHk51TyPwNsdvo6cDEv8F+xeGcazpPHFGqqjJ++NJSnsvaxE1nHM6tMbzSiPTVYwfTq0tqTBc/zN5YyPA+XejZxbdnd841nSeOKFRVGT96eRmz5m/ixtMP57azRxHsYht7nVKTueLEYbz9yXZWbd3d5PbMjIU5Bb7/hnMuZjxxNKCqyvjPvy1j5sc5XH/aCL53TvySRrXLTxxKpw7J/O79pl915Ozax869ZT6+4ZyLGU8c9TAzfjJ7Gc/8O4frTh3B7eceEfekAdCzSyrTJg9m9qItbC7c36S2Ppv451cczrnY8MRRBzPjp7OX8+ePcrj21OHcMbV5kka175w8HAOe+GB9k9rJ3lhI144pjPKJf865GIlr4pA0VdIqSWsk3VlHncskrZC0XNKzYdl4SR+GZUskfS2i/tOS1ktaFD7Gxyl2BqZ3YsYpw7lz6uhmTRoAA9M7cdG4Acyan0PhvrJDbic7p4Bxg3uQnNS88Tvn2q64JQ5JycBDwHnAGGC6pDE16owE7gKmmNlY4Jbw0D7g8rBsKnC/pPSIU283s/HhY1G8PsO1p47gh188stmTxmfvP5x9ZZX86cONh3T+vrIKPtm627upnHMxFc8rjsnAGjNbZ2ZlwCzg4hp1rgEeMrMCADPbHv77qZmtDp9vAbYDGXGMtUUa3a87px+RwdP/2kBJeeMXP1y8KZz454nDORdD8UwcA4FNEa9zw7JIo4BRkuZJ+kjS1JqNSJoMpAKRtxj9POzCuk9SrbsSSZohKUtSVn5+ftM+SQJdd+oIdu4t4y9ZmxquXIPv+Oeci4dED46nACOB04DpwOORXVKS+gN/Aq4ys6qw+C5gNHAc0Au4o7aGzewxM5tkZpMyMlrvxcrkzF5MGJLOYx+so6KyquETIizMKWB4RhfSO/vEP+dc7MQzcWwGBke8HhSWRcoFZptZuZmtBz4lSCRI6g7MAX5kZh9Vn2BmeRYoBZ4i6BJrsyRx7Skj2LRrP68t2xr1edU7/h3r3VTOuRiLZ+KYD4yUlCkpFZgGzK5R52WCqw0k9SHouloX1n8J+KOZvRB5QngVgoIR60uAZfH7CC3DOWMOY3hGl0Ytfrhh5z527S1joq+I65yLsbglDjOrAG4E5gIrgefNbLmkeyRdFFabC+yUtAJ4h+BuqZ3AZcApwJW13Hb7jKSlwFKgD/Df8foMLUVSkrj2lOEs31LMP9fsiOqc7I0+8c85Fx+K5fLdLdWkSZMsKysr0WE0SWlFJSf/8h1GHtaVZ75zQoP1f/TSUmYv2sLin55Dks/hcM4dAkkLzGxSzfJED467KHVMSebqkzKZt2YnS3OLGqyfnVPI+CHpnjScczHniaMV+frxQ+jWMYVHG1j8cE9pBau2FvuKuM65uPDE0Yp0T+vAN04YymtL89i4c2+d9ZZsKqTKYKLP33DOxYEnjlbm6inDSElK4rH319VZx3f8c87FkyeOVqZv9zS+fOxA/rIgl/zdpbXWyc4p5PC+XenRuUMzR+ecaw88cbRC15w8nPLKKp7+18FLrlfv+OfdVM65ePHE0QoNz+jKuWP68acPN7KntOJzx9bv2EvBvnKfv+GcixtPHK3UdaeNoLikglkf53yuPDunEMBnjDvn4sYTRys1fnA6Jwzvxe8/WE9ZxWeLH2bnFNAtLYXDM7omMDrnXFvmiaMVu+7UEWwtLuFviz5bOzJ7YwEThvT0iX/OubjxxNGKnToqg9H9uvHY++uoqjL2lFbw6bbdPjDunIsrTxytmCSuO3UEq7fv4e1PtrP4wMQ/H99wzsWPJ45W7oJj+jMwvROPvreWBRsLkGC8X3E45+LIE0crl5KcxDUnZ5K1sYBZH+cwsm9Xuqf5xD/nXPx44mgDLjtuMD07d2BLUYl3Uznn4s4TRxvQOTWFy08cBvj4hnMu/uKaOCRNlbRK0hpJd9ZR5zJJKyQtl/RsRPkVklaHjysiyo+VtDRs88FwC9l27+qTMrlqyjDOHdsv0aE459q4uO0AKCkZ+BQ4G8gl2IN8upmtiKgzEngeOMPMCiT1NbPtknoBWcAkwIAFwLFhnY+Bm4F/A68CD5rZa/XF0hZ2AHTOueaWiB0AJwNrzGydmZUBs4CLa9S5BnjIzAoAzGx7WH4u8IaZ7QqPvQFMldQf6G5mH1mQ8f4IXBLHz+Ccc66GeCaOgcCmiNe5YVmkUcAoSfMkfSRpagPnDgyf19cmAJJmSMqSlJWfn9+Ej+Gccy5SogfHU4CRwGnAdOBxSemxaNjMHjOzSWY2KSMjIxZNOuecI76JYzMwOOL1oLAsUi4w28zKzWw9wZjIyHrO3Rw+r69N55xzcRTPxDEfGCkpU1IqMA2YXaPOywRXG0jqQ9B1tQ6YC5wjqaeknsA5wFwzywOKJZ0Q3k11OfC3OH4G55xzNaTEq2Ezq5B0I0ESSAaeNLPlku4BssxsNp8liBVAJXC7me0EkPQzguQDcI+Z7QqfXw88DXQCXgsfzjnnmkncbsdtSfx2XOeca7xE3I7rnHOuDWoXVxyS8oGNh3h6H2BHDMOJt9YUr8caP60p3tYUK7SueJsa61AzO+i21HaROJpCUlZtl2otVWuK12ONn9YUb2uKFVpXvPGK1buqnHPONYonDuecc43iiaNhjyU6gEZqTfF6rPHTmuJtTbFC64o3LrH6GIdzzrlG8SsO55xzjeKJwznnXKN44qhHNDsYtgSSBkt6J2Inxe8mOqaGSEqWtFDSK4mOpSGS0iW9IOkTSSslnZjomOoi6dbwd2CZpJmS0hIdUyRJT0raLmlZRFkvSW+Eu32+Ea5P1yLUEe+vw9+FJZJeitWK3k1VW6wRx74nycI1AZvME0cdwh0MHwLOA8YA0yWNSWxUdaoAvmdmY4ATgBtacKzVvgusTHQQUXoAeN3MRgPjaKFxSxpIsDvmJDM7imCNuGmJjeogTwNTa5TdCbxlZiOBt8LXLcXTHBzvG8BRZnYMwYredzV3UHV4moNjRdJggoVic2L1Rp446hbNDoYtgpnlmVl2+Hw3wRdbrRtctQSSBgHnA79PdCwNkdQDOAV4AsDMysysMKFB1S8F6CQpBegMbElwPJ9jZu8Du2oUXwz8IXz+B1rQrp61xWtm/zCzivDlR3x+q4eEqeNnC3Af8AOCbbhjwhNH3aLZwbDFkTQMmECwJ3tLdT/BL3JVguOIRiaQDzwVdq39XlKXRAdVGzPbDPyG4C/LPKDIzP6R2Kiicli4ZQLAVuCwRAbTSFfTglfolnQxsNnMFseyXU8cbYikrsCLwC1mVpzoeGoj6QJgu5ktSHQsUUoBJgKPmNkEYC8tqyvlgHBs4GKCZDcA6CLpm4mNqnEsmB/QKuYISPoRQTfxM4mOpTaSOgM/BH4S67Y9cdQtmh0MWwxJHQiSxjNm9tdEx1OPKcBFkjYQdP+dIenPiQ2pXrlArplVX8G9QJBIWqKzgPVmlm9m5cBfgS8kOKZobJPUHyD8d3uC42mQpCuBC4BvWMudDDeC4I+IxeH/b4OAbEn9mtqwJ466RbODYYsQ7ob4BLDSzP4v0fHUx8zuMrNBZjaM4Gf6tpm12L+KzWwrsEnSEWHRmcCKBIZUnxzgBEmdw9+JM2mhA/k1zAauCJ9fQQvf1VPSVIKu1ovMbF+i46mLmS01s75mNiz8/y0XmBj+TjeJJ446hINf1TsYrgSeN7PliY2qTlOAbxH89b4ofHwx0UG1ITcBz0haAowHfpHYcGoXXhW9AGQDSwn+/25Ry2NImgl8CBwhKVfSt4F7gbMlrSa4aro3kTFGqiPe3wLdgDfC/9ceTWiQoTpijc97tdyrLOeccy2RX3E455xrFE8czjnnGsUTh3POuUbxxOGcc65RPHE455xrFE8czjVA0p7w32GSvh7jtn9Y4/W/Ytm+c/HgicO56A0DGpU4wsUG6/O5xGFmrWGmt2vnPHE4F717gZPDSV+3hnuK/FrS/HBvhmsBJJ0m6QNJswlnmUt6WdKCcK+MGWHZvQQr2S6S9ExYVn11o7DtZZKWSvpaRNvvRuwP8kw4SxxJ9yrYk2WJpN80+0/HtRsN/TXknPvMncD3zewCgDABFJnZcZI6AvMkVa9GO5Fgz4b14eurzWyXpE7AfEkvmtmdkm40s/G1vNelBLPUxwF9wnPeD49NAMYSLJk+D5giaSXwJWC0mVlL2VzItU1+xeHcoTsHuFzSIoJl7HsDI8NjH0ckDYCbJS0m2L9hcES9upwEzDSzSjPbBrwHHBfRdq6ZVQGLCLrQioAS4AlJlwItdg0l1/p54nDu0Am4yczGh4/MiP0v9h6oJJ1GsAbTiWY2DlgINGVL19KI55VASri22mSCtaouAF5vQvvO1csTh3PR202wuF21ucB/hEvaI2lUHZs89QAKzGyfpNEE2/tWK68+v4YPgK+F4ygZBLsQflxXYOFeLD3M7FXgVoIuLufiwsc4nIveEqAy7HJ6mmAv8mEEexyIYKfAS2o573XgunAcYhVBd1W1x4AlkrLN7BsR5S8BJwKLCTY2+oGZbQ0TT226AX+TlEZwJXTbIX1C56Lgq+M655xrFO+qcs451yieOJxzzjWKJw7nnHON4onDOedco3jicM451yieOJxzzjWKJw7nnHON8v8B3vGLSaeeiZgAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "acc_per_epoch = [np.mean(acc_per_epoch) for acc_per_epoch in running_test_acc]\n",
+ "display_loss_plot(acc_per_epoch, title=\"Training accuracy\", ylabel=\"Accuracy [%]\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.7979886313948404"
+ ]
+ },
+ "execution_count": 15,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
"source": [
"test(model, test_quantized_loader)"
]
},
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Save the Brevitas model to disk\n",
+ "torch.save(model.state_dict(), \"state_dict_self-trained.pth\")"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
@@ -479,7 +549,7 @@
},
{
"cell_type": "code",
- "execution_count": 24,
+ "execution_count": 17,
"metadata": {},
"outputs": [
{
@@ -488,7 +558,7 @@
"(64, 593)"
]
},
- "execution_count": 24,
+ "execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
@@ -504,7 +574,7 @@
},
{
"cell_type": "code",
- "execution_count": 25,
+ "execution_count": 18,
"metadata": {},
"outputs": [
{
@@ -513,7 +583,7 @@
"(64, 600)"
]
},
- "execution_count": 25,
+ "execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
@@ -528,7 +598,7 @@
},
{
"cell_type": "code",
- "execution_count": 26,
+ "execution_count": 19,
"metadata": {},
"outputs": [
{
@@ -537,7 +607,7 @@
"torch.Size([64, 600])"
]
},
- "execution_count": 26,
+ "execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
@@ -560,7 +630,7 @@
},
{
"cell_type": "code",
- "execution_count": 27,
+ "execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
@@ -588,7 +658,7 @@
},
{
"cell_type": "code",
- "execution_count": 28,
+ "execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
@@ -618,16 +688,16 @@
},
{
"cell_type": "code",
- "execution_count": 29,
+ "execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
- "0.9188772287810328"
+ "0.8083369771170383"
]
},
- "execution_count": 29,
+ "execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
@@ -647,7 +717,7 @@
},
{
"cell_type": "code",
- "execution_count": 30,
+ "execution_count": 23,
"metadata": {
"scrolled": true
},
@@ -695,7 +765,7 @@
},
{
"cell_type": "code",
- "execution_count": 32,
+ "execution_count": 24,
"metadata": {},
"outputs": [
{
@@ -719,10 +789,10 @@
" "
],
"text/plain": [
- "<IPython.lib.display.IFrame at 0x7f4045ac19e8>"
+ "<IPython.lib.display.IFrame at 0x7f3a74f3a5c0>"
]
},
- "execution_count": 32,
+ "execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
diff --git a/notebooks/end2end_example/cybersecurity/2-export-to-finn-and-verify.ipynb b/notebooks/end2end_example/cybersecurity/2-export-to-finn-and-verify.ipynb
index f48cada0d..358615dbe 100644
--- a/notebooks/end2end_example/cybersecurity/2-export-to-finn-and-verify.ipynb
+++ b/notebooks/end2end_example/cybersecurity/2-export-to-finn-and-verify.ipynb
@@ -6,7 +6,9 @@
"source": [
"# Verify Exported ONNX Model in FINN\n",
"\n",
- "**Important: This notebook depends on the 1-train-mlp-with-brevitas notebook, because we are using the ONNX model that was exported there. So please make sure the needed .onnx file is generated before you run this notebook. Also remember to 'close and halt' any other FINN notebooks, since Netron visualizations use the same port.**\n",
+ "**Important: This notebook depends on the 1-train-mlp-with-brevitas notebook, because we are using the ONNX model that was exported there. So please make sure the needed .onnx file is generated before you run this notebook.**\n",
+ "\n",
+ "**Also remember to 'close and halt' any other FINN notebooks, since Netron visualizations use the same port.**\n",
"\n",
"In this notebook we will show how to import the network we trained in Brevitas and verify it in the FINN compiler. \n",
"This verification process can actually be done at various stages in the compiler [as explained in this notebook](../bnn-pynq/tfc_end2end_verification.ipynb) but for this example we'll only consider the first step: verifying the exported high-level FINN-ONNX model.\n",
@@ -122,7 +124,7 @@
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
@@ -150,7 +152,7 @@
},
{
"cell_type": "code",
- "execution_count": 5,
+ "execution_count": 4,
"metadata": {},
"outputs": [
{
@@ -243,7 +245,7 @@
},
{
"cell_type": "code",
- "execution_count": 7,
+ "execution_count": 5,
"metadata": {},
"outputs": [
{
@@ -252,7 +254,7 @@
"torch.Size([100, 593])"
]
},
- "execution_count": 7,
+ "execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
@@ -280,7 +282,7 @@
},
{
"cell_type": "code",
- "execution_count": 8,
+ "execution_count": 6,
"metadata": {},
"outputs": [
{
@@ -289,7 +291,7 @@
"IncompatibleKeys(missing_keys=[], unexpected_keys=[])"
]
},
- "execution_count": 8,
+ "execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
@@ -325,12 +327,15 @@
"# replace this with your trained network checkpoint if you're not\n",
"# using the pretrained weights\n",
"trained_state_dict = torch.load(\"state_dict.pth\")[\"models_state_dict\"][0]\n",
+ "# Uncomment the following line if you previously chose to train the network yourself\n",
+ "#trained_state_dict = torch.load(\"state_dict_self-trained.pth\")\n",
+ "\n",
"brevitas_model.load_state_dict(trained_state_dict, strict=False)"
]
},
{
"cell_type": "code",
- "execution_count": 9,
+ "execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
@@ -360,7 +365,7 @@
},
{
"cell_type": "code",
- "execution_count": 10,
+ "execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
@@ -390,14 +395,14 @@
},
{
"cell_type": "code",
- "execution_count": 11,
+ "execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "ok 100 nok 0: 100%|██████████| 100/100 [00:48<00:00, 2.09it/s]\n"
+ "ok 100 nok 0: 100%|██████████| 100/100 [00:46<00:00, 2.17it/s]\n"
]
}
],
@@ -426,7 +431,7 @@
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": 10,
"metadata": {},
"outputs": [
{
diff --git a/notebooks/end2end_example/cybersecurity/3-build-accelerator-with-finn.ipynb b/notebooks/end2end_example/cybersecurity/3-build-accelerator-with-finn.ipynb
index 1ee1cefbe..75566b1ae 100644
--- a/notebooks/end2end_example/cybersecurity/3-build-accelerator-with-finn.ipynb
+++ b/notebooks/end2end_example/cybersecurity/3-build-accelerator-with-finn.ipynb
@@ -41,7 +41,7 @@
"Since version 0.5b, the FINN compiler has a `build_dataflow` tool. Compared to previous versions which required setting up all the needed transformations in a Python script, it makes experimenting with dataflow architecture generation easier. The core idea is to specify the relevant build info as a configuration `dict`, which invokes all the necessary steps to make the dataflow build happen. It can be invoked either from the [command line](https://finn-dev.readthedocs.io/en/latest/command_line.html) or with a single Python function call\n",
"\n",
"\n",
- "In this notebook, we'll use the Python function call to invoke the builds to stay inside the Jupyter notebook, but feel free to experiment with reproducing what we do here with the `./run-docker.sh build_dataflow` and `./run-docker.sh build_custom` command-line entry points too, as documented [here]((https://finn-dev.readthedocs.io/en/latest/command_line.html))."
+ "In this notebook, we'll use the Python function call to invoke the builds to stay inside the Jupyter notebook, but feel free to experiment with reproducing what we do here with the `./run-docker.sh build_dataflow` and `./run-docker.sh build_custom` command-line entry points too, as documented [here](https://finn-dev.readthedocs.io/en/latest/command_line.html)."
]
},
{
@@ -277,7 +277,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Here, we can see the estimated number of clock cycles each layer will take. Recall that all of these layers will be running in parallel, and the slowest layer will determine the overall throughput of the entire neural network. FINN attempts to parallelize each layer such that they all take a similar number of cycles, and less than the corresponding number of cycles that would be required to meet `target_fps`.\n",
+ "Here, we can see the estimated number of clock cycles each layer will take. Recall that all of these layers will be running in parallel, and the slowest layer will determine the overall throughput of the entire neural network. FINN attempts to parallelize each layer such that they all take a similar number of cycles, and less than the corresponding number of cycles that would be required to meet `target_fps`. Additionally by summing up all layer cycle estimates one can obtain an estimate for the overall latency of the whole network. \n",
"\n",
"Finally, we can see the layer-by-layer resource estimates in the `estimate_layer_resources.json` report:"
]
@@ -341,7 +341,9 @@
"source": [
"## Launch a Build: Stitched IP, out-of-context synth and rtlsim Performance <a id=\"build_ip_synth_rtlsim\"></a>\n",
"\n",
- "Once we have a configuration that gives satisfactory estimates, we can move on to generating the accelerator. We can do this in different ways depending on how we want to integrate the accelerator into a larger system. For instance, if we have a larger streaming system built in Vivado or if we'd like to re-use this generated accelerator as an IP component in other projects, the `STITCHED_IP` output product is a good choice. We can also use the `OOC_SYNTH` output product to get post-synthesis resource and clock frequency numbers for our accelerator."
+ "Once we have a configuration that gives satisfactory estimates, we can move on to generating the accelerator. We can do this in different ways depending on how we want to integrate the accelerator into a larger system. For instance, if we have a larger streaming system built in Vivado or if we'd like to re-use this generated accelerator as an IP component in other projects, the `STITCHED_IP` output product is a good choice. We can also use the `OOC_SYNTH` output product to get post-synthesis resource and clock frequency numbers for our accelerator.\n",
+ "\n",
+ "**NOTE: These next builds will take several minutes since multiple calls to Vivado and a call to the RTL simulator are involved.**"
]
},
{
--
GitLab