diff --git a/notebooks/end2end_example/tfc_end2end_example.ipynb b/notebooks/end2end_example/tfc_end2end_example.ipynb
index 1796faab6166caefc880ab2ec4e29d6abab29dec..2a248138fc97e2ec7c1eaf71c66d5f57ac120cf9 100644
--- a/notebooks/end2end_example/tfc_end2end_example.ipynb
+++ b/notebooks/end2end_example/tfc_end2end_example.ipynb
@@ -6,25 +6,18 @@
"source": [
"# FINN - End-to-End Flow\n",
"-----------------------------------------------------------------\n",
- "This notebook gives an overview about the end to end flow of FINN. From loading an ONNX model from Brevitas, followed by the numerous transformations in FINN and up to the generation of a bitstream that can be used to load an FPGA. \n",
"\n",
- "We'll use the following showSrc function to print the source code for function calls in the Jupyter notebook."
+ "In this notebook, we will show how to take a simple, binarized, fully-connected network trained on the MNIST data set and take it all the way down to a customized bitfile running on an FPGA. \n",
+ "\n",
+ "This notebook is quite lengthy, and some of the cells (involving Vivado synthesis) may take up to an hour to finish running. To let you save and resume your progress, we will save the intermediate ONNX models that are generated in the various steps to disk, so that you can jump back directly to where you left off.\n",
+ "\n"
]
},
{
- "cell_type": "code",
- "execution_count": 1,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "import inspect\n",
- "import netron\n",
- "from finn.util.basic import make_build_dir\n",
- "\n",
- "def showSrc(what):\n",
- " print(\"\".join(inspect.getsourcelines(what)[0]))\n",
- " \n",
- "build_dir = \"/workspace/finn\""
+ "We'll use the following showSrc function to print the source code for function calls in the Jupyter notebook."
]
},
{
@@ -32,7 +25,8 @@
"metadata": {},
"source": [
"## Overview\n",
- "The notebook is based on the following diagram. "
+ "\n",
+ "The FINN compiler comes with many `transformations` that modify the ONNX representation of the network according to certain patterns. This notebook will demonstrate a *possible* sequence of such transformations to take a particular trained network all the way down to hardware, as shown in the figure below."
]
},
{
@@ -46,8 +40,32 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "The diagram visualizes the end-to-end flow of FINN. The cylinder-like fields show the state of the network representation in the respective step. The rectangular fields represent the transformations that are applied to the network to achieve a certain result. The diagram is divided into 5 blocks, each of it includes several flow steps. The flow starts in top left corner with Brevitas export (pink block), followed by the preparation of the network (grey block) for the Vivado HLS and Vivado synthesis (yellow block). There is also a section for testing and verification in software (green block) and the hardware test on the PYNQ board (red block).\n",
- "The diagram leads to the following outline for this Jupyter notebook."
+ "The cylinder-like fields show the state of the network representation in the respective step. The rectangular fields represent the transformations that are applied to the network to achieve a certain result. The diagram is divided into 5 blocks, each of it includes several flow steps. The flow starts in top left corner with Brevitas export (purple block), followed by the preparation of the network (grey block) for the Vivado HLS and Vivado synthesis (yellow block), and testing on a PYNQ board (pink block).\n",
+ "There is an additional section for functional verification (green block), which we will not cover in this notebook.\n",
+ "\n",
+ "\n",
+ "This Jupyter notebook is organized based on the sections described above. We will use the following helper functions, `showSrc` to show source code of FINN library calls and `showInNetron` to show the ONNX model at the current transformation step. The Netron displays are interactive, but they only work when running the notebook actively and not on GitHub (i.e. if you are viewing this on GitHub you'll only see blank squares)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 74,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import inspect\n",
+ "import netron\n",
+ "from finn.util.basic import make_build_dir\n",
+ "from IPython.display import IFrame\n",
+ "\n",
+ "def showSrc(what):\n",
+ " print(\"\".join(inspect.getsourcelines(what)[0]))\n",
+ " \n",
+ "def showInNetron(model_filename):\n",
+ " netron.start(model_filename, port=8081, host=\"0.0.0.0\")\n",
+ " return IFrame(src=\"http://0.0.0.0:8081/\", width=\"100%\", height=400)\n",
+ " \n",
+ "build_dir = \"/workspace/finn\""
]
},
{
@@ -67,11 +85,7 @@
" * Creation of stitched design\n",
" * PYNQ shell project\n",
" * Synthesis, place and route\n",
- "4. [Hardware Test](#hw_test)\n",
- "5. [Simulation & Emulation flows for functional verification](#sim)\n",
- " * Simulation using Python\n",
- " * Simulation (npysim) using C++\n",
- " * Emulation (rtlsim) using PyVerilator"
+ "4. [Deploy and Test on PYNQ](#hw_test)"
]
},
{
@@ -79,14 +93,14 @@
"metadata": {},
"source": [
"## 1. Brevitas export <a id='brev_exp'></a>\n",
- "FINN expects an ONNX model as input. This can be a model trained with [Brevitas](https://github.com/Xilinx/brevitas). Brevitas is a Pytorch library for quantization-aware training and the FINN Docker image comes with several [example Brevitas networks](https://github.com/maltanar/brevitas_cnv_lfc). To show the FINN end-to-end flow, we'll use the TFC-w1a1 model as example network. The Brevitas export is only briefly described here, for details see Jupyter notebook [3-FINN-Brevitas-network-import](3-FINN-Brevitas-network-import.ipynb).\n",
+ "FINN expects an ONNX model as input. This can be a model trained with [Brevitas](https://github.com/Xilinx/brevitas). Brevitas is a PyTorch library for quantization-aware training and the FINN Docker image comes with several [example Brevitas networks](https://github.com/maltanar/brevitas_cnv_lfc). To show the FINN end-to-end flow, we'll use the TFC-w1a1 model as example network. The Brevitas export is only briefly described here, for details see Jupyter notebook [3-FINN-Brevitas-network-import](3-FINN-Brevitas-network-import.ipynb).\n",
"\n",
"First a few things have to be imported. Then the model can be loaded with the pretrained weights."
]
},
{
"cell_type": "code",
- "execution_count": 40,
+ "execution_count": 75,
"metadata": {},
"outputs": [
{
@@ -117,7 +131,7 @@
},
{
"cell_type": "code",
- "execution_count": 41,
+ "execution_count": 82,
"metadata": {},
"outputs": [
{
@@ -128,33 +142,31 @@
"Stopping http://0.0.0.0:8081\n",
"Serving '/workspace/finn/tfc_w1_a1.onnx' at http://0.0.0.0:8081\n"
]
- }
- ],
- "source": [
- "netron.start(build_dir+\"/tfc_w1_a1.onnx\", port=8081, host=\"0.0.0.0\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 42,
- "metadata": {},
- "outputs": [
+ },
{
"data": {
"text/html": [
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>\n"
+ "\n",
+ " <iframe\n",
+ " width=\"100%\"\n",
+ " height=\"400\"\n",
+ " src=\"http://0.0.0.0:8081/\"\n",
+ " frameborder=\"0\"\n",
+ " allowfullscreen\n",
+ " ></iframe>\n",
+ " "
],
"text/plain": [
- "<IPython.core.display.HTML object>"
+ "<IPython.lib.display.IFrame at 0x7fc62d60e5c0>"
]
},
+ "execution_count": 82,
"metadata": {},
- "output_type": "display_data"
+ "output_type": "execute_result"
}
],
"source": [
- "%%html\n",
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>"
+ "showInNetron(build_dir+\"/tfc_w1_a1.onnx\")"
]
},
{
@@ -166,7 +178,7 @@
},
{
"cell_type": "code",
- "execution_count": 43,
+ "execution_count": 83,
"metadata": {},
"outputs": [],
"source": [
@@ -235,7 +247,7 @@
},
{
"cell_type": "code",
- "execution_count": 44,
+ "execution_count": 84,
"metadata": {},
"outputs": [],
"source": [
@@ -250,9 +262,7 @@
"model = model.transform(GiveReadableTensorNames())\n",
"model = model.transform(InferDataTypes())\n",
"\n",
- "\n",
- "# save model with other name for section \"Simulation using Python\"\n",
- "model.save(build_dir+\"/tfc_w1_a1_after_brevitas_export.onnx\")"
+ "model.save(build_dir+\"/tfc_w1_a1_tidy.onnx\")"
]
},
{
@@ -264,7 +274,7 @@
},
{
"cell_type": "code",
- "execution_count": 45,
+ "execution_count": 85,
"metadata": {},
"outputs": [
{
@@ -273,36 +283,33 @@
"text": [
"\n",
"Stopping http://0.0.0.0:8081\n",
- "Serving '/workspace/finn/tfc_w1_a1.onnx' at http://0.0.0.0:8081\n"
+ "Serving '/workspace/finn/tfc_w1_a1_tidy.onnx' at http://0.0.0.0:8081\n"
]
- }
- ],
- "source": [
- "model.save(build_dir+\"/tfc_w1_a1.onnx\")\n",
- "netron.start(build_dir+\"/tfc_w1_a1.onnx\", port=8081, host=\"0.0.0.0\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 46,
- "metadata": {},
- "outputs": [
+ },
{
"data": {
"text/html": [
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>\n"
+ "\n",
+ " <iframe\n",
+ " width=\"100%\"\n",
+ " height=\"400\"\n",
+ " src=\"http://0.0.0.0:8081/\"\n",
+ " frameborder=\"0\"\n",
+ " allowfullscreen\n",
+ " ></iframe>\n",
+ " "
],
"text/plain": [
- "<IPython.core.display.HTML object>"
+ "<IPython.lib.display.IFrame at 0x7fc6c4430828>"
]
},
+ "execution_count": 85,
"metadata": {},
- "output_type": "display_data"
+ "output_type": "execute_result"
}
],
"source": [
- "%%html\n",
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>"
+ "showInNetron(build_dir+\"/tfc_w1_a1_tidy.onnx\")"
]
},
{
@@ -317,7 +324,7 @@
},
{
"cell_type": "code",
- "execution_count": 47,
+ "execution_count": 86,
"metadata": {},
"outputs": [
{
@@ -369,7 +376,7 @@
},
{
"cell_type": "code",
- "execution_count": 48,
+ "execution_count": 88,
"metadata": {},
"outputs": [
{
@@ -380,42 +387,43 @@
"Stopping http://0.0.0.0:8081\n",
"Serving '/workspace/finn/tfc_w1_a1_streamlined.onnx' at http://0.0.0.0:8081\n"
]
- }
- ],
- "source": [
- "model = model.transform(Streamline())\n",
- "model.save(build_dir+\"/tfc_w1_a1_streamlined.onnx\")\n",
- "netron.start(build_dir+\"/tfc_w1_a1_streamlined.onnx\", port=8081, host=\"0.0.0.0\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 49,
- "metadata": {},
- "outputs": [
+ },
{
"data": {
"text/html": [
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>\n"
+ "\n",
+ " <iframe\n",
+ " width=\"100%\"\n",
+ " height=\"400\"\n",
+ " src=\"http://0.0.0.0:8081/\"\n",
+ " frameborder=\"0\"\n",
+ " allowfullscreen\n",
+ " ></iframe>\n",
+ " "
],
"text/plain": [
- "<IPython.core.display.HTML object>"
+ "<IPython.lib.display.IFrame at 0x7fc62d863320>"
]
},
+ "execution_count": 88,
"metadata": {},
- "output_type": "display_data"
+ "output_type": "execute_result"
}
],
"source": [
- "%%html\n",
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>"
+ "model = ModelWrapper(build_dir+\"/tfc_w1_a1_tidy.onnx\")\n",
+ "model = model.transform(Streamline())\n",
+ "model.save(build_dir+\"/tfc_w1_a1_streamlined.onnx\")\n",
+ "showInNetron(build_dir+\"/tfc_w1_a1_streamlined.onnx\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Our example network is a quantized network with 1 bit precision. For this reason, after streamlining, the resulting bipolar matrix multiplications are converted into xnorpopcount operations. This transformation produces operations that are again collapsed and converted into thresholds. This procedure is shown below. \n",
+ "You can see that the network has become simplified considerably compared to the previous step -- a lot of Mul - Add operations have disappeared between the layers, and the `Sign` nodes have been replaced with `MultiThreshold` nodes instead.\n",
+ "\n",
+ "Our example network is a quantized network with 1-bit bipolar (-1, +1 values) precision. For this reason, after streamlining, the resulting bipolar matrix multiplications are converted into xnorpopcount operations. This transformation produces operations that are again collapsed and converted into thresholds. This procedure is shown below. \n",
"In this state the model can still be simulated with Python, even if it no longer contains only standard ONNX nodes. For details, see section [Simulation using Python](#simpy).\n",
"\n",
"After these finishing transformations, the nodes can be converted to HLS layers for further processing."
@@ -423,9 +431,40 @@
},
{
"cell_type": "code",
- "execution_count": 50,
+ "execution_count": 89,
"metadata": {},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "Stopping http://0.0.0.0:8081\n",
+ "Serving '/workspace/finn/tfc_w1a1_ready_for_hls_conversion.onnx' at http://0.0.0.0:8081\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ " <iframe\n",
+ " width=\"100%\"\n",
+ " height=\"400\"\n",
+ " src=\"http://0.0.0.0:8081/\"\n",
+ " frameborder=\"0\"\n",
+ " allowfullscreen\n",
+ " ></iframe>\n",
+ " "
+ ],
+ "text/plain": [
+ "<IPython.lib.display.IFrame at 0x7fc654223ba8>"
+ ]
+ },
+ "execution_count": 89,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
"source": [
"from finn.transformation.bipolar_to_xnor import ConvertBipolarMatMulToXnorPopcount\n",
"import finn.transformation.streamline.absorb as absorb\n",
@@ -434,7 +473,17 @@
"model = model.transform(ConvertBipolarMatMulToXnorPopcount())\n",
"model = model.transform(absorb.AbsorbAddIntoMultiThreshold())\n",
"model = model.transform(absorb.AbsorbMulIntoMultiThreshold())\n",
- "model = model.transform(RoundAndClipThresholds())"
+ "model = model.transform(RoundAndClipThresholds())\n",
+ "\n",
+ "model.save(build_dir+\"/tfc_w1a1_ready_for_hls_conversion.onnx\")\n",
+ "showInNetron(build_dir+\"/tfc_w1a1_ready_for_hls_conversion.onnx\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Observe the pairs of `XnorPopcountmatMul` and `MultiThreshold` layers following each other -- this is the particular pattern that the next step will be looking for in order to convert them to HLS layers."
]
},
{
@@ -449,8 +498,10 @@
},
{
"cell_type": "code",
- "execution_count": 51,
- "metadata": {},
+ "execution_count": 90,
+ "metadata": {
+ "scrolled": false
+ },
"outputs": [
{
"name": "stdout",
@@ -460,47 +511,42 @@
"Stopping http://0.0.0.0:8081\n",
"Serving '/workspace/finn/tfc_w1_a1_hls_layers.onnx' at http://0.0.0.0:8081\n"
]
- }
- ],
- "source": [
- "import finn.transformation.fpgadataflow.convert_to_hls_layers as to_hls\n",
- "model = model.transform(to_hls.InferBinaryStreamingFCLayer())\n",
- "model.save(build_dir+\"/tfc_w1_a1_hls_layers.onnx\")\n",
- "netron.start(build_dir+\"/tfc_w1_a1_hls_layers.onnx\", port=8081, host=\"0.0.0.0\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 52,
- "metadata": {
- "scrolled": true
- },
- "outputs": [
+ },
{
"data": {
"text/html": [
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>\n"
+ "\n",
+ " <iframe\n",
+ " width=\"100%\"\n",
+ " height=\"400\"\n",
+ " src=\"http://0.0.0.0:8081/\"\n",
+ " frameborder=\"0\"\n",
+ " allowfullscreen\n",
+ " ></iframe>\n",
+ " "
],
"text/plain": [
- "<IPython.core.display.HTML object>"
+ "<IPython.lib.display.IFrame at 0x7fc62d60eb70>"
]
},
+ "execution_count": 90,
"metadata": {},
- "output_type": "display_data"
+ "output_type": "execute_result"
}
],
"source": [
- "%%html\n",
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>"
+ "import finn.transformation.fpgadataflow.convert_to_hls_layers as to_hls\n",
+ "model = ModelWrapper(build_dir+\"/tfc_w1a1_ready_for_hls_conversion.onnx\")\n",
+ "model = model.transform(to_hls.InferBinaryStreamingFCLayer())\n",
+ "model.save(build_dir+\"/tfc_w1_a1_hls_layers.onnx\")\n",
+ "showInNetron(build_dir+\"/tfc_w1_a1_hls_layers.onnx\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Each StreamingFCLayer_Batch node has two attributes that specify the degree of folding, PE and SIMD. In all nodes the values for these attributes are set as default to 1, which would correspond to a maximum folding. The user can now adjust the folding as desired. This is described in the next section.\n",
- "\n",
- "At this point the model can also be simulated using C++. The exact procedure is described in section [Simulation using C++](#npysim)."
+ "Each StreamingFCLayer_Batch node has two attributes that specify the degree of folding, PE and SIMD. In all nodes the values for these attributes are set as default to 1, which would correspond to a maximum folding (time multiplexing) and thus minimum performance. The user can now adjust the folding as desired. This is described in the next section."
]
},
{
@@ -514,7 +560,7 @@
},
{
"cell_type": "code",
- "execution_count": 53,
+ "execution_count": 91,
"metadata": {},
"outputs": [
{
@@ -525,39 +571,36 @@
"Stopping http://0.0.0.0:8081\n",
"Serving '/workspace/finn/tfc_w1_a1_dataflow_parent.onnx' at http://0.0.0.0:8081\n"
]
- }
- ],
- "source": [
- "from finn.transformation.fpgadataflow.create_dataflow_partition import CreateDataflowPartition\n",
- "\n",
- "parent_model = model.transform(CreateDataflowPartition())\n",
- "parent_model.save(build_dir+\"/tfc_w1_a1_dataflow_parent.onnx\")\n",
- "netron.start(build_dir+\"/tfc_w1_a1_dataflow_parent.onnx\", port=8081, host=\"0.0.0.0\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 54,
- "metadata": {
- "scrolled": false
- },
- "outputs": [
+ },
{
"data": {
"text/html": [
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>\n"
+ "\n",
+ " <iframe\n",
+ " width=\"100%\"\n",
+ " height=\"400\"\n",
+ " src=\"http://0.0.0.0:8081/\"\n",
+ " frameborder=\"0\"\n",
+ " allowfullscreen\n",
+ " ></iframe>\n",
+ " "
],
"text/plain": [
- "<IPython.core.display.HTML object>"
+ "<IPython.lib.display.IFrame at 0x7fc62d60e1d0>"
]
},
+ "execution_count": 91,
"metadata": {},
- "output_type": "display_data"
+ "output_type": "execute_result"
}
],
"source": [
- "%%html\n",
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>"
+ "from finn.transformation.fpgadataflow.create_dataflow_partition import CreateDataflowPartition\n",
+ "\n",
+ "model = ModelWrapper(build_dir+\"/tfc_w1_a1_hls_layers.onnx\")\n",
+ "parent_model = model.transform(CreateDataflowPartition())\n",
+ "parent_model.save(build_dir+\"/tfc_w1_a1_dataflow_parent.onnx\")\n",
+ "showInNetron(build_dir+\"/tfc_w1_a1_dataflow_parent.onnx\")"
]
},
{
@@ -569,7 +612,7 @@
},
{
"cell_type": "code",
- "execution_count": 55,
+ "execution_count": 92,
"metadata": {},
"outputs": [
{
@@ -578,38 +621,36 @@
"text": [
"\n",
"Stopping http://0.0.0.0:8081\n",
- "Serving '/tmp/finn_maltanar/dataflow_partition_n7ae7i0t/df_model.onnx' at http://0.0.0.0:8081\n"
+ "Serving '/tmp/finn_maltanar/dataflow_partition_l2y9b77c/df_model.onnx' at http://0.0.0.0:8081\n"
]
- }
- ],
- "source": [
- "from finn.custom_op.registry import getCustomOp\n",
- "sdp_node = getCustomOp(parent_model.graph.node[2])\n",
- "dataflow_model_filename = sdp_node.get_nodeattr(\"model\")\n",
- "netron.start(dataflow_model_filename, port=8081, host=\"0.0.0.0\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 56,
- "metadata": {},
- "outputs": [
+ },
{
"data": {
"text/html": [
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>\n"
+ "\n",
+ " <iframe\n",
+ " width=\"100%\"\n",
+ " height=\"400\"\n",
+ " src=\"http://0.0.0.0:8081/\"\n",
+ " frameborder=\"0\"\n",
+ " allowfullscreen\n",
+ " ></iframe>\n",
+ " "
],
"text/plain": [
- "<IPython.core.display.HTML object>"
+ "<IPython.lib.display.IFrame at 0x7fc62d60e320>"
]
},
+ "execution_count": 92,
"metadata": {},
- "output_type": "display_data"
+ "output_type": "execute_result"
}
],
"source": [
- "%%html\n",
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>"
+ "from finn.custom_op.registry import getCustomOp\n",
+ "sdp_node = getCustomOp(parent_model.graph.node[2])\n",
+ "dataflow_model_filename = sdp_node.get_nodeattr(\"model\")\n",
+ "showInNetron(dataflow_model_filename)"
]
},
{
@@ -621,7 +662,7 @@
},
{
"cell_type": "code",
- "execution_count": 57,
+ "execution_count": 93,
"metadata": {},
"outputs": [],
"source": [
@@ -638,7 +679,7 @@
},
{
"cell_type": "code",
- "execution_count": 20,
+ "execution_count": 94,
"metadata": {},
"outputs": [
{
@@ -672,7 +713,7 @@
},
{
"cell_type": "code",
- "execution_count": 21,
+ "execution_count": 95,
"metadata": {},
"outputs": [
{
@@ -704,10 +745,11 @@
" 'executable_path': ('s', False, ''),\n",
" 'ipgen_path': ('s', False, ''),\n",
" 'exec_mode': ('s', False, ''),\n",
- " 'sim_cycles': ('i', False, 0)}"
+ " 'sim_cycles': ('i', False, 0),\n",
+ " 'rtlsim_trace': ('s', False, '')}"
]
},
- "execution_count": 21,
+ "execution_count": 95,
"metadata": {},
"output_type": "execute_result"
}
@@ -725,7 +767,7 @@
},
{
"cell_type": "code",
- "execution_count": 23,
+ "execution_count": 96,
"metadata": {},
"outputs": [],
"source": [
@@ -759,20 +801,52 @@
},
{
"cell_type": "code",
- "execution_count": 24,
+ "execution_count": 98,
"metadata": {},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "Stopping http://0.0.0.0:8081\n",
+ "Serving '/workspace/finn/tfc_w1_a1_set_folding_factors.onnx' at http://0.0.0.0:8081\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ " <iframe\n",
+ " width=\"100%\"\n",
+ " height=\"400\"\n",
+ " src=\"http://0.0.0.0:8081/\"\n",
+ " frameborder=\"0\"\n",
+ " allowfullscreen\n",
+ " ></iframe>\n",
+ " "
+ ],
+ "text/plain": [
+ "<IPython.lib.display.IFrame at 0x7fc654223780>"
+ ]
+ },
+ "execution_count": 98,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
"source": [
"from finn.transformation.fpgadataflow.insert_tlastmarker import InsertTLastMarker\n",
"model = model.transform(InsertTLastMarker())\n",
- "model.save(build_dir+\"/tfc_w1_a1_set_folding_factors.onnx\")"
+ "model.save(build_dir+\"/tfc_w1_a1_set_folding_factors.onnx\")\n",
+ "showInNetron(build_dir+\"/tfc_w1_a1_set_folding_factors.onnx\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "This completes the network preparation and the network can be passed on to the next block *Vivado HLS and Vivado synthesis*. Which is described below."
+ "This completes the network preparation and the network can be passed on to the next block *Vivado HLS and Vivado synthesis*, which is described below."
]
},
{
@@ -791,7 +865,7 @@
},
{
"cell_type": "code",
- "execution_count": 25,
+ "execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
@@ -821,10 +895,11 @@
},
{
"cell_type": "code",
- "execution_count": 26,
+ "execution_count": 99,
"metadata": {},
"outputs": [],
"source": [
+ "model = ModelWrapper(build_dir+\"/tfc_w1_a1_set_folding_factors.onnx\")\n",
"model = model.transform(GiveUniqueNodeNames())\n",
"\n",
"from finn.transformation.fpgadataflow.codegen_ipgen import CodeGen_ipgen\n",
@@ -842,7 +917,7 @@
},
{
"cell_type": "code",
- "execution_count": 27,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@@ -860,7 +935,7 @@
},
{
"cell_type": "code",
- "execution_count": 28,
+ "execution_count": 27,
"metadata": {},
"outputs": [
{
@@ -875,32 +950,7 @@
],
"source": [
"model.save(build_dir+\"/tfc_w1_a1_ipgen.onnx\")\n",
- "netron.start(build_dir+\"/tfc_w1_a1_ipgen.onnx\", port=8081, host=\"0.0.0.0\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 29,
- "metadata": {
- "scrolled": true
- },
- "outputs": [
- {
- "data": {
- "text/html": [
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"800\"></iframe>\n"
- ],
- "text/plain": [
- "<IPython.core.display.HTML object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "%%html\n",
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"800\"></iframe>"
+ "showInNetron(build_dir+\"/tfc_w1_a1_ipgen.onnx\")"
]
},
{
@@ -916,7 +966,7 @@
},
{
"cell_type": "code",
- "execution_count": 30,
+ "execution_count": 29,
"metadata": {},
"outputs": [
{
@@ -945,7 +995,7 @@
},
{
"cell_type": "code",
- "execution_count": 31,
+ "execution_count": 30,
"metadata": {},
"outputs": [
{
@@ -953,8 +1003,8 @@
"output_type": "stream",
"text": [
"#!/bin/bash \r\n",
- "cd /tmp/finn_maltanar/code_gen_ipgen_StreamingFCLayer_Batch_y_fxb2eb\r\n",
- "vivado_hls /tmp/finn_maltanar/code_gen_ipgen_StreamingFCLayer_Batch_y_fxb2eb/hls_syn_StreamingFCLayer_Batch_0.tcl\r\n",
+ "cd /tmp/finn_maltanar/code_gen_ipgen_StreamingFCLayer_Batch_hc367wg4\r\n",
+ "vivado_hls /tmp/finn_maltanar/code_gen_ipgen_StreamingFCLayer_Batch_hc367wg4/hls_syn_StreamingFCLayer_Batch_0.tcl\r\n",
"cd /workspace/finn\r\n"
]
}
@@ -975,7 +1025,7 @@
},
{
"cell_type": "code",
- "execution_count": 32,
+ "execution_count": 31,
"metadata": {},
"outputs": [
{
@@ -985,7 +1035,7 @@
"\r\n",
"set config_proj_name project_StreamingFCLayer_Batch_0\r\n",
"puts \"HLS project: $config_proj_name\"\r\n",
- "set config_hwsrcdir \"/tmp/finn_maltanar/code_gen_ipgen_StreamingFCLayer_Batch_y_fxb2eb\"\r\n",
+ "set config_hwsrcdir \"/tmp/finn_maltanar/code_gen_ipgen_StreamingFCLayer_Batch_hc367wg4\"\r\n",
"puts \"HW source dir: $config_hwsrcdir\"\r\n",
"set config_proj_part \"xczu3eg-sbva484-1-e\"\r\n",
"\r\n",
@@ -1036,7 +1086,7 @@
},
{
"cell_type": "code",
- "execution_count": 36,
+ "execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
@@ -1056,22 +1106,22 @@
},
{
"cell_type": "code",
- "execution_count": 37,
+ "execution_count": 33,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[key: \"vivado_stitch_proj\"\n",
- "value: \"/tmp/finn_maltanar/vivado_stitch_proj_gvhcdxah\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_stitch_proj_n3me5eke\"\n",
", key: \"vivado_stitch_vlnv\"\n",
"value: \"xilinx_finn:finn:finn_design:1.0\"\n",
", key: \"wrapper_filename\"\n",
- "value: \"/tmp/finn_maltanar/vivado_stitch_proj_gvhcdxah/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_stitch_proj_n3me5eke/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
"]"
]
},
- "execution_count": 37,
+ "execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
@@ -1082,16 +1132,16 @@
},
{
"cell_type": "code",
- "execution_count": 38,
+ "execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
- "'/tmp/finn_maltanar/vivado_stitch_proj_gvhcdxah'"
+ "'/tmp/finn_maltanar/vivado_stitch_proj_n3me5eke'"
]
},
- "execution_count": 38,
+ "execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
@@ -1116,7 +1166,7 @@
},
{
"cell_type": "code",
- "execution_count": 39,
+ "execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
@@ -1134,24 +1184,24 @@
},
{
"cell_type": "code",
- "execution_count": 39,
+ "execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[key: \"vivado_stitch_proj\"\n",
- "value: \"/tmp/finn_maltanar/vivado_stitch_proj_ud9yxuzi\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_stitch_proj_n3me5eke\"\n",
", key: \"vivado_stitch_vlnv\"\n",
"value: \"xilinx_finn:finn:finn_design:1.0\"\n",
", key: \"wrapper_filename\"\n",
- "value: \"/tmp/finn_maltanar/vivado_stitch_proj_ud9yxuzi/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_stitch_proj_n3me5eke/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
", key: \"vivado_pynq_proj\"\n",
- "value: \"/tmp/finn_maltanar/vivado_pynq_proj_6rhrsy8m\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_pynq_proj_hqlnpt5q\"\n",
"]"
]
},
- "execution_count": 39,
+ "execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
@@ -1165,7 +1215,7 @@
},
{
"cell_type": "code",
- "execution_count": 40,
+ "execution_count": 37,
"metadata": {},
"outputs": [
{
@@ -1191,7 +1241,7 @@
},
{
"cell_type": "code",
- "execution_count": 41,
+ "execution_count": 38,
"metadata": {},
"outputs": [],
"source": [
@@ -1214,26 +1264,26 @@
},
{
"cell_type": "code",
- "execution_count": 42,
+ "execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[key: \"vivado_stitch_proj\"\n",
- "value: \"/tmp/finn_maltanar/vivado_stitch_proj_ud9yxuzi\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_stitch_proj_n3me5eke\"\n",
", key: \"vivado_stitch_vlnv\"\n",
"value: \"xilinx_finn:finn:finn_design:1.0\"\n",
", key: \"wrapper_filename\"\n",
- "value: \"/tmp/finn_maltanar/vivado_stitch_proj_ud9yxuzi/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_stitch_proj_n3me5eke/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
", key: \"vivado_pynq_proj\"\n",
- "value: \"/tmp/finn_maltanar/vivado_pynq_proj_6rhrsy8m\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_pynq_proj_hqlnpt5q\"\n",
", key: \"vivado_pynq_bitfile\"\n",
- "value: \"/tmp/finn_maltanar/vivado_pynq_proj_6rhrsy8m/resizer.bit\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_pynq_proj_hqlnpt5q/resizer.bit\"\n",
"]"
]
},
- "execution_count": 42,
+ "execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
@@ -1247,7 +1297,7 @@
},
{
"cell_type": "code",
- "execution_count": 48,
+ "execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
@@ -1258,35 +1308,16 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## 4. Hardware test <a id='hw_test'></a>"
+ "## 4. Deploy and Test on PYNQ <a id='hw_test'></a>\n",
+ "\n",
+ "Now that we have synthesized a bitfile for our network, we will generate some Python code for PYNQ that will act as the driver for this bitfile, package everything into a deployment folder and copy that to our PYNQ board."
]
},
{
"cell_type": "code",
- "execution_count": 49,
+ "execution_count": 41,
"metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[key: \"vivado_stitch_proj\"\n",
- "value: \"/tmp/finn_maltanar_22115/vivado_stitch_proj_nfte0nh0\"\n",
- ", key: \"vivado_stitch_vlnv\"\n",
- "value: \"xilinx_finn:finn:finn_design:1.0\"\n",
- ", key: \"vivado_pynq_proj\"\n",
- "value: \"/tmp/finn_maltanar_22115/vivado_pynq_proj_bj_z4tm0\"\n",
- ", key: \"vivado_pynq_bitfile\"\n",
- "value: \"/tmp/finn_maltanar_22115/vivado_pynq_proj_bj_z4tm0/resizer.bit\"\n",
- ", key: \"pynq_driver_dir\"\n",
- "value: \"/tmp/finn_maltanar_22115/pynq_driver_63xiuej8\"\n",
- "]"
- ]
- },
- "execution_count": 49,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
+ "outputs": [],
"source": [
"from finn.transformation.fpgadataflow.make_pynq_driver import MakePYNQDriver\n",
"model = ModelWrapper(build_dir + \"/tfc_w1_a1_post_synthesis.onnx\")\n",
@@ -1297,437 +1328,319 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## 5. Simulation & Emulation flows for functional verification <a id='sim'></a>\n",
- "* [Simulation using Python](#simpy)\n",
- "* [Simulation (npysim) using C++](#npysim)\n",
- "* [Emulation (rtlsim) using PyVerilator](#rtlsim)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Simulation using Python <a id='simpy'></a>\n",
- "If an ONNX model consists of [standard ONNX nodes](https://github.com/onnx/onnx/blob/master/docs/Operators.md) and/or FINN custom operations that do not belong to the fpgadataflow (`backend` $\\neq$ \"fpgadataflow\") this model can be checked for functionality using Python. General information about FINN custom op nodes can be found in Jupyter notebook [7-FINN-CustomOps](7-FINN-CustomOps.ipynb)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "To simulate an ONNX node [ONNX Runtime](https://github.com/microsoft/onnxruntime) is used. ONNX Runtime is an open source tool developed by Microsoft to run standard ONNX nodes. For the FINN custom op nodes execution functions are defined. The following is an example of the execution function of a XNOR popcount node."
+ "The generated driver is placed in a folder that is indicated by the `pynq_driver_dir` top-level metadata. We can examine the generated PYNQ Python driver code as follows:"
]
},
{
"cell_type": "code",
- "execution_count": 25,
+ "execution_count": 42,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "def xnorpopcountmatmul(inp0, inp1):\n",
- " # extract the operand shapes\n",
- " (M, K0) = inp0.shape\n",
- " (K1, N) = inp1.shape\n",
- " # make sure shapes are compatible with matmul\n",
- " assert K0 == K1\n",
- " K = K0\n",
- " # we simulate XNOR-popcount matrix multiplication as a regular bipolar\n",
- " # matrix multiplication followed by some post processing\n",
- " # first, convert binary inputs to bipolar\n",
- " inp0_bipolar = 2.0 * inp0 - 1.0\n",
- " inp1_bipolar = 2.0 * inp1 - 1.0\n",
- " # call regular numpy matrix multiplication\n",
- " out = np.matmul(inp0_bipolar, inp1_bipolar)\n",
- " # XNOR-popcount does not produce the regular dot product result --\n",
- " # it returns the number of +1s after XNOR. let P be the number of +1s\n",
- " # and N be the number of -1s. XNOR-popcount returns P, whereas the\n",
- " # regular dot product result from numpy is P-N, so we need to apply\n",
- " # some correction.\n",
- " # out = P-N\n",
- " # K = P+N\n",
- " # out + K = 2P, so P = (out + K)/2\n",
- " return (out + K) * 0.5\n",
- "\n"
+ "\r\n",
+ "from pynq import Overlay\r\n",
+ "import numpy as np\r\n",
+ "from pynq import allocate\r\n",
+ "from finn.util.data_packing import (\r\n",
+ " finnpy_to_packed_bytearray,\r\n",
+ " packed_bytearray_to_finnpy\r\n",
+ ")\r\n",
+ "from finn.core.datatype import DataType\r\n",
+ "\r\n",
+ "bitfile_path = \"resizer.bit\"\r\n",
+ "ol = Overlay(bitfile_path)\r\n",
+ "dma=ol.axi_dma_0\r\n",
+ "\r\n",
+ "# declare input/output types and shapes for the accelerator\r\n",
+ "# input FINN DataType\r\n",
+ "idt = DataType.BINARY\r\n",
+ "# normal, folded and packed input shapes\r\n",
+ "ishape_normal = (1, 784)\r\n",
+ "ishape_folded = (1, 49, 16)\r\n",
+ "ishape_packed = (1, 49, 2)\r\n",
+ "# output FINN DataType\r\n",
+ "odt = DataType.UINT32\r\n",
+ "# normal, folded and packed output shapes\r\n",
+ "oshape_normal = (1, 10)\r\n",
+ "oshape_folded = (1, 1, 10)\r\n",
+ "oshape_packed = (1, 1, 40)\r\n",
+ "\r\n",
+ "# load desired input .npy file\r\n",
+ "ibuf_normal = np.load(\"input.npy\")\r\n",
+ "# ensure that shape is as expected\r\n",
+ "assert ibuf_normal.shape == ishape_normal\r\n",
+ "# convert to folded form\r\n",
+ "ibuf_folded = ibuf_normal.reshape(ishape_folded)\r\n",
+ "\r\n",
+ "# pack the input buffer, reversing both SIMD dim and endianness\r\n",
+ "ibuf_packed = finnpy_to_packed_bytearray(\r\n",
+ " ibuf_folded, idt, reverse_endian=True, reverse_inner=True\r\n",
+ ")\r\n",
+ "# allocate a PYNQ buffer for the packed input buffer\r\n",
+ "ibuf_packed_device = allocate(shape=ishape_packed, dtype=np.uint8)\r\n",
+ "# copy the packed data into the PYNQ buffer\r\n",
+ "# TODO optimization: pack directly into the PYNQ buffer?\r\n",
+ "np.copyto(ibuf_packed_device, ibuf_packed)\r\n",
+ "\r\n",
+ "# allocate a PYNQ buffer for the returned packed output buffer\r\n",
+ "obuf_packed = allocate(shape=oshape_packed, dtype=np.uint8)\r\n",
+ "\r\n",
+ "# set up the DMA and wait until all transfers complete\r\n",
+ "dma.sendchannel.transfer(ibuf_packed_device)\r\n",
+ "dma.recvchannel.transfer(obuf_packed)\r\n",
+ "dma.sendchannel.wait()\r\n",
+ "dma.recvchannel.wait()\r\n",
+ "\r\n",
+ "# unpack the packed output buffer from accelerator\r\n",
+ "obuf_folded = packed_bytearray_to_finnpy(\r\n",
+ " obuf_packed, odt, oshape_folded, reverse_endian=True, reverse_inner=True\r\n",
+ ")\r\n",
+ "# convert to normal reshape and save\r\n",
+ "obuf_normal = obuf_folded.reshape(oshape_normal)\r\n",
+ "np.save(\"output.npy\", obuf_normal)\r\n"
]
}
],
"source": [
- "from finn.custom_op.xnorpopcount import xnorpopcountmatmul\n",
- "showSrc(xnorpopcountmatmul)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The function contains a description of the behaviour in Python and can thus calculate the result of the node."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This execution function and onnxruntime is used when `execute_onnx` from [onnx_exec](https://github.com/Xilinx/finn/blob/dev/src/finn/core/onnx_exec.py) is applied to the model. The model is then simulated node by node and the result is stored in a context dictionary, which contains the values of each tensor at the end of the execution. To get the result, only the output tensor has to be extracted.\n",
- "\n",
- "The procedure is shown below. We take the model after the Brevitas export and the basic netwok transformations and generate an input tensor to pass to the execution function. The input tensor is generated from the Brevitas example inputs."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "from pkgutil import get_data\n",
- "import onnx\n",
- "import onnx.numpy_helper as np_helper\n",
- "\n",
- "raw_i = get_data(\"finn\", \"data/onnx/mnist-conv/test_data_set_0/input_0.pb\")\n",
- "input_tensor = onnx.load_tensor_from_string(raw_i)\n",
- "input_dict = {\"global_in\": np_helper.to_array(input_tensor)}\n",
- "\n",
- "model_for_sim = ModelWrapper(build_dir+\"/tfc_w1_a1_after_brevitas_export.onnx\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "array([[ 3.3252678 , -2.5652065 , 9.215742 , -1.4251148 , 1.4251148 ,\n",
- " -3.3727715 , 0.28502294, -0.5700459 , 7.07807 , -1.2826033 ]],\n",
- " dtype=float32)"
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import finn.core.onnx_exec as oxe\n",
- "output_dict = oxe.execute_onnx(model_for_sim, input_dict)\n",
- "pysim_output = output_dict[list(output_dict.keys())[0]]\n",
- "pysim_output"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This result can now be compared with a theoretical calculated value for verification. This is not done in this notebook, but there are some tests in the FINN repository that demonstrate such a procedure. They can be found [here](https://github.com/Xilinx/finn/tree/dev/tests)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Simulation (npysim) using C++ <a id='npysim'></a>\n",
- "When dealing with HLS custom op nodes in FINN the simulation using Python is no longer sufficient. After the nodes have been converted to HLS layers, the simulation using C++ can be used. To do this, the input tensor is stored in an .npy file and C++ code is generated that reads the values from the .npy array, streams them to the corresponding finn-hlslib function and writes the result to a new .npy file. This in turn can be read in Python and processed in the FINN flow. For this example the model after the conversion to HLS layers is used."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "model_for_npysim = ModelWrapper(build_dir+\"/tfc_w1_a1_after_conv_to_hls.onnx\")"
+ "driver_dir = model.get_metadata_prop(\"pynq_driver_dir\")\n",
+ "! cat {driver_dir}/driver.py"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "To generate the code for this simulation and to generate the executable two transformations are used:\n",
- " * `CodeGen_npysim` which generates the C++ code for the corresponding hls layer\n",
- " * `Compile` which compiles the C++ code and stores the path to the executable"
+ "We'll now use the `DeployToPYNQ` transformation to create a deployment folder with the bitfile and driver file(s), and copy that to the PYNQ board. You can change the default IP address, username, password and target folder for the PYNQ below."
]
},
{
"cell_type": "code",
- "execution_count": 31,
+ "execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
- "from finn.transformation.fpgadataflow.codegen_npysim import CodeGen_npysim\n",
- "from finn.transformation.fpgadataflow.compile import Compile\n",
- "\n",
- "model_for_npysim = model_for_npysim.transform(CodeGen_npysim())\n",
- "model_for_npysim = model_for_npysim.transform(Compile())"
+ "from finn.transformation.fpgadataflow.make_deployment import DeployToPYNQ\n",
+ "ip = \"192.168.3.1\"\n",
+ "username = \"xilinx\"\n",
+ "password = \"xilinx\"\n",
+ "target_dir = \"/home/xilinx/finn_tfc_end2end_example\"\n",
+ "model = model.transform(DeployToPYNQ(ip, username, password, target_dir))\n",
+ "model.save(build_dir + \"/tfc_w1_a1_pynq_deploy.onnx\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "When we take a look at the model using netron, we can see that the transformations introduced new attributes."
+ "Let's verify that the remote access credentials is saved in the model metadata, and that the deployment folder has been successfully copied to the board:"
]
},
{
"cell_type": "code",
- "execution_count": 34,
+ "execution_count": 49,
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "Stopping http://0.0.0.0:8081\n",
- "Serving 'lfc_w1_a1_after_conv_to_hls.onnx' at http://0.0.0.0:8081\n"
- ]
- }
- ],
- "source": [
- "model_for_npysim.save(build_dir+\"/tfc_w1_a1_after_conv_to_hls.onnx\")\n",
- "netron.start(build_dir+\"/tfc_w1_a1_after_conv_to_hls.onnx\", port=8081, host=\"0.0.0.0\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 35,
- "metadata": {
- "scrolled": true
- },
"outputs": [
{
"data": {
- "text/html": [
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>\n"
- ],
"text/plain": [
- "<IPython.core.display.HTML object>"
+ "[key: \"vivado_stitch_proj\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_stitch_proj_n3me5eke\"\n",
+ ", key: \"vivado_stitch_vlnv\"\n",
+ "value: \"xilinx_finn:finn:finn_design:1.0\"\n",
+ ", key: \"wrapper_filename\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_stitch_proj_n3me5eke/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
+ ", key: \"vivado_pynq_proj\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_pynq_proj_hqlnpt5q\"\n",
+ ", key: \"vivado_pynq_bitfile\"\n",
+ "value: \"/tmp/finn_maltanar/vivado_pynq_proj_hqlnpt5q/resizer.bit\"\n",
+ ", key: \"pynq_driver_dir\"\n",
+ "value: \"/tmp/finn_maltanar/pynq_driver_yu_l_jao\"\n",
+ ", key: \"pynq_ip\"\n",
+ "value: \"192.168.3.1\"\n",
+ ", key: \"pynq_username\"\n",
+ "value: \"xilinx\"\n",
+ ", key: \"pynq_password\"\n",
+ "value: \"xilinx\"\n",
+ ", key: \"pynq_target_dir\"\n",
+ "value: \"/home/xilinx/finn_tfc_end2end_example\"\n",
+ ", key: \"pynq_deployment_dir\"\n",
+ "value: \"/tmp/finn_maltanar/pynq_deployment_1oyo7x66\"\n",
+ ", key: \"pynq_deploy_dir\"\n",
+ "value: \"/tmp/finn_maltanar/pynq_deployment_1oyo7x66\"\n",
+ ", key: \"exec_mode\"\n",
+ "value: \"remote_pynq\"\n",
+ "]"
]
},
+ "execution_count": 49,
"metadata": {},
- "output_type": "display_data"
+ "output_type": "execute_result"
}
],
"source": [
- "%%html\n",
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The following node attributes have been added:\n",
- "* `code_gen_dir_npysim` indicates the directory where the files for the simulation using C++ are stored \n",
- "* `executable_path` specifies the path to the executable\n",
- "\n",
- "We take now a closer look into the files that were generated:"
+ "model.model.metadata_props"
]
},
{
"cell_type": "code",
- "execution_count": 37,
+ "execution_count": 47,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "compile.sh execute_StreamingFCLayer_Batch.cpp\tnode_model params.h thresh.h\r\n"
+ "total 4\r\n",
+ "drwx------ 3 xilinx xilinx 4096 Feb 13 13:36 pynq_deployment_1oyo7x66\r\n"
]
}
],
"source": [
- "fc0 = model_for_npysim.graph.node[2]\n",
- "fc0w = StreamingFCLayer_Batch(fc0)\n",
- "code_gen_dir = fc0w.get_nodeattr(\"code_gen_dir_npysim\")\n",
- "!ls {code_gen_dir}"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Besides the .cpp file, the folder contains .h files with the weights and thresholds. The shell script contains the compile command and *node_model* is the executable generated by compilation. Comparing this with the `executable_path` node attribute, it can be seen that it specifies exactly the path to *node_model*.\n",
- "\n",
- "To simulate the model the *simulation mode*(`sim_mode`) must be set to \"npysim\". This is done using the transformation `SetSimMode`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [],
- "source": [
- "from finn.transformation.fpgadataflow.set_sim_mode import SetSimMode\n",
- "\n",
- "model_for_npysim = model_for_npysim.transform(SetSimMode(\"npysim\"))"
+ "! sshpass -p {password} ssh {username}@{ip} 'ls -l {target_dir}'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Now the model can be executed using `execute_onnx`. The function reads the `sim_mode` and writes the input into the correct directory in a .npy file. To be able to read this in C++, there is an additional .hpp file ([npy2apintstream.hpp](https://github.com/Xilinx/finn/blob/dev/src/finn/data/cpp/npy2apintstream.hpp)) in FINN, which uses [cnpy](https://github.com/rogersce/cnpy) to read .npy files and convert them into streams, or to read a stream and write it into an .npy. cnpy is a helper to read and write .npy and .npz formates in C++. "
+ "We only have two more steps to be able to remotely execute the deployed bitfile with some test data from the MNIST dataset. Let's load up some test data that comes bundled with FINN."
]
},
{
"cell_type": "code",
- "execution_count": 44,
+ "execution_count": 52,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
- "array([[ 3.3252678 , -2.5652065 , 9.215742 , -1.4251148 , 1.4251148 ,\n",
- " -3.3727715 , 0.28502294, -0.5700459 , 7.07807 , -1.2826033 ]],\n",
- " dtype=float32)"
+ "<matplotlib.image.AxesImage at 0x7fc62d83dbe0>"
]
},
- "execution_count": 44,
+ "execution_count": 52,
"metadata": {},
"output_type": "execute_result"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAARX0lEQVR4nO3dfYyVZXrH8d/FoDAw8iYRCaisG/5QqmUbgk1KyOKmxlUMbKJm/aPauAmarMmqTVqz/UOSaqJVa/pH3YStL9CsmiWoq0a7a82mWo1GNFQQW1CULGR4E5H3t+HqH/NgZ3We6549z3nOc9z7+0kmM3Ouec65OTM/zsv13Pdt7i4Af/xGNT0AAJ1B2IFMEHYgE4QdyARhBzIxupM3Zma89Z+ZUaPKH09OnTpV23VXvf6enp6wPjAw0PJ1183dbbjLK4XdzK6U9M+SeiT9q7vfV+X6cmU27O/mS6k/6ip/eKNHx38CqcCk6r29vaW1Q4cOhcem9PX1hfUDBw6U1lIt50mTJoX1zz77LKx3o5afxptZj6R/kfR9SRdLusHMLm7XwAC0V5XX7PMlfeTuW9z9uKSnJS1pz7AAtFuVsM+Q9Lsh328rLvs9ZrbMzNaa2doKtwWgotrfoHP3FZJWSLxBBzSpyiP7dknnDfl+ZnEZgC5UJezvSJptZt8yszMl/VDS8+0ZFoB2a/lpvLufNLPbJP1ag623x9z9g7aNLCPjx48P6wcPHmz5useMGRPWjx07FtZTbcFx48aF9ai9lmoppqSOj9prqT76vn37WhpTN6v0mt3dX5L0UpvGAqBGnC4LZIKwA5kg7EAmCDuQCcIOZIKwA5mwTq4um+vpsqled6qXffTo0bA+duzYlo9Nia676vWfffbZYb3qNNLofp06dWp47O7du8N6amrwyZMnw3qdyuaz88gOZIKwA5kg7EAmCDuQCcIOZIKwA5mg9fYNkGrNVfkd1nnddUtNDa6yem1q6m5qanCTS03TegMyR9iBTBB2IBOEHcgEYQcyQdiBTBB2IBP02TvgrLPOCuvRbqOSNHHixLB+4sSJ0lpqN9LUFNbPP/88rC9YsCCs33rrraW1VC/6jjvuCOtbt24N601OM20SfXYgc4QdyARhBzJB2IFMEHYgE4QdyARhBzJBn/0b4JFHHgnrUS871Wuuuox1b29vWI+ktk2+5JJLwvqmTZvC+vHjx0trZ5xxRnhsdO6ClP53HzlyJKzXqazPXmnLZjP7VNIBSQOSTrr7vCrXB6A+lcJeWOTue9pwPQBqxGt2IBNVw+6SfmNm75rZsuF+wMyWmdlaM1tb8bYAVFD1afwCd99uZudIesXM/sfdXxv6A+6+QtIKiTfogCZVemR39+3F512SnpU0vx2DAtB+LYfdzMab2Vmnv5Z0haQN7RoYgPaq8jR+mqRniz7taElPuvu/t2VUf2RSWzYvWrQorF922WVhPeqVHzx4MDw21W/u6+sL66nzNKI566m11x999NGWr1uS7rzzztLaW2+9FR5b93bSTWg57O6+RdKftnEsAGpE6w3IBGEHMkHYgUwQdiAThB3IBFNcu0Bqqubs2bPD+v79+0trEyZMCI+NpoFK6SmwVbZ8TrX9UlJLcO/du7e0tnTp0vDYdevWhfVUSzLV8qwTS0kDmSPsQCYIO5AJwg5kgrADmSDsQCYIO5CJdiw42TFRT7fOfnBK6thU/ZZbbgnrq1atCuszZ85s+bZTffZ77rknrK9evTqsn3nmmaW1K664Ijz2wQcfDOuprbCj2168eHF47LZt28L6nj3fvDVWeWQHMkHYgUwQdiAThB3IBGEHMkHYgUwQdiATHZ/Pnup3Rzo51naqOvd54cKFYf2iiy4qrY0bNy48dvTo+FSLNWvWhPUtW7aE9SpSyz3PmTMnrKfu90jq75T57AC6FmEHMkHYgUwQdiAThB3IBGEHMkHYgUx0vM8+alT5/y9V54XXqcpc+lOnTlW67eg+S9VPnjwZHjt+/PiwfujQobCe2o46+p2l5tJfffXVYf3pp58O61X67Kk17VP3a5Na7rOb2WNmtsvMNgy5bIqZvWJmm4vPk9s5WADtN5Kn8U9IuvIrl90l6VV3ny3p1eJ7AF0sGXZ3f03SV/fRWSJpZfH1SknxXjoAGtfqGnTT3L2/+HqHpGllP2hmyyQta/F2ALRJ5QUn3d2jDRvdfYWkFRIbOwJNarX1ttPMpktS8XlX+4YEoA6thv15STcVX98k6VftGQ6AuiT77Gb2lKTvSpoqaaekuyU9J+mXks6XtFXS9e5evhn2/19XbU/jq64bX7UeSfVkU3uoR/uvV9Xb2xvWjxw5EtZT5wBUOcfgwgsvDOsff/xxy9edGldqTfqUw4cPVzq+irI+e/I1u7vfUFL6XqURAegoTpcFMkHYgUwQdiAThB3IBGEHMsGWzYVUC3JgYCCsR3p6esJ61WWHozZRqsWUmsKakrr+aNvkqCZJixYtamlMp0W/0xMnToTHpqa4Vvl7aAqP7EAmCDuQCcIOZIKwA5kg7EAmCDuQCcIOZKKr+ux1budcdTnnKuq+7QMHDpTWUv3iVK87dXyqTx8tF51axvq6664L60ePHg3rY8eOLa2l+uyp31mTWzK3ikd2IBOEHcgEYQcyQdiBTBB2IBOEHcgEYQcy0fE+ezS3u5t75dGSyanllFPq3Fb50ksvDY+dM2dOWE8tJf3cc8+F9UjUB5ekhQsXhvUqW3inlqGOzl2Qqi/B3QQe2YFMEHYgE4QdyARhBzJB2IFMEHYgE4QdyETH++zRnPU6++ipufKped1RT3j06PhuXLp0aVhPHb9kyZKwPmbMmNLa3Llzw2MnTZoU1lO97Ndff73l42fPnh0em1qbPdXrXr9+fWnt8ssvD4+N7lOpO/voKclHdjN7zMx2mdmGIZctN7PtZrau+Liq3mECqGokT+OfkHTlMJc/7O5zi4+X2jssAO2WDLu7vyZpbwfGAqBGVd6gu83M3i+e5k8u+yEzW2Zma81sbYXbAlBRq2H/maRvS5orqV/SQ2U/6O4r3H2eu89r8bYAtEFLYXf3ne4+4O6nJP1c0vz2DgtAu7UUdjObPuTbH0jaUPazALqDpfqoZvaUpO9Kmippp6S7i+/nSnJJn0q6xd37kzdmFt5Yqt+cmvcdmTVrVli/5pprwvrixYtLa6l516l526m509H+61K8hnlfX194bErVed3R7/SLL74Ij504cWJYT9m8eXNpbdWqVeGxDz1U+spUUnf32d192JNKkifVuPsNw1z8aOURAegoTpcFMkHYgUwQdiAThB3IBGEHMpFsvbX1xsw8Wna5zimud999d1hfvnx5WN+zZ09pberUqa0M6UuprYf37o2nJkT1Cy64IDw21RZMbdmccuzYsdJaahpp6u8h1YqNpi2ntlx++eWXw/rNN98c1pvc0rms9cYjO5AJwg5kgrADmSDsQCYIO5AJwg5kgrADmeh4nz2qV9maODXVMtX3rLLt8q5du8L61q1bw/oDDzwQ1levXh3W580rXwTo4YcfDo9Nbdk8eXLpimOSpG3btoX16Hf6xBNPhMd+8sknYf3aa68N69HU46rTa1988cWwnpoyXSf67EDmCDuQCcIOZIKwA5kg7EAmCDuQCcIOZKKjffZRo0Z5ND/6+PHj4fHnnHNOaW337t3hsak+e2rudNQvTm0HvWnTprA+ZcqUsJ5atjha7vn8888Pj03NZ08t771v376wfuONN5bWXnjhhfDYlNQ6AtFy0YsWLQqPTa0xkLpfUst/14k+O5A5wg5kgrADmSDsQCYIO5AJwg5kgrADmeiq+exVpPqeK1euDOvXX399y9d/+PDh8Nhx48aF9dS2yKl5/gMDA6W11Lrvb775Zlh/8sknw/q6devC+htvvFFaS51fkOrhp37n0Xkb8+fPD499++23w/rjjz8e1lPrytep5T67mZ1nZr81s41m9oGZ/aS4fIqZvWJmm4vP8SoHABo1kqfxJyX9jbtfLOnPJf3YzC6WdJekV919tqRXi+8BdKlk2N29393fK74+IOlDSTMkLZF0+rnxSklL6xokgOriFz1fYWazJH1H0tuSprl7f1HaIWlayTHLJC1rfYgA2mHE78abWZ+kNZJud/f9Q2s++C7fsG++ufsKd5/n7uWrIgKo3YjCbmZnaDDov3D3Z4qLd5rZ9KI+XVK8xCqARiVbbzY4f3OlpL3ufvuQyx+Q9Jm732dmd0ma4u5/m7iu8MbOPffccCw7duwI65Fo+15JmjlzZli/9957S2szZswIj01tuZzaujjaLlqS7r///tLaxo0bw2NTU1xT2yKnpKYtR1JtwxMnToT1aOpx6u9+woQJYb3qlOk6lbXeRvKa/S8k/ZWk9WZ2uqn6U0n3Sfqlmf1I0lZJcaMaQKOSYXf3/5JU9l/k99o7HAB14XRZIBOEHcgEYQcyQdiBTBB2IBMdneLa09PjUV83NVU06n3u37+/tCZJfX19YT3VN416vlX6vVK655s6RyDqZad6+MeOHQvrVUW/79Ryzampwam/lyq/s5SqY6sTS0kDmSPsQCYIO5AJwg5kgrADmSDsQCYIO5CJrlpKOjWHOOqlp5YVrjove/r06aW1/v7+0tpI9Pb2hvXUls11XndqGetDhw6F9SpzylNGjYofq6rMKW/6/IQq6LMDmSPsQCYIO5AJwg5kgrADmSDsQCYIO5CJruqzA6iOPjuQOcIOZIKwA5kg7EAmCDuQCcIOZIKwA5lIht3MzjOz35rZRjP7wMx+Uly+3My2m9m64uOq+ocLoFXJk2rMbLqk6e7+npmdJeldSUs1uB/7QXd/cMQ3xkk1QO3KTqoZyf7s/ZL6i68PmNmHkma0d3gA6vYHvWY3s1mSviPp7eKi28zsfTN7zMwmlxyzzMzWmtnaSiMFUMmIz403sz5J/ynpXnd/xsymSdojySX9gwaf6t+cuA6exgM1K3saP6Kwm9kZkl6U9Gt3/6dh6rMkvejuf5K4HsIO1KzliTA2uDzoo5I+HBr04o27034gaUPVQQKoz0jejV8g6XVJ6yWdXpv3p5JukDRXg0/jP5V0S/FmXnRdPLIDNav0NL5dCDtQP+azA5kj7EAmCDuQCcIOZIKwA5kg7EAmCDuQCcIOZIKwA5kg7EAmCDuQCcIOZIKwA5kg7EAmkgtOttkeSVuHfD+1uKwbdevYunVcEmNrVTvHdkFZoaPz2b9242Zr3X1eYwMIdOvYunVcEmNrVafGxtN4IBOEHchE02Ff0fDtR7p1bN06LomxtaojY2v0NTuAzmn6kR1AhxB2IBONhN3MrjSz/zWzj8zsribGUMbMPjWz9cU21I3uT1fsobfLzDYMuWyKmb1iZpuLz8PusdfQ2LpiG+9gm/FG77umtz/v+Gt2M+uRtEnSX0raJukdSTe4+8aODqSEmX0qaZ67N34ChpktlHRQ0qrTW2uZ2T9K2uvu9xX/UU5297/rkrEt1x+4jXdNYyvbZvyv1eB9187tz1vRxCP7fEkfufsWdz8u6WlJSxoYR9dz99ck7f3KxUskrSy+XqnBP5aOKxlbV3D3fnd/r/j6gKTT24w3et8F4+qIJsI+Q9Lvhny/Td2137tL+o2ZvWtmy5oezDCmDdlma4ekaU0OZhjJbbw76SvbjHfNfdfK9udV8Qbd1y1w9z+T9H1JPy6ernYlH3wN1k29059J+rYG9wDsl/RQk4MpthlfI+l2d98/tNbkfTfMuDpyvzUR9u2Szhvy/czisq7g7tuLz7skPavBlx3dZOfpHXSLz7saHs+X3H2nuw+4+ylJP1eD912xzfgaSb9w92eKixu/74YbV6futybC/o6k2Wb2LTM7U9IPJT3fwDi+xszGF2+cyMzGS7pC3bcV9fOSbiq+vknSrxocy+/plm28y7YZV8P3XePbn7t7xz8kXaXBd+Q/lvT3TYyhZFwXSvrv4uODpscm6SkNPq07ocH3Nn4k6WxJr0raLOk/JE3porH9mwa39n5fg8Ga3tDYFmjwKfr7ktYVH1c1fd8F4+rI/cbpskAmeIMOyARhBzJB2IFMEHYgE4QdyARhBzJB2IFM/B+tIjCppYWKvAAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
}
],
"source": [
- "output_dict = oxe.execute_onnx(model_for_npysim, input_dict)\n",
- "npysim_output = output_dict[list(output_dict.keys())[0]]\n",
- "npysim_output"
+ "from pkgutil import get_data\n",
+ "import onnx.numpy_helper as nph\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "raw_i = get_data(\"finn\", \"data/onnx/mnist-conv/test_data_set_0/input_0.pb\")\n",
+ "x = nph.to_array(onnx.load_tensor_from_string(raw_i))\n",
+ "plt.imshow(x.reshape(28,28), cmap='gray')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "This result can now be compared with the previous simulation using Python."
+ "Recall that we partitioned our original network into a parent graph that contained the non-synthesizable nodes and a child graph that contained the bulk of the network, which we turned into a bitfile. We'll load up the parent graph, modify the `StreamingDataflowPartition` node so that it points to the deployed ONNX graph."
]
},
{
"cell_type": "code",
- "execution_count": 47,
+ "execution_count": 55,
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Results are the same\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
- "if (pysim_output == npysim_output).all():\n",
- " print(\"Results are the same\")\n",
- "else:\n",
- " raise Exception"
+ "parent_model = ModelWrapper(build_dir+\"/tfc_w1_a1_dataflow_parent.onnx\")\n",
+ "sdp_node = parent_model.graph.node[2]\n",
+ "remote_exec_model = build_dir + \"/tfc_w1_a1_pynq_deploy.onnx\"\n",
+ "getCustomOp(sdp_node).set_nodeattr(\"model\", remote_exec_model)\n",
+ "parent_model.save(build_dir+\"/tfc_w1_a1_dataflow_parent_with_remote_bitfile_exec.onnx\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Emulation (rtlsim) using Pyverilator <a id='rtlsim'></a>\n",
- "The emulation using [PyVerilator](https://github.com/maltanar/pyverilator) can be done after IP blocks are generated from the corresponding HLS layers. Pyverilator is a tool which makes it possible to simulate verilog files using verilator via a python interface.\n",
- "\n",
- "In the first step we load the model after IP blocks have been created from the layers and set `sim_mode` to \"rtlsim\"."
+ "Finally, we can call `execute_onnx` on the parent graph, which will internally call remote execution with the bitfile once the `StreamingDataflowPartition` node is reached, grab the results, then continue executing the last portion of the network. "
]
},
{
"cell_type": "code",
- "execution_count": 9,
+ "execution_count": 61,
"metadata": {},
"outputs": [],
"source": [
- "model_for_rtlsim = ModelWrapper(build_dir+\"/tfc_w1_a1_after_hls_ip_per_layer.onnx\")\n",
- "\n",
- "model_for_rtlsim = model_for_rtlsim.transform(SetSimMode(\"rtlsim\"))"
+ "import numpy as np\n",
+ "from finn.core.onnx_exec import execute_onnx\n",
+ "iname = parent_model.graph.input[0].name\n",
+ "oname = parent_model.graph.output[0].name\n",
+ "ishape = parent_model.get_tensor_shape(iname)\n",
+ "input_dict = {iname: x.reshape(ishape)}\n",
+ "ret = execute_onnx(parent_model, input_dict, True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Because the necessary files for the emulation are already generated in section [](), in the next step the execution of the model can be done directly."
+ "We'll pass the output of the network through a softmax function to interpret it as probabilities, and plot the per-class probabilities as a bar chart."
]
},
{
"cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "Stopping http://0.0.0.0:8081\n",
- "Serving 'lfc_w1_a1_after_hls_ip_per_layer.onnx' at http://0.0.0.0:8081\n"
- ]
- }
- ],
- "source": [
- "model_for_rtlsim.save(build_dir+\"/tfc_w1_a1_after_hls_ip_per_layer.onnx\")\n",
- "netron.start(build_dir+\"/tfc_w1_a1_after_hls_ip_per_layer.onnx\", port=8081, host=\"0.0.0.0\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
+ "execution_count": 62,
"metadata": {},
"outputs": [
{
"data": {
- "text/html": [
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>\n"
- ],
"text/plain": [
- "<IPython.core.display.HTML object>"
+ "<BarContainer object of 10 artists>"
]
},
+ "execution_count": 62,
"metadata": {},
+ "output_type": "execute_result"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAMoUlEQVR4nO3cf6jd913H8edryercD1sxV9AkLgEzNQyl5dJVC1pshbSV5A9FGqjoKMs/y6yuKJlKHfWfzcn8gXUa5xzO2azWIcFGI7iKILbkdp11SYxcstrcrNK7rtYfQ7Pg2z/uiZzd3ptzkp57T/u+zwcEzvf7/XC+75ObPDn3e36kqpAkvfa9btoDSJImw6BLUhMGXZKaMOiS1IRBl6QmNk/rxFu2bKkdO3ZM6/SS9Jr05JNPfqmqZlY6NrWg79ixg7m5uWmdXpJek5L8y2rHvOQiSU0YdElqwqBLUhMjg57kY0meT/L5VY4nyW8mmU/ydJIbJj+mJGmUcZ6hfxzYc5njtwO7Bn8OAB955WNJkq7UyKBX1d8CX77Mkn3AH9aSx4HrknzLpAaUJI1nEtfQtwLnhrYXBvteJsmBJHNJ5hYXFydwaknSJev6omhVHa6q2aqanZlZ8X3xkqSrNImgnwe2D21vG+yTJK2jSXxS9ChwMMkR4B3AS1X13ATuV8vsOPTomp/jmQ/cuebnkLQ2RgY9yUPALcCWJAvALwGvB6iq3wGOAXcA88BXgHeu1bCSpNWNDHpV7R9xvIB3T2wiSdJV8ZOiktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1MRYQU+yJ8mZJPNJDq1w/NuSPJbkqSRPJ7lj8qNKki5nZNCTbAIeBG4HdgP7k+xetuwXgYer6nrgLuC3Jz2oJOnyxnmGfiMwX1Vnq+oCcATYt2xNAd8wuH0t8MXJjShJGsc4Qd8KnBvaXhjsG/Z+4O4kC8Ax4D0r3VGSA0nmkswtLi5exbiSpNVM6kXR/cDHq2obcAfwiSQvu++qOlxVs1U1OzMzM6FTS5JgvKCfB7YPbW8b7Bt2D/AwQFX9PfAGYMskBpQkjWecoJ8AdiXZmeQall70PLpszbPArQBJvouloHtNRZLW0cigV9VF4CBwHDjN0rtZTiZ5IMnewbL7gHcl+QfgIeAnq6rWamhJ0sttHmdRVR1j6cXO4X33D90+Bdw82dEkSVfCT4pKUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSE2MFPcmeJGeSzCc5tMqaH0tyKsnJJH882TElSaNsHrUgySbgQeCHgAXgRJKjVXVqaM0u4H3AzVX1YpJvXquBJUkrG+cZ+o3AfFWdraoLwBFg37I17wIerKoXAarq+cmOKUkaZZygbwXODW0vDPYNexvwtiR/l+TxJHtWuqMkB5LMJZlbXFy8uoklSSua1Iuim4FdwC3AfuD3kly3fFFVHa6q2aqanZmZmdCpJUkwXtDPA9uHtrcN9g1bAI5W1Ver6gvAP7MUeEnSOhkn6CeAXUl2JrkGuAs4umzNn7H07JwkW1i6BHN2gnNKkkYYGfSquggcBI4Dp4GHq+pkkgeS7B0sOw68kOQU8Bjws1X1wloNLUl6uZFvWwSoqmPAsWX77h+6XcB7B38kSVPgJ0UlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpibGCnmRPkjNJ5pMcusy6H0lSSWYnN6IkaRwjg55kE/AgcDuwG9ifZPcK694C3As8MekhJUmjjfMM/UZgvqrOVtUF4Aiwb4V1vwx8EPjvCc4nSRrTOEHfCpwb2l4Y7Pt/SW4AtlfVo5e7oyQHkswlmVtcXLziYSVJq3vFL4omeR3wYeC+UWur6nBVzVbV7MzMzCs9tSRpyDhBPw9sH9reNth3yVuAtwN/k+QZ4CbgqC+MStL6GifoJ4BdSXYmuQa4Czh66WBVvVRVW6pqR1XtAB4H9lbV3JpMLEla0cigV9VF4CBwHDgNPFxVJ5M8kGTvWg8oSRrP5nEWVdUx4NiyffevsvaWVz6WJOlK+UlRSWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJamKsoCfZk+RMkvkkh1Y4/t4kp5I8neSvk7x18qNKki5nZNCTbAIeBG4HdgP7k+xetuwpYLaqvht4BPiVSQ8qSbq8cZ6h3wjMV9XZqroAHAH2DS+oqseq6iuDzceBbZMdU5I0yjhB3wqcG9peGOxbzT3AX6x0IMmBJHNJ5hYXF8efUpI00kRfFE1yNzALfGil41V1uKpmq2p2ZmZmkqeWpA1v8xhrzgPbh7a3DfZ9jSS3Ab8A/EBV/c9kxpMkjWucZ+gngF1Jdia5BrgLODq8IMn1wO8Ce6vq+cmPKUkaZWTQq+oicBA4DpwGHq6qk0keSLJ3sOxDwJuBP0nyuSRHV7k7SdIaGeeSC1V1DDi2bN/9Q7dvm/BckqQr5CdFJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqYmxgp5kT5IzSeaTHFrh+Ncl+dTg+BNJdkx6UEnS5Y0MepJNwIPA7cBuYH+S3cuW3QO8WFXfDvwa8MFJDypJurzNY6y5EZivqrMASY4A+4BTQ2v2Ae8f3H4E+K0kqaqa4Kyaoh2HHl3zczzzgTvX/ByvNWv99+7feS/jBH0rcG5oewF4x2prqupikpeAbwK+NLwoyQHgwGDzP5OcuZqhr9KW5fNsEFf0uDPF360mfG5/3mOY5s97wjbSz/utqx0YJ+gTU1WHgcPrec5LksxV1ew0zj1NPu6Nxce9sY3zouh5YPvQ9rbBvhXXJNkMXAu8MIkBJUnjGSfoJ4BdSXYmuQa4Czi6bM1R4CcGt38U+IzXzyVpfY285DK4Jn4QOA5sAj5WVSeTPADMVdVR4PeBTySZB77MUvRfbaZyqedVwMe9sfi4N7D4RFqSevCTopLUhEGXpCbaB33U1xZ0lGR7kseSnEpyMsm9055pPSXZlOSpJH8+7VnWU5LrkjyS5J+SnE7yvdOeaT0k+ZnBv/PPJ3koyRumPdO0tA76mF9b0NFF4L6q2g3cBLx7gzzuS+4FTk97iCn4DeAvq+o7ge9hA/wdJNkK/BQwW1VvZ+mNG6/GN2Wsi9ZBZ+hrC6rqAnDpawtaq6rnquqzg9v/wdJ/7K3TnWp9JNkG3Al8dNqzrKck1wLfz9I7zqiqC1X1b9Odat1sBr5+8BmYNwJfnPI8U9M96Ct9bcGGCNslg2++vB54YrqTrJtfB34O+N9pD7LOdgKLwB8MLjd9NMmbpj3UWquq88CvAs8CzwEvVdVfTXeq6eke9A0tyZuBPwV+uqr+fdrzrLUkPww8X1VPTnuWKdgM3AB8pKquB/4LaP+aUZJvZOm37p3AtwJvSnL3dKeanu5BH+drC1pK8nqWYv7Jqvr0tOdZJzcDe5M8w9LltR9M8kfTHWndLAALVXXpN7FHWAp8d7cBX6iqxar6KvBp4PumPNPUdA/6OF9b0E6SsHQt9XRVfXja86yXqnpfVW2rqh0s/aw/U1Ub4tlaVf0rcC7Jdwx23crXfsV1V88CNyV54+Df/a1sgBeDV7Ou37a43lb72oIpj7UebgZ+HPjHJJ8b7Pv5qjo2xZm09t4DfHLw5OUs8M4pz7PmquqJJI8An2Xp3V1PsYG/BsCP/ktSE90vuUjShmHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUxP8BwjHuoBhu1y0AAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
"output_type": "display_data"
}
],
"source": [
- "%%html\n",
- "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>"
+ "def softmax(x):\n",
+ " \"\"\"Compute softmax values for each sets of scores in x.\"\"\"\n",
+ " e_x = np.exp(x - np.max(x))\n",
+ " return e_x / e_x.sum()\n",
+ "\n",
+ "logits = ret[oname].flatten()\n",
+ "prob = softmax(logits)\n",
+ "\n",
+ "plt.bar(np.arange(10), prob)"
]
},
{
- "cell_type": "code",
- "execution_count": 12,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "MultiThreshold\n",
- "StreamingFCLayer_Batch\n"
- ]
- },
- {
- "ename": "Exception",
- "evalue": "Found no verilog files for this node,\n did you run the codegen_ipgen transformation?",
- "output_type": "error",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mException\u001b[0m Traceback (most recent call last)",
- "\u001b[0;32m<ipython-input-12-96db733648ff>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0moutput_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moxe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecute_onnx\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_for_rtlsim\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_dict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0mrtlsim_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moutput_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput_dict\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mrtlsim_output\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m/workspace/finn/src/finn/core/onnx_exec.py\u001b[0m in \u001b[0;36mexecute_onnx\u001b[0;34m(model, input_dict, return_full_exec_context)\u001b[0m\n\u001b[1;32m 118\u001b[0m \u001b[0;31m# topologically sorted\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 119\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mnode\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mgraph\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 120\u001b[0;31m \u001b[0mexecute_node\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexecution_context\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgraph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 121\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mreturn_full_exec_context\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 122\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mexecution_context\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m/workspace/finn/src/finn/core/onnx_exec.py\u001b[0m in \u001b[0;36mexecute_node\u001b[0;34m(node, context, graph)\u001b[0m\n\u001b[1;32m 41\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdomain\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"finn\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 42\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 43\u001b[0;31m \u001b[0mex_cu_node\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecute_custom_node\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontext\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgraph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 44\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 45\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m/workspace/finn/src/finn/core/execute_custom_node.py\u001b[0m in \u001b[0;36mexecute_custom_node\u001b[0;34m(node, context, graph)\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0minst\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mregistry\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcustom_op\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mop_type\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mop_type\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0minst\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecute_node\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcontext\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgraph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 13\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0;31m# exception if op_type is not supported\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m/workspace/finn/src/finn/custom_op/fpgadataflow/streamingfclayer_batch.py\u001b[0m in \u001b[0;36mexecute_node\u001b[0;34m(self, context, graph)\u001b[0m\n\u001b[1;32m 496\u001b[0m raise Exception(\n\u001b[1;32m 497\u001b[0m \"\"\"Found no verilog files for this node,\n\u001b[0;32m--> 498\u001b[0;31m did you run the codegen_ipgen transformation?\"\"\"\n\u001b[0m\u001b[1;32m 499\u001b[0m )\n\u001b[1;32m 500\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;31mException\u001b[0m: Found no verilog files for this node,\n did you run the codegen_ipgen transformation?"
- ]
- }
- ],
"source": [
- "output_dict = oxe.execute_onnx(model_for_rtlsim, input_dict)\n",
- "rtlsim_output = output_dict[list(output_dict.keys())[0]]\n",
- "rtlsim_output"
+ "We see that the network correctly predicts this as a digit 2 with high probability. This concludes our tutorial on how to take a simple fully-connected BNN all the way down to hardware with FINN, and execute it remotely on a PYNQ board."
]
},
{