"/workspace/brevitas/brevitas_examples/bnn_pynq/models/TFC.py:80: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.\n",
"Downloading: \"https://github.com/Xilinx/brevitas/releases/download/bnn_pynq-r0/tfc_1w1a-ff8140dc.pth\" to /home/maltanar/.cache/torch/checkpoints/tfc_1w1a-ff8140dc.pth\n",
"/workspace/brevitas/brevitas_examples/bnn_pynq/models/FC.py:83: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.\n",
" x = 2.0 * x - torch.tensor([1.0], device=x.device)\n"
]
}
...
...
@@ -132,7 +134,7 @@
" "
],
"text/plain": [
"<IPython.lib.display.IFrame at 0x7f98457e4048>"
"<IPython.lib.display.IFrame at 0x7f1b8e6db128>"
]
},
"execution_count": 3,
...
...
@@ -293,7 +295,7 @@
" "
],
"text/plain": [
"<IPython.lib.display.IFrame at 0x7f97d7ff7cf8>"
"<IPython.lib.display.IFrame at 0x7f1add273cf8>"
]
},
"execution_count": 6,
...
...
@@ -402,7 +404,7 @@
" "
],
"text/plain": [
"<IPython.lib.display.IFrame at 0x7f97d46a65f8>"
"<IPython.lib.display.IFrame at 0x7f1adc25df60>"
]
},
"execution_count": 8,
...
...
@@ -456,7 +458,7 @@
" "
],
"text/plain": [
"<IPython.lib.display.IFrame at 0x7f97d46ad240>"
"<IPython.lib.display.IFrame at 0x7f1adc2548d0>"
]
},
"execution_count": 9,
...
...
@@ -532,7 +534,7 @@
" "
],
"text/plain": [
"<IPython.lib.display.IFrame at 0x7f97d46a6668>"
"<IPython.lib.display.IFrame at 0x7f1adc254630>"
]
},
"execution_count": 10,
...
...
@@ -592,7 +594,7 @@
" "
],
"text/plain": [
"<IPython.lib.display.IFrame at 0x7f9872793940>"
"<IPython.lib.display.IFrame at 0x7f1add27eba8>"
]
},
"execution_count": 11,
...
...
@@ -627,7 +629,7 @@
"text": [
"\n",
"Stopping http://0.0.0.0:8081\n",
"Serving '/tmp/finn_dev_maltanar/dataflow_partition0_cuza0or8/df_model.onnx' at http://0.0.0.0:8081\n"
"Serving '/tmp/finn_dev_maltanar/dataflow_partition0_8y5bzo4x/df_model.onnx' at http://0.0.0.0:8081\n"
In this notebook, we will show how to take a simple, binarized, fully-connected network trained on the MNIST data set and take it all the way down to a customized bitfile running on a PYNQ board.
This notebook is quite lengthy, and some of the cells (involving Vivado synthesis) may take up to an hour to finish running. To let you save and resume your progress, we will save the intermediate ONNX models that are generated in the various steps to disk, so that you can jump back directly to where you left off.
%% Cell type:markdown id: tags:
## Overview
The FINN compiler comes with many *transformations* that modify the ONNX representation of the network according to certain patterns. This notebook will demonstrate a *possible* sequence of such transformations to take a particular trained network all the way down to hardware, as shown in the figure below.
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
The white fields show the state of the network representation in the respective step. The colored fields represent the transformations that are applied to the network to achieve a certain result. The diagram is divided into 5 sections represented by a different color, each of it includes several flow steps. The flow starts in top left corner with Brevitas export (green section), followed by the preparation of the network (blue section) for the Vivado HLS synthesis and Vivado IPI stitching (orange section), and finally building a PYNQ overlay bitfile and testing it on a PYNQ board (yellow section).
There is an additional section for functional verification (red section) on the right side of the diagram, which we will not cover in this notebook. For details please take a look in the verification notebook which you can find [here](tfc_end2end_verification.ipynb)
This Jupyter notebook is organized based on the sections described above. We will use the following helper functions, `showSrc` to show source code of FINN library calls and `showInNetron` to show the ONNX model at the current transformation step. The Netron displays are interactive, but they only work when running the notebook actively and not on GitHub (i.e. if you are viewing this on GitHub you'll only see blank squares).
4.[PYNQ hardware generation and deployment](#hw_test)
%% Cell type:markdown id: tags:
## 1. Brevitas export <a id='brev_exp'></a>
FINN expects an ONNX model as input. This can be a model trained with [Brevitas](https://github.com/Xilinx/brevitas). Brevitas is a PyTorch library for quantization-aware training and the FINN Docker image comes with several [example Brevitas networks](https://github.com/Xilinx/brevitas/tree/master/brevitas_examples/bnn_pynq). To show the FINN end-to-end flow, we'll use the TFC-w1a1 model as example network.
First a few things have to be imported. Then the model can be loaded with the pretrained weights.
/workspace/brevitas/brevitas_examples/bnn_pynq/models/TFC.py:80: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
Downloading: "https://github.com/Xilinx/brevitas/releases/download/bnn_pynq-r0/tfc_1w1a-ff8140dc.pth" to /home/maltanar/.cache/torch/checkpoints/tfc_1w1a-ff8140dc.pth
/workspace/brevitas/brevitas_examples/bnn_pynq/models/FC.py:83: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
x = 2.0 * x - torch.tensor([1.0], device=x.device)
%% Cell type:markdown id: tags:
The model was now exported, loaded with the pretrained weights and saved under the name "lfc_w1_a1.onnx".
To visualize the exported model, Netron can be used. Netron is a visualizer for neural networks and allows interactive investigation of network properties. For example, you can click on the individual nodes and view the properties.
%% Cell type:code id: tags:
``` python
showInNetron(build_dir+"/tfc_w1_a1.onnx")
```
%% Output
Serving '/workspace/finn/tfc_w1_a1.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7f98457e4048>
<IPython.lib.display.IFrame at 0x7f1b8e6db128>
%% Cell type:markdown id: tags:
Now that we have the model in .onnx format, we can work with it using FINN. For that FINN `ModelWrapper` is used. It is a wrapper around the ONNX model which provides several helper functions to make it easier to work with the model.
%% Cell type:code id: tags:
``` python
fromfinn.core.modelwrapperimportModelWrapper
model=ModelWrapper(build_dir+"/tfc_w1_a1.onnx")
```
%% Cell type:markdown id: tags:
Now the model is prepared and could be simulated using Python. How this works is described in the Jupyter notebook about verification and can be found [here](tfc_end2end_verification.ipynb#simpy).
The model can now also be processed in different ways. The principle of FINN are analysis and transformation passes, which can be applied to the model. An analysis pass extracts specific information about the model and returns it to the user in the form of a dictionary. A transformation pass changes the model and returns the changed model back to the FINN flow.
Since the goal in this notebook is to process the model to such an extent that a bitstream can be generated from it, the focus is on the transformations that are necessary for this. In the next section these are discussed in more detail.
*[Creating a Dataflow Partition](#dataflow_partition)
*[Folding and Datawidth Converter, FIFO and TLastMarker Insertion](#folding)
In this section, we will put the network through a series of transformations that puts it in a form that can be stitched together to form a FINN-style dataflow architecture, yielding a high-performance, high-efficiency FPGA accelerator.
We start with a quick recap of FINN-style dataflow architectures. The key idea in such architectures is to parallelize across layers as well as within layers by dedicating a proportionate amount of compute resources to each layer, as illustrated in the figure below taken from the [FINN-R paper](https://arxiv.org/pdf/1809.04570.pdf):
In practice, the compute arrays are instantiated by function calls to optimized Vivado HLS building blocks from the [finn-hlslib](https://github.com/Xilinx/finn-hlslib) library. As these function calls can only handle certain patterns/cases, we need to transform the network into an appropriate form so that we can replace network layers with these function calls, which is the goal of the network preparation process.
This section deals with some basic transformations, which are applied to the model like a kind of "tidy-up" to make it easier to be processed. They do not appear in the diagram above, but they are applied in many steps in the FINN flow to postprocess the model after a transformation and/or to prepare it for the next transformation.
%% Cell type:markdown id: tags:
These transformations are:
* GiveUniqueNodeNames
* GiveReadableTensorNames
* InferShapes
* InferDataTypes
* FoldConstants
%% Cell type:markdown id: tags:
In the first two transformations (`GiveUniqueNodeNames`, `GiveReadableTensorNames`) the nodes in the graph are first given unique (by enumeration) names, then the tensors are given human-readable names (based on the node names). The following two transformations (`InferShapes`, `InferDataTypes`) derive the shapes and data types of the tensors from the model properties and set them in the `ValueInfo` of the model. These transformations can almost always be applied without negative effects and do not affect the structure of the graph, ensuring that all the information needed is available.
The last listed transformation is `FoldConstants`, which performs constant folding. It identifies a node with constant inputs and determines its output. The result is then set as constant-only inputs for the following node and the old node is removed. Although this transformation changes the structure of the model, it is a transformation that is usually always desired and can be applied to any model.
%% Cell type:markdown id: tags:
These transformations can be imported and applied as follows.
The result of these transformations can be viewed with netron after the model has been saved again. By clicking on the individual nodes, it can now be seen, for example, that each node has been given a name. Also the whole upper area could be folded, so that now the first node is "Reshape".
%% Cell type:code id: tags:
``` python
showInNetron(build_dir+"/tfc_w1_a1_tidy.onnx")
```
%% Output
Stopping http://0.0.0.0:8081
Serving '/workspace/finn/tfc_w1_a1_tidy.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7f97d7ff7cf8>
<IPython.lib.display.IFrame at 0x7f1add273cf8>
%% Cell type:markdown id: tags:
### Streamlining <a id='streamline'></a>
Streamlining is a transformation containing several sub-transformations. The goal of streamlining is to eliminate floating point operations by moving them around, then collapsing them into one operation and in the last step transform them into multi-thresholding nodes. For more information on the theoretical background of this, see [this paper](https://arxiv.org/pdf/1709.04060).
Let's have a look at which sub-transformations `Streamline` consists of:
"""Apply the streamlining transform, see arXiv:1709.04060."""
def apply(self, model):
streamline_transformations = [
ConvertSubToAdd(),
ConvertDivToMul(),
BatchNormToAffine(),
ConvertSignToThres(),
AbsorbSignBiasIntoMultiThreshold(),
MoveAddPastMul(),
MoveScalarAddPastMatMul(),
MoveAddPastConv(),
MoveScalarMulPastMatMul(),
MoveScalarMulPastConv(),
MoveAddPastMul(),
CollapseRepeatedAdd(),
CollapseRepeatedMul(),
AbsorbAddIntoMultiThreshold(),
FactorOutMulSignMagnitude(),
AbsorbMulIntoMultiThreshold(),
Absorb1BitMulIntoMatMul(),
Absorb1BitMulIntoConv(),
RoundAndClipThresholds(),
]
for trn in streamline_transformations:
model = model.transform(trn)
model = model.transform(RemoveIdentityOps())
model = model.transform(GiveUniqueNodeNames())
model = model.transform(GiveReadableTensorNames())
model = model.transform(InferDataTypes())
return (model, False)
%% Cell type:markdown id: tags:
As can be seen, several transformations are involved in the streamlining transformation. There are move and collapse transformations. In the last step the operations are transformed into multithresholds. The involved transformations can be viewed in detail [here](https://github.com/Xilinx/finn/tree/master/src/finn/transformation/streamline). After each transformation, three of the tidy-up transformations (`GiveUniqueNodeNames`, `GiveReadableTensorNames` and `InferDataTypes`) are applied to the model.
Serving '/workspace/finn/tfc_w1_a1_streamlined.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7f97d46a65f8>
<IPython.lib.display.IFrame at 0x7f1adc25df60>
%% Cell type:markdown id: tags:
You can see that the network has become simplified considerably compared to the previous step -- a lot of nodes have disappeared between the `MatMul` layers, and the `Sign` nodes have been replaced with `MultiThreshold` nodes instead.
**The current implementation of streamlining is highly network-specific and may not work for your network if its topology is very different than the example network here. We hope to rectify this in future releases.**
Our example network is a quantized network with 1-bit bipolar (-1, +1 values) precision, and we want FINN to implement them as XNOR-popcount operations [as described in the original FINN paper](https://arxiv.org/pdf/1612.07119). For this reason, after streamlining, the resulting bipolar matrix multiplications are converted into xnorpopcount operations. This transformation produces operations that are again collapsed and converted into thresholds. This procedure is shown below.
Serving '/workspace/finn/tfc_w1a1_ready_for_hls_conversion.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7f97d46ad240>
<IPython.lib.display.IFrame at 0x7f1adc2548d0>
%% Cell type:markdown id: tags:
Observe the pairs of `XnorPopcountmatMul` and `MultiThreshold` layers following each other -- this is the particular pattern that the next step will be looking for in order to convert them to HLS layers.
%% Cell type:markdown id: tags:
### Conversion to HLS layers <a id='hls_layers'></a>
Converts the nodes to HLS layers that correspond to the functions in [finn-hls library](https://finn-hlslib.readthedocs.io/en/latest/). In our case this transformation converts pairs of binary XnorPopcountMatMul layers to StreamingFCLayer_Batch layers. Any immediately following MultiThreshold layers will also be absorbed into the MVTU.
Below is the code for the transformation and the network is visualized using netron to create the new structure with `StreamingFCLayer_Batch` nodes, which will correspond to a function call from the [finn-hlslib](https://finn-hlslib.readthedocs.io/en/latest/library/fclayer.html#_CPPv4I_j_j_j_j000_i_i000E22StreamingFCLayer_BatchvRN3hls6streamI7ap_uintI9InStreamWEEERN3hls6streamI7ap_uintI10OutStreamWEEERK2TWRK2TAKjRK1R) library.
%% Cell type:markdown id: tags:
**Note:** The transformation `to_hls.InferBinaryStreamingFCLayer` gets the string "decoupled" as argument, this indicates the `mem_mode` for the weights. In FINN there are different options to set the way the weights are stored and accessed. For details please have a look on the [FINN readthedocs website](https://finn.readthedocs.io/) under Internals.
Serving '/workspace/finn/tfc_w1_a1_hls_layers.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7f97d46a6668>
<IPython.lib.display.IFrame at 0x7f1adc254630>
%% Cell type:markdown id: tags:
Each StreamingFCLayer_Batch node has two attributes that specify the degree of folding, PE and SIMD. In all nodes the values for these attributes are set as default to 1, which would correspond to a maximum folding (time multiplexing) and thus minimum performance. We will shortly cover how these can be adjusted, but first we want to separate the HLS layers from the non-HLS layers in this network.
%% Cell type:markdown id: tags:
### Creating a Dataflow Partition <a id='dataflow_partition'></a>
In the graph above, you can see that there is a mixture of FINN HLS layers (StreamingFCLayer_Batch) with regular ONNX layers (Reshape, Mul, Add). To create a bitstream, FINN needs a model with only HLS layers. In order to achieve this, we will use the `CreateDataflowPartition` transformation to create a "dataflow partition" in this graph, separating out the HLS layers into another model, and replacing them with a placeholder layer called StreamingDataflowPartition:
Serving '/workspace/finn/tfc_w1_a1_dataflow_parent.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7f9872793940>
<IPython.lib.display.IFrame at 0x7f1add27eba8>
%% Cell type:markdown id: tags:
We can see that the StreamingFCLayer instances have all been replaced with a single `StreamingDataflowPartition`, which has an attribute `model` that points to the extracted, HLS dataflow-only graph:
Serving '/tmp/finn_dev_maltanar/dataflow_partition0_cuza0or8/df_model.onnx' at http://0.0.0.0:8081
Serving '/tmp/finn_dev_maltanar/dataflow_partition0_8y5bzo4x/df_model.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7f987278b9e8>
<IPython.lib.display.IFrame at 0x7f1b7af8d240>
%% Cell type:markdown id: tags:
We can see all the extracted `StreamingFCLayer` instances have been moved to the child (dataflow) model. We will load the child model with `ModelWrapper` and continue working on it.
%% Cell type:code id: tags:
``` python
model=ModelWrapper(dataflow_model_filename)
```
%% Cell type:markdown id: tags:
### Folding: Adjusting the Parallelism <a id='folding'></a>
*Folding* in FINN describes how much a layer is time-multiplexed in terms of execution resources. There are several *folding factors* for each layer, controlled by the PE (parallelization over outputs) and SIMD (parallelization over inputs) parameters as described by the original [FINN paper](https://arxiv.org/pdf/1612.07119). The higher the PE and SIMD values are set, the faster the generated accelerator will run, and the more FPGA resources it will consume.
Since the folding parameters are node attributes, they can be easily accessed and changed using a helper function of the `ModelWrapper`. But first we take a closer look at one of the nodes that implement a StreamingFCLayer_Batch operation. This is where the Netron visualization helps us, in the above diagram we can see that the first four nodes are StreamingFCLayer_Batch. So as an example we extract the first node.
%% Cell type:markdown id: tags:
We can use the higher-level [HLSCustomOp](https://github.com/Xilinx/finn/blob/master/src/finn/custom_op/fpgadataflow/__init__.py) wrappers for this node. These wrappers provide easy access to specific properties of these nodes, such as the folding factors (PE and SIMD). Let's have a look at which node attributes are defined by the CustomOp wrapper, and adjust the SIMD and PE attributes.
%% Cell type:code id: tags:
``` python
fc0=model.graph.node[0]
fc0w=getCustomOp(fc0)
print("CustomOp wrapper is of class "+fc0w.__class__.__name__)
fc0w.get_nodeattr_types()
```
%% Output
CustomOp wrapper is of class StreamingFCLayer_Batch
{'PE': ('i', True, 0),
'SIMD': ('i', True, 0),
'MW': ('i', True, 0),
'MH': ('i', True, 0),
'resType': ('s', True, ''),
'ActVal': ('i', False, 0),
'inputDataType': ('s', True, ''),
'weightDataType': ('s', True, ''),
'outputDataType': ('s', True, ''),
'accDataType': ('s', False, 'INT32'),
'binaryXnorMode': ('i', False, 0),
'noActivation': ('i', False, 0),
'numInputVectors': ('ints', False, [1]),
'mem_mode': ('s', False, 'const'),
'ram_style': ('s', False, 'auto'),
'backend': ('s', True, 'fpgadataflow'),
'code_gen_dir_cppsim': ('s', False, ''),
'code_gen_dir_ipgen': ('s', False, ''),
'executable_path': ('s', False, ''),
'ipgen_path': ('s', False, ''),
'ip_path': ('s', False, ''),
'ip_vlnv': ('s', False, ''),
'exec_mode': ('s', False, ''),
'cycles_rtlsim': ('i', False, 0),
'cycles_estimate': ('i', False, 0),
'rtlsim_trace': ('s', False, ''),
'res_estimate': ('s', False, ''),
'res_hls': ('s', False, ''),
'res_synth': ('s', False, ''),
'rtlsim_so': ('s', False, ''),
'partition_id': ('i', False, 0),
'inFIFODepth': ('i', False, 2),
'outFIFODepth': ('i', False, 2)}
%% Cell type:markdown id: tags:
We can see that the PE and SIMD are listed as node attributes, as well as the depths of the FIFOs that will be inserted between consecutive layers, and all can be adjusted using `set_nodeattr` subject to certain constraints.
**In this notebook we are setting the folding factors and FIFO depths manually, but in a future version we will support determining the folding factors given an FPGA resource budget according to the analytical model from the [FINN-R paper](https://arxiv.org/pdf/1809.04570).**
We are setting PE and SIMD so that each layer has a total folding of 16.
%% Cell type:markdown id: tags:
Besides PE and SIMD three other node attributes are set. `ram_style` specifies how the weights are to be stored (BRAM, LUTRAM, and so on). It can be selected explicitly or with the option `auto` you can let Vivado decide.
`inFIFODepth` and `outFIFODepth` specifies the FIFO depths that is needed by the node from the surrounding FIFOs. These attributes are used in the transformation 'InsertFIFO' to insert the appropriate FIFOs between the nodes, which will be automatically called as part of the hardware build process.
In previous versions of FINN we had to call transformations to insert data width converters, FIFOs and `TLastMarker` manually at this step. This is no longer needed, as all this is taken care of by the `ZynqBuild` or `VitisBuild` transformations.
Serving '/workspace/finn/tfc_w1_a1_set_folding_factors.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7f97d464ca20>
<IPython.lib.display.IFrame at 0x7f1adc266ba8>
%% Cell type:markdown id: tags:
This completes the network preparation and the network can be passed on to the next block *Vivado HLS and IPI*, which is described below.
%% Cell type:markdown id: tags:
## 3. Hardware Build <a id='vivado'></a>
We're finally ready to start generating hardware from our network. Depending on whether you want to target a Zynq or Alveo platform, FINN offers two transformations to build the accelerator, integrate into an appropriate shell and build a bitfile. These are `ZynqBuild` and `VitisBuild` for Zynq and Alveo, respectively. In this notebook we'll demonstrate the `ZynqBuild` as these boards are more common and it's much faster to complete bitfile generation for the smaller FPGAs found on them.
As we will be dealing with FPGA synthesis tools in these tasks, we'll define two helper variables that describe the Xilinx FPGA part name and the PYNQ board name that we are targeting.
# change this if you have a different PYNQ board, see list above
pynq_board="Pynq-Z1"
fpga_part=pynq_part_map[pynq_board]
target_clk_ns=10
```
%% Cell type:markdown id: tags:
In previous versions of FINN, we had to manually go through several steps to generate HLS code, stitch IP, create a PYNQ project and run synthesis. All these steps are now performed by the `ZynqBuild` transform (or the `VitisBuild` transform for Alveo). **As this involves calling HLS synthesis and Vivado synthesis, this transformation will run for some time (up to half an hour depending on your PC).**
### Examining the generated outputs <a id='gen_outputs'></a>
TODO
%% Cell type:markdown id: tags:
## 4. PYNQ deployment <a id='hw_test'></a>
*[Deployment and Remote Execution](#deploy)
*[Throughput Test on PYNQ Board](#throughput)
We are almost done preparing our hardware design. We'll now put it in a form suitable for use as a PYNQ overlay, synthesize and deploy it.
%% Cell type:markdown id: tags:
### Deployment and Remote Execution <a id='deploy'></a>
We'll now use the `DeployToPYNQ` transformation to create a deployment folder with the bitfile and driver file(s), and copy that to the PYNQ board. You can change the default IP address, username, password and target folder for the PYNQ below.
Let's verify that the remote access credentials is saved in the model metadata, and that the deployment folder has been successfully copied to the board:
We only have two more steps to be able to remotely execute the deployed bitfile with some test data from the MNIST dataset. Let's load up some test data that comes bundled with FINN.
Recall that we partitioned our original network into a parent graph that contained the non-synthesizable nodes and a child graph that contained the bulk of the network, which we turned into a bitfile. We'll load up the parent graph, modify the `StreamingDataflowPartition` node so that it points to the deployed ONNX graph.
Finally, we can call `execute_onnx` on the parent graph, which will internally call remote execution with the bitfile once the `StreamingDataflowPartition` node is reached, grab the results, then continue executing the last portion of the network.
%% Cell type:code id: tags:
``` python
importnumpyasnp
fromfinn.core.onnx_execimportexecute_onnx
iname=parent_model.graph.input[0].name
oname=parent_model.graph.output[0].name
ishape=parent_model.get_tensor_shape(iname)
input_dict={iname:x.reshape(ishape)}
ret=execute_onnx(parent_model,input_dict,True)
```
%% Cell type:markdown id: tags:
We'll pass the output of the network through a softmax function to interpret it as probabilities, and plot the per-class probabilities as a bar chart.
%% Cell type:code id: tags:
``` python
defsoftmax(x):
"""Compute softmax values for each sets of scores in x."""
e_x=np.exp(x-np.max(x))
returne_x/e_x.sum()
logits=ret[oname].flatten()
prob=softmax(logits)
plt.bar(np.arange(10),prob)
```
%% Output
<BarContainer object of 10 artists>
%% Cell type:markdown id: tags:
We see that the network correctly predicts this as a digit 2 with high probability. This concludes our tutorial on how to take a simple fully-connected BNN all the way down to hardware with FINN, and execute it remotely on a PYNQ board.
%% Cell type:markdown id: tags:
### Throughput Test on PYNQ Board <a id='throughput'></a>
In addition to the functional verification, FINN also offers the possibility to measure the network performance directly on the PYNQ board. This can be done using the core function `throughput_test`. In the next section we import the function and execute it.
First we extract the `remote_exec_model` again and pass it to the function. The function returns the metrics of the network as dictionary.
Together with the values for folding we can evaluate the performance of our accelerator. Each layer has a total folding factor of 64 and because the network is fully pipelined, it follows: `II = 64`. II is the initiation interval and indicates how many cycles are needed for one input to be processed.
%% Cell type:code id: tags:
``` python
II=64
# frequency in MHz
f_MHz=100
# expected throughput in MFPS
expected_throughput=f_MHz/II
# measured throughput (FPS) from throughput test, converted to MFPS
print("We reach approximately "+str(round((measured_throughput/expected_throughput)*100))+"% of the ideal performance.")
```
%% Output
We reach approximately 86% of the ideal performance.
%% Cell type:markdown id: tags:
The measured values were recorded with a batch size of 10000 and at a frequency of 100 MHz. We will be improving the efficiency of the generated accelerator examples in the coming FINN releases.
**Important: This notebook depends on the tfc_end2end_example notebook, because we are using models that are available at intermediate steps in the end-to-end flow. So please make sure the needed .onnx files are generated to run this notebook.**
In this notebook, we will show how to take the intermediate results of the end-to-end tfc example and verify their functionality with different methods. In the following picture you can see the section in the end-to-end flow about the *Simulation & Emulation Flows*. Besides the methods in this notebook, there is another one that is covered in the Jupyter notebook [tfc_end2end_example](tfc_end2end_example.ipynb): remote execution. The remote execution allows functional verification directly on the PYNQ board, for details please have a look at the mentioned Jupyter notebook.
We will use the following helper functions, `showSrc` to show source code of FINN library calls and `showInNetron` to show the ONNX model at the current transformation step. The Netron displays are interactive, but they only work when running the notebook actively and not on GitHub (i.e. if you are viewing this on GitHub you'll only see blank squares).
To verify the simulations, a "golden" output is calculated as a reference. This is calculated directly from the Brevitas model using PyTorch, by running some example data from the MNIST dataset through the trained model.
If an ONNX model consists of [standard ONNX](https://github.com/onnx/onnx/blob/master/docs/Operators.md) nodes and/or FINN custom operations that do not belong to the fpgadataflow (backend $\neq$ "fpgadataflow") this model can be checked for functionality using Python.
To simulate a standard ONNX node [onnxruntime](https://github.com/microsoft/onnxruntime) is used. onnxruntime is an open source tool developed by Microsoft to run standard ONNX nodes. For the FINN custom op nodes execution functions are defined. The following is an example of the execution function of a XNOR popcount node.
"""Simulates XNOR-popcount matrix multiplication as a regular bipolar
matrix multiplication followed by some post processing."""
# extract the operand shapes
# (M, K0) = inp0.shape
# (K1, N) = inp1.shape
K0 = inp0.shape[-1]
K1 = inp1.shape[0]
# make sure shapes are compatible with matmul
assert K0 == K1, "Matrix shapes are not compatible with matmul."
K = K0
# convert binary inputs to bipolar
inp0_bipolar = 2.0 * inp0 - 1.0
inp1_bipolar = 2.0 * inp1 - 1.0
# call regular numpy matrix multiplication
out = np.matmul(inp0_bipolar, inp1_bipolar)
# XNOR-popcount does not produce the regular dot product result --
# it returns the number of +1s after XNOR. let P be the number of +1s
# and N be the number of -1s. XNOR-popcount returns P, whereas the
# regular dot product result from numpy is P-N, so we need to apply
# some correction.
# out = P-N
# K = P+N
# out + K = 2P, so P = (out + K)/2
return (out + K) * 0.5
%% Cell type:markdown id: tags:
The function contains a description of the behaviour in Python and can thus calculate the result of the node.
This execution function and onnxruntime is used when `execute_onnx` from `onnx_exec` is applied to the model. The model is then simulated node by node and the result is stored in a context dictionary, which contains the values of each tensor at the end of the execution. To get the result, only the output tensor has to be extracted.
The procedure is shown below. We take the model right before the nodes should be converted into HLS layers and generate an input tensor to pass to the execution function. The input tensor is generated from the Brevitas example inputs.
The result is compared with the theoretical "golden" value for verification.
%% Cell type:markdown id: tags:
## Simulation (cppsim) using C++
When dealing with HLS custom op nodes in FINN the simulation using Python is no longer sufficient. After the nodes have been converted to HLS layers, the simulation using C++ can be used. To do this, the input tensor is stored in an .npy file and C++ code is generated that reads the values from the .npy array, streams them to the corresponding finn-hlslib function and writes the result to a new .npy file. This in turn can be read in Python and processed in the FINN flow. For this example the model after setting the folding factors in the HLS layers is used, please be aware that this is not the full model, but the dataflow partition, so before executing at the end of this section we have to integrate the model back into the parent model.
Besides the .cpp file, the folder contains .h files with the weights and thresholds. The shell script contains the compile command and *node_model* is the executable generated by compilation. Comparing this with the `executable_path` node attribute, it can be seen that it specifies exactly the path to *node_model*.
%% Cell type:markdown id: tags:
To simulate the model the execution mode(exec_mode) must be set to "cppsim". This is done using the transformation SetExecMode.
Before the model can be executed using `execute_onnx`, we integrate the child model in the parent model. The function reads then the `exec_mode` and writes the input into the correct directory in a .npy file. To be able to read this in C++, there is an additional .hpp file ([npy2apintstream.hpp](https://github.com/Xilinx/finn/blob/master/src/finn/data/cpp/npy2apintstream.hpp)) in FINN, which uses cnpy to read .npy files and convert them into streams, or to read a stream and write it into an .npy. [cnpy](https://github.com/rogersce/cnpy) is a helper to read and write .npy and .npz formates in C++.
The result is again compared to the "golden" output.
The emulation using [PyVerilator](https://github.com/maltanar/pyverilator) can be done after IP blocks are generated from the corresponding HLS layers. Pyverilator is a tool which makes it possible to simulate verilog files using verilator via a python interface.
We have two ways to use rtlsim, one is to run the model node-by-node as with the simulation methods, but if the model is in the form of the dataflow partition, the part of the graph that consist of only HLS nodes could also be executed as whole.
%% Cell type:markdown id: tags:
Because at the point where we want to grab and verify the model, the model is already in split form (parent graph consisting of non-hls layers and child graph consisting only of hls layers) we first have to reference the child graph within the parent graph. This is done using the node attribute `model` for the `StreamingDataflowPartition` node.
First the procedure is shown, if the child graph has ip blocks corresponding to the individual layers, then the procedure is shown, if the child graph already has a stitched IP.
%% Cell type:markdown id: tags:
### Emulation of model node-by-node
The child model is loaded and the `exec_mode` for each node is set. To prepare the node-by-node emulation the transformation `PrepareRTLSim` is applied to the child model. With this transformation the emulation files are created for each node and can be used directly when calling `execute_onnx()`. Each node has a new node attribute "rtlsim_so" after transformation, which contains the path to the corresponding emulation files. Then it is saved in a new .onnx file so that the changed model can be referenced in the parent model.
The next step is to load the parent model and set the node attribute `model` in the StreamingDataflowPartition node (`sdp_node`). Afterwards the `exec_mode` is set in the parent model in each node.
Because the necessary files for the emulation are already generated in Jupyter notebook [tfc_end2end_example](tfc_end2end_example.ipynb), in the next step the execution of the model can be done directly.
Here we use the same procedure. First the child model is loaded, but in contrast to the layer-by-layer emulation, the metadata property `exec_mode` is set to "rtlsim" for the whole child model. When the model is integrated and executed in the last step, the verilog files of the stitched IP of the child model are used.
