"**Note:** The transformation `to_hls.InferBinaryStreamingFCLayer` gets the string \"decoupled\" as argument, this indicates the `mem_mode` for the weights. In FINN there are different options to set the way the weights are stored and accessed. For details please see the corresponding FINN readthedocs website."
"**Note:** The transformation `to_hls.InferBinaryStreamingFCLayer` gets the string \"decoupled\" as argument, this indicates the `mem_mode` for the weights. In FINN there are different options to set the way the weights are stored and accessed. For details please have a look on the [FINN readthedocs website](https://finn.readthedocs.io/) under Internals."
]
},
{
...
...
@@ -1055,7 +1055,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Directory *project_StreamingFCLayer_Batch_0* contains the project created by Vivado HLS into which the IP Block is exported, along with other files generated by Vivado HLS. If we compare it to the above visualization of the network with netron, this is exactly the name of the folder stored in the node attribute `ipgen_path`. The .cpp code that is passed to Vivado HLS can be found in the file *top_StreamingFCLayer_Batch_0.cpp*. The file *thresh.h* belongs to that as well, it contains the value for the thresholds. The weights are stored as .npy file and as .dat file (*memblock_0.dat*). *vivado_hls.log* is the log file from Vivado HLS. Besides these files, the folder contains *ipgen.sh* and *hls_syn_StreamingFCLayer_Batch_0.tcl* and because we use the StreamingFCLayer in \"decoupled\" mode a verilog wrapper (*StreamingFCLayer_Batch_0_memstream.v*) is produced, for more details on \"decoupled\" and \"const\" mode please see on the readthedocs website. \n",
"Directory *project_StreamingFCLayer_Batch_0* contains the project created by Vivado HLS into which the IP Block is exported, along with other files generated by Vivado HLS. If we compare it to the above visualization of the network with netron, this is exactly the name of the folder stored in the node attribute `ipgen_path`. The .cpp code that is passed to Vivado HLS can be found in the file *top_StreamingFCLayer_Batch_0.cpp*. The file *thresh.h* belongs to that as well, it contains the value for the thresholds. The weights are stored as .npy file and as .dat file (*memblock_0.dat*). *vivado_hls.log* is the log file from Vivado HLS. Besides these files, the folder contains *ipgen.sh* and *hls_syn_StreamingFCLayer_Batch_0.tcl* and because we use the StreamingFCLayer in \"decoupled\" mode a verilog wrapper (*StreamingFCLayer_Batch_0_memstream.v*) is produced, for more details on \"decoupled\" and \"const\" mode please see on the [FINN readthedocs website](https://finn.readthedocs.io/) under Internals. \n",
"\n",
"In the following we take a closer look at the two generated scripts. We start with *ipgen.sh*."
In this notebook, we will show how to take a simple, binarized, fully-connected network trained on the MNIST data set and take it all the way down to a customized bitfile running on a PYNQ board.
This notebook is quite lengthy, and some of the cells (involving Vivado synthesis) may take up to an hour to finish running. To let you save and resume your progress, we will save the intermediate ONNX models that are generated in the various steps to disk, so that you can jump back directly to where you left off.
%% Cell type:markdown id: tags:
## Overview
The FINN compiler comes with many *transformations* that modify the ONNX representation of the network according to certain patterns. This notebook will demonstrate a *possible* sequence of such transformations to take a particular trained network all the way down to hardware, as shown in the figure below.
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
The white fields show the state of the network representation in the respective step. The colored fields represent the transformations that are applied to the network to achieve a certain result. The diagram is divided into 5 sections represented by a different color, each of it includes several flow steps. The flow starts in top left corner with Brevitas export (green section), followed by the preparation of the network (blue section) for the Vivado HLS synthesis and Vivado IPI stitching (orange section), and finally building a PYNQ overlay bitfile and testing it on a PYNQ board (yellow section).
There is an additional section for functional verification (red section) on the right side of the diagram, which we will not cover in this notebook. For details please take a look in the verification notebook which you can find [here](tfc_end2end_verification.ipynb)
This Jupyter notebook is organized based on the sections described above. We will use the following helper functions, `showSrc` to show source code of FINN library calls and `showInNetron` to show the ONNX model at the current transformation step. The Netron displays are interactive, but they only work when running the notebook actively and not on GitHub (i.e. if you are viewing this on GitHub you'll only see blank squares).
4.[PYNQ hardware generation and deployment](#hw_test)
%% Cell type:markdown id: tags:
## 1. Brevitas export <a id='brev_exp'></a>
FINN expects an ONNX model as input. This can be a model trained with [Brevitas](https://github.com/Xilinx/brevitas). Brevitas is a PyTorch library for quantization-aware training and the FINN Docker image comes with several [example Brevitas networks](https://github.com/maltanar/brevitas_cnv_lfc). To show the FINN end-to-end flow, we'll use the TFC-w1a1 model as example network.
First a few things have to be imported. Then the model can be loaded with the pretrained weights.
/workspace/brevitas_cnv_lfc/training_scripts/models/TFC.py:85: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
x = 2.0 * x - torch.tensor([1.0]).to(self.device)
%% Cell type:markdown id: tags:
The model was now exported, loaded with the pretrained weights and saved under the name "lfc_w1_a1.onnx".
To visualize the exported model, Netron can be used. Netron is a visualizer for neural networks and allows interactive investigation of network properties. For example, you can click on the individual nodes and view the properties.
%% Cell type:code id: tags:
``` python
showInNetron(build_dir+"/tfc_w1_a1.onnx")
```
%% Output
Serving '/workspace/finn/tfc_w1_a1.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7fe1ad0b6e80>
%% Cell type:markdown id: tags:
Now that we have the model in .onnx format, we can work with it using FINN. For that FINN `ModelWrapper` is used. It is a wrapper around the ONNX model which provides several helper functions to make it easier to work with the model.
%% Cell type:code id: tags:
``` python
fromfinn.core.modelwrapperimportModelWrapper
model=ModelWrapper(build_dir+"/tfc_w1_a1.onnx")
```
%% Cell type:markdown id: tags:
Now the model is prepared and could be simulated using Python. How this works is described in the Jupyter notebook about verification and can be found [here](tfc_end2end_verification.ipynb#simpy).
The model can now also be processed in different ways. The principle of FINN are analysis and transformation passes, which can be applied to the model. An analysis pass extracts specific information about the model and returns it to the user in the form of a dictionary. A transformation pass changes the model and returns the changed model back to the FINN flow.
Since the goal in this notebook is to process the model to such an extent that a bitstream can be generated from it, the focus is on the transformations that are necessary for this. In the next section these are discussed in more detail.
*[Creating a Dataflow Partition](#dataflow_partition)
*[Folding and Datawidth Converter, FIFO and TLastMarker Insertion](#folding)
In this section, we will put the network through a series of transformations that puts it in a form that can be stitched together to form a FINN-style dataflow architecture, yielding a high-performance, high-efficiency FPGA accelerator.
We start with a quick recap of FINN-style dataflow architectures. The key idea in such architectures is to parallelize across layers as well as within layers by dedicating a proportionate amount of compute resources to each layer, as illustrated in the figure below taken from the [FINN-R paper](https://arxiv.org/pdf/1809.04570.pdf):
In practice, the compute arrays are instantiated by function calls to optimized Vivado HLS building blocks from the [finn-hlslib](https://github.com/Xilinx/finn-hlslib) library. As these function calls can only handle certain patterns/cases, we need to transform the network into an appropriate form so that we can replace network layers with these function calls, which is the goal of the network preparation process.
This section deals with some basic transformations, which are applied to the model like a kind of "tidy-up" to make it easier to be processed. They do not appear in the diagram above, but they are applied in many steps in the FINN flow to postprocess the model after a transformation and/or to prepare it for the next transformation.
%% Cell type:markdown id: tags:
These transformations are:
* GiveUniqueNodeNames
* GiveReadableTensorNames
* InferShapes
* InferDataTypes
* FoldConstants
%% Cell type:markdown id: tags:
In the first two transformations (`GiveUniqueNodeNames`, `GiveReadableTensorNames`) the nodes in the graph are first given unique (by enumeration) names, then the tensors are given human-readable names (based on the node names). The following two transformations (`InferShapes`, `InferDataTypes`) derive the shapes and data types of the tensors from the model properties and set them in the `ValueInfo` of the model. These transformations can almost always be applied without negative effects and do not affect the structure of the graph, ensuring that all the information needed is available.
The last listed transformation is `FoldConstants`, which performs constant folding. It identifies a node with constant inputs and determines its output. The result is then set as constant-only inputs for the following node and the old node is removed. Although this transformation changes the structure of the model, it is a transformation that is usually always desired and can be applied to any model.
%% Cell type:markdown id: tags:
These transformations can be imported and applied as follows.
The result of these transformations can be viewed with netron after the model has been saved again. By clicking on the individual nodes, it can now be seen, for example, that each node has been given a name. Also the whole upper area could be folded, so that now the first node is "Reshape".
%% Cell type:code id: tags:
``` python
showInNetron(build_dir+"/tfc_w1_a1_tidy.onnx")
```
%% Output
Stopping http://0.0.0.0:8081
Serving '/workspace/finn/tfc_w1_a1_tidy.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7fe1ad0639e8>
%% Cell type:markdown id: tags:
### Streamlining <a id='streamline'></a>
Streamlining is a transformation containing several sub-transformations. The goal of streamlining is to eliminate floating point operations by moving them around, then collapsing them into one operation and in the last step transform them into multi-thresholding nodes. For more information on the theoretical background of this, see [this paper](https://arxiv.org/pdf/1709.04060).
Let's have a look at which sub-transformations `Streamline` consists of:
"""Apply the streamlining transform, see arXiv:1709.04060."""
def apply(self, model):
streamline_transformations = [
ConvertSubToAdd(),
ConvertDivToMul(),
BatchNormToAffine(),
ConvertSignToThres(),
MoveAddPastMul(),
MoveScalarAddPastMatMul(),
MoveScalarAddPastConv(),
MoveScalarMulPastMatMul(),
MoveScalarMulPastConv(),
MoveAddPastMul(),
CollapseRepeatedAdd(),
CollapseRepeatedMul(),
AbsorbAddIntoMultiThreshold(),
FactorOutMulSignMagnitude(),
AbsorbMulIntoMultiThreshold(),
Absorb1BitMulIntoMatMul(),
Absorb1BitMulIntoConv(),
RoundAndClipThresholds(),
]
for trn in streamline_transformations:
model = model.transform(trn)
model = model.transform(GiveUniqueNodeNames())
model = model.transform(GiveReadableTensorNames())
model = model.transform(InferDataTypes())
return (model, False)
%% Cell type:markdown id: tags:
As can be seen, several transformations are involved in the streamlining transformation. There are move and collapse transformations. In the last step the operations are transformed into multithresholds. The involved transformations can be viewed in detail [here](https://github.com/Xilinx/finn/tree/master/src/finn/transformation/streamline). After each transformation, three of the tidy-up transformations (`GiveUniqueNodeNames`, `GiveReadableTensorNames` and `InferDataTypes`) are applied to the model.
Serving '/workspace/finn/tfc_w1_a1_streamlined.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7fe1346e4ef0>
%% Cell type:markdown id: tags:
You can see that the network has become simplified considerably compared to the previous step -- a lot of nodes have disappeared between the `MatMul` layers, and the `Sign` nodes have been replaced with `MultiThreshold` nodes instead.
**The current implementation of streamlining is highly network-specific and may not work for your network if its topology is very different than the example network here. We hope to rectify this in future releases.**
Our example network is a quantized network with 1-bit bipolar (-1, +1 values) precision, and we want FINN to implement them as XNOR-popcount operations [as described in the original FINN paper](https://arxiv.org/pdf/1612.07119). For this reason, after streamlining, the resulting bipolar matrix multiplications are converted into xnorpopcount operations. This transformation produces operations that are again collapsed and converted into thresholds. This procedure is shown below.
Serving '/workspace/finn/tfc_w1a1_ready_for_hls_conversion.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7fe1346f7780>
%% Cell type:markdown id: tags:
Observe the pairs of `XnorPopcountmatMul` and `MultiThreshold` layers following each other -- this is the particular pattern that the next step will be looking for in order to convert them to HLS layers.
%% Cell type:markdown id: tags:
### Conversion to HLS layers <a id='hls_layers'></a>
Converts the nodes to HLS layers that correspond to the functions in [finn-hls library](https://finn-hlslib.readthedocs.io/en/latest/). In our case this transformation converts pairs of binary XnorPopcountMatMul layers to StreamingFCLayer_Batch layers. Any immediately following MultiThreshold layers will also be absorbed into the MVTU.
Below is the code for the transformation and the network is visualized using netron to create the new structure with `StreamingFCLayer_Batch` nodes, which will correspond to a function call from the [finn-hlslib](https://finn-hlslib.readthedocs.io/en/latest/library/fclayer.html#_CPPv4I_j_j_j_j000_i_i000E22StreamingFCLayer_BatchvRN3hls6streamI7ap_uintI9InStreamWEEERN3hls6streamI7ap_uintI10OutStreamWEEERK2TWRK2TAKjRK1R) library.
%% Cell type:markdown id: tags:
**Note:** The transformation `to_hls.InferBinaryStreamingFCLayer` gets the string "decoupled" as argument, this indicates the `mem_mode` for the weights. In FINN there are different options to set the way the weights are stored and accessed. For details please see the corresponding FINN readthedocs website.
**Note:** The transformation `to_hls.InferBinaryStreamingFCLayer` gets the string "decoupled" as argument, this indicates the `mem_mode` for the weights. In FINN there are different options to set the way the weights are stored and accessed. For details please have a look on the [FINN readthedocs website](https://finn.readthedocs.io/) under Internals.
Serving '/workspace/finn/tfc_w1_a1_hls_layers.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7fe1346f1080>
%% Cell type:markdown id: tags:
Each StreamingFCLayer_Batch node has two attributes that specify the degree of folding, PE and SIMD. In all nodes the values for these attributes are set as default to 1, which would correspond to a maximum folding (time multiplexing) and thus minimum performance. We will shortly cover how these can be adjusted, but first we want to separate the HLS layers from the non-HLS layers in this network.
%% Cell type:markdown id: tags:
### Creating a Dataflow Partition <a id='dataflow_partition'></a>
In the graph above, you can see that there is a mixture of FINN HLS layers (StreamingFCLayer_Batch) with regular ONNX layers (Reshape, Mul, Add). To create a bitstream, FINN needs a model with only HLS layers. In order to achieve this, we will use the `CreateDataflowPartition` transformation to create a "dataflow partition" in this graph, separating out the HLS layers into another model, and replacing them with a placeholder layer called StreamingDataflowPartition:
Serving '/workspace/finn/tfc_w1_a1_dataflow_parent.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7fe1ad0b6e48>
%% Cell type:markdown id: tags:
We can see that the StreamingFCLayer instances have all been replaced with a single `StreamingDataflowPartition`, which has an attribute `model` that points to the extracted, HLS dataflow-only graph:
Serving '/tmp/finn_dev_jakobap/dataflow_partition_pbrjefjg/df_model.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7fe1346f3550>
%% Cell type:markdown id: tags:
We can see all the extracted `StreamingFCLayer` instances have been moved to the child (dataflow) model. We will load the child model with `ModelWrapper` and continue working on it.
%% Cell type:code id: tags:
``` python
model=ModelWrapper(dataflow_model_filename)
```
%% Cell type:markdown id: tags:
### Folding and Datawidth Converter, FIFO and TLastMarker Insertion <a id='folding'></a>
*Folding* in FINN describes how much a layer is time-multiplexed in terms of execution resources. There are several *folding factors* for each layer, controlled by the PE (parallelization over outputs) and SIMD (parallelization over inputs) parameters as described by the original [FINN paper](https://arxiv.org/pdf/1612.07119). The higher the PE and SIMD values are set, the faster the generated accelerator will run, and the more FPGA resources it will consume.
Since the folding parameters are node attributes, they can be easily accessed and changed using a helper function of the `ModelWrapper`. But first we take a closer look at one of the nodes that implement a StreamingFCLayer_Batch operation. This is where the Netron visualization helps us, in the above diagram we can see that the first four nodes are StreamingFCLayer_Batch. So as an example we extract the first node.
%% Cell type:markdown id: tags:
We can use the higher-level [HLSCustomOp](https://github.com/Xilinx/finn/blob/master/src/finn/custom_op/fpgadataflow/__init__.py) wrappers for this node. These wrappers provide easy access to specific properties of these nodes, such as the folding factors (PE and SIMD). Let's have a look at which node attributes are defined by the CustomOp wrapper, and adjust the SIMD and PE attributes.
%% Cell type:code id: tags:
``` python
fc0=model.graph.node[0]
fc0w=getCustomOp(fc0)
print("CustomOp wrapper is of class "+fc0w.__class__.__name__)
fc0w.get_nodeattr_types()
```
%% Output
CustomOp wrapper is of class StreamingFCLayer_Batch
{'PE': ('i', True, 0),
'SIMD': ('i', True, 0),
'MW': ('i', True, 0),
'MH': ('i', True, 0),
'resType': ('s', True, ''),
'ActVal': ('i', False, 0),
'inputDataType': ('s', True, ''),
'weightDataType': ('s', True, ''),
'outputDataType': ('s', True, ''),
'binaryXnorMode': ('i', False, 0),
'noActivation': ('i', False, 0),
'numInputVectors': ('ints', False, [1]),
'mem_mode': ('s', False, 'const'),
'ram_style': ('s', False, 'auto'),
'backend': ('s', True, 'fpgadataflow'),
'code_gen_dir_cppsim': ('s', False, ''),
'code_gen_dir_ipgen': ('s', False, ''),
'executable_path': ('s', False, ''),
'ipgen_path': ('s', False, ''),
'ip_path': ('s', False, ''),
'ip_vlnv': ('s', False, ''),
'exec_mode': ('s', False, ''),
'sim_cycles': ('i', False, 0),
'rtlsim_trace': ('s', False, ''),
'res_estimate': ('s', False, ''),
'res_hls': ('s', False, ''),
'res_synth': ('s', False, ''),
'rtlsim_so': ('s', False, ''),
'inFIFODepth': ('i', False, 2),
'outFIFODepth': ('i', False, 2)}
%% Cell type:markdown id: tags:
We can see that the PE and SIMD are listed as node attributes, as well as the depths of the FIFOs that will be inserted between consecutive layers, and all can be adjusted using `set_nodeattr` subject to certain constraints.
**In this notebook we are setting the folding factors and FIFO depths manually, but in a future version we will support determining the folding factors given an FPGA resource budget according to the analytical model from the [FINN-R paper](https://arxiv.org/pdf/1809.04570).**
We are setting PE and SIMD so that each layer has a total folding of 16.
%% Cell type:markdown id: tags:
Besides PE and SIMD three other node attributes are set. `ram_style` specifies how the weights are to be stored (BRAM, LUTRAM, and so on). It can be selected explicitly or with the option `auto` you can let Vivado decide.
`inFIFODepth` and `outFIFODepth` specifies the FIFO depths that is needed by the node from the surrounding FIFOs. These attributes are used in the transformation 'InsertFIFO' to insert the appropriate FIFOs between the nodes.
But before FIFOs can be added, it must be determined whether datawidth converters (DWC) are required and they must be inserted correctly. Because by setting the folding, the folded output shape of one node may not match the folded input shape of the next node.
In the following, first DWCs and then FIFOs are inserted using the corresponding transformations in FINN.
Finally, we will run the `InsertTLastMarker` transformation to get a `TLastMarker` node at the output of this graph, which is necessary to run the DMA engines correctly. Using netron we can observe that now the nodes contain the set folding, if necessary a DWC is inserted, inbetween the nodes are FIFOs inserted and the last node is the `TLastMarker` node we insert in the following.
Serving '/workspace/finn/tfc_w1_a1_set_folding_factors.onnx' at http://0.0.0.0:8081
<IPython.lib.display.IFrame at 0x7fe135b84780>
%% Cell type:markdown id: tags:
This completes the network preparation and the network can be passed on to the next block *Vivado HLS and IPI*, which is described below.
%% Cell type:markdown id: tags:
## 3. Vivado HLS and IPI <a id='vivado'></a>
*[Generating HLS Code](#hls_per_layer)
*[Synthesizing HLS to IP Blocks](#hls_synth)
*[IP Stitching](#ip_stitching)
As we will be dealing with FPGA synthesis tools in these tasks, we'll define two helper variables that describe the Xilinx FPGA part name and the PYNQ board name that we are targeting.
This section deals with the generation of an IP block from the different layers. These can then be stitched to a block design that corresponds to the complete model. The single conversion into IP blocks allows a good transparency and we can check the functionality of each IP block and compare it with the behaviour of the corresponding ONNX node.
%% Cell type:markdown id: tags:
Two transformations are required to generate HLS IP blocks for each layer:
*`PrepareIP` which generates the HLS C++ code for the node and a tcl-script which starts the HLS synthesis and exports the design as IP.
*`HLSSynthIP` which passes the tcl-script to Vivado HLS and thus performs the actual IP generation.
We start off by giving unique node names using the basic transformation `GiveUniqueNodeNames`, and then proceed with the HLS C++ code generation with `PrepareIP`.
### Synthesizing HLS to IP Blocks <a id='hls_synth'></a>
Now that we have generated the HLS code for each layer, we can call the `HLSSynthIP` transformation to convert the generated HLS into Vivado IP blocks. **As this involves calling HLS synthesis, this transformation will run for some time (several minutes).**
Directory *project_StreamingFCLayer_Batch_0* contains the project created by Vivado HLS into which the IP Block is exported, along with other files generated by Vivado HLS. If we compare it to the above visualization of the network with netron, this is exactly the name of the folder stored in the node attribute `ipgen_path`. The .cpp code that is passed to Vivado HLS can be found in the file *top_StreamingFCLayer_Batch_0.cpp*. The file *thresh.h* belongs to that as well, it contains the value for the thresholds. The weights are stored as .npy file and as .dat file (*memblock_0.dat*). *vivado_hls.log* is the log file from Vivado HLS. Besides these files, the folder contains *ipgen.sh* and *hls_syn_StreamingFCLayer_Batch_0.tcl* and because we use the StreamingFCLayer in "decoupled" mode a verilog wrapper (*StreamingFCLayer_Batch_0_memstream.v*) is produced, for more details on "decoupled" and "const" mode please see on the readthedocs website.
Directory *project_StreamingFCLayer_Batch_0* contains the project created by Vivado HLS into which the IP Block is exported, along with other files generated by Vivado HLS. If we compare it to the above visualization of the network with netron, this is exactly the name of the folder stored in the node attribute `ipgen_path`. The .cpp code that is passed to Vivado HLS can be found in the file *top_StreamingFCLayer_Batch_0.cpp*. The file *thresh.h* belongs to that as well, it contains the value for the thresholds. The weights are stored as .npy file and as .dat file (*memblock_0.dat*). *vivado_hls.log* is the log file from Vivado HLS. Besides these files, the folder contains *ipgen.sh* and *hls_syn_StreamingFCLayer_Batch_0.tcl* and because we use the StreamingFCLayer in "decoupled" mode a verilog wrapper (*StreamingFCLayer_Batch_0_memstream.v*) is produced, for more details on "decoupled" and "const" mode please see on the [FINN readthedocs website](https://finn.readthedocs.io/) under Internals.
In the following we take a closer look at the two generated scripts. We start with *ipgen.sh*.
%% Cell type:code id: tags:
``` python
shell_script=code_gen_dir+"/ipgen.sh"
!cat{shell_script}
```
%% Output
#!/bin/bash
cd /tmp/finn_dev_jakobap/code_gen_ipgen_StreamingFCLayer_Batch_0_edb__5oc
The script consists only of two framing `cd` commands and a command to pass the tcl script to *vivado_hls*. The directory has to be changed to create the files in the correct folder and will then be changed back to the original directory.
Below is the tcl script which is passed to *vivado_hls*.
In the first part of the script the project is configured. For example the FPGA part and the clock are set. Then the project is opened and the files are added. The toplevel function is set and after creating a clock, the design is first synthesized with `csynth` and then exported as an IP block.
Now that all IP blocks are in place, they can be stitched together to create an IP design that matches the ONNX model. This is covered in the next section.
%% Cell type:markdown id: tags:
### IP Stitching <a id='ip_stitching'></a>
We now have IP blocks for each of our layers, and will stitch them together into a larger IP that implements the whole network using the `CreateStitchedIP` transformation. Bear in mind that this transformation can only be applied on a graph that only contains HLS nodes that already have been through the `HLSSynthIP` transformation, which is the last step we performed. Prior to calling IP stitching, we'll also use the `ReplaceVerilogRelPaths` transformation to convert any relative `$readmemh` paths in the generated IP blocks to absolute ones, which prevents errors later on. **This step invokes Vivado and may take a few minutes to run.**
If you examine the nodes themselves on the transformed model you won't see a difference, because the IP stitching adds model-level metadata to the graph. This can be accessed using the `.model.metadata_props`, the `get_metadata_prop` function in `ModelWrapper`, or by clicking on the global input/output tensors in Netron.
If you navigate to the folder above (remember the /tmp/finn_xxx folder is mounted on the host as well as inside Docker) you can open the Vivado project (.xpr) file there using Vivado, and view the following stitched IP block design:
%% Cell type:markdown id: tags:
%% Cell type:code id: tags:
``` python
model.save(build_dir+"/tfc_w1_a1_ipstitch.onnx")
```
%% Cell type:markdown id: tags:
At this point, one could take the generated stitched IP and integrate it into your own project using Vivado IP Integrator if desired. Here, we will continue the tutorial by assuming that we want to do a stand-alone deployment for this accelerator for a PYNQ board.
%% Cell type:markdown id: tags:
## 4. PYNQ hardware generation and deployment <a id='hw_test'></a>
*[Inserting the IP into a PYNQ Overlay Shell](#pynq_shell)
*[Synthesis, Place and Route](#synth_pl_ro)
*[Driver Generation](#driver_gen)
*[Deployment and Remote Execution](#deploy)
*[Throughput Test on PYNQ Board](#throughput)
We are almost done preparing our hardware design. We'll now put it in a form suitable for use as a PYNQ overlay, synthesize and deploy it.
%% Cell type:markdown id: tags:
### Inserting the IP into a PYNQ Overlay Shell <a id='pynq_shell'></a>
To deploy our accelerator on a PYNQ platform, it needs to be put inside an appropriate *shell* that bridges it with the interfaces that the underlying system exposes. FINN makes it easy to create a PYNQ-compatible overlay by inserting the stitched IP into an appropriate PYNQ shell with the `MakePYNQProject` transformation, and view the created PYNQ shell project directory using the `metadata_props`. **This invokes Vivado and may take a few minutes to run.**
If we open the created Vivado project (.xpr) under the `vivado_pynq_proj` directory above, we can see the system-level block design as below, with the FINN-generated part of the design highlighted. Various other components, such as the DMA engine and data width converters, have also been instantiated.
### Synthesis, Place and Route <a id='synth_pl_ro'></a>
%% Cell type:markdown id: tags:
We are now ready for the final hardware generation step, which is synthesis, place and route to generate an FPGA bitfile. This can be done by either running the `synth_project.sh` script in the generated Vivado PYNQ project directory inside Docker, or by executing the `SynthPYNQProject` transformation. **This step involves launching Vivado for synthesis and may take a few hours.**
Now that we have synthesized a bitfile for our network, we will generate some Python code for PYNQ that will act as the driver for this bitfile, package everything into a deployment folder and copy that to our PYNQ board.
The generated driver is placed in a folder that is indicated by the `pynq_driver_dir` top-level metadata. We can examine the generated PYNQ Python driver code as follows:
We can see that in the generated driver a class is implemented which implements the FINN accelerator. The constructor gets the batchsize (N) as integer and the bitfile as string. It also contains the expected input/output shapes, and takes care of the instantiation of the accelerator by loading the bitfile and setting up dma and buffer. Several member functions take care of the data folding and packing. The function `copy_input_data_to_device` copies the input data into the PYNQ buffer and `execute` sets up the dma channels and waits until the transfer is completed. This class is used in the main function. But first the arguments are parsed, which are passed to the script. The driver can be used in two modes: "execute" and "throughput_test". By default all arguments are set to "execute" mode. In this mode the batch size is 1, and the passed files are set to the names used by the FINN transformations.
In the "execute" mode works as follows:
1. the data is loaded from the "inputfile"
2. the data is folded using `fold_input`
3. the data is packed using `pack_input`
4. the data is copied to the device using `copy_input_data_to_device`
5. FINNAccelDriver is executed using `execute`
6. the data is unpacked using `unpack_output`
7. the data is unfolded using `unfold_output`
8. the data is stored in the "outputfile"
If "throughput_test" is selected as `exec_mode`, no actual data needs to be loaded. The batchsize N should be set to a high value (i.e. 1000) and a time measurement is implemented in python. An empty dictionary (`res`) is created and after running the accelerator with the measured runtime it is filled with the metrics and saved in a .txt file.
You can build your own applications around the accelerator by modifying the driver, or use the remote execution capabilities that FINN provides just to check if it is working, which will be our next step.
%% Cell type:markdown id: tags:
### Deployment and Remote Execution <a id='deploy'></a>
We'll now use the `DeployToPYNQ` transformation to create a deployment folder with the bitfile and driver file(s), and copy that to the PYNQ board. You can change the default IP address, username, password and target folder for the PYNQ below.
Let's verify that the remote access credentials is saved in the model metadata, and that the deployment folder has been successfully copied to the board:
-rw-r--r-- 1 xilinx xilinx 6363 May 7 15:05 driver.py
drwxr-xr-x 4 xilinx xilinx 4096 May 7 15:05 finn
-rw-r--r-- 1 xilinx xilinx 3264 May 7 15:06 input.npy
-rw-r--r-- 1 root root 172 May 7 15:11 nw_metrics.txt
-rw-r--r-- 1 root root 120 May 7 15:06 output.npy
-rw-r--r-- 1 xilinx xilinx 4045675 May 7 15:05 resizer.bit
-rw-r--r-- 1 xilinx xilinx 302015 May 7 15:05 resizer.hwh
-rw-r--r-- 1 root root 32 May 7 15:11 sds_trace_data.dat
%% Cell type:markdown id: tags:
We only have two more steps to be able to remotely execute the deployed bitfile with some test data from the MNIST dataset. Let's load up some test data that comes bundled with FINN.
Recall that we partitioned our original network into a parent graph that contained the non-synthesizable nodes and a child graph that contained the bulk of the network, which we turned into a bitfile. We'll load up the parent graph, modify the `StreamingDataflowPartition` node so that it points to the deployed ONNX graph.
Finally, we can call `execute_onnx` on the parent graph, which will internally call remote execution with the bitfile once the `StreamingDataflowPartition` node is reached, grab the results, then continue executing the last portion of the network.
%% Cell type:code id: tags:
``` python
importnumpyasnp
fromfinn.core.onnx_execimportexecute_onnx
iname=parent_model.graph.input[0].name
oname=parent_model.graph.output[0].name
ishape=parent_model.get_tensor_shape(iname)
input_dict={iname:x.reshape(ishape)}
ret=execute_onnx(parent_model,input_dict,True)
```
%% Cell type:markdown id: tags:
We'll pass the output of the network through a softmax function to interpret it as probabilities, and plot the per-class probabilities as a bar chart.
%% Cell type:code id: tags:
``` python
defsoftmax(x):
"""Compute softmax values for each sets of scores in x."""
e_x=np.exp(x-np.max(x))
returne_x/e_x.sum()
logits=ret[oname].flatten()
prob=softmax(logits)
plt.bar(np.arange(10),prob)
```
%% Output
<BarContainer object of 10 artists>
%% Cell type:markdown id: tags:
We see that the network correctly predicts this as a digit 2 with high probability. This concludes our tutorial on how to take a simple fully-connected BNN all the way down to hardware with FINN, and execute it remotely on a PYNQ board.
%% Cell type:markdown id: tags:
### Throughput Test on PYNQ Board <a id='throughput'></a>
In addition to the functional verification, FINN also offers the possibility to measure the network performance directly on the PYNQ board. This can be done using the core function `throughput_test`. In the next section we import the function and execute it.
First we extract the `remote_exec_model` again and pass it to the function. The function returns the metrics of the network as dictionary.
Together with the values for folding we can evaluate the performance of our accelerator. Each layer has a total folding factor of 64 and because the network is fully pipelined, it follows: `II = 64`. II is the initiation interval and indicates how many cycles are needed for one input to be processed.
%% Cell type:code id: tags:
``` python
II=64
# frequency in MHz
f_MHz=100
# expected throughput in MFPS
expected_throughput=f_MHz/II
# measured throughput (FPS) from throughput test, converted to MFPS
print("We reach approximately "+str(round((measured_throughput/expected_throughput)*100))+"% of the ideal performance.")
```
%% Output
We reach approximately 43% of the ideal performance.
%% Cell type:markdown id: tags:
The measured values were recorded with a batch size of 1000 and at a frequency of 100 MHz. We will be improving the efficiency of the generated accelerator examples in the coming FINN releases.
