"<font size=\"3\">**`compile_singlenode_code()`**: To compile the generated code, the compile command must be built. This is done in this function. It creates an instance of the `CppBuilder()` class and assembles the various components for the function. The `.build` function creates the executable and then sets the corresponding attribute. The class `CppBuilder` is a transformation and a more detailed description can be found in Jupyter notebook [FINN-CodeGenerationAndCompilation](github.com/Xilinx/finn/blob/dev/notebooks/FINN-CodeGenerationAndCompilation.ipynb).\n",
"<font size=\"3\">**`compile_singlenode_code()`**: To compile the generated code, the compile command must be built. This is done in this function. It creates an instance of the `CppBuilder()` class and assembles the various components for the function. The `.build` function creates the executable and then sets the corresponding attribute. The class `CppBuilder` is a transformation and a more detailed description can be found in Jupyter notebook [FINN-CodeGenerationAndCompilation](FINN-CodeGenerationAndCompilation.ipynb).\n",
<fontsize="3">This notebook should give a more detailed insight into FINN custom operation nodes. </font>
%% Cell type:markdown id: tags:
<fontsize="3">Following showSrc function is used to print the source code of function calls in the Jupyter notebook: </font>
%% Cell type:code id: tags:
``` python
importinspect
defshowSrc(what):
print("".join(inspect.getsourcelines(what)[0]))
```
%% Cell type:markdown id: tags:
<fontsize="3">FINN uses many custom operations (`op_type` in ONNX NodeProto) that are not defined in the ONNX operator schema. These custom nodes are marked with `domain="finn"` in the protobuf to identify them as such. These nodes can represent specific operations that we need for low-bit networks, or operations that are specific to a particular hardware backend.
A very abstract version of a custom op node representing a streaming fc layer is shown below. </font>
%% Cell type:markdown id: tags:
## Outline
---------------------------
*<fontsize="3">FINN-ONNX node</font>
*<fontsize="3">CustomOp class</font>
*<fontsize="3">HLSCustomOp class</font>
%% Cell type:markdown id: tags:
## FINN-ONNX node
<fontsize="3">To create a FINN-ONNX node you can use the helper function of ONNX. Because it is an ONNX NodeProtobuf, but with several additional attributes. </font>
`FCLayer_node = helper.make_node(
"StreamingFCLayer_Batch",
node_inp_list,
node_outp_list,
domain="finn",
backend="fpgadataflow",
code_gen_dir="",
executable_path="",
resType="ap_resource_lut()",
MW=mw,
MH=mh,
SIMD=simd,
PE=pe,
inputDataType=<FINN DataType>,
weightDataType=<FINN DataType>,
outputDataType=<FINN DataType>,
ActVal=actval,
binaryXnorMode=<0/1>,
noActivation=<0/1>
)`
%% Cell type:markdown id: tags:
<fontsize="3">`"StreamingFCLayer_Batch"` describes the op_type, then the inputs and outputs are declared. This is still like building a default onnx node without additional attributes. But since this is a custom op node of FINN, the attribute `domain="finn"` must be set. The streaming fc layer is a custom op from the finn-hls library, this information is set in the node using the `backend` attribute. To execute a custom op from the finn-hls library, the corresponding c++ code must be created and an executable must be produced. Where the generated code is stored is specified in the `code_gen_dir` attribute and `executable_path` specifies the path to the produced executable. In addition to the data types of the input and output tensors, the node also contains various other attributes resulting from the parameters of the corresponding finn-hls library function. More detailed information can be found in the documentation of [finn-hlslib](github.com/Xilinx/finn-hlslib).</font>
%% Cell type:markdown id: tags:
## CustomOp class
<fontsize="3">Custom Ops are represented in FINN as ONNX nodes on the one hand and by a CustomOp class on the other hand. This allows easier access to different attributes and introduces special custom op functions. See below for the standard CustomOp class.</font>
%% Cell type:code id: tags:
``` python
fromfinn.custom_opimportCustomOp
showSrc(CustomOp)
```
%% Output
class CustomOp(ABC):
def __init__(self, onnx_node):
super().__init__()
self.onnx_node = onnx_node
def get_nodeattr(self, name):
"""Get a node attribute by name. Data is stored inside the ONNX node's
AttributeProto container. Attribute must be part of get_nodeattr_types.
Default value is returned if attribute is not set."""
- dtype indicates which member of the ONNX AttributeProto
will be utilized
- require indicates whether this attribute is required
- default_val indicates the default value that will be used if the
attribute is not set
"""
pass
@abstractmethod
def make_shape_compatible_op(self):
"""Returns a standard ONNX op which is compatible with this CustomOp
for performing shape inference."""
pass
@abstractmethod
def infer_node_datatype(self, model):
"""Set the DataType annotations corresponding to the outputs of this
node."""
pass
@abstractmethod
def execute_node(self, context, graph):
"""Execute this CustomOp instance, given the execution context and
ONNX graph."""
pass
%% Cell type:markdown id: tags:
<fontsize="3">When instantiating the class, the ONNX node is passed to access all attributes of the node within the class. This is accompanied by the functions `get_nodeattr()`and `set_nodeattr()`, which each instance of this class has. Furthermore 4 abstract methods are implemented, which are described in more detail in the comments in the code. </font>
%% Cell type:markdown id: tags:
## HLSCustomOp class
<fontsize="3">If it is a node from the finn-hls library another class is used which is derived from the CustomOp class:</font>
<fontsize="3">When creating an instance of this class, a template is introduced, which forms the layout for the c++ code to execute the node. It has some general constructs, like the inclusion of bnn-library.h, which contains the references to the finn-hls library, and of cnpy.h and npy2apintstream.hpp, which support the transfer of python numpy arrays in c++. The idea of this template is to replace the variables marked with `$ $` with c++ calls during code generation. Then the template can be written into a .cpp file and be compiled.
**`get_nodeattr_types()`**: each instance of the HLSCustomOp class must have the attributes `code_gen_dir` and `executable_path`, since to execute these nodes c++ code must be generated and correspondingly the executables.
</font>
%% Cell type:markdown id: tags:
<fontsize="3">**`code_generation(model)`**: all functions required for code generation are called and the `$ $` variables in the template are replaced accordingly and written into a .cpp file. Almost all of these subfunctions are implemented as abstract methods in the class, so they are completely customized for each custom op node. A special function is `generate_params()`. This is not implemented as an abstract method, but as a normal function, but contains by default only `pass`. This is because some custom op nodes do not have parameters that need to be generated and in this way the function is skipped. For example for a streaming fc layer node a parameter generation is necessary. How such a parameter generation can look like is described in more detail in the course of this notebook.
</font>
%% Cell type:markdown id: tags:
<fontsize="3">**`compile_singlenode_code()`**: To compile the generated code, the compile command must be built. This is done in this function. It creates an instance of the `CppBuilder()` class and assembles the various components for the function. The `.build` function creates the executable and then sets the corresponding attribute. The class `CppBuilder` is a transformation and a more detailed description can be found in Jupyter notebook [FINN-CodeGenerationAndCompilation](github.com/Xilinx/finn/blob/dev/notebooks/FINN-CodeGenerationAndCompilation.ipynb).
<fontsize="3">**`compile_singlenode_code()`**: To compile the generated code, the compile command must be built. This is done in this function. It creates an instance of the `CppBuilder()` class and assembles the various components for the function. The `.build` function creates the executable and then sets the corresponding attribute. The class `CppBuilder` is a transformation and a more detailed description can be found in Jupyter notebook [FINN-CodeGenerationAndCompilation](FINN-CodeGenerationAndCompilation.ipynb).
</font>
%% Cell type:markdown id: tags:
<fontsize="3">**`dynamic_input_to_npy(context, count)`**: creates a .npy file for all inputs of the node. These files will be stored in the directory specified by code_gen_dir. The argument `count` must be used to specify the number of inputs. `context` contains the values for the inputs.</font>
%% Cell type:markdown id: tags:
<fontsize="3">**`npy_to_dynamic_output(context)`**: reads the output values and sets `context` dictionary accordingly. When executing the c++ executable of the node, the output values are written to a .npy file. </font>
%% Cell type:markdown id: tags:
<fontsize="3">**`exec_precompiled_singlenode_model()`**: executes precompiled executable which is specified in `executable_path`</font>
%% Cell type:markdown id: tags:
<fontsize="3">**`execute_node(context,graph)`**: calls first `dynamic_input_to_npy()`, then executes the executable using `exec_precompiled_singlenode_model()` and at the end reads the output .npy file with `npy_to_dynamic_output`</font>
%% Cell type:markdown id: tags:
#### Generate Parameter
<fontsize="3">Parameters have to be generated for specific types of HLSCustomOps. For example if the node is a streaming fc layer, there are weights and activation values, which are written to separate .h files and added to the template using `#include`. For streaming fc layer the parameter generation looks like this:
<fontsize="3">First, the values for the weights are extracted with `get_initializer()` using the ModelWrapper. At this point it is assumed that the second input of the streamingfclayer specifies the weights. After a few manipulations the weights are written in `params.h`. If there are threshold values, they will be prepared and written to `thresh.h`. </font>
"<font size=\"3\">ModelWrapper contains more useful functions, if you are interested please have a look at the [Python code](https://github.com/Xilinx/finn/blob/dev/src/finn/core/modelwrapper.py) directly. Additionally, in the folder notebooks/ a Jupyter notebook about transformation passes and one about analysis passes can be found.</font>"
"<font size=\"3\">ModelWrapper contains more useful functions, if you are interested please have a look at the [Python code](https://github.com/Xilinx/finn/blob/dev/src/finn/core/modelwrapper.py) directly. Additionally, in the folder notebooks/ a Jupyter notebook about transformation passes [FINN-HowToTransformationPass](FINN-HowToTransformationPass.ipynb) and one about analysis passes [FINN-HowToAnalysisPass](FINN-HowToAnalysisPass.ipynb) can be found.</font>"
]
},
{
...
...
%% Cell type:markdown id: tags:
# FINN - ModelWrapper
--------------------------------------
<fontsize="3"> This notebook is about the ModelWrapper class within FINN.
Following showSrc function is used to print the source code of function calls in the Jupyter notebook:</font>
%% Cell type:code id: tags:
``` python
importinspect
defshowSrc(what):
print("".join(inspect.getsourcelines(what)[0]))
```
%% Cell type:markdown id: tags:
## General Information
------------------------------
*<fontsize="3"> wrapper around ONNX ModelProto that exposes several utility
functions for graph manipulation and exploration </font>
*<fontsize="3"> ModelWrapper instance takes onnx model proto and `make_deepcopy` flag as input </font>
*<fontsize="3"> onnx model proto can either be a string with the path to a stored .onnx file on disk, or serialized bytes </font>
*<fontsize="3">`make_deepcopy` is by default False but can be set to True if a (deep) copy should be created </font>
%% Cell type:markdown id: tags:
### Create a ModelWrapper instance
<fontsize="3">Here we use a premade ONNX file on disk to load up the ModelWrapper, but this could have been produced from e.g. a trained Brevitas PyTorch model. See [this notebook](brevitas-network-import.ipynb) for more details.</font>
%% Cell type:code id: tags:
``` python
fromfinn.core.modelwrapperimportModelWrapper
model=ModelWrapper("LFCW1A1.onnx")
```
%% Cell type:markdown id: tags:
### Access the ONNX GraphProto through ModelWrapper
<fontsize="3">ModelWrapper is a thin wrapper around the ONNX protobuf, and it offers a range of helper functions as well as direct access to the underlying protobuf. The `.model` member gives access to the full ONNX ModelProto, whereas `.graph` gives access to the GraphProto, as follows:</font>
%% Cell type:code id: tags:
``` python
# access the ONNX ModelProto
modelproto=model.model
print("ModelProto IR version is %d"%modelproto.ir_version)
# the graph
graphproto=model.graph
print("GraphProto top-level outputs are %s"%str(graphproto.output))
# the node list
nodes=model.graph.node
print("There are %d nodes in the graph"%len(nodes))
print("The first node is \n%s"%str(nodes[0]))
```
%% Output
ModelProto IR version is 4
GraphProto top-level outputs are [name: "60"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 10
}
}
}
}
]
There are 29 nodes in the graph
The first node is
input: "0"
output: "21"
op_type: "Shape"
%% Cell type:markdown id: tags:
### Helper functions for tensors
<fontsize="3"> Every input and output of every node in the onnx model is represented as tensor with several properties (i.e. name, shape, data type). ModelWrapper provides some utility functions to work with the tensors. </font>
%% Cell type:markdown id: tags:
##### Get all tensor names
<fontsize="3">Produces a list of all tensor names (inputs, activations, weights, outputs...) in the graph.</font>
<fontsize="3">A tensor can have a producer node and/or a consumer node in the onnx model. ModelWrapper provides two helper functions to access these nodes, they are shown in the following.
It may be that a tensor does not have a producer or consumer node, for example if the tensor represents a constant that is already set. In that case `None` will be returned.</font>
%% Cell type:code id: tags:
``` python
# get random tensor and find producer and consumer (returns node)
tensor_name=tensor_list[25]
print("Producer node of tensor {}:".format(tensor_name))
print(model.find_producer(tensor_name))
tensor_name=tensor_list[0]
print("Consumer node of tensor {}:".format(tensor_name))
print(model.find_consumer(tensor_name))
print("Producer of tensor 0: %s"%str(model.find_producer("0")))
```
%% Output
Producer node of tensor 60:
input: "59"
input: "58"
output: "60"
op_type: "Mul"
Consumer node of tensor 0:
input: "0"
output: "21"
op_type: "Shape"
Producer of tensor 0: None
%% Cell type:markdown id: tags:
##### Tensor shape
<fontsize="3">Each tensor has a specific shape which can be accessed with the following ModelWrapper helper functions.</font>
%% Cell type:code id: tags:
``` python
# get tensor_shape
print("Shape of tensor 0 is %s"%str(model.get_tensor_shape("0")))
```
%% Output
Shape of tensor 0 is [1, 1, 28, 28]
%% Cell type:markdown id: tags:
<fontsize="3">It is also possible to set the tensor shape with a helper function. The syntax would be the following:
Optionally, the dtype (container datatype) of the tensor can also be specified as third argument. By default it is set to TensorProto.FLOAT.
**Important:** dtype should not be confused with FINN data type, which specifies the quantization annotation. See the remarks about FINN-ONNX in [this notebook](finn-basics.ipynb). It is safest to use floating point tensors as the container data type for best compatibility inside FINN.</font>
%% Cell type:markdown id: tags:
##### Tensor FINN DataType
%% Cell type:markdown id: tags:
<fontsize="3">FINN introduces its [own data types](https://github.com/Xilinx/finn/blob/dev/src/finn/core/datatype.py) because ONNX does not natively support precisions less than 8 bits. FINN is about quantized neural networks, so precision of i.e. 4 bits, 3 bits, 2 bits or 1 bit are of interest. To represent the data within FINN, float tensors are used with additional annotation to specify the quantized data type of a tensor. The following helper functions are about this quantization annotation.</font>
%% Cell type:code id: tags:
``` python
# get tensor data type (FINN data type)
print("The FINN DataType of tensor 0 is "+str(model.get_tensor_datatype("0")))
print("The FINN DataType of tensor 32 is "+str(model.get_tensor_datatype("32")))
```
%% Output
The FINN DataType of tensor 0 is DataType.FLOAT32
The FINN DataType of tensor 32 is DataType.BIPOLAR
%% Cell type:markdown id: tags:
<fontsize="3">In addition to the get_tensor_datatatype() function, the (FINN) datatype of a tensor can be set using the `set_tensor_datatype(tensor_name, datatype)` function.</font>
%% Cell type:markdown id: tags:
##### Tensor initializers
<fontsize="3">Some tensors have *initializers*, like tensors that represent constants or i.e. the trained weight values.
ModelWrapper contains two helper functions for this case, one to determine the current initializer and one to set the initializer of a tensor. If there is no initializer, `None` is returned.</font>
%% Cell type:code id: tags:
``` python
# get tensor initializer
tensor_name=tensor_list[1]
print("Initializer for tensor 33:\n"+str(model.get_initializer("33")))
print("Initializer for tensor 0:\n"+str(model.get_initializer("0")))
```
%% Output
Initializer for tensor 33:
[[ 1. 1. 1. ... 1. 1. -1.]
[ 1. 1. -1. ... 1. 1. -1.]
[-1. 1. -1. ... -1. 1. -1.]
...
[-1. 1. -1. ... -1. -1. 1.]
[ 1. 1. -1. ... 1. 1. -1.]
[-1. 1. 1. ... -1. -1. 1.]]
Initializer for tensor 0:
None
%% Cell type:markdown id: tags:
<fontsize="3">Like for the other tensor helper functions there is a `set_initializer(tensor_name, tensor_value)` function.</font>
%% Cell type:markdown id: tags:
### More helper functions
<fontsize="3">ModelWrapper contains more useful functions, if you are interested please have a look at the [Python code](https://github.com/Xilinx/finn/blob/dev/src/finn/core/modelwrapper.py) directly. Additionally, in the folder notebooks/ a Jupyter notebook about transformation passes and one about analysis passes can be found.</font>
<fontsize="3">ModelWrapper contains more useful functions, if you are interested please have a look at the [Python code](https://github.com/Xilinx/finn/blob/dev/src/finn/core/modelwrapper.py) directly. Additionally, in the folder notebooks/ a Jupyter notebook about transformation passes [FINN-HowToTransformationPass](FINN-HowToTransformationPass.ipynb) and one about analysis passes [FINN-HowToAnalysisPass](FINN-HowToAnalysisPass.ipynb) can be found.</font>
