<fontsize="3">This notebook is about code generation and compilation to enable execution of FINN custom operation nodes.
Following showSrc function is used to print the source code of function calls in the Jupyter notebook:</font>
%% Cell type:code id: tags:
``` python
importinspect
defshowSrc(what):
print("".join(inspect.getsourcelines(what)[0]))
```
%% Cell type:markdown id: tags:
## Outline
-------------
*<fontsize="3">Example model</font>
*<fontsize="3">Code generation</font>
%% Cell type:markdown id: tags:
### Example model
<fontsize="3">To show the code generation and compilation of a node, an example model with a streaming fclayer node is first created. To learn more about FINN custom operation nodes, please take a look in notebook *FINN-CustomOps*.
First TensorProto and helper are imported from ONNX. These functions can be used to create tensors, nodes, graphs and models in ONNX. Additional functions from `util` and the classes `DataType` and `ModelWrapper` are needed. More information about `DataType` and `ModelWrapper` can be found in Jupyter notebook *FINN-ModelWrapper*.</font>
%% Cell type:code id: tags:
``` python
fromonnximportTensorProto,helper
importfinn.core.utilsasutil
fromfinn.core.datatypeimportDataType
fromfinn.core.modelwrapperimportModelWrapper
```
%% Cell type:markdown id: tags:
<fontsize="3">Then all parameters, that are needed to create a streaming fclayer, are set. To keep the example clear small values are chosen. </font>
%% Cell type:code id: tags:
``` python
idt=wdt=odt=DataType.BIPOLAR
mw=8
mh=8
pe=4
simd=4
wmem=mw*mh//(pe*simd)
nf=mh//pe
sf=mw//simd
```
%% Cell type:markdown id: tags:
<fontsize="3">A `tensor_value_info` is created for all tensors involved. In this case there is one tensor for the weights besides the input and output tensors. Then an input list is created containing the two inputs (`"inp"`and `"weights"`).</font>
<fontsize="3">Now the node can be created. The operation type is set to `"StreamingFCLayer_Batch"` and the rest of the attributes are set appropriately. The relevant attributes for the activation of the code generation and compilation are:</font>
*<fontsize="3">**`domain="finn"`**: specifies that the created node is a FINN-Custom Op</font>
*<fontsize="3">**`backend="fpgadataflow"`**: specifies that it is a node that corresponds to a function in the finn-hls library</font>
*<fontsize="3">**`code_gen_dir"`**: specifies the path to the directory where the generated c++ files are (is set during code generation)</font>
*<fontsize="3">**`executable_path"`**: specifies the path to the executable created after compilation (is set during compilation)</font>
%% Cell type:code id: tags:
``` python
FCLayer_node=helper.make_node(
"StreamingFCLayer_Batch",
node_inp_list,
["outp"],
domain="finn",
backend="fpgadataflow",
code_gen_dir="",
executable_path="",
resType="ap_resource_lut()",
MW=mw,
MH=mh,
SIMD=simd,
PE=pe,
WMEM=wmem,
TMEM=0,
noActivation=1,
binaryXnorMode=1,
inputDataType=idt.name,
weightDataType=wdt.name,
outputDataType=odt.name,
)
```
%% Cell type:markdown id: tags:
<fontsize="3"> The node is packed into a graph environment and the inputs and outputs are set.</font>
<fontsize="3">A model is now created from the graph, which is then converted into a ModelWrapper object for further processing in FINN. Afterwards the ModelWrapper internal functions can be used to set the FINN data types and the initializer for the weights. Since this is an example, the weights are not taken from the training, but random values are generated using the utility function `gen_finn_dt_tensor()`. This function gets a FINN datatype and a shape and generates a tensor with values of this datatype in the desired shape.</font>
<fontsize="3">Code generation is a transformation that can be applied to the model. For more information about transformation passes, see Jupyter Notebook *FINN-HowToTransformPass*.
The code generation transformation is shown below.</font>
<fontsize="3">The transformation passes iterates over all nodes in the model and if `domain="finn"` and `backend="fpgadataflow"` the function `_codegen_single_node()` is executed which is also part of the transformation pass and is shown below. </font>
# ensure that there is generated code inside the dir
inst.code_generation(model)
except KeyError:
# exception if op_type is not supported
raise Exception("Custom op_type %s is currently not supported." % op_type)
%% Cell type:markdown id: tags:
<fontsize="3">An instance of the node is created and checked for the attribute `code_gen_dir`. If the attribute is not set, a temporary directory is created and the attribute is set accordingly.
Then the `code_generation()` function of the instance is called. If an error occurs during this process, this is probably due to the fact that the selected CustomOp is not yet supported.</font>
%% Cell type:markdown id: tags:
<fontsize="3">The following description of the code generation within the CustomOp instance may lead to overlaps with the Jupyter notebook *FINN-CustomOps*. </font>
"<font size=\"3\">Custom Ops are represented in Finn as ONNX nodes on the one hand and by a CustomOp class on the other hand. This allows easier access to the different attributes and introduces special custom op functions. See below for the standard CustomOp class.</font>"
"<font size=\"3\">Custom Ops are represented in FINN as ONNX nodes on the one hand and by a CustomOp class on the other hand. This allows easier access to the different attributes and introduces special custom op functions. See below for the standard CustomOp class.</font>"
<fontsize="3">This notebook should give a more detailed insight into FINN custom operation nodes. </font>
%% Cell type:markdown id: tags:
<fontsize="3">Following showSrc function is used to print the source code of function calls in the Jupyter notebook: </font>
%% Cell type:code id: tags:
``` python
importinspect
defshowSrc(what):
print("".join(inspect.getsourcelines(what)[0]))
```
%% Cell type:markdown id: tags:
## FINN Custom Ops
---------------------------
<fontsize="3">FINN uses many custom operations (`op_type` in ONNX NodeProto) that are not defined in the ONNX operator schema. These custom nodes are marked with `domain="finn"` in the protobuf to identify them as such. These nodes can represent specific operations that we need for low-bit networks, or operations that are specific to a particular hardware backend.
A very abstract version of a custom op node representing a streaming fc layer is shown below. </font>
%% Cell type:markdown id: tags:
`FCLayer_node = helper.make_node(
"StreamingFCLayer_Batch",
node_inp_list,
node_outp_list,
domain="finn",
backend="fpgadataflow",
code_gen_dir="",
executable_path="",
resType="ap_resource_lut()",
MW=mw,
MH=mh,
SIMD=simd,
PE=pe,
WMEM=wmem,
TMEM=tmem,
inputDataType=FINN-DataType,
weightDataType=FINN-DataType,
outputDataType=FINN-DataType,
inputDataType=<FINN DataType>,
weightDataType=<FINN DataType>,
outputDataType=<FINN DataType>,
ActVal=actval,
binaryXnorMode=<0/1>,
noActivation=<0/1>
)`
%% Cell type:markdown id: tags:
<fontsize="3">Unlike standard nodes, the custom op nodes has several additional attributes. The node is created using the helper function of ONNX. `"StreamingFCLayer_Batch"` describes the op_type, then the inputs and outputs are declared. Since this is a custom op node of FINN, the attribute `domain="finn"` must be set. The streaming fc layer is a custom op from the finn-hls library, this is set in the node using the `backend` attribute. To execute a custom op from the finn-hls library, the corresponding c++ code must be created and an executable must be produced. Where the generated code is stored is specified in the `code_gen_dir` attribute and `executable_path` specifies the path to the produced executable. In addition to the data types of the input and output tensors, the node also contains various other attributes resulting from the parameters of the corresponding finn-hls library function. This will not be discussed here.</font>
%% Cell type:markdown id: tags:
<fontsize="3">Custom Ops are represented in Finn as ONNX nodes on the one hand and by a CustomOp class on the other hand. This allows easier access to the different attributes and introduces special custom op functions. See below for the standard CustomOp class.</font>
<fontsize="3">Custom Ops are represented in FINN as ONNX nodes on the one hand and by a CustomOp class on the other hand. This allows easier access to the different attributes and introduces special custom op functions. See below for the standard CustomOp class.</font>
%% Cell type:code id: tags:
``` python
fromfinn.custom_opimportCustomOp
showSrc(CustomOp)
```
%% Output
class CustomOp(ABC):
def __init__(self, onnx_node):
super().__init__()
self.onnx_node = onnx_node
def get_nodeattr(self, name):
"""Get a node attribute by name. Data is stored inside the ONNX node's
AttributeProto container. Attribute must be part of get_nodeattr_types.
Default value is returned if attribute is not set."""
- dtype indicates which member of the ONNX AttributeProto
will be utilized
- require indicates whether this attribute is required
- default_val indicates the default value that will be used if the
attribute is not set
"""
pass
@abstractmethod
def make_shape_compatible_op(self):
"""Returns a standard ONNX op which is compatible with this CustomOp
for performing shape inference."""
pass
@abstractmethod
def infer_node_datatype(self, model):
"""Set the DataType annotations corresponding to the outputs of this
node."""
pass
@abstractmethod
def execute_node(self, context, graph):
"""Execute this CustomOp instance, given the execution context and
ONNX graph."""
pass
%% Cell type:markdown id: tags:
<fontsize="3">When instantiating the class, the ONNX node is passed to access all attributes of the node within the class. This is accompanied by the functions `get_nodeattr()`and `set_nodeattr()`, which each instance of this class has. Furthermore 4 abstract methods are implemented, which are described in more detail in the comments in the code. </font>
%% Cell type:markdown id: tags:
<fontsize="3">If it is a node from the finn-hls library another class is used which is derived from the CustomOp class:</font>
<fontsize="3">When creating an instance of this class, a template is introduced, which forms the layout for the c++ code to execute the node. It has some general constructs, like the inclusion of bnn-library.h, which contains the references to the finn-hls library, and of cnpy.h and npy2apintstream.hpp, which support the transfer of python numpy arrays in c++. The idea of this template is to replace the variables marked with `$ $` with c++ calls during code generation. Then the template can be written into a .cpp file and be compiled.
**`get_nodeattr_types()`**: each instance of the HLSCustomOp class must have the attributes `code_gen_dir` and `executable_path`, since to execute these nodes c++ code must be generated and correspondingly the executables.
</font>
%% Cell type:markdown id: tags:
<fontsize="3">**`code_generation(model)`**: all functions required for code generation are called and the `$ $` variables in the template are replaced accordingly and written into a .cpp file. Almost all of these subfunctions are implemented as abstract methods in the class, so they are completely customized for each custom op node. A special function is `generate_params()`. This is not implemented as an abstract method, but as a normal function, but contains by default only `pass`. This is because some custom op nodes do not have parameters that need to be generated and in this way the function is skipped. For example for a streaming fc layer node a parameter generation is necessary. How such a parameter generation can look like is described in more detail in the course of this notebook.
</font>
%% Cell type:markdown id: tags:
<fontsize="3">**`compile_singlenode_code()`**: To compile the generated code, the compile command must be built. This is done in this function. It creates an instance of the `CppBuilder()` class and assembles the various components for the function. The `.build` function creates the executable and then sets the corresponding attribute. The class `CppBuilder` is a transformation and a more detailed description can be found in Jupyter notebook *FINN-CodeGenerationAndCompilation*.
<fontsize="3">Parameters have to be generated for specific types of HLSCustomOps. For example if the node is a streaming fc layer, there are weights and activation values, which are written to separate .h files and added to the template using `#include`. For streaming fc layer the parameter generation looks like this:
