From a65e08b0210e89b6d09aaf7a84e4a18c927566bd Mon Sep 17 00:00:00 2001
From: Yaman Umuroglu <yamanu@xilinx.com>
Date: Mon, 22 Feb 2021 00:14:14 +0100
Subject: [PATCH] Custom op notebook (#283)
* [Notebook] start custom op nb
* [Notebook] update custom_op notebook
* [Notebook] update the custom op notebook to reflect new reg system
* [Notebook] Added more descriptive text into the notebook (#278)
* [Web] update publications
* Build step: measure rtlsim performance (#262)
* [Build] introduce step_measure_rtlsim_performance
* [Docs] document the rtlsim performance step
* [Docs] add comments on CPU/RAM/storage recommandations
* [Build, Test] enable rtlsim perf as part of build test too
* [Build] print where intermediate outputs are generated
* [Docs] remove ghpages content, add note
* [Docker] use -c continue for build_custom mode
* [Docs] minor fixes
* Driver data packing + improvements (#261)
* [Driver] add driver_base.py as own template file + comments
* [Driver] also move validation to won template + use in transform
* [Driver] more comments
* [Driver] suggested updates from PYNQ team + async mode exec_on_buffers
* [Driver] allow smaller batchsize in execute_on_buffers
* [Driver] optimize buffer alloc a bit
* [Driver] wait condition fix
* [Deps] update finn-base
* [Deps] update finn-base
* [Driver] enable fast_mode, expose more benchmarks
* [Deps] update finn-base
* [Driver] handle case with no rt weights
* [Build] auto-exit build_custom if no errors
* [Driver] get Alveo clock during test
* Add report_utilization to stitched project tcl script (#243)
* [Vitis] use -mode batch for report gen Vivado launch
* Feature/cybersecurity notebook (#259)
* Created cybersecurity notebook and downloaded data with wget
* Reorganized the cybersecurity notebook
* Read the entire dataset as a pytorch tensor and trained a simple MLP on the UNSW_NB15 dataset
* small changes to markdown
* Training and testing have the same integer encoding
* Added debugger tool
* added loss visualization
* aded 1hot encoder and separated dataloader
Added 1-hot encoder with scikit learn.
Seperated the dataloader into a python file.
* changes to get 99.998% accuracy
* changed loss plot
* accuracy at 70% after 50 epochs
However, loss is not ok
* got 75% accuracy
* see loss
* added scheduled statistics
* added iterator over all possible parameters
* updates on automation
debugging model error
* Delete cybersecurity-checkpoint.ipynb
* Delete cybersecurity_2-checkpoint.ipynb
* Delete exemplofixe-checkpoint.ipynb
* Delete UNSW_NB15_testing-set.csv
* Delete UNSW_NB15_train.csv
* Delete UNSW_NB15_training-set.csv
* Delete UNSW_NB15_val.csv
* general cleanup
* general cleanup
* updates with 75.7238% accuracy
* added quantization of the dataset
* debugging the quantization of the dataset
* added plots for the debugging
* debugging the quantization. Show the differences
* update debugging
* updates on debugging
* updates on debugging quantization
* updates on debug, uint32 df added
* updates on debugging
* added 2 pictures for the debugging
* Added quantization of the dataset
* aded results for training the model with the quantized dataset
* added quantization of the dataset
* added new notebook with model definition with brevitas
* cleaning up documents
* modified the model definition
* Changed the loss function
* Added debug with pdb
* Successfully created neural network with Brevitas
* Correctly quantize the dataset
* Added export of onnx model
* Added FINN validation of the Brevitas model
* improved dataloader
* general cleanup
* General Cleanup
* General Cleanup
* Completed verifying the FINN model against Brevitas
* Added new layer to MLP - Debugging
* verified that the MLP model with new layer (QuantIdentity) outputs the same in Brevitas and in FINN for all 82332 test inputs
* verified that the MLP model with new layer (QuantIdentity) outputs the same in Brevitas and in FINN for all 82332 test inputs
* verified that model with new layer (QuantIdentity) outputs the same in Brevitas and in FINN for all 82332 inputs
* verified that model with new layer and input shifted to accept {-1,1}, outputs the same in Brevitas (input is in {-1,+1}) and in FINN (input is in {-1,+1}) for the 82332 inputs
* General cleanup and added text
* General cleanup: improved text
* General cleanup: fixed text typos
* General cleanup: Added text
* Delete cybersecurity.ipynb
* Delete dataloader.py
* Rename cybersecurity_Brevitas_1bit.ipynb to 1-cybersecurity-Brevitas-1bit.ipynb
* Rename cybersecurity_Brevitas_Verification.ipynb to 2-cybersecurity-finn-verification.ipynb
* Added the last notebook
* Added last notebook describing the finn build
* Added changed parameters to see differences
* Added changed parameters and added text
* Fixed typo
* [Notebooks] reorganize into folders, add README for cybsec
* [Notebook] add license header and refs to cybsec dataset quantizer
* [Notebooks] rename cybsec notebook files
* [Notebook] first pass thru cybsec part 1
* [Notebook] refactor more of cybsec part 1
* [Notebook] add option to use pretrained weights
* fixed outline and typo
* [Notebook] update cybsec notebook #2 and gitignore
* [Notebook] start refactoring cybsec part 3
* [ConvertToHLS] allow out_scale=2 for bipolar MT
* [Build] add alternative set of steps for estimation only
* [Transform] attempt to handle padding for IODMAs
* [Transform] explicitly ignore IODMA nodes for InsertDWC
* [Notebook] full pass over cybsec notebook 3
* [Util] move vivado utils into finn-base
* fixed outline
* [HLSCustomOp] better err msg on ipgen failure
Co-authored-by: Alina Vasilciuc <alinav@xlnx.xilinx.com>
Co-authored-by: Yaman Umuroglu <yamanu@xilinx.com>
* [Notebook] add README, remove mobilenet notebook
* [Docs] round of updates
* [Docs] make stack image local
* [Docs] apidocs updates
* [Docs] update README for v0.5b
* [Release] merge dev into master for v0.5b
* [Docs] bring back missing img
* [Docs] bring back missing images
* Switch hlslib comparison functions (#263)
* [Test] add lfc to end2end tests
* [Deps] update hlslib to get comp:: fxns
* [HLS] use comp:: comparators instead of std::
* [VVAU] hlslib now uses inner prod dim instead of K
* [Thres] manually workaround vivado_hls bug for T[0][0]=0
* [Infra] add .vscode to .gitignore
* [Build] separate HLS codegen and ipgen steps (#265)
* [Docs] first version of developer docs
* [Notebook] fix broken resource paths
* Fix Im2Col attributes for newer finn-base (#276)
* [Deps] update finn-base
* [Refactor] fix im2col props for updated finn-base
* [Docs ] Add FAQ page to the Documentation (#273)
* [Docs] Add FAQ page into the Docs
* [Docs] some minor text format changes.
* [Docs] updates to FAQ
Co-authored-by: Yaman Umuroglu <yaman.umuroglu@xilinx.com>
* [Notebook] Added more descriptive text into the notebook
* Fix ZCU102 support in Vivado shell project (#280)
* [ConvertToHLS] fix conversion for bipolar outputs
* [Notebook] updates to custom op notebook
Co-authored-by: Yaman Umuroglu <yamanu@xilinx.com>
Co-authored-by: Tobi-Alonso <tobi.alonso@gmail.com>
Co-authored-by: alinavalinav <60705229+alinavalinav@users.noreply.github.com>
Co-authored-by: Alina Vasilciuc <alinav@xlnx.xilinx.com>
Co-authored-by: Yaman Umuroglu <yaman.umuroglu@xilinx.com>
Co-authored-by: Felix Jentzsch <45395194+fpjentzsch@users.noreply.github.com>
Co-authored-by: jalezeta <51440887+jalezeta@users.noreply.github.com>
Co-authored-by: Tobi-Alonso <tobi.alonso@gmail.com>
Co-authored-by: alinavalinav <60705229+alinavalinav@users.noreply.github.com>
Co-authored-by: Alina Vasilciuc <alinav@xlnx.xilinx.com>
Co-authored-by: Felix Jentzsch <45395194+fpjentzsch@users.noreply.github.com>
---
notebooks/advanced/2_custom_op.ipynb | 923 +++++++++++++++++++++++++++
1 file changed, 923 insertions(+)
create mode 100644 notebooks/advanced/2_custom_op.ipynb
diff --git a/notebooks/advanced/2_custom_op.ipynb b/notebooks/advanced/2_custom_op.ipynb
new file mode 100644
index 000000000..7d7bc5c50
--- /dev/null
+++ b/notebooks/advanced/2_custom_op.ipynb
@@ -0,0 +1,923 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Introduction to custom ops in FINN\n",
+ "\n",
+ "Suppose that you want to introduce a new (custom) operation type into the FINN. Custom operations in FINN are useful for a variety of things ranging from code generation to functional verification. This is achieved by creating a new Python module for your custom operation that fulfills certain interface specifications.\n",
+ "\n",
+ "One thing to point out before we start is that **these custom operations are generic** and not really tied to e.g. Vivado HLS or few-bit quantization. As you will see in this notebook, it's possible to provide arbitrary Python/C/C++/... execution and code generation paths for custom nodes.\n",
+ "\n",
+ "## The CustomOp base class\n",
+ "\n",
+ "Subclasses of `CustomOp` provide a way of providing custom functionality for ONNX nodes in FINN.\n",
+ "This is the base class for every custom op node used in the framework, so you must create subclasses of `CustomOp` to provide execution, code generation and other functionalities in FINN.\n",
+ "\n",
+ "Let's start by looking at the `CustomOp` base class itself, which lives in the `finn-base` repository. You can view it [here](https://github.com/Xilinx/finn-base/blob/dev/src/finn/custom_op/base.py). Note that the `finn` Docker container already has `finn-base` set up as a dependency.\n",
+ "\n",
+ "Some points of importance:\n",
+ "\n",
+ "1. `CustomOp` instances (in Python) are not meant to store any data, only provide functionality on top of data stored in ONNX. Each `CustomOp` instance has a member `self.onnx_node` which gives access to the ONNX `NodeProto` with attributes. There is also a custom attribute setter/getter system in `CustomOp` to make this process easier.\n",
+ "\n",
+ "2. `CustomOp` subclasses need to implement the methods below (those not starting with underscore).\n",
+ "\n",
+ "3. To be discoverable in the custom op register, `CustomOp` subclasses must set the `domain` field to the name of the Python module they appear in. For instance, to use the custom `Im2Col` op type from [here](https://github.com/Xilinx/finn-base/blob/dev/src/finn/custom_op/general/im2col.py), the ONNX node must use `domain=finn.custom_op.general` since its module is located at `finn/custom_op/general/im2col.py`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "['__abstractmethods__',\n",
+ " '__class__',\n",
+ " '__delattr__',\n",
+ " '__dict__',\n",
+ " '__dir__',\n",
+ " '__doc__',\n",
+ " '__eq__',\n",
+ " '__format__',\n",
+ " '__ge__',\n",
+ " '__getattribute__',\n",
+ " '__gt__',\n",
+ " '__hash__',\n",
+ " '__init__',\n",
+ " '__init_subclass__',\n",
+ " '__le__',\n",
+ " '__lt__',\n",
+ " '__module__',\n",
+ " '__ne__',\n",
+ " '__new__',\n",
+ " '__reduce__',\n",
+ " '__reduce_ex__',\n",
+ " '__repr__',\n",
+ " '__setattr__',\n",
+ " '__sizeof__',\n",
+ " '__str__',\n",
+ " '__subclasshook__',\n",
+ " '__weakref__',\n",
+ " '_abc_cache',\n",
+ " '_abc_negative_cache',\n",
+ " '_abc_negative_cache_version',\n",
+ " '_abc_registry',\n",
+ " 'execute_node',\n",
+ " 'get_nodeattr',\n",
+ " 'get_nodeattr_allowed_values',\n",
+ " 'get_nodeattr_def',\n",
+ " 'get_nodeattr_types',\n",
+ " 'infer_node_datatype',\n",
+ " 'make_shape_compatible_op',\n",
+ " 'set_nodeattr',\n",
+ " 'verify_node']"
+ ]
+ },
+ "execution_count": 1,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from finn.custom_op.base import CustomOp\n",
+ "dir(CustomOp)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## A Simple CustomOp Example\n",
+ "\n",
+ "Let's make a simple CustomOp that raises its input to a given exponent (specified as attribute). For now it'll only work in Python, but later we'll add C++ execution capability too."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from onnx import helper\n",
+ "import numpy as np\n",
+ "\n",
+ "class MyPythonPowerOp(CustomOp):\n",
+ " \n",
+ " # here we use the CustomOp attribute system to make it easier\n",
+ " # to set/get custom attributes on this node\n",
+ " def get_nodeattr_types(self):\n",
+ " return {\n",
+ " # each entry is:\n",
+ " # name of attribute : (dtype, required, default value)\n",
+ " # dtype follows the ONNX attribute protobuf so\n",
+ " # \"i\" is int, \"s\" is string, \"f\" is float,\n",
+ " # \"ints\" is a list of integers...\n",
+ " # also good practice to document what each attribute does here:\n",
+ " \n",
+ " # which integer power to raise the input to\n",
+ " \"exponent\" : (\"i\", True, 0),\n",
+ " # execution mode : currently only python\n",
+ " \"exec_mode\" : (\"s\", True, \"python\"),\n",
+ " }\n",
+ " \n",
+ " # return an ONNX node that has the same shape inference behavior\n",
+ " # here we want in shape = out shape, so we use the ONNX ReLU\n",
+ " # node to mimic its shape inference behavior\n",
+ " # we have access to the entire ModelWrapper to help make this decision\n",
+ " # (the parameter called model)\n",
+ " def make_shape_compatible_op(self, model):\n",
+ " node = self.onnx_node\n",
+ " # make a Relu node connected to the same in-out tensors to get\n",
+ " # shape inference\n",
+ " # a general-purpose alternative is to use a Constant node that \n",
+ " # produces the desired shape\n",
+ " return helper.make_node(\"Relu\", [node.input[0]], [node.output[0]])\n",
+ "\n",
+ " # used for FINN DataType inference: set the output tensors' datatypes\n",
+ " # accordingly for this node\n",
+ " # here we assume input datatype = output datatype\n",
+ " # we have access to the entire ModelWrapper to help make this decision\n",
+ " # (the parameter called model)\n",
+ " def infer_node_datatype(self, model):\n",
+ " node = self.onnx_node\n",
+ " # data type stays the same\n",
+ " dtype = model.get_tensor_datatype(node.input[0])\n",
+ " model.set_tensor_datatype(node.output[0], dtype)\n",
+ " \n",
+ " # execute this node\n",
+ " # context: used for both input and output, dictionary of named\n",
+ " # tensors\n",
+ " # graph: the ONNX GraphProto (ModelWrapper.graph), generally \n",
+ " # not needed to execute a single node\n",
+ " def execute_node(self, context, graph):\n",
+ " exec_mode = self.get_nodeattr(\"exec_mode\")\n",
+ " if exec_mode == \"python\":\n",
+ " # get names of node input and output tensors\n",
+ " i_name = self.onnx_node.input[0]\n",
+ " o_name = self.onnx_node.output[0]\n",
+ " # grab input tensor from context\n",
+ " i_tensor = context[i_name]\n",
+ " # get which power to raise to from attribute\n",
+ " expnt = self.get_nodeattr(\"exponent\")\n",
+ " # compute and put output into context\n",
+ " o_tensor = np.power(i_tensor, expnt)\n",
+ " context[o_name] = o_tensor\n",
+ " else:\n",
+ " raise Exception(\"Only python exec_mode is supported\")\n",
+ " \n",
+ " # can use to do a sanity check of all the node's properties\n",
+ " # optional, not implemented here\n",
+ " def verify_node(self):\n",
+ " pass\n",
+ " \n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "To make sure our custom op is available, it needs to be registered. The best practice for this is to create a submodule under `finn.custom_op` which includes a `custom_op` dictionary that maps strings (op names) to classes (op implementations). Since we're in a Jupyter notebook we'll just hijack it at runtime like this:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import finn.custom_op.general as general\n",
+ "general.custom_op[\"MyPythonPowerOp\"] = MyPythonPowerOp"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can see which custom ops are registered under this submodule by looking at the dictionary:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'DebugMarker': finn.custom_op.general.debugmarker.DebugMarker,\n",
+ " 'QuantAvgPool2d': finn.custom_op.general.quantavgpool2d.QuantAvgPool2d,\n",
+ " 'MaxPoolNHWC': finn.custom_op.general.maxpoolnhwc.MaxPoolNHWC,\n",
+ " 'StreamingDataflowPartition': finn.custom_op.general.streamingdataflowpartition.StreamingDataflowPartition,\n",
+ " 'MultiThreshold': finn.custom_op.general.multithreshold.MultiThreshold,\n",
+ " 'XnorPopcountMatMul': finn.custom_op.general.xnorpopcount.XnorPopcountMatMul,\n",
+ " 'Im2Col': finn.custom_op.general.im2col.Im2Col,\n",
+ " 'MyPythonPowerOp': __main__.MyPythonPowerOp}"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "general.custom_op"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Let's Try Out our CustomOp\n",
+ "\n",
+ "We'll manually build a small ONNX graph containing our node in order to try out some of the functionality. This would normally go into the unit test for this CustomOp. \n",
+ "\n",
+ "The graph is built by first specifying the input/output tensor information (name, type, shape). Then,the custom node is generated; which is later used to generate the graph along the input/output tensor information. The model is built using the graph. Finally, the model is wrapped around using the ModelWrapper function from FINN."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from finn.core.modelwrapper import ModelWrapper\n",
+ "from onnx import TensorProto\n",
+ "\n",
+ "def make_graph(ishape, exp, op_type = \"MyPythonPowerOp\"):\n",
+ " inp = helper.make_tensor_value_info(\n",
+ " \"inp\", TensorProto.FLOAT, ishape\n",
+ " )\n",
+ " outp = helper.make_tensor_value_info(\n",
+ " \"outp\", TensorProto.FLOAT, ishape\n",
+ " )\n",
+ "\n",
+ " custom_node = helper.make_node(\n",
+ " # op type string in ONNX, what we used to register the custom op\n",
+ " op_type,\n",
+ " # name of input tensor\n",
+ " [\"inp\"],\n",
+ " # name of output tensor\n",
+ " [\"outp\"],\n",
+ " # specify domain s.t. FINN can find our op under this submodule\n",
+ " domain=\"finn.custom_op.general\",\n",
+ " # set up attributes\n",
+ " exponent = int(exp),\n",
+ " exec_mode = \"python\"\n",
+ " )\n",
+ "\n",
+ " graph = helper.make_graph(\n",
+ " nodes=[custom_node], name=\"custom_graph\", inputs=[inp], outputs=[outp]\n",
+ " )\n",
+ " model = helper.make_model(graph, producer_name=\"custom-model\")\n",
+ " return ModelWrapper(model)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, we specify the input tensor shape and we generate the graph using the function we have just created. The input tensor shape and the exponent value are passed as parameters. These parameters are used to generate our model, graph and custom node using the `MyPythonPowerOp` operation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[input: \"inp\"\n",
+ "output: \"outp\"\n",
+ "op_type: \"MyPythonPowerOp\"\n",
+ "attribute {\n",
+ " name: \"exec_mode\"\n",
+ " s: \"python\"\n",
+ " type: STRING\n",
+ "}\n",
+ "attribute {\n",
+ " name: \"exponent\"\n",
+ " i: 2\n",
+ " type: INT\n",
+ "}\n",
+ "domain: \"finn.custom_op.general\"\n",
+ "]"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# generate a small graph with our custom op\n",
+ "input_shape = (1, 2, 4)\n",
+ "ret_model = make_graph(input_shape, 2)\n",
+ "ret_model.model.graph.node"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We generate a random tensor based on the `input_shape` defined before. See the shape and values of the `random_input` below and the datatype. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[-6., 2., -3., -6.],\n",
+ " [-6., 0., 1., -2.]]], dtype=float32)"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from finn.core.datatype import DataType\n",
+ "from finn.util.basic import gen_finn_dt_tensor\n",
+ "\n",
+ "# generate a random input of e.g signed 4-bit values\n",
+ "random_input = gen_finn_dt_tensor(DataType.INT4, input_shape)\n",
+ "random_input\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Just generate an Input Dictionary with the random values just generated. Then we execute the model using the model and random values just generated. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'outp': array([[[36., 4., 9., 36.],\n",
+ " [36., 0., 1., 4.]]], dtype=float32)}"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from finn.core.onnx_exec import execute_onnx\n",
+ "\n",
+ "# run with FINN's execute_onnx\n",
+ "inp_dict = {\"inp\" : random_input}\n",
+ "ret = execute_onnx(ret_model, inp_dict)\n",
+ "ret"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Done! We have just executed the model that uses our custom operation. The result should be the input number to the power of 2."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## A CustomOp with C++ Generation\n",
+ "\n",
+ "We can write our CustomOps in C++ for instance and generate a model the same way we have done it previously. This can be done through python bindings that let us call C++ code from python. In fact, we will compile the C++ code and execute it from python. \n",
+ "\n",
+ "The following class is based on the `MyPythonPowerOp` class previously written. We are adding a new attribute `codegen_dir` into the `get_nodeattr_types` function that specifies the directory for the generated C++ code, building script and executable application.\n",
+ "\n",
+ "We define a new function that `my_custom_cpp_gen` that writes the C++ code into a file and builds it. Finally the `execute_node` function is modified to support the C++ execution of the CustomOp. The `c++` branch of the if-else statements first flattens the input tensor and writes it into the \"input.txt\" file. Then the C++ compiled application is executed using bash commands. The application reads the \".txt\" file, calculates the power value based on the exponent, and writes the result back into the \"output.txt\" file. Then the result of the ouput file is read and reshaped back into the original shape. Finally, the result is written into the `context` dictionary"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from finn.util.basic import make_build_dir, CppBuilder\n",
+ "import subprocess\n",
+ "\n",
+ "# derive from our previous example\n",
+ "class MyMixedPowerOp(MyPythonPowerOp):\n",
+ " \n",
+ " # here we use the CustomOp attribute system to make it easier\n",
+ " # to set/get custom attributes on this node\n",
+ " def get_nodeattr_types(self):\n",
+ " return {\n",
+ " # each entry is:\n",
+ " # name of attribute : (dtype, required, default value)\n",
+ " # dtype follows the ONNX attribute protobuf so\n",
+ " # \"i\" is int, \"s\" is string, \"f\" is float,\n",
+ " # \"ints\" is a list of integers...\n",
+ " # also good practice to document what each attribute does here:\n",
+ " \n",
+ " # which integer power to raise the input to\n",
+ " \"exponent\" : (\"i\", True, 0),\n",
+ " # execution mode : python or c++\n",
+ " \"exec_mode\" : (\"s\", True, \"python\"),\n",
+ " # code generation directory\n",
+ " \"codegen_dir\" : (\"s\", False, \"\"),\n",
+ " }\n",
+ " \n",
+ " def my_custom_cpp_gen(self):\n",
+ " codegen_dir = make_build_dir(prefix=\"my_custom_op\")\n",
+ " # set attribute for codegen dir\n",
+ " self.set_nodeattr(\"codegen_dir\", codegen_dir)\n",
+ " # generate some C++ code\n",
+ " cpp_code = \"\"\"\n",
+ "#include <iostream>\n",
+ "#include <fstream>\n",
+ "using namespace std;\n",
+ "#define EXPONENT %d\n",
+ "\n",
+ "int main(int argc, char **argv) {\n",
+ " ifstream infile(\"input.txt\");\n",
+ " ofstream outfile(\"output.txt\");\n",
+ " \n",
+ " float elem;\n",
+ " while (infile >> elem)\n",
+ " {\n",
+ " float res = 1.0;\n",
+ " for(int i=0; i < EXPONENT; i++) {\n",
+ " res *= elem;\n",
+ " }\n",
+ " outfile << res << \"\\\\n\";\n",
+ " }\n",
+ "\n",
+ " return 0;\n",
+ "}\n",
+ " \"\"\" % (self.get_nodeattr(\"exponent\"))\n",
+ " with open(codegen_dir+\"/top.cpp\", \"w\") as f:\n",
+ " f.write(cpp_code)\n",
+ " builder = CppBuilder()\n",
+ " # to enable additional debug features please uncommand the next line\n",
+ " builder.append_includes(\"--std=c++11\")\n",
+ " builder.append_includes(\"-O3\")\n",
+ " builder.append_sources(codegen_dir + \"/*.cpp\")\n",
+ " builder.set_executable_path(codegen_dir + \"/node_model\")\n",
+ " builder.build(codegen_dir)\n",
+ " \n",
+ " # execute this node\n",
+ " # context: used for both input and output, dictionary of named\n",
+ " # tensors\n",
+ " # graph: the ONNX GraphProto (ModelWrapper.graph), generally \n",
+ " # not needed to execute a single node\n",
+ " def execute_node(self, context, graph):\n",
+ " exec_mode = self.get_nodeattr(\"exec_mode\")\n",
+ " # get names of node input and output tensors\n",
+ " i_name = self.onnx_node.input[0]\n",
+ " o_name = self.onnx_node.output[0]\n",
+ " # grab input tensor from context\n",
+ " i_tensor = context[i_name]\n",
+ " # get which power to raise to from attribute\n",
+ " expnt = self.get_nodeattr(\"exponent\")\n",
+ " if exec_mode == \"python\":\n",
+ " # compute and put output into context\n",
+ " o_tensor = np.power(i_tensor, expnt)\n",
+ " context[o_name] = o_tensor\n",
+ " elif exec_mode == \"c++\":\n",
+ " build_dir = self.get_nodeattr(\"codegen_dir\")\n",
+ " # save input as txt, could preprocess, change layout etc..\n",
+ " np.savetxt(build_dir+\"/input.txt\", i_tensor.flatten())\n",
+ " bash_command = [\"./node_model\"]\n",
+ " proc_run = subprocess.Popen(bash_command, cwd=build_dir, stdout=subprocess.PIPE)\n",
+ " proc_run.communicate()\n",
+ " o_tensor = np.loadtxt(build_dir+\"/output.txt\")\n",
+ " o_tensor = o_tensor.reshape(i_tensor.shape)\n",
+ " context[o_name] = o_tensor\n",
+ " else:\n",
+ " raise Exception(\"Only python and c++ exec_mode is supported\")\n",
+ " \n",
+ " # can use to do a sanity check of all the node's properties\n",
+ " # optional, not implemented here\n",
+ " def verify_node(self):\n",
+ " pass\n",
+ " \n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We just register the new CustomOp the same way as we did before. Then, we create another graph using the same function `make_graph` as before. We can see the node containing the custom operation printed below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[input: \"inp\"\n",
+ "output: \"outp\"\n",
+ "op_type: \"MyMixedPowerOp\"\n",
+ "attribute {\n",
+ " name: \"exec_mode\"\n",
+ " s: \"python\"\n",
+ " type: STRING\n",
+ "}\n",
+ "attribute {\n",
+ " name: \"exponent\"\n",
+ " i: 2\n",
+ " type: INT\n",
+ "}\n",
+ "domain: \"finn.custom_op.general\"\n",
+ "]"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# register our new op\n",
+ "general.custom_op[\"MyMixedPowerOp\"] = MyMixedPowerOp\n",
+ "\n",
+ "# make graph with new op\n",
+ "mixedop_graph = make_graph(input_shape, 2, op_type = \"MyMixedPowerOp\")\n",
+ "mixedop_graph.graph.node"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We just print all the functions inside the CustomOp, the default C++ code directory and the `exec_mode` attribute."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Available functions: ['__abstractmethods__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_abc_cache', '_abc_negative_cache', '_abc_negative_cache_version', '_abc_registry', 'execute_node', 'get_nodeattr', 'get_nodeattr_allowed_values', 'get_nodeattr_def', 'get_nodeattr_types', 'infer_node_datatype', 'make_shape_compatible_op', 'my_custom_cpp_gen', 'onnx_node', 'set_nodeattr', 'verify_node']\n",
+ "codegen_dir: \n",
+ "exec_mode: python\n"
+ ]
+ }
+ ],
+ "source": [
+ "from finn.custom_op.registry import getCustomOp\n",
+ "\n",
+ "# get FINN wrapper for this node, with all the functionality\n",
+ "op_inst = getCustomOp(mixedop_graph.model.graph.node[0])\n",
+ "print(\"Available functions: \" + str(dir(op_inst)))\n",
+ "# query some attributes\n",
+ "print(\"codegen_dir: \" + op_inst.get_nodeattr(\"codegen_dir\"))\n",
+ "print(\"exec_mode: \" + op_inst.get_nodeattr(\"exec_mode\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Implement a code generation transformation\n",
+ "\n",
+ "We define a local transformation function that transforms a specific model by accessing and modifying the attributes of the specified node. It will execute the `my_custom_cpp_gen` function from the node \"MyMixedPowerOp\" if the \"codegen_dir\" is not present."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#from finn.transformation.base import Transformation\n",
+ "# can derive from NodeLocalTransformation for faster (parallel) execution\n",
+ "from finn.transformation.base import NodeLocalTransformation\n",
+ "import os\n",
+ "\n",
+ "class MyNodeLocalCodeGen(NodeLocalTransformation):\n",
+ " \n",
+ " # will get called (possibly in parallel) for each node\n",
+ " def applyNodeLocal(self, node):\n",
+ " # keep track whether we changed anything\n",
+ " modified_graph = False\n",
+ " # check node type before we do anything\n",
+ " if node.op_type == \"MyMixedPowerOp\":\n",
+ " # get FINN wrapper for this node, with all the functions\n",
+ " op_inst = getCustomOp(node)\n",
+ " if not os.path.isdir(op_inst.get_nodeattr(\"codegen_dir\")):\n",
+ " # call the codegen function we defined\n",
+ " # this will modify the underlying node by setting attribute\n",
+ " op_inst.my_custom_cpp_gen()\n",
+ " # codegen function modifies attribute\n",
+ " modified_graph = True\n",
+ " # important: must return modified_graph = False at some point\n",
+ " # otherwise transformation will run in infinite loop!\n",
+ " return (node, modified_graph)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Apply the transformation into the model we had before. The returned model is the same input model after applying the specified transformation. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "mixedop_graph_new = mixedop_graph.transform(MyNodeLocalCodeGen())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Print the \"codegen_dir\" attribute from CustomOp node."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "/tmp/finn_dev_jalezeta/my_custom_oppaxpincq\n"
+ ]
+ }
+ ],
+ "source": [
+ "new_op_inst = getCustomOp(mixedop_graph_new.graph.node[0])\n",
+ "codegen_dir = new_op_inst.get_nodeattr(\"codegen_dir\")\n",
+ "print(codegen_dir)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can see that the `codegen_dir` folder contains the compile script, compiled application and the C++ source file:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "compile.sh node_model\ttop.cpp\r\n"
+ ]
+ }
+ ],
+ "source": [
+ "! ls {codegen_dir}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's view the content of the C++ source file:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\r\n",
+ "#include <iostream>\r\n",
+ "#include <fstream>\r\n",
+ "using namespace std;\r\n",
+ "#define EXPONENT 2\r\n",
+ "\r\n",
+ "int main(int argc, char **argv) {\r\n",
+ " ifstream infile(\"input.txt\");\r\n",
+ " ofstream outfile(\"output.txt\");\r\n",
+ " \r\n",
+ " float elem;\r\n",
+ " while (infile >> elem)\r\n",
+ " {\r\n",
+ " float res = 1.0;\r\n",
+ " for(int i=0; i < EXPONENT; i++) {\r\n",
+ " res *= elem;\r\n",
+ " }\r\n",
+ " outfile << res << \"\\n\";\r\n",
+ " }\r\n",
+ "\r\n",
+ " return 0;\r\n",
+ "}\r\n",
+ " "
+ ]
+ }
+ ],
+ "source": [
+ "! cat {codegen_dir}/top.cpp"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Manually generate input and run C++ node model\n",
+ "\n",
+ "We will now manually generate the input data and write it into the `input.txt` file. Then, we manually execute the compiled application and finally see the result in the `output.txt` file. \n",
+ "\n",
+ "The purpose of this is mostly to show that there is no \"magic\" happening when FINN is executing our custom op; it's just launching a program. When debugging the execution of your custom op, it's a good idea to keep this in mind -- for instance, you can use `gdb` to debug the internals of the C++ node model here."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! echo \"7.0 8.0 9.0\" > {codegen_dir}/input.txt"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! cd {codegen_dir}; ./node_model"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "49\r\n",
+ "64\r\n",
+ "81\r\n"
+ ]
+ }
+ ],
+ "source": [
+ "! cat {codegen_dir}/output.txt"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! rm {codegen_dir}/*.txt"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Use FINN execution flow\n",
+ "\n",
+ "We'll now trigger the custom node execution from inside FINN, via the custom ONNX execution capabilities which will automatically launch the appropriate handler when a custom node is encountered inside the ONNX graph, in this case launching the compiled C++ program. To do this, we will first generate a random tensor with a pre-specified tensor shape and print it. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[[-8., 4., 7., 2.],\n",
+ " [-5., -1., 2., 0.]]], dtype=float32)"
+ ]
+ },
+ "execution_count": 21,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# generate a random input of e.g signed 4-bit values\n",
+ "random_input = gen_finn_dt_tensor(DataType.INT4, input_shape)\n",
+ "random_input"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We set the CustomOp node attribute to execute in \"Python\" mode. Then, generate an input dictionay with the random input tensor and execute the transformed model using the `execute_onnx`. We print the output to see the results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'outp': array([[[64., 16., 49., 4.],\n",
+ " [25., 1., 4., 0.]]], dtype=float32)}"
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# run with FINN's execute_onnx, custom node will use Python execution\n",
+ "new_op_inst.set_nodeattr(\"exec_mode\", \"python\")\n",
+ "inp_dict = {\"inp\" : random_input}\n",
+ "ret = execute_onnx(mixedop_graph_new, inp_dict)\n",
+ "ret"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We repeat the previous process in \"c++\" execution mode. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'outp': array([[[64., 16., 49., 4.],\n",
+ " [25., 1., 4., 0.]]])}"
+ ]
+ },
+ "execution_count": 23,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# run with FINN's execute_onnx, custom node will use c++ execution\n",
+ "new_op_inst.set_nodeattr(\"exec_mode\", \"c++\")\n",
+ "ret = execute_onnx(mixedop_graph_new, inp_dict)\n",
+ "ret"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.8"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
--
GitLab