[notebook - code gen] Added first information about code generation (code gen trafo)

f048807f · auphelia · cbcd38cc · f048807f · f048807f
Commit f048807f authored 5 years ago by auphelia
--- a/notebooks/FCLayer_graph.onnx
+++ b/notebooks/FCLayer_graph.onnx
--- a/notebooks/FINN-CodeGenerationAndCompilation.ipynb
+++ b/notebooks/FINN-CodeGenerationAndCompilation.ipynb
@@ -6,7 +6,21 @@
   "source": [
    "# FINN - Code Generation and Compilation\n",
    "-----------------------------------------------------------------\n",
-    "<font size=\"3\">This notebook is about code generation and compilation to enable execution of FINN custom operation nodes. </font>"
+    "<font size=\"3\">This notebook is about code generation and compilation to enable execution of FINN custom operation nodes. \n",
+    "\n",
+    "Following showSrc function is used to print the source code of function calls in the Jupyter notebook:</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import inspect\n",
+    "\n",
+    "def showSrc(what):\n",
+    "    print(\"\".join(inspect.getsourcelines(what)[0]))"
   ]
  },
  {
@@ -15,7 +29,8 @@
   "source": [
    "## Outline\n",
    "-------------\n",
-    "* <font size=\"3\">Example model</font>\n"
+    "* <font size=\"3\">Example model</font>\n",
+    "* <font size=\"3\">Code generation</font>"
   ]
  },
  {
@@ -30,7 +45,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
@@ -49,7 +64,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
@@ -72,7 +87,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
@@ -95,7 +110,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
@@ -129,7 +144,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
@@ -147,7 +162,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
@@ -170,7 +185,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
@@ -179,7 +194,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
@@ -199,7 +214,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
@@ -220,6 +235,104 @@
    "<iframe src=\"http://0.0.0.0:8081/\" style=\"position: relative; width: 100%;\" height=\"400\"></iframe>"
   ]
  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Code Generation\n",
+    "<font size=\"3\">Code generation is a transformation that can be applied to the model. For more information about transformation passes, see Jupyter Notebook *FINN-HowToTransformPass*.\n",
+    "\n",
+    "The code generation transformation is shown below.</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "class CodeGen(Transformation):\n",
+      "    \"\"\"Code generation for all nodes in model\"\"\"\n",
+      "\n",
+      "    def apply(self, model):\n",
+      "        for node in model.graph.node:\n",
+      "            if node.domain == \"finn\":\n",
+      "                backend_attribute = get_by_name(node.attribute, \"backend\")\n",
+      "                backend_value = backend_attribute.s.decode(\"UTF-8\")\n",
+      "                if backend_value == \"fpgadataflow\":\n",
+      "                    _codegen_single_node(node, model)\n",
+      "        return (model, False)\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "from finn.transformation.fpgadataflow.codegen import CodeGen\n",
+    "showSrc(CodeGen)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font size=\"3\">The transformation passes iterates over all nodes in the model and if `domain=\"finn\"` and `backend=\"fpgadataflow\"` the function `_codegen_single_node()` is executed which is also part of the transformation pass and is shown below. </font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "def _codegen_single_node(node, model):\n",
+      "    \"\"\"Call custom implementation to generate code for single custom node\n",
+      "    and create folder that contains all the generated files\"\"\"\n",
+      "    op_type = node.op_type\n",
+      "    try:\n",
+      "        # lookup op_type in registry of CustomOps\n",
+      "        inst = registry.custom_op[op_type](node)\n",
+      "        # get the path of the code generation directory\n",
+      "        code_gen_dir = inst.get_nodeattr(\"code_gen_dir\")\n",
+      "        # ensure that there is a directory\n",
+      "        if code_gen_dir == \"\" or not os.path.isdir(code_gen_dir):\n",
+      "            code_gen_dir = tmp.mkdtemp(prefix=\"code_gen_\" + str(node.op_type) + \"_\")\n",
+      "            inst.set_nodeattr(\"code_gen_dir\", code_gen_dir)\n",
+      "        # ensure that there is generated code inside the dir\n",
+      "        inst.code_generation(model)\n",
+      "    except KeyError:\n",
+      "        # exception if op_type is not supported\n",
+      "        raise Exception(\"Custom op_type %s is currently not supported.\" % op_type)\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "from finn.transformation.fpgadataflow.codegen import _codegen_single_node\n",
+    "showSrc(_codegen_single_node)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font size=\"3\">An instance of the node is created and checked for the attribute `code_gen_dir`. If the attribute is not set, a temporary directory is created and the attribute is set accordingly. \n",
+    "\n",
+    "Then the `code_generation()` function of the instance is called. If an error occurs during this process, this is probably due to the fact that the selected CustomOp is not yet supported.</font>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font size=\"3\">The following description of the code generation within the CustomOp instance may lead to overlaps with the Jupyter notebook *FINN-CustomOps*. </font>"
+   ]
+  },
  {
   "cell_type": "code",
   "execution_count": null,

 %% Cell type:markdown id: tags:

 # FINN - Code Generation and Compilation
 -----------------------------------------------------------------
-<font size="3">This notebook is about code generation and compilation to enable execution of FINN custom operation nodes. </font>
+<font size="3">This notebook is about code generation and compilation to enable execution of FINN custom operation nodes.
+
+Following showSrc function is used to print the source code of function calls in the Jupyter notebook:</font>
+
+%% Cell type:code id: tags:
+
+``` python
+import inspect
+
+def showSrc(what):
+    print("".join(inspect.getsourcelines(what)[0]))
+```

 %% Cell type:markdown id: tags:

 ## Outline
 -------------
 * <font size="3">Example model</font>
+* <font size="3">Code generation</font>

 %% Cell type:markdown id: tags:

 ### Example model
 <font size="3">To show the code generation and compilation of a node, an example model with a streaming fclayer node is first created. To learn more about FINN custom operation nodes, please take a look in notebook *FINN-CustomOps*.

 First TensorProto and helper are imported from ONNX. These functions can be used to create tensors, nodes, graphs and models in ONNX. Additional functions from `util` and the classes `DataType` and `ModelWrapper` are needed. More information about `DataType` and `ModelWrapper` can be found in Jupyter notebook *FINN-ModelWrapper*.</font>

 %% Cell type:code id: tags:

 ``` python
 from onnx import TensorProto, helper
 import finn.core.utils as util
 from finn.core.datatype import DataType
 from finn.core.modelwrapper import ModelWrapper
 ```

 %% Cell type:markdown id: tags:

 <font size="3">Then all parameters, that are needed to create a streaming fclayer, are set. To keep the example clear small values are chosen. </font>

 %% Cell type:code id: tags:

 ``` python
 idt = wdt = odt = DataType.BIPOLAR
 mw = 8
 mh = 8
 pe = 4
 simd = 4
 wmem = mw * mh // (pe * simd)
 nf = mh // pe
 sf = mw // simd
 ```

 %% Cell type:markdown id: tags:

 <font size="3">A `tensor_value_info` is created for all tensors involved. In this case there is one tensor for the weights besides the input and output tensors. Then an input list is created containing the two inputs (`"inp"`and `"weights"`).</font>

 %% Cell type:code id: tags:

 ``` python
 inp = helper.make_tensor_value_info("inp", TensorProto.FLOAT, [1, sf, simd])
 weights = helper.make_tensor_value_info("weights", TensorProto.FLOAT, [mw, mh])
 outp = helper.make_tensor_value_info("outp", TensorProto.FLOAT, [1, nf, pe])
 node_inp_list = ["inp", "weights"]
 ```

 %% Cell type:markdown id: tags:

 <font size="3">Now the node can be created. The operation type is set to `"StreamingFCLayer_Batch"` and the rest of the attributes are set appropriately. The relevant attributes for the activation of the code generation and compilation are:</font>
 * <font size="3">**`domain="finn"`**: specifies that the created node is a FINN-Custom Op</font>
 * <font size="3">**`backend="fpgadataflow"`**: specifies that it is a node that corresponds to a function in the finn-hls library</font>
 * <font size="3">**`code_gen_dir"`**: specifies the path to the directory where the generated c++ files are (is set during code generation)</font>
 * <font size="3">**`executable_path"`**: specifies the path to the executable created after compilation (is set during compilation)</font>

 %% Cell type:code id: tags:

 ``` python
 FCLayer_node = helper.make_node(
        "StreamingFCLayer_Batch",
        node_inp_list,
        ["outp"],
        domain="finn",
        backend="fpgadataflow",
        code_gen_dir="",
        executable_path="",
        resType="ap_resource_lut()",
        MW=mw,
        MH=mh,
        SIMD=simd,
        PE=pe,
        WMEM=wmem,
        TMEM=0,
        inputDataType=idt.name,
        weightDataType=wdt.name,
        outputDataType=odt.name,
 )
 ```

 %% Cell type:markdown id: tags:

 <font size="3"> The node is packed into a graph environment and the inputs and outputs are set.</font>

 %% Cell type:code id: tags:

 ``` python
 graph = helper.make_graph(
        nodes=[FCLayer_node], name="fclayer_graph", inputs=[inp], outputs=[outp]
    )
 ```

 %% Cell type:markdown id: tags:

 <font size="3">A model is now created from the graph, which is then converted into a ModelWrapper object for further processing in FINN. Afterwards the ModelWrapper internal functions can be used to set the FINN data types and the initializer for the weights. Since this is an example, the weights are not taken from the training, but random values are generated using the utility function `gen_finn_dt_tensor()`. This function gets a FINN datatype and a shape and generates a tensor with values of this datatype in the desired shape.</font>

 %% Cell type:code id: tags:

 ``` python
 model = helper.make_model(graph, producer_name="fclayer-model")
 model = ModelWrapper(model)

 model.set_tensor_datatype("inp", idt)
 model.set_tensor_datatype("outp", odt)
 model.set_tensor_datatype("weights", wdt)
 W = util.gen_finn_dt_tensor(wdt, (mw, mh))
 model.set_initializer("weights", W)
 ```

 %% Cell type:markdown id: tags:

 <font size="3">The model is saved and then netron is used to visualize the resulting model. </font>

 %% Cell type:code id: tags:

 ``` python
 model.save("FCLayer_graph.onnx")
 ```

 %% Cell type:code id: tags:

 ``` python
 import netron
 netron.start('FCLayer_graph.onnx', port=8081, host="0.0.0.0")
 ```

 %% Output

    
    Stopping http://0.0.0.0:8081
    Serving 'FCLayer_graph.onnx' at http://0.0.0.0:8081

 %% Cell type:code id: tags:

 ``` python
 %%html
 <iframe src="http://0.0.0.0:8081/" style="position: relative; width: 100%;" height="400"></iframe>
 ```

 %% Output


+%% Cell type:markdown id: tags:
+
+### Code Generation
+<font size="3">Code generation is a transformation that can be applied to the model. For more information about transformation passes, see Jupyter Notebook *FINN-HowToTransformPass*.
+
+The code generation transformation is shown below.</font>
+
+%% Cell type:code id: tags:
+
+``` python
+from finn.transformation.fpgadataflow.codegen import CodeGen
+showSrc(CodeGen)
+```
+
+%% Output
+
+    class CodeGen(Transformation):
+        """Code generation for all nodes in model"""
+    
+        def apply(self, model):
+            for node in model.graph.node:
+                if node.domain == "finn":
+                    backend_attribute = get_by_name(node.attribute, "backend")
+                    backend_value = backend_attribute.s.decode("UTF-8")
+                    if backend_value == "fpgadataflow":
+                        _codegen_single_node(node, model)
+            return (model, False)
+    
+
+%% Cell type:markdown id: tags:
+
+<font size="3">The transformation passes iterates over all nodes in the model and if `domain="finn"` and `backend="fpgadataflow"` the function `_codegen_single_node()` is executed which is also part of the transformation pass and is shown below. </font>
+
+%% Cell type:code id: tags:
+
+``` python
+from finn.transformation.fpgadataflow.codegen import _codegen_single_node
+showSrc(_codegen_single_node)
+```
+
+%% Output
+
+    def _codegen_single_node(node, model):
+        """Call custom implementation to generate code for single custom node
+        and create folder that contains all the generated files"""
+        op_type = node.op_type
+        try:
+            # lookup op_type in registry of CustomOps
+            inst = registry.custom_op[op_type](node)
+            # get the path of the code generation directory
+            code_gen_dir = inst.get_nodeattr("code_gen_dir")
+            # ensure that there is a directory
+            if code_gen_dir == "" or not os.path.isdir(code_gen_dir):
+                code_gen_dir = tmp.mkdtemp(prefix="code_gen_" + str(node.op_type) + "_")
+                inst.set_nodeattr("code_gen_dir", code_gen_dir)
+            # ensure that there is generated code inside the dir
+            inst.code_generation(model)
+        except KeyError:
+            # exception if op_type is not supported
+            raise Exception("Custom op_type %s is currently not supported." % op_type)
+    
+
+%% Cell type:markdown id: tags:
+
+<font size="3">An instance of the node is created and checked for the attribute `code_gen_dir`. If the attribute is not set, a temporary directory is created and the attribute is set accordingly.
+
+Then the `code_generation()` function of the instance is called. If an error occurs during this process, this is probably due to the fact that the selected CustomOp is not yet supported.</font>
+
+%% Cell type:markdown id: tags:
+
+<font size="3">The following description of the code generation within the CustomOp instance may lead to overlaps with the Jupyter notebook *FINN-CustomOps*. </font>
+
 %% Cell type:code id: tags:

 ``` python
 ```