diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 5a7f70f8f69293d8dcef9b64c763aa606d5d73f5..126a4ac4b2bee7f3eaaf610646855b48d07b9e32 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -51,7 +51,7 @@ repos:
args: ['--fix=no']
- repo: https://github.com/PyCQA/isort
- rev: 5.10.1
+ rev: 5.12.0
hooks:
- id: isort
diff --git a/fetch-repos.sh b/fetch-repos.sh
index 7078b284a9bbfdebc6bfe5bd8f7d577bdfcacabc..5b060f5bc83b9d9709fc7e34e855543b4690af5a 100755
--- a/fetch-repos.sh
+++ b/fetch-repos.sh
@@ -27,7 +27,7 @@
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-QONNX_COMMIT="ce321742d98f23909a890ed680a9c99640d7aaab"
+QONNX_COMMIT="dd35a8ff49d7225a07ffceeebe25a6361df48349"
FINN_EXP_COMMIT="9cbd2787b5160e2b44e0e8164a0df1457dbd5366"
BREVITAS_COMMIT="a5b71d6de1389d3e7db898fef72e014842670f03"
PYVERILATOR_COMMIT="766e457465f5c0dd315490d7b9cc5d74f9a76f4f"
diff --git a/notebooks/advanced/0_custom_analysis_pass.ipynb b/notebooks/advanced/0_custom_analysis_pass.ipynb
index a4ad32ed7f547a4c035b5cbe4da11ebe2565883a..f8444520c3ded795702420d7f86335d0048ef043 100644
--- a/notebooks/advanced/0_custom_analysis_pass.ipynb
+++ b/notebooks/advanced/0_custom_analysis_pass.ipynb
@@ -137,7 +137,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
diff --git a/notebooks/advanced/1_custom_transformation_pass.ipynb b/notebooks/advanced/1_custom_transformation_pass.ipynb
index e40a534af56352712f20bfb250112aeacfee278f..391e852a71e1109b376abd7bb5d5f9d264d06498 100644
--- a/notebooks/advanced/1_custom_transformation_pass.ipynb
+++ b/notebooks/advanced/1_custom_transformation_pass.ipynb
@@ -233,7 +233,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
diff --git a/notebooks/advanced/2_custom_op.ipynb b/notebooks/advanced/2_custom_op.ipynb
index 051a406708ee7b4bcbd548b39acac000b473c7cf..636da64dd52fab81f8d6a763d199e8e13e9e3cc0 100644
--- a/notebooks/advanced/2_custom_op.ipynb
+++ b/notebooks/advanced/2_custom_op.ipynb
@@ -8,14 +8,14 @@
"\n",
"Suppose that you want to introduce a new (custom) operation type into the FINN compiler. Custom operations in FINN are useful for a variety of things ranging from code generation to functional verification. This is achieved by creating a new Python module for your custom operation that fulfills certain interface specifications.\n",
"\n",
- "One thing to point out before we start is that **these custom operations are generic** and not really tied to e.g. Vivado HLS or few-bit quantization. As you will see in this notebook, it's possible to provide arbitrary Python/C/C++/... execution and code generation paths for custom nodes.\n",
+ "One thing to point out before we start is that **these custom operations are generic** and not really tied to e.g. Vitis HLS or few-bit quantization. As you will see in this notebook, it's possible to provide arbitrary Python/C/C++/... execution and code generation paths for custom nodes.\n",
"\n",
"## The CustomOp base class\n",
"\n",
"Subclasses of `CustomOp` provide a way of providing custom functionality for ONNX nodes in FINN.\n",
"This is the base class for every custom op node used in the framework, so you must create subclasses of `CustomOp` to provide execution, code generation and other functionalities in FINN.\n",
"\n",
- "Let's start by looking at the `CustomOp` base class itself, which lives in the `finn-base` repository. You can view it [here](https://github.com/Xilinx/finn-base/blob/dev/src/finn/custom_op/base.py). Note that the `finn` Docker container already has `finn-base` set up as a dependency.\n",
+ "Let's start by looking at the `CustomOp` base class itself, which lives in the `qonnx` repository. You can view it [here](https://github.com/fastmachinelearning/qonnx/blob/main/src/qonnx/custom_op/base.py). Note that the `finn` Docker container already has `qonnx` set up as a dependency.\n",
"\n",
"Some points of importance:\n",
"\n",
@@ -23,7 +23,7 @@
"\n",
"2. `CustomOp` subclasses need to implement the methods below (those not starting with underscore).\n",
"\n",
- "3. To be discoverable in the custom op register, `CustomOp` subclasses must set the `domain` field to the name of the Python module they appear in. For instance, to use the custom `Im2Col` op type from [here](https://github.com/Xilinx/finn-base/blob/dev/src/finn/custom_op/general/im2col.py), the ONNX node must use `domain=qonnx.custom_op.general` since its module is located at `finn/custom_op/general/im2col.py`."
+ "3. To be discoverable in the custom op register, `CustomOp` subclasses must set the `domain` field to the name of the Python module they appear in. For instance, to use the custom `Im2Col` op type from [here](https://github.com/fastmachinelearning/qonnx/blob/main/src/qonnx/custom_op/general/im2col.py), the ONNX node must use `domain=qonnx.custom_op.general` since its module is located at `qonnx/custom_op/general/im2col.py`."
]
},
{
@@ -130,7 +130,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "To make sure our custom op is available, it needs to be registered. The best practice for this is to create a submodule under `finn.custom_op` which includes a `custom_op` dictionary that maps strings (op names) to classes (op implementations). Since we're in a Jupyter notebook we'll just hijack it at runtime like this:"
+ "To make sure our custom op is available, it needs to be registered. The best practice for this is to create a submodule under `qonnx.custom_op` which includes a `custom_op` dictionary that maps strings (op names) to classes (op implementations). Since we're in a Jupyter notebook we'll just hijack it at runtime like this:"
]
},
{
@@ -658,7 +658,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
diff --git a/notebooks/basics/0_how_to_work_with_onnx.ipynb b/notebooks/basics/0_how_to_work_with_onnx.ipynb
index b6a5a0481574928ef490d8bb55bbbe2bb882b951..35a83ea97b87bbe78ae1ff58a5ee50a0b0420a8f 100644
--- a/notebooks/basics/0_how_to_work_with_onnx.ipynb
+++ b/notebooks/basics/0_how_to_work_with_onnx.ipynb
@@ -24,7 +24,7 @@
"source": [
"### How to create a simple ONNX model\n",
"\n",
- "To explain how to create an ONNX model a simple example with mathematical operations is used. All nodes are from the [standard operations library of ONNX](https://github.com/onnx/onnx/blob/master/docs/Operators.md).\n",
+ "To explain how to create an ONNX model a simple example with mathematical operations is used. All nodes are from the [standard operations library of ONNX](https://github.com/onnx/onnx/blob/main/docs/Operators.md).\n",
"\n",
"First ONNX is imported, then the helper function can be used to make a node."
]
@@ -305,7 +305,7 @@
"source": [
"### How to manipulate an ONNX model\n",
"\n",
- "In the model there are two successive adder nodes. An adder node in ONNX can only add two inputs, but there is also the [**sum**](https://github.com/onnx/onnx/blob/master/docs/Operators.md#Sum) node, which can process more than two inputs. So it would be a reasonable change of the graph to combine the two successive adder nodes to one sum node."
+ "In the model there are two successive adder nodes. An adder node in ONNX can only add two inputs, but there is also the [**sum**](https://github.com/onnx/onnx/blob/main/docs/Operators.md#Sum) node, which can process more than two inputs. So it would be a reasonable change of the graph to combine the two successive adder nodes to one sum node."
]
},
{
@@ -599,7 +599,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
diff --git a/notebooks/basics/1_brevitas_network_import.ipynb b/notebooks/basics/1_brevitas_network_import.ipynb
index 5fb29754dc0ad56c2d07c783cf43102975b1621b..a884e90d7572789fc64cf9b953b5730590d4e8f1 100644
--- a/notebooks/basics/1_brevitas_network_import.ipynb
+++ b/notebooks/basics/1_brevitas_network_import.ipynb
@@ -297,7 +297,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
diff --git a/notebooks/end2end_example/bnn-pynq/cnv_end2end_example.ipynb b/notebooks/end2end_example/bnn-pynq/cnv_end2end_example.ipynb
index 28155d6f3eacd4dfd77aefbc73fc4ed3ef12f1dd..8ea6a3500955736ea3aaa803684b0e403b06a866 100644
--- a/notebooks/end2end_example/bnn-pynq/cnv_end2end_example.ipynb
+++ b/notebooks/end2end_example/bnn-pynq/cnv_end2end_example.ipynb
@@ -46,7 +46,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "The white fields show the state of the network representation in the respective step. The colored fields represent the transformations that are applied to the network to achieve a certain result. The diagram is divided into 5 sections represented by a different color, each of it includes several flow steps. The flow starts in top left corner with Brevitas export (green section), followed by the preparation of the network (blue section) for the Vivado HLS synthesis and Vivado IPI stitching (orange section), and finally building a PYNQ overlay bitfile and testing it on a PYNQ board (yellow section).\n",
+ "The white fields show the state of the network representation in the respective step. The colored fields represent the transformations that are applied to the network to achieve a certain result. The diagram is divided into 5 sections represented by a different color, each of it includes several flow steps. The flow starts in top left corner with Brevitas export (green section), followed by the preparation of the network (blue section) for the Vitis HLS synthesis and Vivado IPI stitching (orange section), and finally building a PYNQ overlay bitfile and testing it on a PYNQ board (yellow section).\n",
"There is an additional section for functional verification (red section) on the left side of the diagram, which we will not cover in this notebook. For details please take a look in the verification notebook which you can find [here](tfc_end2end_verification.ipynb)\n",
"\n",
"\n",
@@ -199,7 +199,7 @@
"\n",
"\n",
"\n",
- "Note how the convolution layer looks very similar to the fully connected one in terms of the matrix-vector-threshold unit (MVTU), but now the MVTU is preceded by a sliding window unit that produces the matrix from the input image. All of these building blocks, including the `MaxPool` layer you see in this figure, exist as templated Vivado HLS C++ functions in [finn-hlslib](https://github.com/Xilinx/finn-hlslib).\n",
+ "Note how the convolution layer looks very similar to the fully connected one in terms of the matrix-vector-threshold unit (MVTU), but now the MVTU is preceded by a sliding window unit that produces the matrix from the input image. All of these building blocks, including the `MaxPool` layer you see in this figure, exist as templated Vitis HLS C++ functions in [finn-hlslib](https://github.com/Xilinx/finn-hlslib).\n",
"\n",
"\n",
"To target this kind of hardware architecture with our network we'll apply a convolution lowering transformation, in addition to streamlining. You may recall the *streamlining transformation* that we applied to the TFC-w1a1 network, which is a series of mathematical simplifications that allow us to get rid of floating point scaling operations by implementing few-bit activations as thresholding operations. \n",
@@ -462,11 +462,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## 5. Deployment and Remote Execution\n",
+ "## 5. Deployment and Execution\n",
"\n",
- "Now that we're done with the hardware generation, we can copy the necessary files onto our PYNQ board.\n",
- "\n",
- "**Make sure you've [set up the SSH keys for your PYNQ board](https://finn-dev.readthedocs.io/en/latest/getting_started.html#pynq-board-first-time-setup) before executing this step.**"
+ "The bitfile and generated driver files(s) will be copied into a deployment folder which then can be used to run the network on the PYNQ board."
]
},
{
@@ -475,33 +473,33 @@
"metadata": {},
"outputs": [],
"source": [
- "import os\n",
+ "from shutil import copy\n",
+ "from distutils.dir_util import copy_tree\n",
+ "\n",
+ "# create directory for deployment files\n",
+ "deployment_dir = make_build_dir(prefix=\"pynq_deployment_\")\n",
+ "model.set_metadata_prop(\"pynq_deployment_dir\", deployment_dir)\n",
"\n",
- "# set up the following values according to your own environment\n",
- "# FINN will use ssh to deploy and run the generated accelerator\n",
- "ip = \"192.168.2.99\"\n",
- "username = os.getenv(\"PYNQ_USERNAME\", \"xilinx\")\n",
- "password = os.getenv(\"PYNQ_PASSWORD\", \"xilinx\")\n",
- "port = os.getenv(\"PYNQ_PORT\", 22)\n",
- "target_dir = os.getenv(\"PYNQ_TARGET_DIR\", \"/home/xilinx/finn_cnv_end2end_example\")\n",
- "# set up ssh options to only allow publickey authentication\n",
- "options = \"-o PreferredAuthentications=publickey -o PasswordAuthentication=no\"\n",
+ "# get and copy necessary files\n",
+ "# .bit and .hwh file\n",
+ "bitfile = model.get_metadata_prop(\"bitfile\")\n",
+ "hwh_file = model.get_metadata_prop(\"hw_handoff\")\n",
+ "deploy_files = [bitfile, hwh_file]\n",
"\n",
- "# test access to PYNQ board\n",
- "! ssh {options} {username}@{ip} -p {port} cat /var/run/motd.dynamic"
+ "for dfile in deploy_files:\n",
+ " if dfile is not None:\n",
+ " copy(dfile, deployment_dir)\n",
+ "\n",
+ "# driver.py and python libraries\n",
+ "pynq_driver_dir = model.get_metadata_prop(\"pynq_driver_dir\")\n",
+ "copy_tree(pynq_driver_dir, deployment_dir)"
]
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "from finn.transformation.fpgadataflow.make_deployment import DeployToPYNQ\n",
- "\n",
- "model = ModelWrapper(build_dir + \"/end2end_cnv_w1a1_synth.onnx\")\n",
- "model = model.transform(DeployToPYNQ(ip, port, username, password, target_dir))\n",
- "model.save(build_dir + \"/end2end_cnv_w1a1_pynq_deploy.onnx\")"
+ "Next to these files, we will also need an example numpy array to test the network on the PYNQ board. (*and before you ask, that's supposed to be a cat (CIFAR-10 class number 3)*) Recall that we partitioned our original network into a parent graph that contained the non-synthesizable nodes and a child graph that contained the bulk of the network, which we turned into a bitfile. The only operator left outside the FPGA partition was a `Transpose` to convert NCHW images into NHWC ones. Thus, we can skip the execution in the parent as long as we ensure our image has the expected data layout. The example numpy array can then be saved as .npy file."
]
},
{
@@ -510,8 +508,14 @@
"metadata": {},
"outputs": [],
"source": [
- "target_dir_pynq = target_dir + \"/\" + model.get_metadata_prop(\"pynq_deployment_dir\").split(\"/\")[-1]\n",
- "target_dir_pynq"
+ "import pkg_resources as pk\n",
+ "import matplotlib.pyplot as plt\n",
+ "import numpy as np\n",
+ "\n",
+ "fn = pk.resource_filename(\"finn.qnn-data\", \"cifar10/cifar10-test-data-class3.npz\")\n",
+ "x = np.load(fn)[\"arr_0\"]\n",
+ "x = x.reshape(3, 32,32).transpose(1, 2, 0)\n",
+ "plt.imshow(x)"
]
},
{
@@ -520,14 +524,19 @@
"metadata": {},
"outputs": [],
"source": [
- "! ssh {options} {username}@{ip} -p {port} 'ls -l {target_dir_pynq}'"
+ "model = ModelWrapper(build_dir + \"/end2end_cnv_w1a1_pynq_deploy.onnx\")\n",
+ "iname = model.graph.input[0].name\n",
+ "ishape = model.get_tensor_shape(iname)\n",
+ "np.save(deployment_dir + \"/input.npy\", x.reshape(ishape))"
]
},
{
- "cell_type": "markdown",
+ "cell_type": "code",
+ "execution_count": null,
"metadata": {},
+ "outputs": [],
"source": [
- "We only have two more steps to be able to remotely execute the deployed bitfile with some test data from the CIFAR-10 dataset. Let's load up some test data that comes bundled with FINN -- *and before you ask, that's supposed to be a cat (CIFAR-10 class number 3)*."
+ "! ls {deployment_dir}"
]
},
{
@@ -536,54 +545,34 @@
"metadata": {},
"outputs": [],
"source": [
- "import pkg_resources as pk\n",
- "import matplotlib.pyplot as plt\n",
- "import numpy as np\n",
- "\n",
- "fn = pk.resource_filename(\"finn.qnn-data\", \"cifar10/cifar10-test-data-class3.npz\")\n",
- "x = np.load(fn)[\"arr_0\"]\n",
- "x = x.reshape(3, 32,32).transpose(1, 2, 0)\n",
- "plt.imshow(x)"
+ "from shutil import make_archive\n",
+ "make_archive('deploy-on-pynq-cnv', 'zip', deployment_dir)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Recall that we partitioned our original network into a parent graph that contained the non-synthesizable nodes and a child graph that contained the bulk of the network, which we turned into a bitfile. The only operator left outside the FPGA partition was a `Transpose` to convert NCHW images into NHWC ones. Thus, we can skip the execution in the parent as long as we ensure our image has the expected data layout, which we have done above."
+ "You can now download the created zipfile (File -> Open, mark the checkbox next to the deploy-on-pynq-tfc.zip and select Download from the toolbar), then copy it to your PYNQ board (for instance via scp or rsync). Then, run the following commands on the PYNQ board to extract the archive and run the execution:"
]
},
{
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "from finn.core.onnx_exec import execute_onnx\n",
- "\n",
- "model = ModelWrapper(build_dir + \"/end2end_cnv_w1a1_pynq_deploy.onnx\")\n",
- "iname = model.graph.input[0].name\n",
- "oname = model.graph.output[0].name\n",
- "ishape = model.get_tensor_shape(iname)\n",
- "input_dict = {iname: x.astype(np.float32).reshape(ishape)}\n",
- "ret = execute_onnx(model, input_dict, True)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "ret[oname]"
+ "```shell\n",
+ "unzip deploy-on-pynq-cnv.zip -d finn-cnv-demo\n",
+ "cd finn-cnv-demo\n",
+ "sudo python3 -m pip install bitstring\n",
+ "sudo python3 driver.py --exec_mode=execute --batchsize=1 --bitfile=resizer.bit --inputfile=input.npy\n",
+ "```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "We see that the network correctly predicts this as a class 3 (\"cat\"). "
+ "The output will be saved on the PYNQ board as `output.npy` and can be copied to the host and opened with `np.load()`."
]
},
{
@@ -592,7 +581,7 @@
"source": [
"### Validating the Accuracy on a PYNQ Board <a id='validation'></a>\n",
"\n",
- "All the command line prompts here are meant to be executed with `sudo` on the PYNQ board, so we'll use a workaround (`echo password | sudo -S command`) to get that working from this notebook running on the host computer.\n",
+ "All the command line prompts here are meant to be executed with `sudo` on the PYNQ board.\n",
"\n",
"**Ensure that your PYNQ board has a working internet connecting for the next steps, since some there is some downloading involved.**\n",
"\n",
@@ -601,16 +590,9 @@
"\n",
"Command to execute on PYNQ:\n",
"\n",
- "```pip3 install git+https://github.com/fbcotter/dataset_loading.git@0.0.4#egg=dataset_loading```"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "! ssh {options} -t {username}@{ip} -p {port} 'echo {password} | sudo -S pip3 install git+https://github.com/fbcotter/dataset_loading.git@0.0.4#egg=dataset_loading'"
+ "```shell\n",
+ "sudo pip3 install git+https://github.com/fbcotter/dataset_loading.git@0.0.4#egg=dataset_loading\n",
+ "```"
]
},
{
@@ -621,16 +603,9 @@
"\n",
"Command to execute on PYNQ:\n",
"\n",
- "`python3.6 validate.py --dataset cifar10 --batchsize 1000`"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "! ssh {options} -t {username}@{ip} -p {port} 'cd {target_dir_pynq}; echo {password} | sudo -S python3.6 validate.py --dataset cifar10 --batchsize 1000'"
+ "```shell\n",
+ "sudo python3 validate.py --dataset cifar10 --batchsize 1000\n",
+ "```"
]
},
{
@@ -643,7 +618,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
diff --git a/notebooks/end2end_example/bnn-pynq/tfc_end2end_example.ipynb b/notebooks/end2end_example/bnn-pynq/tfc_end2end_example.ipynb
index c4fc92b97c91d6b1dfadc41ac3c23d014bd9fada..7e9980cf2abf40219847eab2c0cd381cc32f6682 100644
--- a/notebooks/end2end_example/bnn-pynq/tfc_end2end_example.ipynb
+++ b/notebooks/end2end_example/bnn-pynq/tfc_end2end_example.ipynb
@@ -33,7 +33,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "The white fields show the state of the network representation in the respective step. The colored fields represent the transformations that are applied to the network to achieve a certain result. The diagram is divided into 5 sections represented by a different color, each of it includes several flow steps. The flow starts in top left corner with Brevitas export (green section), followed by the preparation of the network (blue section) for the Vivado HLS synthesis and Vivado IPI stitching (orange section), and finally building a PYNQ overlay bitfile and testing it on a PYNQ board (yellow section).\n",
+ "The white fields show the state of the network representation in the respective step. The colored fields represent the transformations that are applied to the network to achieve a certain result. The diagram is divided into 5 sections represented by a different color, each of it includes several flow steps. The flow starts in top left corner with Brevitas export (green section), followed by the preparation of the network (blue section) for the Vitis HLS synthesis and Vivado IPI stitching (orange section), and finally building a PYNQ overlay bitfile and testing it on a PYNQ board (yellow section).\n",
"There is an additional section for functional verification (red section) on the right side of the diagram, which we will not cover in this notebook. For details please take a look in the verification notebook which you can find [here](tfc_end2end_verification.ipynb)\n",
"\n",
"\n",
@@ -161,7 +161,7 @@
"\n",
"\n",
"\n",
- "In practice, the compute arrays are instantiated by function calls to optimized Vivado HLS building blocks from the [finn-hlslib](https://github.com/Xilinx/finn-hlslib) library. As these function calls can only handle certain patterns/cases, we need to transform the network into an appropriate form so that we can replace network layers with these function calls, which is the goal of the network preparation process."
+ "In practice, the compute arrays are instantiated by function calls to optimized Vitis HLS building blocks from the [finn-hlslib](https://github.com/Xilinx/finn-hlslib) library. As these function calls can only handle certain patterns/cases, we need to transform the network into an appropriate form so that we can replace network layers with these function calls, which is the goal of the network preparation process."
]
},
{
@@ -248,7 +248,7 @@
"\n",
"In FINN, we can bake some of these pre/postprocessing operatings into the graph, and in some cases these can be highly beneficial for performance by allowing our accelerator to directly consume raw data instead of going through CPU preprocessing. \n",
"\n",
- "We'll demonstrate this for our small image classification network as follows. Brevitas preprocesses BNN-PYNQ network inputs with `torchvision.transforms.ToTensor()` [prior to training](https://github.com/Xilinx/brevitas/blob/master/src/brevitas_examples/bnn_pynq/trainer.py#L104), which converts 8-bit RGB values into floats between 0 and 1 by dividing the input by 255. We can achieve the same effect in FINN by exporting a single-node ONNX graph for division by 255 (which already exists as `finn.util.pytorch.ToTensor` and merging this with our original model. Finally, we're going to mark our input tensor as 8-bit to let FINN know which level of precision to use."
+ "We'll demonstrate this for our small image classification network as follows. Brevitas preprocesses BNN-PYNQ network inputs with `torchvision.transforms.ToTensor()` [prior to training](https://github.com/Xilinx/brevitas/blob/master/src/brevitas_examples/bnn_pynq/trainer.py#L86), which converts 8-bit RGB values into floats between 0 and 1 by dividing the input by 255. We can achieve the same effect in FINN by exporting a single-node ONNX graph for division by 255 (which already exists as `finn.util.pytorch.ToTensor` and merging this with our original model. Finally, we're going to mark our input tensor as 8-bit to let FINN know which level of precision to use."
]
},
{
@@ -343,7 +343,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "As can be seen, several transformations are involved in the streamlining transformation. There are move and collapse transformations. In the last step the operations are transformed into multithresholds. The involved transformations can be viewed in detail [here](https://github.com/Xilinx/finn/tree/master/src/finn/transformation/streamline). After each transformation, three of the tidy-up transformations (`GiveUniqueNodeNames`, `GiveReadableTensorNames` and `InferDataTypes`) are applied to the model.\n",
+ "As can be seen, several transformations are involved in the streamlining transformation. There are move and collapse transformations. In the last step the operations are transformed into multithresholds. The involved transformations can be viewed in detail [here](https://github.com/Xilinx/finn/tree/main/src/finn/transformation/streamline). After each transformation, three of the tidy-up transformations (`GiveUniqueNodeNames`, `GiveReadableTensorNames` and `InferDataTypes`) are applied to the model.\n",
"\n",
"After streamlining the network looks as follows:"
]
@@ -525,7 +525,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "We can use the higher-level [HLSCustomOp](https://github.com/Xilinx/finn/blob/main/src/finn/custom_op/fpgadataflow/__init__.py) wrappers for this node. These wrappers provide easy access to specific properties of these nodes, such as the folding factors (PE and SIMD). Let's have a look at which node attributes are defined by the CustomOp wrapper, and adjust the SIMD and PE attributes."
+ "We can use the higher-level [HLSCustomOp](https://github.com/Xilinx/finn/blob/main/src/finn/custom_op/fpgadataflow/hlscustomop.py) wrappers for this node. These wrappers provide easy access to specific properties of these nodes, such as the folding factors (PE and SIMD). Let's have a look at which node attributes are defined by the CustomOp wrapper, and adjust the SIMD and PE attributes."
]
},
{
@@ -547,7 +547,7 @@
"metadata": {},
"source": [
"We can see that the PE and SIMD are listed as node attributes, as well as the depths of the FIFOs that will be inserted between consecutive layers, and all can be adjusted using `set_nodeattr` subject to certain constraints. There are also a lot of additional attributes that can be set for this node type.\n",
- "**In this notebook we are setting the folding factors and FIFO depths manually, but in a future version we will support determining the folding factors given an FPGA resource budget according to the analytical model from the [FINN-R paper](https://arxiv.org/pdf/1809.04570).**"
+ "**In this notebook we are setting the folding factors and FIFO depths manually but it is possible to use FINN transformations for this ([SetFolding](https://github.com/Xilinx/finn/blob/main/src/finn/transformation/fpgadataflow/set_folding.py) and [InsertAndSetFIFODepths](https://github.com/Xilinx/finn/blob/main/src/finn/transformation/fpgadataflow/set_fifo_depths.py)).**"
]
},
{
@@ -609,7 +609,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "This completes the network preparation and the network can be passed on to the next block *Vivado HLS and IPI*, which is described below."
+ "This completes the network preparation and the network can be passed on to the next block *Vitis HLS and IPI*, which is described below."
]
},
{
@@ -798,23 +798,21 @@
"source": [
"## 4. PYNQ deployment <a id='hw_test'></a>\n",
"\n",
- "* [Deployment and Remote Execution](#deploy)\n",
+ "* [Deployment](#deploy)\n",
"* [Validation on PYNQ Board](#validation)\n",
"* [Throughput Test on PYNQ Board](#throughput)\n",
"\n",
"\n",
- "We are almost done preparing our hardware design. We'll now put it in a form suitable for use as a PYNQ overlay, synthesize and deploy it."
+ "The bitfile and generated driver will be copied together with some necessary files for execution into a deployment folder which then can be used to run the network on the PYNQ board."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Deployment and Remote Execution <a id='deploy'></a>\n",
+ "### Deployment <a id='deploy'></a>\n",
"\n",
- "We'll now use the `DeployToPYNQ` transformation to create a deployment folder with the bitfile and driver file(s), and copy that to the PYNQ board. You can change the default IP address, username, password and target folder for the PYNQ below.\n",
- "\n",
- "**Make sure you've [set up the SSH keys for your PYNQ board](https://finn-dev.readthedocs.io/en/latest/getting_started.html#pynq-board-first-time-setup) before executing this step.**"
+ "We'll now create a deployment folder with the bitfile and driver file(s), we zip it and afterwards it can be copied to the PYNQ board for execution and validation."
]
},
{
@@ -823,74 +821,33 @@
"metadata": {},
"outputs": [],
"source": [
- "import os\n",
+ "from shutil import copy\n",
+ "from distutils.dir_util import copy_tree\n",
"\n",
- "# set up the following values according to your own environment\n",
- "# FINN will use ssh to deploy and run the generated accelerator\n",
- "ip = \"192.168.2.99\"\n",
- "username = os.getenv(\"PYNQ_USERNAME\", \"xilinx\")\n",
- "password = os.getenv(\"PYNQ_PASSWORD\", \"xilinx\")\n",
- "port = os.getenv(\"PYNQ_PORT\", 22)\n",
- "target_dir = os.getenv(\"PYNQ_TARGET_DIR\", \"/home/xilinx/finn_tfc_end2end_example\")\n",
- "# set up ssh options to only allow publickey authentication\n",
- "options = \"-o PreferredAuthentications=publickey -o PasswordAuthentication=no\"\n",
+ "# create directory for deployment files\n",
+ "deployment_dir = make_build_dir(prefix=\"pynq_deployment_\")\n",
+ "model.set_metadata_prop(\"pynq_deployment_dir\", deployment_dir)\n",
"\n",
- "# test access to PYNQ board\n",
- "! ssh {options} {username}@{ip} -p {port} cat /var/run/motd.dynamic"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from finn.transformation.fpgadataflow.make_deployment import DeployToPYNQ\n",
+ "# get and copy necessary files\n",
+ "# .bit and .hwh file\n",
+ "bitfile = model.get_metadata_prop(\"bitfile\")\n",
+ "hwh_file = model.get_metadata_prop(\"hw_handoff\")\n",
+ "deploy_files = [bitfile, hwh_file]\n",
"\n",
- "model = model.transform(DeployToPYNQ(ip, port, username, password, target_dir))\n",
- "model.save(build_dir + \"/tfc_w1_a1_pynq_deploy.onnx\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's verify that the remote access credentials is saved in the model metadata, and that the deployment folder has been successfully copied to the board:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "model.model.metadata_props"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "target_dir_pynq = target_dir + \"/\" + model.get_metadata_prop(\"pynq_deployment_dir\").split(\"/\")[-1]\n",
- "target_dir_pynq"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "! ssh {options} {username}@{ip} -p {port} 'ls -l {target_dir_pynq}'"
+ "for dfile in deploy_files:\n",
+ " if dfile is not None:\n",
+ " copy(dfile, deployment_dir)\n",
+ "\n",
+ "# driver.py and python libraries\n",
+ "pynq_driver_dir = model.get_metadata_prop(\"pynq_driver_dir\")\n",
+ "copy_tree(pynq_driver_dir, deployment_dir)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "We only have two more steps to be able to remotely execute the deployed bitfile with some test data from the MNIST dataset. Let's load up some test data that comes bundled with FINN."
+ "Next to these files, we will also need an example numpy array to test the network on the PYNQ board. You may recall that one \"reshape\" node was left out of the StreamingDataflowPartition. We'll do that manually with a numpy function call when passing in the input, but everything else in the network ended up inside the StreamingDataflowPartition so that's all we need to do. The example numpy array can then be saved as .npy file. "
]
},
{
@@ -914,18 +871,23 @@
"metadata": {},
"outputs": [],
"source": [
+ "import numpy as np\n",
+ "\n",
"model = ModelWrapper(build_dir + \"/tfc_w1_a1_pynq_deploy.onnx\")\n",
"iname = model.graph.input[0].name\n",
"oname = parent_model.graph.output[0].name\n",
"ishape = model.get_tensor_shape(iname)\n",
- "print(\"Expected network input shape is \" + str(ishape))"
+ "print(\"Expected network input shape is \" + str(ishape))\n",
+ "np.save(deployment_dir + \"/input.npy\", x.reshape(ishape))"
]
},
{
- "cell_type": "markdown",
+ "cell_type": "code",
+ "execution_count": null,
"metadata": {},
+ "outputs": [],
"source": [
- "Finally, we can call `execute_onnx` on the graph, which will internally call remote execution with the bitfile, grab the results and return a numpy array. You may recall that one \"reshape\" node was left out of the StreamingDataflowPartition. We'll do that manually with a numpy function call when passing in the input, but everything else in the network ended up inside the StreamingDataflowPartition so that's all we need to do."
+ "! ls {deployment_dir}"
]
},
{
@@ -934,27 +896,34 @@
"metadata": {},
"outputs": [],
"source": [
- "import numpy as np\n",
- "from finn.core.onnx_exec import execute_onnx\n",
- "\n",
- "input_dict = {iname: x.reshape(ishape)}\n",
- "ret = execute_onnx(model, input_dict)"
+ "from shutil import make_archive\n",
+ "make_archive('deploy-on-pynq-tfc', 'zip', deployment_dir)"
]
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You can now download the created zipfile (**File -> Open**, mark the checkbox next to the `deploy-on-pynq-tfc.zip` and select Download from the toolbar), then copy it to your PYNQ board (for instance via `scp` or `rsync`). Then, run the following commands **on the PYNQ board** to extract the archive and run the execution:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "ret[oname]"
+ "```shell\n",
+ "unzip deploy-on-pynq-tfc.zip -d finn-tfc-demo\n",
+ "cd finn-tfc-demo\n",
+ "sudo python3 -m pip install bitstring\n",
+ "sudo python3 driver.py --exec_mode=execute --batchsize=1 --bitfile=resizer.bit --inputfile=input.npy\n",
+ "```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "We see that the network correctly predicts this as a digit 2."
+ "The output will be saved on the PYNQ board as `output.npy` and can be copied to the host and opened with `np.load()`."
]
},
{
@@ -963,25 +932,16 @@
"source": [
"### Validating the Accuracy on a PYNQ Board <a id='validation'></a>\n",
"\n",
- "All the command line prompts here are meant to be executed with `sudo` on the PYNQ board, so we'll use a workaround (`echo password | sudo -S command`) to get that working from this notebook running on the host computer.\n",
- "\n",
"**Ensure that your PYNQ board has a working internet connecting for the next steps, since there is some downloading involved.**\n",
"\n",
"To validate the accuracy, we first need to install the [`dataset-loading`](https://github.com/fbcotter/dataset_loading) Python package to the PYNQ board. This will give us a convenient way of downloading and accessing the MNIST dataset.\n",
"\n",
"\n",
- "Command to execute on PYNQ:\n",
+ "Command to execute on PYNQ board:\n",
"\n",
- "```sudo pip3 install git+https://github.com/fbcotter/dataset_loading.git@0.0.4#egg=dataset_loading```"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "! ssh {options} -t {username}@{ip} -p {port} 'echo {password} | sudo -S pip3 install git+https://github.com/fbcotter/dataset_loading.git@0.0.4#egg=dataset_loading'"
+ "```shell\n",
+ "sudo pip3 install git+https://github.com/fbcotter/dataset_loading.git@0.0.4#egg=dataset_loading\n",
+ "```"
]
},
{
@@ -990,18 +950,11 @@
"source": [
"We can now use the `validate.py` script that was generated together with the driver to measure top-1 accuracy on the MNIST dataset.\n",
"\n",
- "Command to execute on PYNQ:\n",
+ "Command to execute on PYNQ board:\n",
"\n",
- "`python3.6 validate.py --dataset mnist --batchsize 1000`"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "! ssh {options} -t {username}@{ip} -p {port} 'cd {target_dir_pynq}; echo {password} | sudo -S python3.6 validate.py --dataset mnist --batchsize 1000'"
+ "```shell\n",
+ "sudo python3 validate.py --dataset mnist --batchsize 1000\n",
+ "```"
]
},
{
@@ -1016,60 +969,30 @@
"metadata": {},
"source": [
"### Throughput Test on PYNQ Board <a id='throughput'></a>\n",
- "In addition to the functional verification, FINN also offers the possibility to measure the network performance directly on the PYNQ board. This can be done using the core function `throughput_test`. In the next section we import the function and execute it.\n",
- "First we extract the `remote_exec_model` again and pass it to the function. The function returns the metrics of the network as dictionary. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from finn.core.throughput_test import throughput_test_remote\n",
- "\n",
- "model = ModelWrapper(build_dir + \"/tfc_w1_a1_pynq_deploy.onnx\")\n",
- "res = throughput_test_remote(model, 10000)\n",
- "print(\"Network metrics:\")\n",
- "for key in res:\n",
- " print(str(key) + \": \" + str(res[key]))"
+ "In addition to the functional verification, FINN also offers the possibility to measure the network performance directly on the PYNQ board. This can be done setting the `exec_mode` to `throughput_test`. \n",
+ "Command to execute on PYNQ board:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Together with the values for folding we can evaluate the performance of our accelerator. Each layer has a total folding factor of 64 and because the network is fully pipelined, it follows: `II = 64`. II is the initiation interval and indicates how many cycles are needed for one input to be processed. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "II = 64\n",
- "# frequency in MHz\n",
- "f_MHz = 100\n",
- "# expected throughput in MFPS\n",
- "expected_throughput = f_MHz / II\n",
- "# measured throughput (FPS) from throughput test, converted to MFPS\n",
- "measured_throughput = res[\"throughput[images/s]\"] * 0.000001\n",
- "# peformance\n",
- "print(\"We reach approximately \" + str(round((measured_throughput / expected_throughput)*100)) + \"% of the ideal performance.\")"
+ "```shell\n",
+ "sudo python3 driver.py --exec_mode=throughput_test --batchsize=1000 --bitfile=resizer.bit\n",
+ "```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "The measured values were recorded with a batch size of 10000 and at a frequency of 100 MHz. We will be improving the efficiency of the generated accelerator examples in the coming FINN releases."
+ "The network metrics from the throughput test are saved in a file called `nw_metrics.txt` on the PYNQ board. Which can be investigated after running the command above."
]
}
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
diff --git a/notebooks/end2end_example/bnn-pynq/tfc_end2end_verification.ipynb b/notebooks/end2end_example/bnn-pynq/tfc_end2end_verification.ipynb
index 813127197e07e4ddb5ec5ff39aed0278e117babc..6c3b7965098e013fa35ac5f5b2b481e678d68f5d 100644
--- a/notebooks/end2end_example/bnn-pynq/tfc_end2end_verification.ipynb
+++ b/notebooks/end2end_example/bnn-pynq/tfc_end2end_verification.ipynb
@@ -61,7 +61,7 @@
"fc = get_test_model_trained(\"TFC\", 1, 1)\n",
"raw_i = get_data(\"qonnx.data\", \"onnx/mnist-conv/test_data_set_0/input_0.pb\")\n",
"input_tensor = onnx.load_tensor_from_string(raw_i)\n",
- "input_brevitas = torch.from_numpy(nph.to_array(input_tensor)).float()\n",
+ "input_brevitas = torch.from_numpy(nph.to_array(input_tensor).copy()).float()\n",
"output_golden = fc.forward(input_brevitas).detach().numpy()\n",
"output_golden"
]
@@ -72,7 +72,7 @@
"source": [
"## Simulation using Python <a id='simpy'></a>\n",
"\n",
- "If an ONNX model consists of [standard ONNX](https://github.com/onnx/onnx/blob/master/docs/Operators.md) nodes and/or FINN custom operations that do not belong to the fpgadataflow (`backend` $\\neq$ `fpgadataflow`) this model can be checked for functionality using Python.\n",
+ "If an ONNX model consists of [standard ONNX](https://github.com/onnx/onnx/blob/main/docs/Operators.md) nodes and/or FINN custom operations that do not belong to the fpgadataflow (`backend` $\\neq$ `fpgadataflow`) this model can be checked for functionality using Python.\n",
"\n",
"To simulate a standard ONNX node [onnxruntime](https://github.com/microsoft/onnxruntime) is used. onnxruntime is an open source tool developed by Microsoft to run standard ONNX nodes. For the FINN custom op nodes execution, functions are defined. The following is an example of the execution function of a XNOR popcount node.\n"
]
@@ -383,7 +383,15 @@
"\n",
"child_model = ModelWrapper(build_dir + \"/tfc_w1_a1_dataflow_child.onnx\")\n",
"child_model = child_model.transform(InsertDWC())\n",
- "child_model = child_model.transform(InsertFIFO())\n",
+ "\n",
+ "# set all impl_styles of the DWCs to hls to enable emulation\n",
+ "dwc_nodes = child_model.get_nodes_by_op_type(\"StreamingDataWidthConverter_Batch\")\n",
+ "for dwc in dwc_nodes:\n",
+ " dwc_inst = getCustomOp(dwc)\n",
+ " dwc_inst.set_nodeattr(\"impl_style\", \"hls\")\n",
+ " \n",
+ "child_model = child_model.transform(InsertFIFO(create_shallow_fifos=True))\n",
+ "child_model.save(build_dir + \"/test.onnx\");\n",
"child_model = child_model.transform(GiveUniqueNodeNames())\n",
"child_model = child_model.transform(PrepareIP(test_fpga_part, target_clk_ns))\n",
"child_model = child_model.transform(HLSSynthIP())\n",
@@ -431,7 +439,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
diff --git a/notebooks/end2end_example/cybersecurity/1-train-mlp-with-brevitas.ipynb b/notebooks/end2end_example/cybersecurity/1-train-mlp-with-brevitas.ipynb
index 5625a6f1c20ee5e4a66df28931a6a891f699a738..3d77586258b9ddb64985e7f7b7a2215565839c50 100644
--- a/notebooks/end2end_example/cybersecurity/1-train-mlp-with-brevitas.ipynb
+++ b/notebooks/end2end_example/cybersecurity/1-train-mlp-with-brevitas.ipynb
@@ -741,7 +741,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
diff --git a/notebooks/end2end_example/cybersecurity/2-import-into-finn-and-verify.ipynb b/notebooks/end2end_example/cybersecurity/2-import-into-finn-and-verify.ipynb
index 370312c77e90c67a3095e0800ad0c6046bfd75f4..e4848a1f40bed5865eccc1d831a634ac5f54e965 100644
--- a/notebooks/end2end_example/cybersecurity/2-import-into-finn-and-verify.ipynb
+++ b/notebooks/end2end_example/cybersecurity/2-import-into-finn-and-verify.ipynb
@@ -381,7 +381,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
diff --git a/notebooks/end2end_example/cybersecurity/3-build-accelerator-with-finn.ipynb b/notebooks/end2end_example/cybersecurity/3-build-accelerator-with-finn.ipynb
index 33adb68dc8ddfff1b427d82e4666a70e883bf2c8..a18cafd6044328d53139acafb2be2cf73a4ec9b6 100644
--- a/notebooks/end2end_example/cybersecurity/3-build-accelerator-with-finn.ipynb
+++ b/notebooks/end2end_example/cybersecurity/3-build-accelerator-with-finn.ipynb
@@ -624,7 +624,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},