Getting started with eIQ Neutron NPU ExecuTorch backend

Prerequisities

Hardware

For this tutorial, you will need a Linux machine with x86_64 processor architecture. This tutorial demonstrates the use of the eIQ Neutron backend with the Neutron behavioral simulator, called NSYS. You don’t need any specific development board for this tutorial.

Software

First you need to have Python 3.10 - 3.12 installed.

You need to install the ExecuTorch. Please follow the tutorial to install the ExecuTorch Setting Up ExecuTorch

In addition to this, you will need to install the eIQ Neutron Simulator, called NSYS, and the Neutron Converter for generating the byte-code for the eIQ Neutron NPU, during the model conversion in ExecuTorch AoT flow. To install the eIQ Neutron dependencies, run:

examples/nxp/setup.sh

This will install:

• Neutron Converter, for converting the Neutron IR to Neutron byte-code

• eIQ Neutron SDK, containing the eIQ Neutron runtimes (driver and firmware) for various NXP SoC and simulator

• eIQ NSYS, the Neutron behavioral simulator

Preparing a Model for NXP eIQ Neutron Backend

This guide demonstrating the use of ExecuTorch AoT flow to convert a PyTorch model to ExecuTorch format and delegate the model computation to eIQ Neutron NPU using the eIQ Neutron Backend.

Step 1: Environment Setup

This tutorial is intended to be run from a Linux and uses Conda or Virtual Env for Python environment management. For full setup details and system requirements, see Getting Started with ExecuTorch.

Create a Conda environment and install the ExecuTorch Python package.

conda create -y --name executorch python=3.12
conda activate executorch

and install the ExecuTorch Python package, either a prebuilt one:

conda install executorch

or build from source:

./install_executorch.sh

Also run the setup.sh script to install the eIQ Neutron dependencies:

$ ./examples/nxp/setup.sh

Step 2: Model Preparation and Running the Model on Target

See the example aot_neutron_compile.py and its README file.

For the purpose of this tutorial we will use a simple image classification model CifarNet10.

python -m examples.nxp.aot_neutron_compile --quantize \
    --delegate -m "cifar10"

Also, we will dump few of the images from the Cifar 10 dataset to a folder:

python -m examples.nxp.experimental.cifar_net.cifar_net --store-test-data

The destination folder is ./cifar10_test_data.

Runtime with NSYS

nxp_executor_runner example

The end-to-end example illustrating the use of the eIQ Neutron NPU is located in examples/nxp/executor_runner. Before we proceed with the build and execution, let’s stop to briefly introduce the example.

The application runs the executorch on a particular model. If a delegated model is provided to the runner, as in this tutorial, the instruction DelegateCall having the NeutronBackend ID is present in the model. This instruction is recognized by ExecuTorch runtime and handed over to the NeutronBackend. NeutronBackend runs it on the eIQ Neutron NPU, simulated by NSYS.

The NeutronDriver’s API (NeutronDriver.h and NeutronErrors.h) is the same, regardless of whether it is a physical IP or NSYS. What is specific, is the API provided by the NeutronEnvConfig.h, to set up the paths to:

• the NSYS,

• the Neutron Firmware to run and

• the NSYS configuration (.ini file):

    storeNsysConfigPath(FLAGS_nsys_config.c_str());
    storeFirmwarePath(FLAGS_firmware.c_str());
    storeNsysPath(FLAGS_nsys.c_str());

What corresponds to nxp_executor_runner CLI options, --firmware, --nsys, --nsys_config or provided by environmental variable:

CLI Option	Environmental variable	Description
--firmware	NSYS_FIRMWARE_PATH	Path to the NSYS firmware. The NSYS firmware for each supported platform is provided by the eIQ Neutron SDK package.
--nsys_config	NSYS_CONFIG_PATH	Path to the NSYS configuration. For i.MXRT700 SoC available in examples/nxp/exector_runner/neutron-imxrt700.ini. For other platform not required.
--nsys	N/A	Path to the eIQ NSYS simulator executable (which nsys)

For further details about eiq_nsys refer for the provided user guide in the python package.

Build the nxp_executor_runner

Use the provided examples/nxp/executor_runner/CMakeLists.txt. It can be build separately:

mkdir ./examples/nxp/executor_runner/build 
pushd ./examples/nxp/executor_runner/build
cmake ..
make nxp_executor_runner
popd

or as part of the ExecuTorch build:

cd <executorch_root_dir>
mkdir build
cmake .. \
  -DEXECUTORCH_BUILD_NXP_NEUTRON=ON \
  -DEXECUTORCH_BUILD_NXP_NEUTRON_RUNNER=ON \
  -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
  -DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=ON \
make nxp_executorch_runner

Run the nxp_executor_runner

./examples/nxp/executor_runner/build/nxp_executor_runner \
    --firmware `make -C ./examples/nxp/executor_runner/build locate_neutron_firmware | grep "NeutronFirmware.elf" ` \
    --nsys `which nsys` \
    --nsys_config ./examples/nxp/executor_runner/neutron-imxrt700.ini \
    --model ./cifar10_nxp_delegate.pte \
    --dataset ./cifar10_test_data \
    --output ./cifar10_test_output

Takeways

For the convenience, you can run the provided utility script doing all the above-mentioned steps for detailed inspection:

cd <executorch-root>
./examples/nxp/run.sh

FAQs

If you encounter any bugs or issues following this tutorial please file a bug/issue here on Github and label as module: nxp.