Vulkan Backend#
The ExecuTorch Vulkan (ET-VK) backend enables ExecuTorch models to execute on GPUs via the cross-platform Vulkan API. Although the Vulkan API support is almost ubiquitous among modern GPUs, the ExecuTorch Vulkan backend is developed with a focus on Android GPUs, and support for desktop platforms is experimental.
Features#
Wide operator support via an in-tree GLSL compute shader library
Support for models that require dynamic shapes
Support for FP32 and FP16 inference modes
Support for quantized linear layers with 8-bit/4-bit weights and 8-bit dynamically quantized activations
Support for quantized linear layers with 8-bit/4-bit weights and FP32/FP16 activations
Note that the Vulkan backend is under active development, and its GLSL compute shader library is being consistently expanded over time. Additional support for quantized operators (i.e. quantized convolution) and additional quantization modes is on the way.
Target Requirements#
Supports Vulkan 1.1
Development Requirements#
To build the Vulkan delegate, install the
Vulkan SDK 1.4.341.1 or newer.
After installation, the glslc binary must be found in your PATH in order
to compile Vulkan shaders. This can be checked by running
glslc --version
If this is not the case after completing the Vulkan SDK installation, you may have to
go into ~/VulkanSDK/<version>/ and run
source setup-env.sh
or alternatively,
python install_vulkan.py
To target Android, also install a current Android NDK. ExecuTorch CI uses NDK r28c.
Using the Vulkan Backend#
To lower a model to the Vulkan backend during the export and lowering process,
pass an instance of VulkanPartitioner to to_edge_transform_and_lower. The
example below demonstrates this process using the MobileNet V2 model from
torchvision.
import torch
import torchvision.models as models
from executorch.backends.vulkan.partitioner.vulkan_partitioner import VulkanPartitioner
from executorch.exir import to_edge_transform_and_lower
from torchvision.models.mobilenetv2 import MobileNet_V2_Weights
mobilenet_v2 = models.mobilenetv2.mobilenet_v2(
weights=MobileNet_V2_Weights.DEFAULT
).eval()
sample_inputs = (torch.randn(1, 3, 224, 224),)
exported_program = torch.export.export(mobilenet_v2, sample_inputs)
etvk_program = to_edge_transform_and_lower(
exported_program,
partitioner=[VulkanPartitioner()],
).to_executorch()
with open("mv2_vulkan.pte", "wb") as file:
etvk_program.write_to_file(file)
See Partitioner API for a reference on available partitioner options.
Quantization#
The Vulkan delegate currently supports execution of quantized linear layers. See Quantization for more information on available quantization schemes and APIs.
Runtime Integration#
To run the model on-device, use the standard ExecuTorch runtime APIs.
For integration in Android applications, the Vulkan backend is included in the executorch-android-vulkan package.
When building from source, pass -DEXECUTORCH_BUILD_VULKAN=ON when configuring
the CMake build to compile the Vulkan backend. See Running on Device
for more information.
To link against the backend, use the vulkan_backend CMake target. The target
propagates the platform-specific linker options needed to retain the static
initializers that register Vulkan compute shaders and operators.
# CMakeLists.txt
find_package(executorch CONFIG REQUIRED COMPONENTS vulkan_backend)
target_link_libraries(
my_target
PRIVATE
executorch
vulkan_backend
)
No additional steps are necessary to use the backend beyond linking the target. Any Vulkan-delegated .pte file will automatically run on the registered backend.
