Rate this Page

★ ★ ★ ★ ★

Runtime API Reference#

The ExecuTorch C++ API provides an on-device execution framework for exported PyTorch models.

For a tutorial style introduction to the runtime API, check out the using executorch with cpp tutorial and its simplified version.

For detailed information on how APIs evolve and the deprecation process, please refer to the ExecuTorch API Life Cycle and Deprecation Policy.

Model Loading and Execution#

class Program#

A deserialized ExecuTorch program binary.

Public Types

enum class Verification : uint8_t#

Types of validation that the Program can do before parsing the data.

Values:

enumerator Minimal#

Do minimal verification of the data, ensuring that the header appears correct.

Has minimal runtime overhead.

enumerator InternalConsistency#

When ET_ENABLE_PROGRAM_VERIFICATION is enabled, Do full verification of the data, ensuring that internal pointers are self-consistent and that the data has not been truncated or obviously corrupted. May not catch all types of corruption, but should guard against illegal memory operations during parsing. Also performs additional semantic validation such as:

Tensor numel overflow checks (ensuring size calculations don’t overflow)
List element type validation

Will have higher runtime overhead, scaling with the complexity of the proram data.

enum HeaderStatus#

Describes the presence of an ExecuTorch program header.

Values:

enumerator CompatibleVersion#: An ExecuTorch program header is present, and its version is compatible with this version of the runtime.

enumerator IncompatibleVersion#: An ExecuTorch program header is present, but its version is not compatible with this version of the runtime.

enumerator NotPresent#: An ExecuTorch program header is not present.

enumerator ShortData#: The data provided was too short to find the program header.

Public Functions

Result<const void*> get_constant_buffer_data(size_t buffer_idx, size_t nbytes) const#

Get the constant buffer inside Program with index buffer_idx.

Parameters:

buffer_idx – [in] the index of the buffer in the constant_buffer.
nbytes – [in] the number of bytes to read from the buffer.

Returns:

The buffer with corresponding index.

Result<const NamedDataMap*> get_named_data_map() const#

Get the named data map from the program.

Returns:: The named data map.

size_t num_methods() const#: Returns the number of methods in the program.

Result<const char*> get_method_name(size_t method_index) const#

Returns the name of the method at particular index.

Parameters:: method_index – [in] The index of the method name to retrieve. Must be less than the value returned by num_methods().
Returns:: The name of the requested method. The pointer is owned by the Program, and has the same lifetime as the Program.

Result<Method> load_method(const char *method_name, MemoryManager *memory_manager, EventTracer *event_tracer = nullptr, const NamedDataMap *named_data_map = nullptr, const LoadBackendOptionsMap *backend_options = nullptr, Span<const Kernel> kernel_registry = {}) const#

Loads the named method and prepares it for execution.

Parameters:

method_name – [in] The name of the method to load.
memory_manager – [in] The allocators to use during initialization and execution of the loaded method. If memory_manager.temp_allocator() is null, the runtime will allocate temp memory using et_pal_allocate().
event_tracer – [in] The event tracer to use for this method run.
named_data_map – [in] An optional map of {name, blob} used to resolve data that is external to the PTE, if any.
backend_options – [in] An optional map of per-backend load-time options (RuntimeSpecs). Each backend will receive its corresponding options during initialization.

Returns:

The loaded method on success, or an error on failure.

Result<MethodMeta> method_meta(const char *method_name) const#

Gathers metadata for the named method.

Parameters:: method_name – [in] The name of the method to get metadata for.

ET_DEPRECATED Result< const char * > get_output_flattening_encoding (const char *method_name="forward") const

DEPRECATED: Get the pytree encoding string for the output. Deprecated as this functionality will eventually move out of the core program into a higher level structure, but that does not exist at this time.

Parameters:: method_name – [in] The name of the method to get the encoding for.
Returns:: The pytree encoding string for the output

Public Static Functions

static ET_NODISCARD Result< Program > load (DataLoader *loader, Verification verification=Verification::Minimal)

Loads a Program from the provided loader. The Program will hold a pointer to the loader, which must outlive the returned Program instance.

Parameters:

loader – [in] The source to load program data from. The Program will hold a pointer to this loader, which must outlive the returned Program instance.
verification – [in] The type of verification to do before returning success.

static inline ET_DEPRECATED ET_NODISCARD Result< Program > Load (DataLoader *loader, Verification verification=Verification::Minimal): DEPRECATED: Use the lowercase load() instead.

static HeaderStatus check_header(const void *data, size_t size)#

Looks for an ExecuTorch program header in the provided data.

Parameters:

data – [in] The data from the beginning of a file that might contain an ExecuTorch program.
size – [in] The size of data in bytes. Must be >= kMinHeadBytes.

Returns:

A value describing the presence of a header in the data.

Public Static Attributes

static constexpr uint32_t kMaxSupportedSchemaVersion = 0#

The highest program schema version that this runtime can read.

Keep in sync with EXECUTORCH_SCHEMA_VERSION in //executorch/exir/version.py, which is the version that the exporter stamps into every PTE file.

static constexpr size_t kMinHeadBytes = 64#: The minimum number of bytes necessary for calls to check_header.

class Method#

An executable method of an executorch program. Maps to a python method like forward() on the original nn.Module.

Public Functions

inline Method(Method &&rhs) noexcept#: Move ctor. Takes ownership of resources previously owned by rhs, and leaves rhs in an uninitialized state.

ET_NODISCARD Error set_input (const EValue &input_evalue, size_t input_idx)

Sets the internal input value to be equivalent to the to the provided value.

Parameters:

input_evalue – [in] The evalue to copy into the method input. If the evalue is a tensor, the data is copied in most cases, so the tensor passed in here does not always need to outlive this call. But there is a case where the Method will keep a pointer to the tensor’s data. Based on the memory plan of the method, the inputs may not have buffer space pre-allocated for them. In this case the executor will alias the memory of the tensors provided as inputs here rather then deepcopy the input into the memory planned arena.
input_idx – [in] Zero-based index of the input to set. Must be less than the value returned by inputs_size().

Returns:

Error::Ok on success, non-Ok on failure.

ET_NODISCARD Error set_inputs (const executorch::aten::ArrayRef< EValue > &input_evalues)

Sets the values of all method inputs.

See set_input() for a more detailed description of the behavior.

Parameters:: input_evalues – [in] The new values for all of the method inputs. The type of each element must match the type of corresponding input. If the value of an element is a tensor, attempts to allow dynamic shape, but the dtype must always agree.
Returns:: Error::Ok on success, non-Ok on failure.

ET_NODISCARD Error set_output_data_ptr (void *buffer, size_t size, size_t output_idx)

Sets the data buffer of the specified method output to the provided value.

NOTE: Based on the memory plan of the method, the output tensors may not have buffer space pre-allocated for them, in this case the executor will point those tensors to the buffer provided here, so the user should take care that the life span of this memory outlasts the executor forward.

Parameters:

buffer – [in] The block of memory to point the specified tensor at.
size – [in] the length of buffer in bytes, must be >= the nbytes of the specified tensor.
output_idx – [in] The index of the output to set the data_ptr for. Must correspond to a tensor, and that tensor must not have had a buffer allocated by the memory plan.

Returns:

Error::Ok on success, non-Ok on failure.

ET_NODISCARD Error get_outputs (EValue *output_evalues, size_t length)

Copies the method’s outputs into the provided array.

WARNING: The output contains shallow copies of internal tensor outputs. Please do not mutate returned Tensor elements.

TODO(T139259264): Add checks to detect output mutation, or deep-copy outputs.

Parameters:

output_evalues – [in] The array to copy the outputs into. The first outputs_size() elements will be set to the corresponding output values. The rest of the array will be set to the EValue value None.
length – [in] The size of the output_evalues array in elements. Must be greater than or equal to outputs_size().

Returns:

Error::Ok on success, non-Ok on failure.

ET_DEPRECATED ET_NODISCARD Error get_inputs (EValue *input_evalues, size_t length)

DEPRECATED: Use MethodMeta instead to access metadata, and set_input to update Method inputs.

Copies the method’s inputs into the provided array.

WARNING: The input contains shallow copies of internal tensor inputs. Please do not mutate returned Tensor elements.

Parameters:

input_evalues – [in] The array to copy the inputs into. The first inputs_size() elements will be set to the corresponding input values. The rest of the array will be set to the EValue value None.
length – [in] The size of the input_evalues array in elements. Must be greater than or equal to inputs_size().

Returns:

Error::Ok on success, non-Ok on failure.

ET_NODISCARD Result< executorch::aten::Tensor > get_attribute (std::string_view name)

Retrieves the attribute tensor associated with the given name.

Parameters:: name – [in] The name of the attribute tensor to retrieve.
Returns:: Result containing the attribute tensor on success, non-Ok on failure.

ET_NODISCARD Error execute ()

Execute the method.

NOTE: Will fail with Error::InvalidState if the method has been partially executed using the step() api. Call reset_execution() after step() reaches EndOfMethod before calling execute().

If execute() encounters an error mid-execution, the state is automatically reset to allow retry attempts.

Returns:: Error::Ok on success, non-Ok on failure.

ET_EXPERIMENTAL ET_NODISCARD Error step ()

EXPERIMENTAL: Advances/executes a single instruction in the method.

Return values:

Error::Ok – step succeeded
non-Ok – step failed
Error::EndOfMethod – method finished executing successfully

ET_DEPRECATED ET_NODISCARD Error experimental_step (): DEPRECATED: Use step() instead.

ET_EXPERIMENTAL ET_NODISCARD bool in_progress () const

EXPERIMENTAL: Returns true if method execution is in progress.

Execution is considered “in progress” when step() has been called and execution is mid-way through the instruction sequence.

Note

execute() automatically resets state on error, so in_progress() returns false immediately after execute() fails.

Return values:

true – if execution is in progress (not at initial state, not at end)
false – if at initial state, after completion, or after any error

ET_EXPERIMENTAL ET_NODISCARD Error reset_execution ()

EXPERIMENTAL: Resets execution state to the start of the Method. For use with the step() API.

Return values:

Error::Ok – on success
Error::InvalidState – if called before step-based execution reached the end of the Method. When using step(), you must step through to EndOfMethod before resetting. Note: execute() handles its own error recovery and does not require manual reset.

ET_DEPRECATED ET_NODISCARD Error experimental_reset_execution (): DEPRECATED: Use reset_execution() instead.

MethodMeta method_meta() const#: Returns the MethodMeta that corresponds to the calling Method.

size_t inputs_size() const#: Returns the number of inputs the Method expects.

size_t outputs_size() const#: Returns the number of outputs the Method returns.

const EValue &get_output(size_t i) const#: Retrieves the output at the specified index.

ET_DEPRECATED const EValue & get_input (size_t i) const: DEPRECATED: Use MethodMeta instead to access metadata, and set_input to update Method inputs.

ET_DEPRECATED EValue & mutable_input (size_t i): DEPRECATED: Use MethodMeta instead to access metadata, and set_input to update Method inputs.

ET_DEPRECATED EValue & mutable_output (size_t i): DEPRECATED: Use MethodMeta instead to access metadata, and get_output to retrieve Method outputs.

class MethodMeta#

Describes a a method in an ExecuTorch program.

The program used to create a MethodMeta object must outlive the MethodMeta. It is separate from Method so that this information can be accessed without paying the initialization cost of loading the full Method.

Public Functions

const char *name() const#

Get the name of this method.

Returns:: The method name.

size_t num_inputs() const#

Get the number of inputs to this method.

Returns:: The number of inputs.

Result<Tag> input_tag(size_t index) const#

Get the tag of the specified input.

Parameters:: index – [in] The index of the input to look up.
Returns:: The tag of input, can only be [Tensor, Int, Bool, Double, String].

Result<TensorInfo> input_tensor_meta(size_t index) const#

Get metadata about the specified input.

Parameters:: index – [in] The index of the input to look up.
Returns:: The metadata on success, or an error on failure. Only valid for tag::Tensor

size_t num_outputs() const#

Get the number of outputs to this method.

Returns:: The number of outputs.

Result<Tag> output_tag(size_t index) const#

Get the tag of the specified output.

Parameters:: index – [in] The index of the output to look up.
Returns:: The tag of output, can only be [Tensor, Int, Bool, Double, String].

Result<TensorInfo> output_tensor_meta(size_t index) const#

Get metadata about the specified output.

Parameters:: index – [in] The index of the output to look up.
Returns:: The metadata on success, or an error on failure. Only valid for tag::Tensor

size_t num_attributes() const#

Get the number of attribute tensors in this method.

Returns:: The number of attribute tensors.

Result<TensorInfo> attribute_tensor_meta(size_t index) const#

Get metadata about the specified attribute tensor.

Parameters:: index – [in] The index of the attribute tensor to look up.
Returns:: The metadata on success, or an error on failure.

size_t num_memory_planned_buffers() const#

Get the number of memory-planned buffers this method requires.

Returns:: The number of memory-planned buffers.

Result<int64_t> memory_planned_buffer_size(size_t index) const#

Get the size in bytes of the specified memory-planned buffer.

Parameters:: index – [in] The index of the buffer to look up.
Returns:: The size in bytes on success, or an error on failure.

Result<etensor::Device> memory_planned_buffer_device(size_t index) const#

Get the device placement for the specified memory-planned buffer.

For CPU-only programs (no non_const_buffer_device in the PTE), all buffers default to Device{CPU, 0}. For programs with device annotations, returns the device type and index that the buffer should be allocated on.

Parameters:: index – [in] The index of the buffer to look up (0-based, same indexing as memory_planned_buffer_size()).
Returns:: The Device on success, or an error on failure.

bool uses_backend(const char *backend_name) const#

Check to see if a backend is used in this method.

Parameters:: backend_name – [in] The name of the backend to search for.
Returns:: true if a backend is used in this method, otherwise false.

size_t num_backends() const#

Get the number of backends used in this method.

Returns:: The total number of backend names.

Result<const char*> get_backend_name(size_t index) const#

Get the backend name at the given index.

Parameters:: index – [in] The index of the backend name.
Returns:: A Result wrapping the backend name as a C-style string on success, or an error if the index is invalid.

ET_EXPERIMENTAL size_t num_instructions () const

Get the number of instructions in this method.

Returns:: The number of instructions.

inline ET_DEPRECATED size_t num_non_const_buffers () const: DEPRECATED: Use num_memory_planned_buffers() instead.

inline Result<int64_t> non_const_buffer_size(size_t index) const#: DEPRECATED: Use memory_planned_buffer_size() instead.

Public Static Functions

static Result<MethodMeta> from_serialized_execution_plan(const void *data, size_t size)#

Creates metadata over a serialized execution plan after validating it.

data and its backing storage must outlive the returned view. This is intended for format adapters that synthesize an ExecuTorch metadata view.

class DataLoader#

Loads from a data source.

See //executorch/extension/data_loader for common implementations.

Public Functions

virtual ET_NODISCARD Result< FreeableBuffer > load (size_t offset, size_t size, const SegmentInfo &segment_info) const =0

Loads data from the underlying data source.

NOTE: This must be thread-safe. If this call modifies common state, the implementation must do its own locking.

Parameters:

offset – The byte offset in the data source to start loading from.
size – The number of bytes to load.
segment_info – Information about the segment being loaded.

Returns:

a FreeableBuffer that owns the loaded data.

inline virtual ET_NODISCARD Error load_into (size_t offset, size_t size, const SegmentInfo &segment_info, void *buffer) const

Loads data from the underlying data source into the provided buffer.

NOTE: This must be thread-safe. If this call modifies common state, the implementation must do its own locking.

Parameters:

offset – The byte offset in the data source to start loading from.
size – The number of bytes to load.
segment_info – Information about the segment being loaded.
buffer – The buffer to load data into. Must point to at least size bytes of memory.

Returns:

an Error indicating if the load was successful.

virtual ET_NODISCARD Result< size_t > size () const =0: Returns the length of the underlying data source, typically the file size.

inline virtual ET_NODISCARD Result< FreeableBuffer > load_at_offset (uint64_t offset, uint64_t size, const SegmentInfo &segment_info) const: Loads data using a 64-bit source offset and size.

inline virtual ET_NODISCARD Error load_into_at_offset (uint64_t offset, size_t size, const SegmentInfo &segment_info, void *buffer) const: Loads data from a 64-bit source offset into the provided buffer.

inline virtual ET_NODISCARD Result< uint64_t > source_size () const

Returns the 64-bit length of the underlying data source.

The default implementation widens the result from size(). Implementations that wrap sources larger than size_t can represent must override this method.

struct SegmentInfo#

Describes the content of the segment.

Public Types

enum class Type#

Represents the purpose of the segment.

Values:

enumerator Program#: Data for the actual program.

enumerator Constant#: Holds constant tensor data.

enumerator Backend#: Data used for initializing a backend.

enumerator Mutable#: Data used for initializing mutable tensors.

enumerator External#: Data used for initializing external tensors.

Public Members

Type segment_type#: Type of the segment.

size_t segment_index#: Index of the segment within the segment list. Undefined for program segments.

const char *descriptor#: An optional, null-terminated string describing the segment. For Backend segments, this is the backend ID. Null for other segment types.

class MemoryAllocator#

A class that does simple allocation based on a size and returns the pointer to the memory address. It bookmarks a buffer with certain size. The allocation is simply checking space and growing the cur_ pointer with each allocation request.

Simple example:

// User allocates a 100 byte long memory in the heap. uint8_t* memory_pool = malloc(100 * sizeof(uint8_t)); MemoryAllocator allocator(100, memory_pool) // Pass allocator object in the Executor

Underneath the hood, ExecuTorch will call allocator.allocate() to keep iterating cur_ pointer

Subclassed by executorch::runtime::internal::PlatformMemoryAllocator

Public Functions

inline MemoryAllocator(uint32_t size, uint8_t *base_address)#

Constructs a new memory allocator of a given size, starting at the provided base_address.

Parameters:

size – [in] The size in bytes of the buffer at base_address.
base_address – [in] The buffer to allocate from. Does not take ownership of this buffer, so it must be valid for the lifetime of of the MemoryAllocator.

inline virtual void *allocate(size_t size, size_t alignment = kDefaultAlignment)#

Allocates size bytes of memory.

Parameters:

size – [in] Number of bytes to allocate.
alignment – [in] Minimum alignment for the returned pointer. Must be a power of 2.

Return values:

nullptr – Not enough memory, or alignment was not a power of 2.

Returns:

Aligned pointer to the allocated memory on success.

template<typename T> inline T *allocateInstance(size_t alignment = alignof(T))#

Allocates a buffer large enough for an instance of type T. Note that the memory will not be initialized.

Example:

auto p = memory_allocator->allocateInstance<MyType>();

Parameters:: alignment – [in] Minimum alignment for the returned pointer. Must be a power of 2. Defaults to the natural alignment of T.
Return values:: nullptr – Not enough memory, or alignment was not a power of 2.
Returns:: Aligned pointer to the allocated memory on success.

template<typename T> inline T *allocateList(size_t size, size_t alignment = alignof(T))#

Allocates size number of chunks of type T, where each chunk is of size equal to sizeof(T) bytes.

Parameters:

size – [in] Number of memory chunks to allocate.
alignment – [in] Minimum alignment for the returned pointer. Must be a power of 2. Defaults to the natural alignment of T.

Return values:

nullptr – Not enough memory, or alignment was not a power of 2.

Returns:

Aligned pointer to the allocated memory on success.

Public Static Attributes

static constexpr size_t kDefaultAlignment = alignof(void*)#: Default alignment of memory returned by this class. Ensures that pointer fields of structs will be aligned. Larger types like long double may not be, however, depending on the toolchain and architecture.

class HierarchicalAllocator#

A group of buffers that can be used to represent a device’s memory hierarchy.

Public Functions

inline explicit HierarchicalAllocator(Span<Span<uint8_t>> buffers)#

Constructs a new hierarchical allocator with the given array of buffers.

Memory IDs are based on the index into buffers: buffers[N] will have a memory ID of N.
buffers.size() must be >= MethodMeta::num_non_const_buffers().
buffers[N].size() must be >= MethodMeta::non_const_buffer_size(N).

inline HierarchicalAllocator(Span<Span<uint8_t>> buffers, Span<const etensor::Device> planned_buffer_devices)#

Constructs a new hierarchical allocator with per-buffer device metadata.

Parameters:

buffers – [in] Same as above. May contain a mix of CPU and device pointers — HierarchicalAllocator only does pointer arithmetic, so device pointers are valid.
planned_buffer_devices – [in] One entry per buffer (same count as buffers), indicating the Device (type + index) for each buffer. Different buffers can target the same device type but different indices (e.g., cuda:0 vs cuda:1). For CPU-only programs, use the single-arg constructor instead.

inline ET_DEPRECATED HierarchicalAllocator(uint32_t n_allocators, MemoryAllocator *allocators)#: DEPRECATED: Use spans instead.

inline ET_NODISCARD Result< void * > get_offset_address (uint32_t memory_id, size_t offset_bytes, size_t size_bytes)

Returns the address at the byte offset offset_bytes from the given buffer’s base address, which points to at least size_bytes of memory.

Parameters:

memory_id – [in] The ID of the buffer in the hierarchy.
offset_bytes – [in] The offset in bytes into the specified buffer.
size_bytes – [in] The amount of memory that should be available at the offset.

Returns:

On success, the address of the requested byte offset into the specified buffer. On failure, a non-Ok Error.

inline Span<const etensor::Device> planned_buffer_devices() const#: Returns per-buffer device metadata. One entry per buffer, same count as the buffers passed to the constructor. Each entry is a Device carrying both type and index, so callers can distinguish e.g. cuda:0 from cuda:1. Empty if no device metadata was provided (CPU-only program).

class MemoryManager#

A container class for allocators used during Method load and execution.

This class consolidates all dynamic memory needs for Method load and execution. This can allow for heap-based as well as heap-less execution (relevant to some embedded scenarios), and overall provides more control over memory use.

This class, however, cannot ensure all allocation is accounted for since kernel and backend implementations are free to use a separate way to allocate memory (e.g., for things like scratch space). But we do suggest that backends and kernels use these provided allocators whenever possible.

Public Functions

inline explicit MemoryManager(MemoryAllocator *method_allocator, HierarchicalAllocator *planned_memory = nullptr, MemoryAllocator *temp_allocator = nullptr)#

Constructs a new MemoryManager.

Parameters:

method_allocator – [in] The allocator to use when loading a Method and allocating its internal structures. Must outlive the Method that uses it.
planned_memory – [in] The memory-planned buffers to use for mutable tensor data when executing a Method. Must outlive the Method that uses it. May be nullptr if the Method does not use any memory-planned tensor data. The sizes of the buffers in this HierarchicalAllocator must agree with the corresponding MethodMeta::num_memory_planned_buffers() and MethodMeta::memory_planned_buffer_size(N) values, which are embedded in the Program. For device-aware programs, the per-buffer device metadata is owned by the HierarchicalAllocator as well.
temp_allocator – [in] The allocator to use when allocating temporary data during kernel or delegate execution. Must outlive the Method that uses it. May be nullptr if the Method does not use kernels or delegates that allocate temporary data. This allocator will be reset after every kernel or delegate call during execution.

inline ET_DEPRECATED MemoryManager(MemoryAllocator *constant_allocator, HierarchicalAllocator *non_constant_allocator, MemoryAllocator *runtime_allocator, MemoryAllocator *temporary_allocator)#

DEPRECATED: Use the constructor without constant_allocator instead.

TODO(T162089316): Remove this once all users migrate to the new ctor.

inline MemoryAllocator *method_allocator() const#: Returns the allocator that the runtime will use to allocate internal structures while loading a Method. Must not be used after its associated Method has been loaded.

inline HierarchicalAllocator *planned_memory() const#: Returns the memory-planned buffers to use for mutable tensor data.

inline MemoryAllocator *temp_allocator() const#

Returns the allocator to use for allocating temporary data during kernel or delegate execution.

This allocator will be reset after every kernel or delegate call during execution.

inline Span<const etensor::Device> planned_buffer_devices() const#

Returns per-buffer device metadata. One entry per planned memory buffer, same count as planned_memory buffers. Empty if no device metadata was provided (CPU-only program) or if planned_memory is null.

This is a thin wrapper around HierarchicalAllocator::planned_buffer_devices().

inline bool has_device_memory() const#: Returns true if any planned buffer has device metadata attached. When false, the memory setup is CPU-only.

Values#

struct EValue#

Public Functions

inline EValue(executorch::aten::Scalar s)#: Construct an EValue using the implicit value of a Scalar.

template<typename T> Result<T> tryTo() const#: Result-returning equivalent of to<T>(). Tag mismatch returns Error::InvalidType; a null list/string payload returns Error::InvalidState. Specializations are defined below via EVALUE_DEFINE_TRY_TO.

template<typename T> inline std::optional<T> toOptional() const#: Converts the EValue to an optional object that can represent both T and an uninitialized state.

template<typename T> inline Result<std::optional<T>> tryToOptional() const#: Result-returning equivalent of toOptional<T>(). None maps to an empty optional; any other tag that doesn’t match T propagates tryTo<T>()’s error (Error::InvalidType).

union Payload#

union TriviallyCopyablePayload#

class Tensor#

A minimal Tensor type whose API is a source compatible subset of at::Tensor.

NOTE: Instances of this class do not own the TensorImpl given to it, which means that the caller must guarantee that the TensorImpl lives longer than any Tensor instances that point to it.

See the documention on TensorImpl for details about the return/parameter types used here and how they relate to at::Tensor.

Public Types

using SizesType = TensorImpl::SizesType#: The type used for elements of sizes().

using DimOrderType = TensorImpl::DimOrderType#: The type used for elements of dim_order().

using StridesType = TensorImpl::StridesType#: The type used for elements of strides().

Public Functions

inline TensorImpl *unsafeGetTensorImpl() const#

Returns a pointer to the underlying TensorImpl.

NOTE: Clients should be wary of operating on the TensorImpl directly instead of the Tensor. It is easy to break things.

inline size_t nbytes() const#

Returns the size of the tensor in bytes.

NOTE: Only the alive space is returned not the total capacity of the underlying data blob.

inline ssize_t size(ssize_t dim) const#

Returns the size of the tensor at the given dimension.

NOTE: that size() intentionally does not return SizeType even though it returns an element of an array of SizeType. This is to help make calls of this method more compatible with at::Tensor, and more consistent with the rest of the methods on this class and in ETensor.

inline ssize_t dim() const#: Returns the tensor’s number of dimensions.

inline ssize_t numel() const#: Returns the number of elements in the tensor.

inline ScalarType scalar_type() const#: Returns the type of the elements in the tensor (int32, float, bool, etc).

inline ssize_t element_size() const#: Returns the size in bytes of one element of the tensor.

inline const ArrayRef<SizesType> sizes() const#: Returns the sizes of the tensor at each dimension.

inline const ArrayRef<DimOrderType> dim_order() const#: Returns the order the dimensions are laid out in memory.

inline const ArrayRef<StridesType> strides() const#: Returns the strides of the tensor at each dimension.

inline TensorShapeDynamism shape_dynamism() const#: Returns the mutability of the shape of the tensor.

inline Device device() const#: Returns the device where tensor data resides.

inline DeviceType device_type() const#: Returns the type of device where tensor data resides.

inline DeviceIndex device_index() const#: Returns the device index, or 0 if default/unspecified.

template<typename T> inline const T *const_data_ptr() const#: Returns a pointer of type T to the constant underlying data blob.

inline const void *const_data_ptr() const#: Returns a pointer to the constant underlying data blob.

template<typename T> inline T *mutable_data_ptr() const#: Returns a pointer of type T to the mutable underlying data blob.

inline void *mutable_data_ptr() const#: Returns a pointer to the mutable underlying data blob.

template<typename T> inline ET_DEPRECATED T * data_ptr () const: DEPRECATED: Use const_data_ptr or mutable_data_ptr instead.

inline ET_DEPRECATED void * data_ptr () const: DEPRECATED: Use const_data_ptr or mutable_data_ptr instead.

inline ET_DEPRECATED void set_data (void *ptr) const: DEPRECATED: Changes the data_ptr the tensor aliases. Does not free the previously pointed to data, does not assume ownership semantics of the new ptr. This api does not exist in at::Tensor so kernel developers should avoid it.

Module Extension#

The Module extension provides a higher-level C++ facade for loading programs, setting inputs and outputs, and executing methods with common runtime defaults.

class Module#

A facade class for loading programs and executing methods within them.

Subclassed by executorch::extension::bundled_module::BundledModule

Public Types

enum class LoadMode#

Enum to define loading behavior.

Values:

enumerator File#: Load the whole file as a buffer.

enumerator Mmap#: Use mmap to load pages into memory.

enumerator MmapUseMlock#: Use memory locking and handle errors.

enumerator MmapUseMlockIgnoreErrors#: Use memory locking and ignore errors.

enumerator MmapUseMadvise#: Use mmap with madvise(MADV_WILLNEED | MADV_SEQUENTIAL) hints.

enum class Format#

Serialized program format selected after loading.

Values:

enumerator Pte#

enumerator Ptn#

Public Functions

explicit Module(const std::string &file_path, const LoadMode load_mode = LoadMode::File, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, bool share_memory_arenas = false)#

Constructs an instance by loading a program from a file with specified memory locking behavior.

Parameters:

file_path – [in] The path to the ExecuTorch program file to load.
load_mode – [in] The loading mode to use.
event_tracer – [in] A EventTracer used for tracking and logging events.
share_memory_arenas – [in] When true, all methods loaded by this Module share the same memory-planned buffers for mem_id=1 (activation memory) and mem_id=2 (shared mutable buffer memory), sized to the max across all methods. mem_id>2 indicates a custom memory plan, and those receive fresh memory buffers. share_memory_arenas is required for models exported with share_mutable_buffers=true, where methods access shared mutable state (e.g., set/get state). When enabled, outputs from one method may be invalidated by executing another method, since their output tensors can alias the same underlying buffer. Consume or copy outputs before calling execute again. NOTE: This class is not thread-safe and performs no internal synchronization. Calling execute concurrently on the same Module instance from multiple threads is unsafe, regardless of whether share_memory_arenas is true or false. When share_memory_arenas is true, methods may overwrite each other’s data in the shared memory arenas, increasing aliasing and the risk of unintended overwrites.

explicit Module(const std::string &file_path, const std::string &data_map_path, const LoadMode load_mode = LoadMode::File, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, bool share_memory_arenas = false)#

Constructs an instance by loading a program from a file with specified memory locking behavior.

Parameters:

file_path – [in] The path to the ExecuTorch program file to load.
data_map_path – [in] The path to a .ptd file.
load_mode – [in] The loading mode to use.
event_tracer – [in] A EventTracer used for tracking and logging events.
share_memory_arenas – [in] When true, all methods loaded by this Module share a single set of memory-planned buffers.

explicit Module(const std::string &file_path, std::vector<std::string> data_files, const LoadMode load_mode = LoadMode::File, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, bool share_memory_arenas = false)#

Constructs an instance by loading a program from a file with specified memory locking behavior.

Parameters:

file_path – [in] The path to the ExecuTorch program file to load.
data_files – [in] The path to one or more .ptd file/s.
load_mode – [in] The loading mode to use.
event_tracer – [in] A EventTracer used for tracking and logging events.
share_memory_arenas – [in] When true, all methods loaded by this Module share a single set of memory-planned buffers.

explicit Module(std::unique_ptr<runtime::DataLoader> data_loader, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::DataLoader> data_map_loader = nullptr, bool share_memory_arenas = false)#

Constructs an instance with the provided data loader and memory allocator.

Parameters:

data_loader – [in] A DataLoader used for loading program data.
memory_allocator – [in] A MemoryAllocator used for memory management.
temp_allocator – [in] A MemoryAllocator to use when allocating temporary data during kernel or delegate execution.
event_tracer – [in] A EventTracer used for tracking and logging events.
data_map_loader – [in] A DataLoader used for loading external weights.
share_memory_arenas – [in] When true, all methods loaded by this Module share a single set of memory-planned buffers.

explicit Module(std::shared_ptr<Program> program, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::DataLoader> data_map_loader = nullptr, bool share_memory_arenas = false)#

Constructs an instance using an existing shared program.

Parameters:

program – [in] The shared program to use. It’s required the data loader the program uses is valid for the lifetime of the program.
memory_allocator – [in] A MemoryAllocator used for memory management.
temp_allocator – [in] A MemoryAllocator to use when allocating temporary data.
event_tracer – [in] A EventTracer used for tracking and logging events.
data_map_loader – [in] A DataLoader used for loading external weights.
share_memory_arenas – [in] When true, all methods loaded by this Module share a single set of memory-planned buffers.

virtual ET_NODISCARD runtime::Error load (const Program::Verification verification=Program::Verification::Minimal)

Loads the program if needed.

Parameters:: verification – [in] The type of verification to do before returning success.
Returns:: An Error to indicate success or failure of the loading process.

virtual ET_NODISCARD runtime::Error load (const LoadBackendOptionsMap &backend_options, const Program::Verification verification=Program::Verification::Minimal)

Loads the program with per-delegate runtime options.

The Module deep-copies backend_options into internal storage, so the caller may release the input (and any backing BackendOption arrays its Spans referenced) immediately after this call returns. Future lazy load_method calls (e.g. triggered by forward) consume the Module-owned copy.

Transactional: on failure, the previously-installed backend options (if any) are left in place; the input is not committed.

Parameters:

backend_options – [in] A LoadBackendOptionsMap containing per-delegate load-time configuration options. Deep-copied into the Module on success; not retained on failure.
verification – [in] The type of verification to do before returning success.

Returns:

An Error to indicate success or failure of the loading process.

inline const LoadBackendOptionsMap &backend_options() const#

Returns the deep-copied LoadBackendOptionsMap most recently installed via load(LoadBackendOptionsMap, ...). The returned reference is owned by the Module and remains valid until the next call to load(LoadBackendOptionsMap, ...) or until the Module is destroyed.

If load(LoadBackendOptionsMap, ...) has never been called, returns a default-constructed (empty, size() == 0) map.

Returns:: Const reference to the Module-owned LoadBackendOptionsMap.

inline virtual bool is_loaded() const#

Checks if the program is loaded.

Returns:: true if the program is loaded, false otherwise.

runtime::Result<Format> format()#: Loads the program if needed and returns its serialized format.

inline std::shared_ptr<Program> program() const#

Get the program. The data loader used by the program is guaranteed to be valid for the lifetime of the program.

Returns:: Shared pointer to the PTE program, or nullptr if the Module is not loaded or contains a different format.

runtime::Result<size_t> num_methods()#

Get the number of methods available in the loaded program.

Returns:: A Result object containing either the number of methods available or an error to indicate failure.

runtime::Result<std::unordered_set<std::string>> method_names()#

Get a list of method names available in the loaded program. Loads the program and method if needed.

Returns:: A set of strings containing the names of the methods, or an error if the program or method failed to load.

ET_NODISCARD runtime::Error load_method (const std::string &method_name, runtime::HierarchicalAllocator *planned_memory=nullptr, torch::executor::EventTracer *event_tracer=nullptr, const LoadBackendOptionsMap *backend_options=nullptr, std::vector< Kernel > kernel_registry={})

Load a specific method from the program and set up memory management if needed. The loaded method is cached to reuse the next time it’s executed.

Parameters:

method_name – [in] The name of the method to load.
planned_memory – [in] The memory-planned buffers to use for mutable tensor data when executing a method.
event_tracer – [in] Per-method event tracer to profile/trace methods individually. When not given, the event tracer passed to the Module constructor is used. Otherwise, this per-method event tracer takes precedence.

Returns:

An Error to indicate success or failure.

bool unload_method(const std::string &method_name)#

Unload a specific method from the program.

Parameters:: method_name – [in] The name of the method to unload.
Returns:: True if the method is unloaded, false if no-op.

ET_DEPRECATED ET_NODISCARD runtime::Result< Method * > method (const std::string &method_name)

DEPRECATED: Module manages each Method exclusively.

Get a method by it’s name. Not recommended to use this method directly as an end user. It’s exposed to allow for composability of module in apis that operate on method.

Parameters:: method_name – [in] The name of the method to get.
Returns:: A Result object containing either a pointer to the requested method or an error to indicate failure.

inline ET_NODISCARD runtime::Error load_forward (runtime::HierarchicalAllocator *planned_memory=nullptr, torch::executor::EventTracer *event_tracer=nullptr, const LoadBackendOptionsMap *backend_options=nullptr, std::vector< Kernel > kernel_registry={})

Load the ‘forward’ method from the program and set up memory management if needed. The loaded method is cached to reuse the next time it’s executed.

Parameters:

planned_memory – [in] The memory-planned buffers to use for mutable tensor data when executing the ‘forward’ method.
event_tracer – [in] An event tracer used for tracking and logging events.

Returns:

An Error to indicate success or failure.

inline bool unload_forward()#

Unload the ‘forward’ method from the program.

Returns:: True if the ‘forward’ method is unloaded, false if no-op.

bool is_method_loaded(const std::string &method_name) const#

Checks if a specific method is loaded.

Parameters:: method_name – [in] The name of the method to check.
Returns:: true if the method specified by method_name is loaded, false otherwise.

runtime::Result<MethodMeta> method_meta(const std::string &method_name)#

Get a method metadata struct by method name. Loads the program if needed.

Parameters:: method_name – [in] The name of the method to get the metadata for.
Returns:: A method metadata, or an error if the program or method failed to load.

virtual ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name, const std::vector< runtime::EValue > &input_values)

Execute a specific method with the given input values and retrieve the output values. Loads the program and method before executing if needed.

Parameters:

method_name – [in] The name of the method to execute.
input_values – [in] A vector of input values to be passed to the method.

Returns:

A Result object containing either a vector of output values from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name, const runtime::EValue &input_value)

Execute a specific method with a single input value. Loads the program and method before executing if needed.

Parameters:

method_name – [in] The name of the method to execute.
input_value – [in] A value to be passed to the method.

Returns:

A Result object containing either a vector of output values from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name)

Execute a specific method without any input values. Loads the program and method before executing if needed.

Parameters:: method_name – [in] The name of the method to execute.
Returns:: A Result object containing either a vector of output values from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< runtime::EValue > get (const std::string &method_name, const std::vector< runtime::EValue > &input_values)

Retrieve the output value of a specific method with the given input values. Loads the program and method before execution if needed.

Parameters:

method_name – [in] The name of the method to execute.
input_values – [in] A vector of input values to be passed to the method.

Returns:

A Result object containing either the first output value from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< runtime::EValue > get (const std::string &method_name, const runtime::EValue &input_value)

Retrieve the output value of a specific method with a single input value. Loads the program and method before execution if needed.

Parameters:

method_name – [in] The name of the method to execute.
input_value – [in] A value to be passed to the method.

Returns:

A Result object containing either the first output value from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< runtime::EValue > get (const std::string &method_name)

Retrieve the output value of a specific method without any input values. Loads the program and method before execution if needed.

Parameters:: method_name – [in] The name of the method to execute.
Returns:: A Result object containing either the first output value from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > forward (const std::vector< runtime::EValue > &input_values)

Execute the ‘forward’ method with the given input values and retrieve the output values. Loads the program and method before executing if needed.

Parameters:: input_values – [in] A vector of input values for the ‘forward’ method.
Returns:: A Result object containing either a vector of output values from the ‘forward’ method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > forward (const runtime::EValue &input_value)

Execute the ‘forward’ method with a single value. Loads the program and method before executing if needed.

Parameters:: input_value – [in] A value for the ‘forward’ method.
Returns:: A Result object containing either a vector of output values from the ‘forward’ method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > forward ()

Execute the ‘forward’ method without any input values. Loads the program and method before executing if needed.

Returns:: A Result object containing either a vector of output values from the ‘forward’ method or an error to indicate failure.

ET_NODISCARD runtime::Error set_input (const std::string &method_name, const runtime::EValue &input_value, size_t input_index)

Sets a single input value for a specific method.

Parameters:

method_name – [in] The name of the method.
input_value – [in] The EValue to set as the method input.
input_index – [in] Zero-based index of the input to set.

Returns:

An Error to indicate success or failure.

inline ET_NODISCARD runtime::Error set_input (const runtime::EValue &input_value, size_t input_index)

Sets a single input value for the “forward” method.

Parameters:

input_value – [in] The EValue to set as the method input.
input_index – [in] Zero-based index of the input to set.

Returns:

An Error to indicate success or failure.

ET_NODISCARD runtime::Error set_inputs (const std::string &method_name, const std::vector< runtime::EValue > &input_values)

Sets all input values for a specific method.

Parameters:

method_name – [in] The name of the method.
input_values – [in] A vector of EValues to set as the method inputs.

Returns:

An Error to indicate success or failure.

inline ET_NODISCARD runtime::Error set_inputs (const std::vector< runtime::EValue > &input_values)

Sets all input values for the “forward” method.

Parameters:: input_values – [in] A vector of EValues to set as the method inputs.
Returns:: An Error to indicate success or failure.

ET_NODISCARD runtime::Error set_output (const std::string &method_name, runtime::EValue output_value, size_t output_index=0)

Sets the output tensor for a specific method.

Note

Only Tensor outputs are currently supported for setting.

Parameters:

method_name – [in] The name of the method.
output_value – [in] The EValue containing the Tensor to set as the method output.
output_index – [in] Zero-based index of the output to set.

Returns:

An Error to indicate success or failure.

inline ET_NODISCARD runtime::Error set_output (runtime::EValue output_value, size_t output_index=0)

Sets the output tensor for the “forward” method.

Note

Only Tensor outputs are currently supported for setting.

Parameters:

output_value – [in] The EValue containing the Tensor to set as the method output.
output_index – [in] Zero-based index of the output to set.

Returns:

An Error to indicate success or failure.

ET_NODISCARD runtime::Error set_outputs (const std::string &method_name, const std::vector< runtime::EValue > &output_values)

Sets all output tensors for a specific method.

Loads the program and method if needed, and for each output uses the provided tensor’s data buffer as the method’s output buffer.

Note

Only Tensor outputs are currently supported for setting.

Note

Will fail for outputs that are memory-planned or constants.

Parameters:

method_name – [in] The name of the method.
output_values – [in] A vector of EValues to set as the method outputs.

Returns:

An Error to indicate success or failure.

inline ET_NODISCARD runtime::Error set_outputs (const std::vector< runtime::EValue > &output_values)

Sets all output tensors for the “forward” method.

Note

Only Tensor outputs are currently supported for setting.

Note

Will fail for outputs that are memory-planned or constants.

Parameters:: output_values – [in] A vector of EValues to set as the method outputs.
Returns:: An Error to indicate success or failure.

ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > get_outputs (const std::string &method_name)

Retrieve all current output values of a specific method without executing it. Loads the program and method before retrieval if needed.

Parameters:: method_name – [in] The name of the method.
Returns:: A Result containing the vector of output values, or an error.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > get_outputs ()

Retrieve all current output values of the “forward” method without executing it. Loads the program and method before retrieval if needed.

Returns:: A Result containing the vector of output values, or an error.

ET_NODISCARD runtime::Result< runtime::EValue > get_output (const std::string &method_name, size_t output_index=0)

Retrieve a single current output value of a specific method without executing it. Loads the program and method before retrieval if needed.

Parameters:

method_name – [in] The name of the method.
output_index – [in] Zero-based index of the output to retrieve.

Returns:

A Result containing the requested output value, or an error.

inline ET_NODISCARD runtime::Result< runtime::EValue > get_output (size_t output_index=0)

Retrieve a single current output value of the “forward” method without executing it. Loads the program and method before retrieval if needed.

Parameters:: output_index – [in] Zero-based index of the output to retrieve.
Returns:: A Result containing the requested output value, or an error.

inline runtime::EventTracer *event_tracer() const#

Retrieves the EventTracer instance being used by the Module. EventTracer is used for tracking and logging events during the execution of methods.

Returns:: A pointer to the EventTracer instance. Returns nullptr if no EventTracer is set.

class BundledModule : public executorch::extension::module::Module#

A facade class for loading bundled programs and executing methods within them.

Public Functions

explicit BundledModule(const void *bundled_program_ptr, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::DataLoader> data_map_loader = nullptr)#

Constructs an instance with the bundled program buffer pointer.

This constructor reads the program from bundled program buffer to load the module with data loader. The bundled program pointer is preserved so that the portion outside of program is accessible.

Parameters:

bundled_program_ptr – [in] A DataLoader used for loading program data.
memory_allocator – [in] A MemoryAllocator used for memory management.
temp_allocator – [in] A MemoryAllocator to use when allocating temporary data during kernel or delegate execution.
event_tracer – [in] A EventTracer used for tracking and logging events.
data_map_loader – [in] A DataLoader used for loading external weights.

ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name, const size_t testset_idx)

Execute a specific method with the input value at the given testset_idx from the bundle to the method. Loads the program and method before executing if needed.

This function is a wrapper of load_bundled_input in bundled_program.

Parameters:

method_name – [in] The name of the method to execute.
testset_idx – [in] The index of the input value to be passed to the method.

Returns:

Return Error::Ok on a successful load, or the error happens during execution.

ET_NODISCARD runtime::Error verify_method_outputs (const std::string &method_name, const size_t testset_idx, double rtol=1e-5, double atol=1e-8)

Verify the output of a specific method with the expected output from the program bundle at the given testset_idx.

This function is a wrapper of verify_method_outputs in bundled_program.

Parameters:

method_name – [in] The name of the method to extract outputs from.
testset_idx – [in] The index of expected output needs to be compared.
rtol – [in] Relative tolerance used for data comparsion.
atol – [in] Absolute tolerance used for data comparsion.

Returns:

Return Error::Ok if two outputs match, or the error happens during execution.

ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name, const std::vector< runtime::EValue > &input_values)

Execute a specific method with the given input values and retrieve the output values. Loads the program and method before executing if needed.

Parameters:

method_name – [in] The name of the method to execute.
input_values – [in] A vector of input values to be passed to the method.

Returns:

A Result object containing either a vector of output values from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name, const runtime::EValue &input_value)

Execute a specific method with a single input value. Loads the program and method before executing if needed.

Parameters:

method_name – [in] The name of the method to execute.
input_value – [in] A value to be passed to the method.

Returns:

A Result object containing either a vector of output values from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name)

Execute a specific method without any input values. Loads the program and method before executing if needed.

Parameters:: method_name – [in] The name of the method to execute.
Returns:: A Result object containing either a vector of output values from the method or an error to indicate failure.

Public Static Functions

static ET_NODISCARD runtime::Result< std::unique_ptr< BundledModule > > from_file (const std::string &file_path, std::unique_ptr< runtime::MemoryAllocator > memory_allocator=nullptr, std::unique_ptr< runtime::MemoryAllocator > temp_allocator=nullptr, std::unique_ptr< runtime::EventTracer > event_tracer=nullptr, std::unique_ptr< runtime::DataLoader > data_map_loader=nullptr)

Constructs an instance by loading a bundled program from a file with specified memory locking behavior.

Parameters:

file_path – [in] The path to the ExecuTorch bundled program file to load.
memory_allocator – [in] A MemoryAllocator used for memory management.
temp_allocator – [in] A MemoryAllocator to use when allocating temporary data during kernel or delegate execution.
event_tracer – [in] A EventTracer used for tracking and logging events.
data_map_loader – [in] A DataLoader used for loading external weights.

Tensor Extension#

The Tensor extension provides managed tensor helpers for C++ applications that need to create, alias, resize, or index tensors before passing them to runtime APIs.

using executorch::extension::TensorPtr = std::shared_ptr<executorch::aten::Tensor>#: A smart pointer type for managing the lifecycle of a Tensor.

template<typename T, ssize_t N> executorch::runtime::Result<TensorAccessor<T, N>> executorch::extension::make_tensor_accessor(const executorch::aten::Tensor &tensor)#

Creates a TensorAccessor<T, N> from the given tensor. The number of dimension N and the data type T’s size must match those of the input tensor. For Executorch tensors, non-trivial dimension order is not supported.

Parameters:

tensor – Origin tensor. The TensorImpl inside must outlive the returned TensorAccessor.

Return values:

Error::InvalidArgument – Mismatch on data type or number of dimensions.
Error::NotSupported – Input tensor has non-trivial dimension onrder.

Returns:

TensorAccessor of the input tensor.

TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, void *data, std::vector<executorch::aten::DimOrderType> dim_order, std::vector<executorch::aten::StridesType> strides, const executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::Device device = executorch::aten::Device(executorch::aten::DeviceType::CPU), const executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND, std::function<void(void*)> deleter = nullptr)#

Creates a TensorPtr that manages a Tensor with the specified properties.

The device parameter sets the Tensor’s device location only — no data is allocated or copied. The caller is responsible for ensuring data already lives on the requested device; construct the executorch::aten::Device from the runtime environment and pass it in. To copy CPU data to a device, use clone_tensor_ptr_to instead.

Parameters:

sizes – A vector specifying the size of each dimension.
data – A pointer to the data buffer (CPU or device, see device).
dim_order – A vector specifying the order of dimensions.
strides – A vector specifying the strides of the tensor.
type – The scalar type of the tensor elements.
device – The device on which data resides (default CPU).
dynamism – Specifies the mutability of the tensor’s shape.
deleter – A custom deleter function for managing the lifetime of the data buffer. If provided, this deleter will be called when the managed Tensor object is destroyed.

Returns:

A TensorPtr that manages the newly created Tensor.

inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, void *data, const executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::Device device = executorch::aten::Device(executorch::aten::DeviceType::CPU), const executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND, std::function<void(void*)> deleter = nullptr)#

Creates a TensorPtr that manages a Tensor with the specified properties.

Convenience overload for the primary factory; see the primary overload for device semantics.

Parameters:

sizes – A vector specifying the size of each dimension.
data – A pointer to the data buffer (CPU or device, see device).
type – The scalar type of the tensor elements.
device – The device on which data resides (default CPU).
dynamism – Specifies the mutability of the tensor’s shape.
deleter – A custom deleter function for managing the lifetime of the data buffer.

Returns:

A TensorPtr that manages the newly created Tensor.

template<typename T, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value> inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, T *data, std::vector<executorch::aten::DimOrderType> dim_order, std::vector<executorch::aten::StridesType> strides, const executorch::aten::ScalarType type = deduced_type, executorch::aten::Device device = executorch::aten::Device(executorch::aten::DeviceType::CPU), const executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND, std::function<void(void*)> deleter = nullptr)#

Creates a TensorPtr that manages a Tensor with the specified properties.

Overload for a typed data pointer. The scalar type is deduced from the pointer’s element type, the same way the vector overload deduces it from the vector’s element type, so a buffer of one type does not become a Tensor of another. Pass type explicitly to reinterpret the buffer as a different type.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the pointer.

Parameters:

sizes – A vector specifying the size of each dimension.
data – A pointer to the data buffer (CPU or device, see device).
dim_order – A vector specifying the order of dimensions.
strides – A vector specifying the strides of the tensor.
type – The scalar type of the tensor elements.
device – The device on which data resides (default CPU).
dynamism – Specifies the mutability of the tensor’s shape.
deleter – A custom deleter function for managing the lifetime of the data buffer.

Returns:

A TensorPtr that manages the newly created Tensor.

template<typename T, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value> inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, T *data, const executorch::aten::ScalarType type = deduced_type, executorch::aten::Device device = executorch::aten::Device(executorch::aten::DeviceType::CPU), const executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND, std::function<void(void*)> deleter = nullptr)#

Creates a TensorPtr that manages a Tensor with the specified properties.

Convenience overload for a typed data pointer; see the typed overload above for type deduction.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the pointer.

Parameters:

sizes – A vector specifying the size of each dimension.
data – A pointer to the data buffer (CPU or device, see device).
type – The scalar type of the tensor elements.
device – The device on which data resides (default CPU).
dynamism – Specifies the mutability of the tensor’s shape.
deleter – A custom deleter function for managing the lifetime of the data buffer.

Returns:

A TensorPtr that manages the newly created Tensor.

template<typename T = float, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value> inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, std::vector<T> data, std::vector<executorch::aten::DimOrderType> dim_order = {}, std::vector<executorch::aten::StridesType> strides = {}, executorch::aten::ScalarType type = deduced_type, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This template overload is specialized for cases where the tensor data is provided as a vector. The scalar type is automatically deduced from the vector’s data type. If the specified type differs from the deduced type of the vector’s elements, and casting is allowed, the data will be cast to the specified type. This allows for flexible creation of tensors with data vectors of one type and a different scalar type.

The result is always a CPU tensor. To move it to a device, use clone_tensor_ptr_to.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the vector.

Parameters:

sizes – A vector specifying the size of each dimension.
data – A vector containing the tensor’s data.
dim_order – A vector specifying the order of dimensions.
strides – A vector specifying the strides of each dimension.
type – The scalar type of the tensor elements. If it differs from the deduced type, the data will be cast to this type if allowed.
dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr that manages the newly created TensorImpl.

template<typename T = float, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value> inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<T> data, executorch::aten::ScalarType type = deduced_type, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This template overload is specialized for cases where the tensor data is provided as a vector. The scalar type is automatically deduced from the vector’s data type.

The result is always a CPU tensor. To move it to a device, use clone_tensor_ptr_to.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the vector.

Parameters:

data – A vector containing the tensor’s data.
type – The scalar type of the tensor elements. If it differs from the deduced type, the data will be cast to this type if allowed.
dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr that manages the newly created TensorImpl.

template<typename T = float, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value> inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, std::initializer_list<T> list, std::vector<executorch::aten::DimOrderType> dim_order = {}, std::vector<executorch::aten::StridesType> strides = {}, executorch::aten::ScalarType type = deduced_type, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This template overload is specialized for cases where the tensor data is provided as an initializer list. The scalar type is automatically deduced from the initializer list’s data type.

The result is always a CPU tensor. To move it to a device, use clone_tensor_ptr_to.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the initializer list.

Parameters:

sizes – A vector specifying the size of each dimension.
list – An initializer list containing the tensor’s data.
dim_order – A vector specifying the order of dimensions.
strides – A vector specifying the strides of each dimension.
type – The scalar type of the tensor elements. If it differs from the deduced type, the data will be cast to this type if allowed.
dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr that manages the newly created TensorImpl.

template<typename T = float, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value> inline TensorPtr executorch::extension::make_tensor_ptr(std::initializer_list<T> list, executorch::aten::ScalarType type = deduced_type, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This template overload allows creating a Tensor from an initializer list of data. The scalar type is automatically deduced from the type of the initializer list’s elements.

The result is always a CPU tensor. To move it to a device, use clone_tensor_ptr_to.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the initializer list.

Parameters:

list – An initializer list containing the tensor’s data.
type – The scalar type of the tensor elements. If it differs from the deduced type, the data will be cast to this type if allowed.
dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr that manages the newly created TensorImpl.

template<typename T> inline TensorPtr executorch::extension::make_tensor_ptr(T value)#

Creates a TensorPtr that manages a Tensor with a single scalar value.

Template Parameters:: T – The C++ type of the scalar value.
Parameters:: value – The scalar value to be used for the Tensor.
Returns:: A TensorPtr that manages the newly created TensorImpl.

TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, std::vector<uint8_t> data, std::vector<executorch::aten::DimOrderType> dim_order, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This overload accepts a raw memory buffer stored in a std::vector<uint8_t> and a scalar type to interpret the data. The vector is managed, and the memory’s lifetime is tied to the TensorImpl. The result is always a CPU tensor.

Parameters:

sizes – A vector specifying the size of each dimension.
data – A vector containing the raw memory for the tensor’s data.
dim_order – A vector specifying the order of dimensions.
strides – A vector specifying the strides of each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr managing the newly created Tensor.

inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, std::vector<uint8_t> data, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

Convenience overload for the raw-buffer factory; see above. The result is always a CPU tensor.

Parameters:

sizes – A vector specifying the size of each dimension.
data – A vector containing the raw memory for the tensor’s data.
type – The scalar type of the tensor elements.
dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr managing the newly created Tensor.

inline TensorPtr executorch::extension::make_tensor_ptr(const executorch::aten::Tensor &tensor, std::vector<executorch::aten::SizesType> sizes = {}, std::vector<executorch::aten::DimOrderType> dim_order = {}, std::vector<executorch::aten::StridesType> strides = {}, std::function<void(void*)> deleter = nullptr)#

Creates a TensorPtr to manage a new Tensor that aliases the given Tensor’s storage, with optional metadata overrides. Shape dynamism is inherited from the source tensor.

If an override is provided (non-empty), it is passed as-is. If an override is empty, the corresponding metadata is reused from the source tensor when it fits; otherwise it is left empty for the core factory to derive a valid configuration. If dim_order is empty but strides is provided, dim_order is left empty so the core may infer it from the provided strides.

This overload always aliases — it never copies. To copy a tensor’s data to a device, use clone_tensor_ptr_to.

Parameters:

tensor – The source tensor to alias.
sizes – Optional sizes override.
dim_order – Optional dimension order override.
strides – Optional strides override.
deleter – A custom deleter function for managing the lifetime of the original Tensor.

Returns:

A TensorPtr aliasing the same storage with requested metadata.

inline TensorPtr executorch::extension::make_tensor_ptr(const TensorPtr &tensor_ptr, std::vector<executorch::aten::SizesType> sizes = {}, std::vector<executorch::aten::DimOrderType> dim_order = {}, std::vector<executorch::aten::StridesType> strides = {})#

Convenience overload identical to make_tensor_ptr(*tensor_ptr, …). Keeps the original TensorPtr alive until the returned TensorPtr is destroyed.

This overload always aliases — it never copies. To copy a tensor’s data to a device, use clone_tensor_ptr_to.

Parameters:

tensor_ptr – The source tensor pointer to alias.
sizes – Optional sizes override.
dim_order – Optional dimension order override.
strides – Optional strides override.

Returns:

A TensorPtr aliasing the same storage with requested metadata.

TensorPtr executorch::extension::clone_tensor_ptr(const executorch::aten::Tensor &tensor, executorch::aten::ScalarType type)#

Creates a TensorPtr that manages a new Tensor with the same properties as the given Tensor, but with a copy of the data owned by the returned TensorPtr, or nullptr if the original data is null.

Parameters:

tensor – The Tensor to clone.
type – The data type for the cloned tensor. The data will be cast from the source tensor’s type.

Returns:

A new TensorPtr that manages a Tensor with the specified type and copied/cast data.

inline TensorPtr executorch::extension::clone_tensor_ptr(const executorch::aten::Tensor &tensor)#

Creates a TensorPtr that manages a new Tensor with the same properties as the given Tensor, but with a copy of the data owned by the returned TensorPtr, or nullptr if the original data is null.

Parameters:: tensor – The Tensor to clone.
Returns:: A new TensorPtr that manages a Tensor with the same properties as the original but with copied data.

inline TensorPtr executorch::extension::clone_tensor_ptr(const TensorPtr &tensor, executorch::aten::ScalarType type)#

Creates a new TensorPtr by cloning the given TensorPtr, copying the underlying data.

Parameters:

tensor – The TensorPtr to clone.
type – The data type for the cloned tensor. The data will be cast from the source tensor’s type.

Returns:

A new TensorPtr that manages a Tensor with the specified type and copied/cast data.

inline TensorPtr executorch::extension::clone_tensor_ptr(const TensorPtr &tensor)#

Creates a new TensorPtr by cloning the given TensorPtr, copying the underlying data.

Parameters:: tensor – The TensorPtr to clone.
Returns:: A new TensorPtr that manages a Tensor with the same properties as the original but with copied data.

ET_NODISCARD runtime::Error executorch::extension::resize_tensor_ptr (TensorPtr &tensor, const std::vector< executorch::aten::SizesType > &sizes)

Resizes the Tensor managed by the provided TensorPtr to the new sizes.

Parameters:

tensor – A TensorPtr managing the Tensor to resize.
sizes – A vector representing the new sizes for each dimension.

Returns:

Error::Ok on success, or an appropriate error code on failure.

TensorPtr executorch::extension::clone_tensor_ptr_to(const TensorPtr &tensor, executorch::aten::Device target)#

Clones a TensorPtr’s data onto the given target device, allocating and copying as needed.

The transfer direction is inferred from the source and target device: host-to-device when target is an accelerator, and device-to-host when target is CPU. Copies use the DeviceAllocator registered for the accelerator side; a device-backed result owns its memory and frees it via that allocator when destroyed.

Source and target must differ in device domain: for a CPU-to-CPU copy use clone_tensor_ptr, and device-to-device transfers are not supported.

Only available in the ExecuTorch portable build: it relies on the ExecuTorch DeviceAllocator, which has no equivalent in USE_ATEN_LIB builds.

Parameters:

tensor – The source tensor whose data will be copied.
target – The destination device (CPU or an accelerator).

Returns:

A TensorPtr backed by target memory containing the copied data.

inline TensorPtrMaker executorch::extension::for_blob(void *data, std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float)#

Creates a TensorPtrMaker instance for building a TensorPtr from a raw data pointer and tensor sizes.

The TensorPtrMaker returned by this function allows for further customization of the tensor’s properties, such as data type, dimension order, strides, and shape dynamism, before finalizing the TensorPtr creation.

Parameters:

data – A pointer to the raw data to be used by the tensor. It must outlive the TensorPtr created by this function.
sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.

Returns:

A TensorPtrMaker instance for creating a TensorPtr.

template<typename T, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value> inline TensorPtrMaker executorch::extension::for_blob(T *data, std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = deduced_type)#

Creates a TensorPtrMaker from a typed data pointer and tensor sizes.

The scalar type is deduced from the pointer’s element type, so a buffer of one type does not become a Tensor of another. Pass type explicitly to reinterpret the buffer as a different type.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the pointer.

Parameters:

data – A pointer to the raw data used by the tensor.
sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.

Returns:

A TensorPtrMaker instance for creating a TensorPtr.

inline TensorPtr executorch::extension::from_blob(void *data, std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::Device device = executorch::aten::Device(executorch::aten::DeviceType::CPU), executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr from a raw data pointer and tensor sizes, with an optional dynamism setting.

This function provides a convenient way to create a tensor from existing data, with the option to specify whether the tensor’s shape is static or dynamic.

Parameters:

data – A pointer to the raw data used by the tensor. The data must outlive the TensorPtr created by this function.
sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.
device – The device on which data resides (default CPU).
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

template<typename T, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value> inline TensorPtr executorch::extension::from_blob(T *data, std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = deduced_type, executorch::aten::Device device = executorch::aten::Device(executorch::aten::DeviceType::CPU), executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr from a typed data pointer and tensor sizes.

The scalar type is deduced from the pointer’s element type; see for_blob.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the pointer.

Parameters:

data – A pointer to the raw data used by the tensor. The data must outlive the TensorPtr created by this function.
sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.
device – The device on which data resides (default CPU).
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::from_blob(void *data, std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::Device device = executorch::aten::Device(executorch::aten::DeviceType::CPU), executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr from a raw data pointer, tensor sizes, and strides, with an optional dynamism setting.

This function allows for the creation of a tensor from existing data, with the option to specify custom strides for each dimension and whether the tensor’s shape is static, dynamic, or bounded.

Parameters:

data – A pointer to the raw data used by the tensor. The data must outlive the TensorPtr created by this function.
sizes – A vector specifying the size of each dimension.
strides – A vector specifying the stride for each dimension.
type – The scalar type of the tensor elements.
device – The device on which data resides (default CPU).
dynamism – Specifies whether the tensor’s shape is static, dynamic, or bounded.

Returns:

A TensorPtr instance managing the newly created Tensor.

template<typename T, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value> inline TensorPtr executorch::extension::from_blob(T *data, std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = deduced_type, executorch::aten::Device device = executorch::aten::Device(executorch::aten::DeviceType::CPU), executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr from a typed data pointer, tensor sizes, and strides.

The scalar type is deduced from the pointer’s element type; see for_blob.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the pointer.

Parameters:

data – A pointer to the raw data used by the tensor. The data must outlive the TensorPtr created by this function.
sizes – A vector specifying the size of each dimension.
strides – A vector specifying the stride for each dimension.
type – The scalar type of the tensor elements.
device – The device on which data resides (default CPU).
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::from_blob(void *data, std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type, executorch::aten::Device device, std::function<void(void*)> &&deleter, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr from a raw data pointer and tensor sizes, with an optional dynamism setting.

This function is a convenient way to create a tensor from existing data, with the option to specify whether the tensor’s shape is static, dynamic, or bounded.

Parameters:

data – A pointer to the raw data to be used by the tensor. It must outlive the TensorPtr created by this function.
sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.
device – The device on which data resides.
deleter – A function to delete the data when it’s no longer needed.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance that manages the newly created Tensor.

inline TensorPtr executorch::extension::from_blob(void *data, std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type, executorch::aten::Device device, std::function<void(void*)> &&deleter, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr from a raw data pointer, tensor sizes, and strides, with an optional dynamism setting.

This function allows for the creation of a tensor from existing data, with the option to specify custom strides for each dimension and whether the tensor’s shape is static, dynamic, or bounded.

Parameters:

data – A pointer to the raw data to be used by the tensor. It must outlive the TensorPtr created by this function.
sizes – A vector specifying the size of each dimension.
strides – A vector specifying the stride for each dimension.
type – The scalar type of the tensor elements.
device – The device on which data resides.
deleter – A function to delete the data when it’s no longer needed.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance that manages the newly created Tensor.

TensorPtr executorch::extension::empty_strided(std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr with the specified sizes, strides, and properties.

This function allocates memory for the tensor elements but does not initialize them with any specific values. The tensor is created with the specified strides.

Parameters:

sizes – A vector specifying the size of each dimension.
strides – A vector specifying the stride for each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::empty_like(const TensorPtr &other, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates an empty TensorPtr with the same size and properties as the given tensor.

This function allocates memory for the tensor elements but does not initialize them with any specific values.

Parameters:

other – A reference to another tensor, whose size and properties are used.
type – The scalar type of the tensor elements. If not provided, the scalar type of the other tensor is used.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::empty(std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates an empty TensorPtr with the specified sizes and properties.

This function allocates memory for the tensor elements but does not initialize them with any specific values.

Parameters:

sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

TensorPtr executorch::extension::full_strided(std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::Scalar fill_value, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with the specified value.

Parameters:

sizes – A vector specifying the size of each dimension.
strides – A vector specifying the stride for each dimension.
fill_value – The value to fill the tensor with.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::full_like(const TensorPtr &other, executorch::aten::Scalar fill_value, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with the specified value, with the same size and properties as another tensor.

Parameters:

other – A reference to another tensor, whose size and properties will be used.
fill_value – The value to fill the tensor with.
type – The scalar type of the tensor elements. If not specified, the scalar type of the other tensor is used.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::full(std::vector<executorch::aten::SizesType> sizes, executorch::aten::Scalar fill_value, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with the specified value.

Parameters:

sizes – A vector specifying the size of each dimension.
fill_value – The value used to fill the tensor.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::scalar_tensor(executorch::aten::Scalar value, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float)#

Creates a TensorPtr holding a scalar value.

Parameters:

value – The scalar value for the tensor.
type – The scalar type of the tensor elements.

Returns:

A TensorPtr instance managing the newly created scalar Tensor.

inline TensorPtr executorch::extension::ones_like(const TensorPtr &other, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with ones, with the same size and properties as another tensor.

Parameters:

other – A reference to another tensor, whose size and properties are used.
type – The scalar type of the tensor elements. If not provided, the scalar type of the other tensor is used.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::ones(std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with ones.

Parameters:

sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::zeros_like(const TensorPtr &other, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with zeros, with the same size and properties as another tensor.

Parameters:

other – A reference to another tensor, whose size and properties will be used.
type – The scalar type of the tensor elements. If not specified, the scalar type of the other tensor is used.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::zeros(std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with zeros.

Parameters:

sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

TensorPtr executorch::extension::rand_strided(std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values between 0 and 1.

Parameters:

sizes – A vector specifying the size of each dimension.
strides – A vector specifying the stride for each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::rand_like(const TensorPtr &other, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values between 0 and 1.

Parameters:

other – A reference to another tensor, whose size and properties will be used.
type – The scalar type of the tensor elements. If not specified, the scalar type of the other tensor is used.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::rand(std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values between 0 and 1.

Parameters:

sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

TensorPtr executorch::extension::randn_strided(std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values from a normal distribution.

Parameters:

sizes – A vector specifying the size of each dimension.
strides – A vector specifying the stride for each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::randn_like(const TensorPtr &other, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values from a normal distribution.

Parameters:

other – A reference to another tensor, whose size and properties will be used.
type – The scalar type of the tensor elements. If not specified, the scalar type of the other tensor is used.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::randn(std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values from a normal distribution.

Parameters:

sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

TensorPtr executorch::extension::randint_strided(int64_t low, int64_t high, std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Int, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random integer values in the given range.

Parameters:

low – The lower bound (inclusive) of the random values.
high – The upper bound (exclusive) of the random values.
sizes – A vector specifying the size of each dimension.
strides – A vector specifying the stride for each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::randint_like(const TensorPtr &other, int64_t low, int64_t high, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random integer values in the given range.

Parameters:

other – A reference to another tensor, whose size and properties will be used.
low – The lower bound (inclusive) of the random values.
high – The upper bound (exclusive) of the random values.
type – The scalar type of the tensor elements. If not specified, the scalar type of the other tensor is used.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::randint(int64_t low, int64_t high, std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Int, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random integer values within the specified range.

Parameters:

low – The inclusive lower bound of the random values.
high – The exclusive upper bound of the random values.
sizes – A vector specifying the size of each dimension.
type – The scalar type of the tensor elements.
dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

template<typename T, ssize_t N> class TensorAccessor : public executorch::extension::internal::TensorAccessorBase<T, N>#

#include <tensor_accessor.h>

TensorAccessor template with data type and rank as template parameters. No public constructors, can only be created using make_tensor_accessor from a given executorch::aten::Tensor. Use operator[] to index and obtain a lower rank accessor or the underlying scalar value.

Public Functions

inline TensorAccessor<T, N - 1> operator[](ssize_t i)#

Index into the the outer most dimension.

Parameters:: i – Index.
Returns:: If N > 1, a TensorAccessor with N-1 dimensions. If N == 1, a reference to the underlying scalar. Refer to the TensorAccessor<T, 1> specialization.

inline const TensorAccessor<T, N - 1> operator[](ssize_t i) const#

Index into the the outer most dimension.

Parameters:: i – Index.
Returns:: If N > 1, a constant TensorAccessor with N-1 dimensions. If N == 1, a constant reference to the underlying scalar. Refer to the TensorAccessor<T, 1> specialization.

template<typename T> class TensorAccessor<T, 1> : public executorch::extension::internal::TensorAccessorBase<T, 1>#

#include <tensor_accessor.h>

TensorAccessor specialization for N == 1, where operator[] returns a reference to the underlying scalar.

Public Functions

inline T &operator[](ssize_t i)#

Index into the the outer most dimension.

Parameters:: i – Index.
Returns:: Reference to the underlying scalar.

inline const T &operator[](ssize_t i) const#

Index into the the outer most dimension.

Parameters:: i – Index.
Returns:: Constant reference to the underlying scalar.

class TensorPtrMaker#

#include <tensor_ptr_maker.h>

A helper class for creating TensorPtr instances from raw data and tensor properties. Note that the TensorPtr created by this class does not own the data, so the data must outlive the TensorPtr.

TensorPtrMaker provides a fluent interface for specifying various tensor properties, such as type, sizes, data pointer, dimension order, strides, and shape dynamism. The final tensor is created by invoking make_tensor_ptr() or by converting TensorPtrMaker to TensorPtr.

Public Functions

inline TensorPtrMaker &&type(executorch::aten::ScalarType type)#

Sets the scalar type of the tensor elements.

Parameters:: type – The scalar type (e.g., float, int, bool).
Returns:: Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtrMaker &&dim_order(std::vector<executorch::aten::DimOrderType> dim_order)#

Sets the order of dimensions in memory.

Parameters:: dim_order – A vector specifying the dimension order.
Returns:: Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtrMaker &&strides(std::vector<executorch::aten::StridesType> strides)#

Sets the strides for each dimension of the tensor.

Parameters:: strides – A vector specifying the stride for each dimension.
Returns:: Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtrMaker &&dynamism(executorch::aten::TensorShapeDynamism dynamism)#

Sets the shape dynamism of the tensor.

Parameters:: dynamism – Specifies whether the tensor’s shape is static, dynamic, or bounded.
Returns:: Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtrMaker &&deleter(std::function<void(void*)> &&deleter)#

Sets a custom deleter function to manage the lifetime of the data buffer.

Parameters:: deleter – A function that will be called to delete the data buffer when the Tensor object managed by the TensorPtr is destroyed. Explicitly consuming an rvalue to avoid unnecessary copies when the deleter is a lambda that has captured some state.
Returns:: Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtrMaker &&device(executorch::aten::Device device)#

Labels the tensor with the device where its buffer already lives.

No data is moved, matching the corresponding make_tensor_ptr parameter.

Parameters:: device – The device on which the data buffer resides. Give the index explicitly: a device constructed without one is index 0 in this build and -1 in the ATen build, where it then compares unequal to the same device carrying an explicit index.
Returns:: Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtr make_tensor_ptr() &&#

Creates and returns a TensorPtr instance using the properties set in this TensorPtrMaker.

Returns:: A TensorPtr instance that manages the newly created Tensor.

inline operator TensorPtr() &&#

Implicit conversion operator to create a TensorPtr.

Returns:: A TensorPtr instance that manages the newly created Tensor.

Runtime API Reference#

Model Loading and Execution#

Values#

Module Extension#

Tensor Extension#

Docs

Tutorials

Resources