Rate this Page
★ ★ ★ ★ ★

Runtime API Reference#

The ExecuTorch C++ API provides an on-device execution framework for exported PyTorch models.

For a tutorial style introduction to the runtime API, check out the using executorch with cpp tutorial and its simplified version.

For detailed information on how APIs evolve and the deprecation process, please refer to the ExecuTorch API Life Cycle and Deprecation Policy.

Model Loading and Execution#

Warning

doxygenclass: Cannot find class “executorch::runtime::Program” in doxygen xml output for project “ExecuTorch” from directory: ../build/xml/

Warning

doxygenclass: Cannot find class “executorch::runtime::Method” in doxygen xml output for project “ExecuTorch” from directory: ../build/xml/

Warning

doxygenclass: Cannot find class “executorch::runtime::MethodMeta” in doxygen xml output for project “ExecuTorch” from directory: ../build/xml/

class DataLoader#

Loads from a data source.

See //executorch/extension/data_loader for common implementations.

Public Functions

virtual ET_NODISCARD Result< FreeableBuffer > load (size_t offset, size_t size, const SegmentInfo &segment_info) const =0

Loads data from the underlying data source.

NOTE: This must be thread-safe. If this call modifies common state, the implementation must do its own locking.

Parameters:

  • offset – The byte offset in the data source to start loading from.

  • size – The number of bytes to load.

  • segment_info – Information about the segment being loaded.

Returns:

a FreeableBuffer that owns the loaded data.

inline virtual ET_NODISCARD Error load_into (size_t offset, size_t size, const SegmentInfo &segment_info, void *buffer) const

Loads data from the underlying data source into the provided buffer.

NOTE: This must be thread-safe. If this call modifies common state, the implementation must do its own locking.

Parameters:

  • offset – The byte offset in the data source to start loading from.

  • size – The number of bytes to load.

  • segment_info – Information about the segment being loaded.

  • buffer – The buffer to load data into. Must point to at least size bytes of memory.

Returns:

an Error indicating if the load was successful.

virtual ET_NODISCARD Result< size_t > size () const =0

Returns the length of the underlying data source, typically the file size.

struct SegmentInfo#

Describes the content of the segment.

Public Types

enum class Type#

Represents the purpose of the segment.

Values:

enumerator Program#

Data for the actual program.

enumerator Constant#

Holds constant tensor data.

enumerator Backend#

Data used for initializing a backend.

enumerator Mutable#

Data used for initializing mutable tensors.

enumerator External#

Data used for initializing external tensors.

Public Members

Type segment_type#

Type of the segment.

size_t segment_index#

Index of the segment within the segment list. Undefined for program segments.

const char *descriptor#

An optional, null-terminated string describing the segment. For Backend segments, this is the backend ID. Null for other segment types.

class MemoryAllocator#

A class that does simple allocation based on a size and returns the pointer to the memory address. It bookmarks a buffer with certain size. The allocation is simply checking space and growing the cur_ pointer with each allocation request.

Simple example:

// User allocates a 100 byte long memory in the heap. uint8_t* memory_pool = malloc(100 * sizeof(uint8_t)); MemoryAllocator allocator(100, memory_pool) // Pass allocator object in the Executor

Underneath the hood, ExecuTorch will call allocator.allocate() to keep iterating cur_ pointer

Public Functions

inline MemoryAllocator(uint32_t size, uint8_t *base_address)#

Constructs a new memory allocator of a given size, starting at the provided base_address.

Parameters:

  • size – [in] The size in bytes of the buffer at base_address.

  • base_address – [in] The buffer to allocate from. Does not take ownership of this buffer, so it must be valid for the lifetime of of the MemoryAllocator.

inline virtual void *allocate(size_t size, size_t alignment = kDefaultAlignment)#

Allocates size bytes of memory.

Parameters:

  • size – [in] Number of bytes to allocate.

  • alignment – [in] Minimum alignment for the returned pointer. Must be a power of 2.

Return values:

nullptr – Not enough memory, or alignment was not a power of 2.

Returns:

Aligned pointer to the allocated memory on success.

template<typename T>
inline T *allocateInstance(size_t alignment = alignof(T))#

Allocates a buffer large enough for an instance of type T. Note that the memory will not be initialized.

Example: 

auto p = memory_allocator->allocateInstance<MyType>();

Parameters:

alignment – [in] Minimum alignment for the returned pointer. Must be a power of 2. Defaults to the natural alignment of T.

Return values:

nullptr – Not enough memory, or alignment was not a power of 2.

Returns:

Aligned pointer to the allocated memory on success.

template<typename T>
inline T *allocateList(size_t size, size_t alignment = alignof(T))#

Allocates size number of chunks of type T, where each chunk is of size equal to sizeof(T) bytes.

Parameters:

  • size – [in] Number of memory chunks to allocate.

  • alignment – [in] Minimum alignment for the returned pointer. Must be a power of 2. Defaults to the natural alignment of T.

Return values:

nullptr – Not enough memory, or alignment was not a power of 2.

Returns:

Aligned pointer to the allocated memory on success.

Public Static Attributes

static constexpr size_t kDefaultAlignment = alignof(void*)#

Default alignment of memory returned by this class. Ensures that pointer fields of structs will be aligned. Larger types like long double may not be, however, depending on the toolchain and architecture.

class HierarchicalAllocator#

A group of buffers that can be used to represent a device’s memory hierarchy.

Public Functions

inline explicit HierarchicalAllocator(Span<Span<uint8_t>> buffers)#

Constructs a new hierarchical allocator with the given array of buffers.

  • Memory IDs are based on the index into buffers: buffers[N] will have a memory ID of N.

  • buffers.size() must be >= MethodMeta::num_non_const_buffers().

  • buffers[N].size() must be >= MethodMeta::non_const_buffer_size(N).

inline ET_DEPRECATED HierarchicalAllocator(uint32_t n_allocators, MemoryAllocator *allocators)#

DEPRECATED: Use spans instead.

inline ET_NODISCARD Result< void * > get_offset_address (uint32_t memory_id, size_t offset_bytes, size_t size_bytes)

Returns the address at the byte offset offset_bytes from the given buffer’s base address, which points to at least size_bytes of memory.

Parameters:

  • memory_id – [in] The ID of the buffer in the hierarchy.

  • offset_bytes – [in] The offset in bytes into the specified buffer.

  • size_bytes – [in] The amount of memory that should be available at the offset.

Returns:

On success, the address of the requested byte offset into the specified buffer. On failure, a non-Ok Error.

class MemoryManager#

A container class for allocators used during Method load and execution.

This class consolidates all dynamic memory needs for Method load and execution. This can allow for heap-based as well as heap-less execution (relevant to some embedded scenarios), and overall provides more control over memory use.

This class, however, cannot ensure all allocation is accounted for since kernel and backend implementations are free to use a separate way to allocate memory (e.g., for things like scratch space). But we do suggest that backends and kernels use these provided allocators whenever possible.

Public Functions

inline explicit MemoryManager(MemoryAllocator *method_allocator, HierarchicalAllocator *planned_memory = nullptr, MemoryAllocator *temp_allocator = nullptr)#

Constructs a new MemoryManager.

Parameters:

  • method_allocator – [in] The allocator to use when loading a Method and allocating its internal structures. Must outlive the Method that uses it.

  • planned_memory – [in] The memory-planned buffers to use for mutable tensor data when executing a Method. Must outlive the Method that uses it. May be nullptr if the Method does not use any memory-planned tensor data. The sizes of the buffers in this HierarchicalAllocator must agree with the corresponding MethodMeta::num_memory_planned_buffers() and MethodMeta::memory_planned_buffer_size(N) values, which are embedded in the Program.

  • temp_allocator – [in] The allocator to use when allocating temporary data during kernel or delegate execution. Must outlive the Method that uses it. May be nullptr if the Method does not use kernels or delegates that allocate temporary data. This allocator will be reset after every kernel or delegate call during execution.

inline ET_DEPRECATED MemoryManager(MemoryAllocator *constant_allocator, HierarchicalAllocator *non_constant_allocator, MemoryAllocator *runtime_allocator, MemoryAllocator *temporary_allocator)#

DEPRECATED: Use the constructor without constant_allocator instead.

TODO(T162089316): Remove this once all users migrate to the new ctor.

inline MemoryAllocator *method_allocator() const#

Returns the allocator that the runtime will use to allocate internal structures while loading a Method. Must not be used after its associated Method has been loaded.

inline HierarchicalAllocator *planned_memory() const#

Returns the memory-planned buffers to use for mutable tensor data.

inline MemoryAllocator *temp_allocator() const#

Returns the allocator to use for allocating temporary data during kernel or delegate execution.

This allocator will be reset after every kernel or delegate call during execution.

Values#

struct EValue#

Public Functions

inline EValue(executorch::aten::Scalar s)#

Construct an EValue using the implicit value of a Scalar.

template<typename T>
Result<T> tryTo() const#

Result-returning equivalent of to<T>(). Tag mismatch returns Error::InvalidType; a null list/string payload returns Error::InvalidState. Specializations are defined below via EVALUE_DEFINE_TRY_TO.

template<typename T>
inline std::optional<T> toOptional() const#

Converts the EValue to an optional object that can represent both T and an uninitialized state.

template<typename T>
inline Result<std::optional<T>> tryToOptional() const#

Result-returning equivalent of toOptional<T>(). None maps to an empty optional; any other tag that doesn’t match T propagates tryTo<T>()’s error (Error::InvalidType).

union Payload#
union TriviallyCopyablePayload#
class Tensor#

A minimal Tensor type whose API is a source compatible subset of at::Tensor.

NOTE: Instances of this class do not own the TensorImpl given to it, which means that the caller must guarantee that the TensorImpl lives longer than any Tensor instances that point to it.

See the documention on TensorImpl for details about the return/parameter types used here and how they relate to at::Tensor.

Public Types

using SizesType = TensorImpl::SizesType#

The type used for elements of sizes().

using DimOrderType = TensorImpl::DimOrderType#

The type used for elements of dim_order().

using StridesType = TensorImpl::StridesType#

The type used for elements of strides().

Public Functions

inline TensorImpl *unsafeGetTensorImpl() const#

Returns a pointer to the underlying TensorImpl.

NOTE: Clients should be wary of operating on the TensorImpl directly instead of the Tensor. It is easy to break things.

inline size_t nbytes() const#

Returns the size of the tensor in bytes.

NOTE: Only the alive space is returned not the total capacity of the underlying data blob.

inline ssize_t size(ssize_t dim) const#

Returns the size of the tensor at the given dimension.

NOTE: that size() intentionally does not return SizeType even though it returns an element of an array of SizeType. This is to help make calls of this method more compatible with at::Tensor, and more consistent with the rest of the methods on this class and in ETensor.

inline ssize_t dim() const#

Returns the tensor’s number of dimensions.

inline ssize_t numel() const#

Returns the number of elements in the tensor.

inline ScalarType scalar_type() const#

Returns the type of the elements in the tensor (int32, float, bool, etc).

inline ssize_t element_size() const#

Returns the size in bytes of one element of the tensor.

inline const ArrayRef<SizesType> sizes() const#

Returns the sizes of the tensor at each dimension.

inline const ArrayRef<DimOrderType> dim_order() const#

Returns the order the dimensions are laid out in memory.

inline const ArrayRef<StridesType> strides() const#

Returns the strides of the tensor at each dimension.

inline TensorShapeDynamism shape_dynamism() const#

Returns the mutability of the shape of the tensor.

template<typename T>
inline const T *const_data_ptr() const#

Returns a pointer of type T to the constant underlying data blob.

inline const void *const_data_ptr() const#

Returns a pointer to the constant underlying data blob.

template<typename T>
inline T *mutable_data_ptr() const#

Returns a pointer of type T to the mutable underlying data blob.

inline void *mutable_data_ptr() const#

Returns a pointer to the mutable underlying data blob.

template<typename T> inline ET_DEPRECATED T * data_ptr () const

DEPRECATED: Use const_data_ptr or mutable_data_ptr instead.

inline ET_DEPRECATED void * data_ptr () const

DEPRECATED: Use const_data_ptr or mutable_data_ptr instead.

inline ET_DEPRECATED void set_data (void *ptr) const

DEPRECATED: Changes the data_ptr the tensor aliases. Does not free the previously pointed to data, does not assume ownership semantics of the new ptr. This api does not exist in at::Tensor so kernel developers should avoid it.

Module Extension#

The Module extension provides a higher-level C++ facade for loading programs, setting inputs and outputs, and executing methods with common runtime defaults.

class Module#

A facade class for loading programs and executing methods within them.

Subclassed by executorch::extension::bundled_module::BundledModule

Public Types

enum class LoadMode#

Enum to define loading behavior.

Values:

enumerator File#

Load the whole file as a buffer.

enumerator Mmap#

Use mmap to load pages into memory.

enumerator MmapUseMlock#

Use memory locking and handle errors.

enumerator MmapUseMlockIgnoreErrors#

Use memory locking and ignore errors.

Public Functions

explicit Module(const std::string &file_path, const LoadMode load_mode = LoadMode::File, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, bool share_memory_arenas = false)#

Constructs an instance by loading a program from a file with specified memory locking behavior.

Parameters:

  • file_path – [in] The path to the ExecuTorch program file to load.

  • load_mode – [in] The loading mode to use.

  • event_tracer – [in] A EventTracer used for tracking and logging events.

  • share_memory_arenas – [in] When true, all methods loaded by this Module share the same memory-planned buffers for mem_id=1 (activation memory) and mem_id=2 (shared mutable buffer memory), sized to the max across all methods. mem_id>2 indicates a custom memory plan, and those receive fresh memory buffers. share_memory_arenas is required for models exported with share_mutable_buffers=true, where methods access shared mutable state (e.g., set/get state). When enabled, outputs from one method may be invalidated by executing another method, since their output tensors can alias the same underlying buffer. Consume or copy outputs before calling execute again. NOTE: This class is not thread-safe and performs no internal synchronization. Calling execute concurrently on the same Module instance from multiple threads is unsafe, regardless of whether share_memory_arenas is true or false. When share_memory_arenas is true, methods may overwrite each other’s data in the shared memory arenas, increasing aliasing and the risk of unintended overwrites.

explicit Module(const std::string &file_path, const std::string &data_map_path, const LoadMode load_mode = LoadMode::File, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, bool share_memory_arenas = false)#

Constructs an instance by loading a program from a file with specified memory locking behavior.

Parameters:

  • file_path – [in] The path to the ExecuTorch program file to load.

  • data_map_path – [in] The path to a .ptd file.

  • load_mode – [in] The loading mode to use.

  • event_tracer – [in] A EventTracer used for tracking and logging events.

  • share_memory_arenas – [in] When true, all methods loaded by this Module share a single set of memory-planned buffers.

explicit Module(const std::string &file_path, std::vector<std::string> data_files, const LoadMode load_mode = LoadMode::File, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, bool share_memory_arenas = false)#

Constructs an instance by loading a program from a file with specified memory locking behavior.

Parameters:

  • file_path – [in] The path to the ExecuTorch program file to load.

  • data_files – [in] The path to one or more .ptd file/s.

  • load_mode – [in] The loading mode to use.

  • event_tracer – [in] A EventTracer used for tracking and logging events.

  • share_memory_arenas – [in] When true, all methods loaded by this Module share a single set of memory-planned buffers.

explicit Module(std::unique_ptr<runtime::DataLoader> data_loader, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::DataLoader> data_map_loader = nullptr, bool share_memory_arenas = false)#

Constructs an instance with the provided data loader and memory allocator.

Parameters:

  • data_loader – [in] A DataLoader used for loading program data.

  • memory_allocator – [in] A MemoryAllocator used for memory management.

  • temp_allocator – [in] A MemoryAllocator to use when allocating temporary data during kernel or delegate execution.

  • event_tracer – [in] A EventTracer used for tracking and logging events.

  • data_map_loader – [in] A DataLoader used for loading external weights.

  • share_memory_arenas – [in] When true, all methods loaded by this Module share a single set of memory-planned buffers.

explicit Module(std::shared_ptr<Program> program, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::DataLoader> data_map_loader = nullptr, bool share_memory_arenas = false)#

Constructs an instance using an existing shared program.

Parameters:

  • program – [in] The shared program to use. It’s required the data loader the program uses is valid for the lifetime of the program.

  • memory_allocator – [in] A MemoryAllocator used for memory management.

  • temp_allocator – [in] A MemoryAllocator to use when allocating temporary data.

  • event_tracer – [in] A EventTracer used for tracking and logging events.

  • data_map_loader – [in] A DataLoader used for loading external weights.

  • share_memory_arenas – [in] When true, all methods loaded by this Module share a single set of memory-planned buffers.

virtual ET_NODISCARD runtime::Error load (const Program::Verification verification=Program::Verification::Minimal)

Loads the program if needed.

Parameters:

verification – [in] The type of verification to do before returning success.

Returns:

An Error to indicate success or failure of the loading process.

virtual ET_NODISCARD runtime::Error load (const LoadBackendOptionsMap &backend_options, const Program::Verification verification=Program::Verification::Minimal)

Loads the program with per-delegate runtime options.

Parameters:

  • backend_options – [in] A LoadBackendOptionsMap containing per-delegate load-time configuration options. The caller must ensure this object outlives any methods loaded with these options.

  • verification – [in] The type of verification to do before returning success.

Returns:

An Error to indicate success or failure of the loading process.

inline virtual bool is_loaded() const#

Checks if the program is loaded.

Returns:

true if the program is loaded, false otherwise.

inline std::shared_ptr<Program> program() const#

Get the program. The data loader used by the program is guaranteed to be valid for the lifetime of the program.

Returns:

Shared pointer to the program or nullptr if it’s not yet loaded.

runtime::Result<size_t> num_methods()#

Get the number of methods available in the loaded program.

Returns:

A Result object containing either the number of methods available or an error to indicate failure.

runtime::Result<std::unordered_set<std::string>> method_names()#

Get a list of method names available in the loaded program. Loads the program and method if needed.

Returns:

A set of strings containing the names of the methods, or an error if the program or method failed to load.

ET_NODISCARD runtime::Error load_method (const std::string &method_name, runtime::HierarchicalAllocator *planned_memory=nullptr, torch::executor::EventTracer *event_tracer=nullptr, const LoadBackendOptionsMap *backend_options=nullptr)

Load a specific method from the program and set up memory management if needed. The loaded method is cached to reuse the next time it’s executed.

Parameters:

  • method_name – [in] The name of the method to load.

  • planned_memory – [in] The memory-planned buffers to use for mutable tensor data when executing a method.

  • event_tracer – [in] Per-method event tracer to profile/trace methods individually. When not given, the event tracer passed to the Module constructor is used. Otherwise, this per-method event tracer takes precedence.

Returns:

An Error to indicate success or failure.

inline bool unload_method(const std::string &method_name)#

Unload a specific method from the program.

Parameters:

method_name – [in] The name of the method to unload.

Returns:

True if the method is unloaded, false if no-op.

ET_DEPRECATED ET_NODISCARD runtime::Result< Method * > method (const std::string &method_name)

DEPRECATED: Module manages each Method exclusively.

Get a method by it’s name. Not recommended to use this method directly as an end user. It’s exposed to allow for composability of module in apis that operate on method.

Parameters:

method_name – [in] The name of the method to get.

Returns:

A Result object containing either a pointer to the requested method or an error to indicate failure.

inline ET_NODISCARD runtime::Error load_forward (runtime::HierarchicalAllocator *planned_memory=nullptr, torch::executor::EventTracer *event_tracer=nullptr, const LoadBackendOptionsMap *backend_options=nullptr)

Load the ‘forward’ method from the program and set up memory management if needed. The loaded method is cached to reuse the next time it’s executed.

Parameters:

  • planned_memory – [in] The memory-planned buffers to use for mutable tensor data when executing the ‘forward’ method.

  • event_tracer – [in] An event tracer used for tracking and logging events.

Returns:

An Error to indicate success or failure.

inline bool unload_forward()#

Unload the ‘forward’ method from the program.

Returns:

True if the ‘forward’ method is unloaded, false if no-op.

inline bool is_method_loaded(const std::string &method_name) const#

Checks if a specific method is loaded.

Parameters:

method_name – [in] The name of the method to check.

Returns:

true if the method specified by method_name is loaded, false otherwise.

runtime::Result<MethodMeta> method_meta(const std::string &method_name)#

Get a method metadata struct by method name. Loads the program if needed.

Parameters:

method_name – [in] The name of the method to get the metadata for.

Returns:

A method metadata, or an error if the program or method failed to load.

virtual ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name, const std::vector< runtime::EValue > &input_values)

Execute a specific method with the given input values and retrieve the output values. Loads the program and method before executing if needed.

Parameters:

  • method_name – [in] The name of the method to execute.

  • input_values – [in] A vector of input values to be passed to the method.

Returns:

A Result object containing either a vector of output values from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name, const runtime::EValue &input_value)

Execute a specific method with a single input value. Loads the program and method before executing if needed.

Parameters:

  • method_name – [in] The name of the method to execute.

  • input_value – [in] A value to be passed to the method.

Returns:

A Result object containing either a vector of output values from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name)

Execute a specific method without any input values. Loads the program and method before executing if needed.

Parameters:

method_name – [in] The name of the method to execute.

Returns:

A Result object containing either a vector of output values from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< runtime::EValue > get (const std::string &method_name, const std::vector< runtime::EValue > &input_values)

Retrieve the output value of a specific method with the given input values. Loads the program and method before execution if needed.

Parameters:

  • method_name – [in] The name of the method to execute.

  • input_values – [in] A vector of input values to be passed to the method.

Returns:

A Result object containing either the first output value from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< runtime::EValue > get (const std::string &method_name, const runtime::EValue &input_value)

Retrieve the output value of a specific method with a single input value. Loads the program and method before execution if needed.

Parameters:

  • method_name – [in] The name of the method to execute.

  • input_value – [in] A value to be passed to the method.

Returns:

A Result object containing either the first output value from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< runtime::EValue > get (const std::string &method_name)

Retrieve the output value of a specific method without any input values. Loads the program and method before execution if needed.

Parameters:

method_name – [in] The name of the method to execute.

Returns:

A Result object containing either the first output value from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > forward (const std::vector< runtime::EValue > &input_values)

Execute the ‘forward’ method with the given input values and retrieve the output values. Loads the program and method before executing if needed.

Parameters:

input_values – [in] A vector of input values for the ‘forward’ method.

Returns:

A Result object containing either a vector of output values from the ‘forward’ method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > forward (const runtime::EValue &input_value)

Execute the ‘forward’ method with a single value. Loads the program and method before executing if needed.

Parameters:

input_value – [in] A value for the ‘forward’ method.

Returns:

A Result object containing either a vector of output values from the ‘forward’ method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > forward ()

Execute the ‘forward’ method without any input values. Loads the program and method before executing if needed.

Returns:

A Result object containing either a vector of output values from the ‘forward’ method or an error to indicate failure.

ET_NODISCARD runtime::Error set_input (const std::string &method_name, const runtime::EValue &input_value, size_t input_index)

Sets a single input value for a specific method.

Parameters:

  • method_name – [in] The name of the method.

  • input_value – [in] The EValue to set as the method input.

  • input_index – [in] Zero-based index of the input to set.

Returns:

An Error to indicate success or failure.

inline ET_NODISCARD runtime::Error set_input (const runtime::EValue &input_value, size_t input_index)

Sets a single input value for the “forward” method.

Parameters:

  • input_value – [in] The EValue to set as the method input.

  • input_index – [in] Zero-based index of the input to set.

Returns:

An Error to indicate success or failure.

ET_NODISCARD runtime::Error set_inputs (const std::string &method_name, const std::vector< runtime::EValue > &input_values)

Sets all input values for a specific method.

Parameters:

  • method_name – [in] The name of the method.

  • input_values – [in] A vector of EValues to set as the method inputs.

Returns:

An Error to indicate success or failure.

inline ET_NODISCARD runtime::Error set_inputs (const std::vector< runtime::EValue > &input_values)

Sets all input values for the “forward” method.

Parameters:

input_values – [in] A vector of EValues to set as the method inputs.

Returns:

An Error to indicate success or failure.

ET_NODISCARD runtime::Error set_output (const std::string &method_name, runtime::EValue output_value, size_t output_index=0)

Sets the output tensor for a specific method.

Note

Only Tensor outputs are currently supported for setting.

Parameters:

  • method_name – [in] The name of the method.

  • output_value – [in] The EValue containing the Tensor to set as the method output.

  • output_index – [in] Zero-based index of the output to set.

Returns:

An Error to indicate success or failure.

inline ET_NODISCARD runtime::Error set_output (runtime::EValue output_value, size_t output_index=0)

Sets the output tensor for the “forward” method.

Note

Only Tensor outputs are currently supported for setting.

Parameters:

  • output_value – [in] The EValue containing the Tensor to set as the method output.

  • output_index – [in] Zero-based index of the output to set.

Returns:

An Error to indicate success or failure.

ET_NODISCARD runtime::Error set_outputs (const std::string &method_name, const std::vector< runtime::EValue > &output_values)

Sets all output tensors for a specific method.

Loads the program and method if needed, and for each output uses the provided tensor’s data buffer as the method’s output buffer.

Note

Only Tensor outputs are currently supported for setting.

Note

Will fail for outputs that are memory-planned or constants.

Parameters:

  • method_name – [in] The name of the method.

  • output_values – [in] A vector of EValues to set as the method outputs.

Returns:

An Error to indicate success or failure.

inline ET_NODISCARD runtime::Error set_outputs (const std::vector< runtime::EValue > &output_values)

Sets all output tensors for the “forward” method.

Note

Only Tensor outputs are currently supported for setting.

Note

Will fail for outputs that are memory-planned or constants.

Parameters:

output_values – [in] A vector of EValues to set as the method outputs.

Returns:

An Error to indicate success or failure.

ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > get_outputs (const std::string &method_name)

Retrieve all current output values of a specific method without executing it. Loads the program and method before retrieval if needed.

Parameters:

method_name – [in] The name of the method.

Returns:

A Result containing the vector of output values, or an error.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > get_outputs ()

Retrieve all current output values of the “forward” method without executing it. Loads the program and method before retrieval if needed.

Returns:

A Result containing the vector of output values, or an error.

ET_NODISCARD runtime::Result< runtime::EValue > get_output (const std::string &method_name, size_t output_index=0)

Retrieve a single current output value of a specific method without executing it. Loads the program and method before retrieval if needed.

Parameters:

  • method_name – [in] The name of the method.

  • output_index – [in] Zero-based index of the output to retrieve.

Returns:

A Result containing the requested output value, or an error.

inline ET_NODISCARD runtime::Result< runtime::EValue > get_output (size_t output_index=0)

Retrieve a single current output value of the “forward” method without executing it. Loads the program and method before retrieval if needed.

Parameters:

output_index – [in] Zero-based index of the output to retrieve.

Returns:

A Result containing the requested output value, or an error.

inline runtime::EventTracer *event_tracer() const#

Retrieves the EventTracer instance being used by the Module. EventTracer is used for tracking and logging events during the execution of methods.

Returns:

A pointer to the EventTracer instance. Returns nullptr if no EventTracer is set.

class BundledModule : public executorch::extension::module::Module#

A facade class for loading bundled programs and executing methods within them.

Public Functions

explicit BundledModule(const void *bundled_program_ptr, std::unique_ptr<runtime::MemoryAllocator> memory_allocator = nullptr, std::unique_ptr<runtime::MemoryAllocator> temp_allocator = nullptr, std::unique_ptr<runtime::EventTracer> event_tracer = nullptr, std::unique_ptr<runtime::DataLoader> data_map_loader = nullptr)#

Constructs an instance with the bundled program buffer pointer.

This constructor reads the program from bundled program buffer to load the module with data loader. The bundled program pointer is preserved so that the portion outside of program is accessible.

Parameters:

  • bundled_program_ptr – [in] A DataLoader used for loading program data.

  • memory_allocator – [in] A MemoryAllocator used for memory management.

  • temp_allocator – [in] A MemoryAllocator to use when allocating temporary data during kernel or delegate execution.

  • event_tracer – [in] A EventTracer used for tracking and logging events.

  • data_map_loader – [in] A DataLoader used for loading external weights.

ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name, const size_t testset_idx)

Execute a specific method with the input value at the given testset_idx from the bundle to the method. Loads the program and method before executing if needed.

This function is a wrapper of load_bundled_input in bundled_program.

Parameters:

  • method_name – [in] The name of the method to execute.

  • testset_idx – [in] The index of the input value to be passed to the method.

Returns:

Return Error::Ok on a successful load, or the error happens during execution.

ET_NODISCARD runtime::Error verify_method_outputs (const std::string &method_name, const size_t testset_idx, double rtol=1e-5, double atol=1e-8)

Verify the output of a specific method with the expected output from the program bundle at the given testset_idx.

This function is a wrapper of verify_method_outputs in bundled_program.

Parameters:

  • method_name – [in] The name of the method to extract outputs from.

  • testset_idx – [in] The index of expected output needs to be compared.

  • rtol – [in] Relative tolerance used for data comparsion.

  • atol – [in] Absolute tolerance used for data comparsion.

Returns:

Return Error::Ok if two outputs match, or the error happens during execution.

ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name, const std::vector< runtime::EValue > &input_values)

Execute a specific method with the given input values and retrieve the output values. Loads the program and method before executing if needed.

Parameters:

  • method_name – [in] The name of the method to execute.

  • input_values – [in] A vector of input values to be passed to the method.

Returns:

A Result object containing either a vector of output values from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name, const runtime::EValue &input_value)

Execute a specific method with a single input value. Loads the program and method before executing if needed.

Parameters:

  • method_name – [in] The name of the method to execute.

  • input_value – [in] A value to be passed to the method.

Returns:

A Result object containing either a vector of output values from the method or an error to indicate failure.

inline ET_NODISCARD runtime::Result< std::vector< runtime::EValue > > execute (const std::string &method_name)

Execute a specific method without any input values. Loads the program and method before executing if needed.

Parameters:

method_name – [in] The name of the method to execute.

Returns:

A Result object containing either a vector of output values from the method or an error to indicate failure.

Public Static Functions

static ET_NODISCARD runtime::Result< std::unique_ptr< BundledModule > > from_file (const std::string &file_path, std::unique_ptr< runtime::MemoryAllocator > memory_allocator=nullptr, std::unique_ptr< runtime::MemoryAllocator > temp_allocator=nullptr, std::unique_ptr< runtime::EventTracer > event_tracer=nullptr, std::unique_ptr< runtime::DataLoader > data_map_loader=nullptr)

Constructs an instance by loading a bundled program from a file with specified memory locking behavior.

Parameters:

  • file_path – [in] The path to the ExecuTorch bundled program file to load.

  • memory_allocator – [in] A MemoryAllocator used for memory management.

  • temp_allocator – [in] A MemoryAllocator to use when allocating temporary data during kernel or delegate execution.

  • event_tracer – [in] A EventTracer used for tracking and logging events.

  • data_map_loader – [in] A DataLoader used for loading external weights.

Tensor Extension#

The Tensor extension provides managed tensor helpers for C++ applications that need to create, alias, resize, or index tensors before passing them to runtime APIs.

using executorch::extension::TensorPtr = std::shared_ptr<executorch::aten::Tensor>#

A smart pointer type for managing the lifecycle of a Tensor.

template<typename T, ssize_t N>
executorch::runtime::Result<TensorAccessor<T, N>> executorch::extension::make_tensor_accessor(const executorch::aten::Tensor &tensor)#

Creates a TensorAccessor<T, N> from the given tensor. The number of dimension N and the data type T’s size must match those of the input tensor. For Executorch tensors, non-trivial dimension order is not supported.

Parameters:

tensor – Origin tensor. The TensorImpl inside must outlive the returned TensorAccessor.

Return values:

  • Error::InvalidArgument – Mismatch on data type or number of dimensions.

  • Error::NotSupported – Input tensor has non-trivial dimension onrder.

Returns:

TensorAccessor of the input tensor.

TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, void *data, std::vector<executorch::aten::DimOrderType> dim_order, std::vector<executorch::aten::StridesType> strides, const executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, const executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND, std::function<void(void*)> deleter = nullptr)#

Creates a TensorPtr that manages a Tensor with the specified properties.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • data – A pointer to the data buffer.

  • dim_order – A vector specifying the order of dimensions.

  • strides – A vector specifying the strides of the tensor.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies the mutability of the tensor’s shape.

  • deleter – A custom deleter function for managing the lifetime of the data buffer. If provided, this deleter will be called when the managed Tensor object is destroyed.

Returns:

A TensorPtr that manages the newly created Tensor.

inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, void *data, const executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, const executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND, std::function<void(void*)> deleter = nullptr)#

Creates a TensorPtr that manages a Tensor with the specified properties.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • data – A pointer to the data buffer.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies the mutability of the tensor’s shape.

  • deleter – A custom deleter function for managing the lifetime of the data buffer. If provided, this deleter will be called when the managed Tensor object is destroyed.

Returns:

A TensorPtr that manages the newly created Tensor.

template<typename T = float, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value>
inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, std::vector<T> data, std::vector<executorch::aten::DimOrderType> dim_order = {}, std::vector<executorch::aten::StridesType> strides = {}, executorch::aten::ScalarType type = deduced_type, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This template overload is specialized for cases where the tensor data is provided as a vector. The scalar type is automatically deduced from the vector’s data type. If the specified type differs from the deduced type of the vector’s elements, and casting is allowed, the data will be cast to the specified type. This allows for flexible creation of tensors with data vectors of one type and a different scalar type.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the vector.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • data – A vector containing the tensor’s data.

  • dim_order – A vector specifying the order of dimensions.

  • strides – A vector specifying the strides of each dimension.

  • type – The scalar type of the tensor elements. If it differs from the deduced type, the data will be cast to this type if allowed.

  • dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr that manages the newly created TensorImpl.

template<typename T = float, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value>
inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<T> data, executorch::aten::ScalarType type = deduced_type, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This template overload is specialized for cases where the tensor data is provided as a vector. The scalar type is automatically deduced from the vector’s data type. If the specified type differs from the deduced type of the vector’s elements, and casting is allowed, the data will be cast to the specified type. This allows for flexible creation of tensors with data vectors of one type and a different scalar type.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the vector.

Parameters:

  • data – A vector containing the tensor’s data.

  • type – The scalar type of the tensor elements. If it differs from the deduced type, the data will be cast to this type if allowed.

  • dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr that manages the newly created TensorImpl.

template<typename T = float, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value>
inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, std::initializer_list<T> list, std::vector<executorch::aten::DimOrderType> dim_order = {}, std::vector<executorch::aten::StridesType> strides = {}, executorch::aten::ScalarType type = deduced_type, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This template overload is specialized for cases where the tensor data is provided as an initializer list. The scalar type is automatically deduced from the initializer list’s data type. If the specified type differs from the deduced type of the initializer list’s elements, and casting is allowed, the data will be cast to the specified type. This allows for flexible creation of tensors with data vectors of one type and a different scalar type.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the initializer list.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • list – An initializer list containing the tensor’s data.

  • dim_order – A vector specifying the order of dimensions.

  • strides – A vector specifying the strides of each dimension.

  • type – The scalar type of the tensor elements. If it differs from the deduced type, the data will be cast to this type if allowed.

  • dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr that manages the newly created TensorImpl.

template<typename T = float, executorch::aten::ScalarType deduced_type = runtime::CppTypeToScalarType<T>::value>
inline TensorPtr executorch::extension::make_tensor_ptr(std::initializer_list<T> list, executorch::aten::ScalarType type = deduced_type, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This template overload allows creating a Tensor from an initializer list of data. The scalar type is automatically deduced from the type of the initializer list’s elements. If the specified type differs from the deduced type of the initializer list’s elements, and casting is allowed, the data will be cast to the specified type. This allows for flexible creation of tensors with data vectors of one type and a different scalar type.

Template Parameters:

T – The C++ type of the tensor elements, deduced from the initializer list.

Parameters:

  • list – An initializer list containing the tensor’s data.

  • type – The scalar type of the tensor elements. If it differs from the deduced type, the data will be cast to this type if allowed.

  • dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr that manages the newly created TensorImpl.

template<typename T>
inline TensorPtr executorch::extension::make_tensor_ptr(T value)#

Creates a TensorPtr that manages a Tensor with a single scalar value.

Template Parameters:

T – The C++ type of the scalar value.

Parameters:

value – The scalar value to be used for the Tensor.

Returns:

A TensorPtr that manages the newly created TensorImpl.

TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, std::vector<uint8_t> data, std::vector<executorch::aten::DimOrderType> dim_order, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This overload accepts a raw memory buffer stored in a std::vector<uint8_t> and a scalar type to interpret the data. The vector is managed, and the memory’s lifetime is tied to the TensorImpl.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • data – A vector containing the raw memory for the tensor’s data.

  • dim_order – A vector specifying the order of dimensions.

  • strides – A vector specifying the strides of each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr managing the newly created Tensor.

inline TensorPtr executorch::extension::make_tensor_ptr(std::vector<executorch::aten::SizesType> sizes, std::vector<uint8_t> data, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr that manages a Tensor with the specified properties.

This overload accepts a raw memory buffer stored in a std::vector<uint8_t> and a scalar type to interpret the data. The vector is managed, and the memory’s lifetime is tied to the TensorImpl.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • data – A vector containing the raw memory for the tensor’s data.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies the mutability of the tensor’s shape.

Returns:

A TensorPtr managing the newly created Tensor.

inline TensorPtr executorch::extension::make_tensor_ptr(const executorch::aten::Tensor &tensor, std::vector<executorch::aten::SizesType> sizes = {}, std::vector<executorch::aten::DimOrderType> dim_order = {}, std::vector<executorch::aten::StridesType> strides = {}, std::function<void(void*)> deleter = nullptr)#

Creates a TensorPtr to manage a new Tensor that aliases the given Tensor’s storage, with optional metadata overrides. Shape dynamism is inherited from the source tensor.

If an override is provided (non-empty), it is passed as-is. If an override is empty, the corresponding metadata is reused from the source tensor when it fits; otherwise it is left empty for the core factory to derive a valid configuration. If dim_order is empty but strides is provided, dim_order is left empty so the core may infer it from the provided strides.

Parameters:

  • tensor – The source tensor to alias.

  • sizes – Optional sizes override.

  • dim_order – Optional dimension order override.

  • strides – Optional strides override.

  • deleter – A custom deleter function for managing the lifetime of the original Tensor.

Returns:

A TensorPtr aliasing the same storage with requested metadata.

inline TensorPtr executorch::extension::make_tensor_ptr(const TensorPtr &tensor_ptr, std::vector<executorch::aten::SizesType> sizes = {}, std::vector<executorch::aten::DimOrderType> dim_order = {}, std::vector<executorch::aten::StridesType> strides = {})#

Convenience overload identical to make_tensor_ptr(*tensor_ptr, …). Keeps the original TensorPtr alive until the returned TensorPtr is destroyed.

Parameters:

  • tensor_ptr – The source tensor pointer to alias.

  • sizes – Optional sizes override.

  • dim_order – Optional dimension order override.

  • strides – Optional strides override.

Returns:

A TensorPtr aliasing the same storage with requested metadata.

TensorPtr executorch::extension::clone_tensor_ptr(const executorch::aten::Tensor &tensor, executorch::aten::ScalarType type)#

Creates a TensorPtr that manages a new Tensor with the same properties as the given Tensor, but with a copy of the data owned by the returned TensorPtr, or nullptr if the original data is null.

Parameters:

  • tensor – The Tensor to clone.

  • type – The data type for the cloned tensor. The data will be cast from the source tensor’s type.

Returns:

A new TensorPtr that manages a Tensor with the specified type and copied/cast data.

inline TensorPtr executorch::extension::clone_tensor_ptr(const executorch::aten::Tensor &tensor)#

Creates a TensorPtr that manages a new Tensor with the same properties as the given Tensor, but with a copy of the data owned by the returned TensorPtr, or nullptr if the original data is null.

Parameters:

tensor – The Tensor to clone.

Returns:

A new TensorPtr that manages a Tensor with the same properties as the original but with copied data.

inline TensorPtr executorch::extension::clone_tensor_ptr(const TensorPtr &tensor, executorch::aten::ScalarType type)#

Creates a new TensorPtr by cloning the given TensorPtr, copying the underlying data.

Parameters:

  • tensor – The TensorPtr to clone.

  • type – The data type for the cloned tensor. The data will be cast from the source tensor’s type.

Returns:

A new TensorPtr that manages a Tensor with the specified type and copied/cast data.

inline TensorPtr executorch::extension::clone_tensor_ptr(const TensorPtr &tensor)#

Creates a new TensorPtr by cloning the given TensorPtr, copying the underlying data.

Parameters:

tensor – The TensorPtr to clone.

Returns:

A new TensorPtr that manages a Tensor with the same properties as the original but with copied data.

ET_NODISCARD runtime::Error executorch::extension::resize_tensor_ptr (TensorPtr &tensor, const std::vector< executorch::aten::SizesType > &sizes)

Resizes the Tensor managed by the provided TensorPtr to the new sizes.

Parameters:

  • tensor – A TensorPtr managing the Tensor to resize.

  • sizes – A vector representing the new sizes for each dimension.

Returns:

Error::Ok on success, or an appropriate error code on failure.

inline TensorPtrMaker executorch::extension::for_blob(void *data, std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float)#

Creates a TensorPtrMaker instance for building a TensorPtr from a raw data pointer and tensor sizes.

The TensorPtrMaker returned by this function allows for further customization of the tensor’s properties, such as data type, dimension order, strides, and shape dynamism, before finalizing the TensorPtr creation.

Parameters:

  • data – A pointer to the raw data to be used by the tensor. It must outlive the TensorPtr created by this function.

  • sizes – A vector specifying the size of each dimension.

  • type – The scalar type of the tensor elements.

Returns:

A TensorPtrMaker instance for creating a TensorPtr.

inline TensorPtr executorch::extension::from_blob(void *data, std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr from a raw data pointer and tensor sizes, with an optional dynamism setting.

This function provides a convenient way to create a tensor from existing data, with the option to specify whether the tensor’s shape is static or dynamic.

Parameters:

  • data – A pointer to the raw data used by the tensor. The data must outlive the TensorPtr created by this function.

  • sizes – A vector specifying the size of each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::from_blob(void *data, std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr from a raw data pointer, tensor sizes, and strides, with an optional dynamism setting.

This function allows for the creation of a tensor from existing data, with the option to specify custom strides for each dimension and whether the tensor’s shape is static, dynamic, or bounded.

Parameters:

  • data – A pointer to the raw data used by the tensor. The data must outlive the TensorPtr created by this function.

  • sizes – A vector specifying the size of each dimension.

  • strides – A vector specifying the stride for each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static, dynamic, or bounded.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::from_blob(void *data, std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type, std::function<void(void*)> &&deleter, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr from a raw data pointer and tensor sizes, with an optional dynamism setting.

This function is a convenient way to create a tensor from existing data, with the option to specify whether the tensor’s shape is static, dynamic, or bounded.

Parameters:

  • data – A pointer to the raw data to be used by the tensor. It must outlive the TensorPtr created by this function.

  • sizes – A vector specifying the size of each dimension.

  • type – The scalar type of the tensor elements.

  • deleter – A function to delete the data when it’s no longer needed.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance that manages the newly created Tensor.

inline TensorPtr executorch::extension::from_blob(void *data, std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type, std::function<void(void*)> &&deleter, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr from a raw data pointer, tensor sizes, and strides, with an optional dynamism setting.

This function allows for the creation of a tensor from existing data, with the option to specify custom strides for each dimension and whether the tensor’s shape is static, dynamic, or bounded.

Parameters:

  • data – A pointer to the raw data to be used by the tensor. It must outlive the TensorPtr created by this function.

  • sizes – A vector specifying the size of each dimension.

  • strides – A vector specifying the stride for each dimension.

  • type – The scalar type of the tensor elements.

  • deleter – A function to delete the data when it’s no longer needed.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance that manages the newly created Tensor.

TensorPtr executorch::extension::empty_strided(std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr with the specified sizes, strides, and properties.

This function allocates memory for the tensor elements but does not initialize them with any specific values. The tensor is created with the specified strides.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • strides – A vector specifying the stride for each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::empty_like(const TensorPtr &other, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates an empty TensorPtr with the same size and properties as the given tensor.

This function allocates memory for the tensor elements but does not initialize them with any specific values.

Parameters:

  • other – A reference to another tensor, whose size and properties are used.

  • type – The scalar type of the tensor elements. If not provided, the scalar type of the other tensor is used.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::empty(std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates an empty TensorPtr with the specified sizes and properties.

This function allocates memory for the tensor elements but does not initialize them with any specific values.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

TensorPtr executorch::extension::full_strided(std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::Scalar fill_value, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with the specified value.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • strides – A vector specifying the stride for each dimension.

  • fill_value – The value to fill the tensor with.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::full_like(const TensorPtr &other, executorch::aten::Scalar fill_value, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with the specified value, with the same size and properties as another tensor.

Parameters:

  • other – A reference to another tensor, whose size and properties will be used.

  • fill_value – The value to fill the tensor with.

  • type – The scalar type of the tensor elements. If not specified, the scalar type of the other tensor is used.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::full(std::vector<executorch::aten::SizesType> sizes, executorch::aten::Scalar fill_value, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with the specified value.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • fill_value – The value used to fill the tensor.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::scalar_tensor(executorch::aten::Scalar value, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float)#

Creates a TensorPtr holding a scalar value.

Parameters:

  • value – The scalar value for the tensor.

  • type – The scalar type of the tensor elements.

Returns:

A TensorPtr instance managing the newly created scalar Tensor.

inline TensorPtr executorch::extension::ones_like(const TensorPtr &other, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with ones, with the same size and properties as another tensor.

Parameters:

  • other – A reference to another tensor, whose size and properties are used.

  • type – The scalar type of the tensor elements. If not provided, the scalar type of the other tensor is used.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::ones(std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with ones.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::zeros_like(const TensorPtr &other, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with zeros, with the same size and properties as another tensor.

Parameters:

  • other – A reference to another tensor, whose size and properties will be used.

  • type – The scalar type of the tensor elements. If not specified, the scalar type of the other tensor is used.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::zeros(std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with zeros.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

TensorPtr executorch::extension::rand_strided(std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values between 0 and 1.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • strides – A vector specifying the stride for each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::rand_like(const TensorPtr &other, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values between 0 and 1.

Parameters:

  • other – A reference to another tensor, whose size and properties will be used.

  • type – The scalar type of the tensor elements. If not specified, the scalar type of the other tensor is used.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::rand(std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values between 0 and 1.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

TensorPtr executorch::extension::randn_strided(std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values from a normal distribution.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • strides – A vector specifying the stride for each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::randn_like(const TensorPtr &other, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values from a normal distribution.

Parameters:

  • other – A reference to another tensor, whose size and properties will be used.

  • type – The scalar type of the tensor elements. If not specified, the scalar type of the other tensor is used.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::randn(std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Float, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random values from a normal distribution.

Parameters:

  • sizes – A vector specifying the size of each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

TensorPtr executorch::extension::randint_strided(int64_t low, int64_t high, std::vector<executorch::aten::SizesType> sizes, std::vector<executorch::aten::StridesType> strides, executorch::aten::ScalarType type = executorch::aten::ScalarType::Int, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random integer values in the given range.

Parameters:

  • low – The lower bound (inclusive) of the random values.

  • high – The upper bound (exclusive) of the random values.

  • sizes – A vector specifying the size of each dimension.

  • strides – A vector specifying the stride for each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::randint_like(const TensorPtr &other, int64_t low, int64_t high, executorch::aten::ScalarType type = executorch::aten::ScalarType::Undefined, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random integer values in the given range.

Parameters:

  • other – A reference to another tensor, whose size and properties will be used.

  • low – The lower bound (inclusive) of the random values.

  • high – The upper bound (exclusive) of the random values.

  • type – The scalar type of the tensor elements. If not specified, the scalar type of the other tensor is used.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

inline TensorPtr executorch::extension::randint(int64_t low, int64_t high, std::vector<executorch::aten::SizesType> sizes, executorch::aten::ScalarType type = executorch::aten::ScalarType::Int, executorch::aten::TensorShapeDynamism dynamism = executorch::aten::TensorShapeDynamism::DYNAMIC_BOUND)#

Creates a TensorPtr filled with random integer values within the specified range.

Parameters:

  • low – The inclusive lower bound of the random values.

  • high – The exclusive upper bound of the random values.

  • sizes – A vector specifying the size of each dimension.

  • type – The scalar type of the tensor elements.

  • dynamism – Specifies whether the tensor’s shape is static or dynamic.

Returns:

A TensorPtr instance managing the newly created Tensor.

template<typename T, ssize_t N>
class TensorAccessor : public executorch::extension::internal::TensorAccessorBase<T, N>#
#include <tensor_accessor.h>

TensorAccessor template with data type and rank as template parameters. No public constructors, can only be created using make_tensor_accessor from a given executorch::aten::Tensor. Use operator[] to index and obtain a lower rank accessor or the underlying scalar value.

Public Functions

inline TensorAccessor<T, N - 1> operator[](ssize_t i)#

Index into the the outer most dimension.

Parameters:

i – Index.

Returns:

If N > 1, a TensorAccessor with N-1 dimensions. If N == 1, a reference to the underlying scalar. Refer to the TensorAccessor<T, 1> specialization.

inline const TensorAccessor<T, N - 1> operator[](ssize_t i) const#

Index into the the outer most dimension.

Parameters:

i – Index.

Returns:

If N > 1, a constant TensorAccessor with N-1 dimensions. If N == 1, a constant reference to the underlying scalar. Refer to the TensorAccessor<T, 1> specialization.

template<typename T>
class TensorAccessor<T, 1> : public executorch::extension::internal::TensorAccessorBase<T, 1>#
#include <tensor_accessor.h>

TensorAccessor specialization for N == 1, where operator[] returns a reference to the underlying scalar.

Public Functions

inline T &operator[](ssize_t i)#

Index into the the outer most dimension.

Parameters:

i – Index.

Returns:

Reference to the underlying scalar.

inline const T &operator[](ssize_t i) const#

Index into the the outer most dimension.

Parameters:

i – Index.

Returns:

Constant reference to the underlying scalar.

class TensorPtrMaker#
#include <tensor_ptr_maker.h>

A helper class for creating TensorPtr instances from raw data and tensor properties. Note that the TensorPtr created by this class does not own the data, so the data must outlive the TensorPtr.

TensorPtrMaker provides a fluent interface for specifying various tensor properties, such as type, sizes, data pointer, dimension order, strides, and shape dynamism. The final tensor is created by invoking make_tensor_ptr() or by converting TensorPtrMaker to TensorPtr.

Public Functions

inline TensorPtrMaker &&type(executorch::aten::ScalarType type)#

Sets the scalar type of the tensor elements.

Parameters:

type – The scalar type (e.g., float, int, bool).

Returns:

Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtrMaker &&dim_order(std::vector<executorch::aten::DimOrderType> dim_order)#

Sets the order of dimensions in memory.

Parameters:

dim_order – A vector specifying the dimension order.

Returns:

Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtrMaker &&strides(std::vector<executorch::aten::StridesType> strides)#

Sets the strides for each dimension of the tensor.

Parameters:

strides – A vector specifying the stride for each dimension.

Returns:

Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtrMaker &&dynamism(executorch::aten::TensorShapeDynamism dynamism)#

Sets the shape dynamism of the tensor.

Parameters:

dynamism – Specifies whether the tensor’s shape is static, dynamic, or bounded.

Returns:

Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtrMaker &&deleter(std::function<void(void*)> &&deleter)#

Sets a custom deleter function to manage the lifetime of the data buffer.

Parameters:

deleter – A function that will be called to delete the data buffer when the Tensor object managed by the TensorPtr is destroyed. Explicitly consuming an rvalue to avoid unnecessary copies when the deleter is a lambda that has captured some state.

Returns:

Rvalue to this TensorPtrMaker for method chaining.

inline TensorPtr make_tensor_ptr() &&#

Creates and returns a TensorPtr instance using the properties set in this TensorPtrMaker.

Returns:

A TensorPtr instance that manages the newly created Tensor.

inline operator TensorPtr() &&#

Implicit conversion operator to create a TensorPtr.

Returns:

A TensorPtr instance that manages the newly created Tensor.

On this page
Edit on GitHub
Show Source