Replay Buffers

Replay buffers are a central part of off-policy RL algorithms. TorchRL provides an efficient implementation of a few, widely used replay buffers:

Core Replay Buffer Classes

ReplayBuffer(*[, storage, sampler, writer, ...])	A generic, composable replay buffer class.
ReplayBufferEnsemble(*rbs[, storages, ...])	An ensemble of replay buffers.
PrioritizedReplayBuffer(*, alpha, beta[, ...])	Prioritized replay buffer.
TensorDictReplayBuffer(*[, priority_key])	TensorDict-specific wrapper around the ReplayBuffer class.
TensorDictPrioritizedReplayBuffer(*, alpha, beta)	TensorDict-specific wrapper around the PrioritizedReplayBuffer class.
RayReplayBuffer(*args, replay_buffer_cls, ...)	A Ray implementation of the Replay Buffer that can be extended and sampled remotely.
RemoteTensorDictReplayBuffer(*args, **kwargs)	A remote invocation friendly ReplayBuffer class.

Composable Replay Buffers

We also give users the ability to compose a replay buffer. We provide a wide panel of solutions for replay buffer usage, including support for almost any data type; storage in memory, on device or on physical memory; several sampling strategies; usage of transforms etc.

Supported data types and choosing a storage

In theory, replay buffers support any data type but we can’t guarantee that each component will support any data type. The most crude replay buffer implementation is made of a ReplayBuffer base with a ListStorage storage. This is very inefficient but it will allow you to store complex data structures with non-tensor data. Storages in contiguous memory include TensorStorage, LazyTensorStorage and LazyMemmapStorage.

Sampling and indexing

Replay buffers can be indexed and sampled. Indexing and sampling collect data at given indices in the storage and then process them through a series of transforms and collate_fn that can be passed to the __init__ function of the replay buffer.