TorchRL

TorchRL is an open-source Reinforcement Learning (RL) library for PyTorch.

You can install TorchRL directly from PyPI (see more about installation instructions in the dedicated section below):

$ pip install torchrl

TorchRL provides pytorch and python-first, low and high level abstractions for RL that are intended to be efficient, modular, documented and properly tested. The code is aimed at supporting research in RL. Most of it is written in python in a highly modular way, such that researchers can easily swap components, transform them or write new ones with little effort.

This repo attempts to align with the existing pytorch ecosystem libraries in that it has a “dataset pillar” (environments), transforms, models, data utilities (e.g. collectors and containers), etc. TorchRL aims at having as few dependencies as possible (python standard library, numpy and pytorch). Common environment libraries (e.g. OpenAI gym) are only optional.

On the low-level end, torchrl comes with a set of highly re-usable functionals for cost functions, returns and data processing.

TorchRL aims at a high modularity and good runtime performance.

To read more about TorchRL philosophy and capabilities beyond this API reference, check the TorchRL paper.

Installation

TorchRL releases are synced with PyTorch, so make sure you always enjoy the latest features of the library with the most recent version of PyTorch (although core features are guaranteed to be backward compatible with pytorch>=2.0). Nightly releases can be installed via

$ pip install tensordict-nightly
$ pip install torchrl-nightly

or via a git clone if you’re willing to contribute to the library:

$ cd path/to/root
$ git clone https://github.com/pytorch/tensordict
$ git clone https://github.com/pytorch/rl
$ cd tensordict
$ python setup.py develop
$ cd ../rl
$ python setup.py develop

Getting started

A series of quick tutorials to get ramped up with the basic features of the library. If you’re in a hurry, you can start by the last item of the series and navigate to the previous ones whenever you want to learn more!

• Get started with Environments, TED and transforms
• Get started with TorchRL’s modules
• Getting started with model optimization
• Get started with data collection and storage
• Get started with logging
• Get started with your own first training loop

Tutorials

Basics

• Reinforcement Learning (PPO) with TorchRL Tutorial
• Pendulum: Writing your environment and transforms with TorchRL
• Introduction to TorchRL

Intermediate

• Multi-Agent Reinforcement Learning (PPO) with TorchRL Tutorial
• TorchRL envs
• Using pretrained models
• Recurrent DQN: Training recurrent policies
• Using Replay Buffers
• Exporting TorchRL modules
• TorchRL LLM: Building Tool-Enabled Environments

Advanced

• Competitive Multi-Agent Reinforcement Learning (DDPG) with TorchRL Tutorial
• Task-specific policy in multi-task environments
• TorchRL objectives: Coding a DDPG loss
• TorchRL trainer: A DQN example

References

• API Reference
  • torchrl.collectors package
    • Collectors and batch size
    • Collectors and policy copies
    • Weight Synchronization using Weight Update Schemes
    • Legacy: Weight Synchronization in Distributed Environments
    • Collectors and replay buffers interoperability
    • Running the Collector Asynchronously
    • Single node data collectors
    • Distributed data collectors
    • Helper functions
  • torchrl.data package
    • Replay Buffers
    • Composable Replay Buffers
    • TorchRL Episode Data Format (TED)
    • Datasets
    • TensorSpec
    • Trees and Forests
    • Large language models and Reinforcement Learning From Human Feedback (RLHF)
    • Utils
  • torchrl.envs package
    • Env specs: locks and batch size
    • Env methods
    • Partial steps and partial resets
    • Vectorized envs
    • Async environments
    • Custom native TorchRL environments
    • Multi-agent environments
    • Auto-resetting Envs
    • Dynamic Specs
    • Transforms
    • Environments with masked actions
    • Recorders
    • Helpers
    • Domain-specific
    • Libraries
  • LLM Interface
    • Data Structures
    • Modules
    • Collectors
    • Environments
    • Objectives
  • torchrl.modules package
    • TensorDict modules: Actors, exploration, value models and generative models
    • Hooks
    • Models
    • Exploration
    • Planners
    • Distributions
    • Utils
  • torchrl.objectives package
    • torch.vmap and randomness
    • Training value functions
    • DQN
    • DDPG
    • SAC
    • REDQ
    • CrossQ
    • IQL
    • CQL
    • GAIL
    • DT
    • TD3
    • TD3+BC
    • PPO
    • A2C
    • Reinforce
    • Dreamer
    • Multi-agent objectives
    • Returns
    • Utils
  • torchrl.trainers package
    • Checkpointing
    • Trainer and hooks
    • Algorithm-specific trainers (Experimental)
    • Builders
    • Utils
    • Loggers
    • Recording utils
  • torchrl._utils package
    • implement_for
    • set_auto_unwrap_transformed_env
    • auto_unwrap_transformed_env
  • TorchRL Configuration System
    • Quick Start with a Simple Example
    • Configuration Categories and Groups
    • More Complex Example: Parallel Environment with Transforms
    • Getting Available Options
    • Complete Training Example
    • Running Experiments
    • Configuration Store Implementation Details
    • Available Configuration Classes
    • Creating Custom Configurations
    • Best Practices
    • Future Extensions

Knowledge Base

• Knowledge Base
  • Contributing
  • Things to consider when debugging RL
  • Working with gym
  • Working with habitat-lab
  • Working with MuJoCo-based environments
  • Common PyTorch errors and solutions
  • Useful resources
  • Versioning Issues
  • Customising Video Renders

Indices and tables

• Index

• Module Index

• Search Page