Tristan Rice (@d4l3k) · July 21, 2026
distributedrdmanetworkinggpudirectstorage
TL;DR - Python can drive GPUDirect RDMA at line rate without a large transfer-engine abstraction. rdma4py provides lightweight, backend-specific bindings for ibverbs, AWS EFA, and NVMe-oF, reaching about 400 gbps in our benchmarks while preserving direct access to the underlying APIs.
Recently, quite a few different “transfer engines” have been developed for fast weight synchronization between machines using RDMA-capable transports and libraries such as NVLink, ibverbs, EFA, NVMe-oF, and SPDK. These include projects such as Mooncake, NIXL, and Uniflow. As with any large project and abstraction, these projects make trade-offs around specific use cases that might not be optimal for yours. …
Continue reading →Michael Lazos (@mlazos) · June 23, 2026
torchtitandistributedmemoryperformancetorch.compile
TL;DR – We added a graph-based CPU activation-offloading pass to torchtitan’s graph_trainer with agent-tunable knobs and a user-customizable offload policy function. On dense models you can reclaim 10% of peak memory for under 1% throughput loss, scaling to 33% (Llama3) / 38% (Qwen3) with 20% throughput loss. Our implementation achieves SOL PCIe transfer bandwidth of ~300 GB/s.
In model training, the forward pass is run followed by the backward pass to perform updates on the model parameters according to the error gradients. The backward pass computes gradients via the chain rule, and to do so it needs the activations the forward pass produced. Every activation must therefore still be …
Continue reading →Tristan Rice (@d4l3k) · May 14, 2026
distributedtorchcommsncclsymmetric-memorytritonprototyping
TL;DR – Modifying the C++ comms layer is a big barrier when researchers want to prototype new collective features. We’ve added Python bindings to torchcomms (#2080) and built two pure-Python backend prototypes — one wrapping NVIDIA’s new nccl4py bindings (#2515) and one built on SymmetricMemory + Triton (#2521) — both passing the core torchcomms integration test suite. Since they plug into torch.distributed, researchers can fork, tweak, and mix them with existing projects like TorchTitan without touching C++.
We’ve been thinking about how to improve overall research and prototyping speed for comms and collective libraries. LLMs have hugely improved prototyping speed for new ideas and …
Continue reading →Lucas Kabela (@lucaskabela), Jiani Wang, Tianyu Liu, Richard Zou (@zou3519), Joe Cummings, Milad Mohammadi · May 6, 2026
torchtitanrltorch.compiledistributedperformance
TL;DR – We enabled torch.compile across the full RL training loop in TorchTitan, achieving a 6x end-to-end speedup (from 446s to 70s) on Qwen3 0.6B for GSM8K. Thanks to TorchTitan RL using a single unified model definition for both training and inference, we can share compiled artifacts across the trainer and generator, reducing startup time while leveraging performance improvements to make this possible.
Most RL frameworks (Verl, OpenRLHF, etc.) maintain separate model definitions for training vs. inference. This means:
Duplicated code to keep in sync Separate optimization paths for each No opportunity to share compilation work TorchTitan RL uses one model definition across both the …
Continue reading →