Ruisi Zhang (@ruisizhang123) · September 2, 2026
torchtitandistributedmodel factory
TL;DR – I have reproduced 23% of Olmo3’s pre-training loss curve with TorchTitan. The training takes 4 days running on 512H100 GPUs. The on-the-fly downstream task evaluations further confirm TorchTitan’s ability to train capable LLMs. This marks the first step in our model factory efforts, which aims to enable agents to use TorchTitan to hillclimb scaling ladders, explore new research ideas, and validate them across increasing model scales. Stay tuned for more updates.
Olmo3 is Ai2’s fully open family of language models (link), with training data, code, checkpoints, and recipes spanning pretraining, mid-training, long-context training, and post-training. This transparency provides a …
Continue reading →Pian Pawakapan (@pianpwk) · August 26, 2026
torchtitandistributeddtensorspmdsharding
TL;DR
TorchTitan now uses spmd_types as its default backend for distributed model computation. Authors specify sharding contracts and collectives explicitly, with optional typechecking to catch distributed-correctness errors during development. At runtime, the typechecking machinery can be erased so forward and backward execute on plain tensors. Coupled FWD-BWD typing and support for both global and local SPMD also make distributed behavior easier to express and reason about. Across our repeated debug-model benchmarks, spmd_types improved eager throughput by up to 46% and Inductor throughput by 5-9% over partial_dtensor, without a meaningful peak-memory increase. FSDP2 continues to use …
Continue reading →Chien-Chin Huang (@fegin) · August 17, 2026
torchtitandistributeddtensorshardingspmdmoe
TL;DR
TorchTitan now adopts a declarative approach: ALL sharding (SPMD parallelization) is expressed in configuration. We created a full DTensor mode where every tensor is a DTensor that shards on all activated mesh axes (DP, CP, TP; EP for MoE experts). Full DTensor removes the ambiguity where a tensor could be a plain local tensor or a DTensor sharded on only some axes - the source of correctness bugs. Available via --parallelism.spmd_backend=full_dtensor, verified bit-identical to the legacy path across FSDP/HSDP/CP/TP/EP, at performance parity. TorchTitan is transitioning to spmd_types; the work here (config-based sharding and full DTensor) is the foundation that makes it possible. …
Continue reading →Tristan Rice (@d4l3k) · July 21, 2026
distributedrdmanetworkinggpudirectstorage
TL;DR - Python can drive GPUDirect RDMA at line rate without a large transfer-engine abstraction. rdma4py provides lightweight, backend-specific bindings for ibverbs, AWS EFA, and NVMe-oF, reaching about 400 gbps in our benchmarks while preserving direct access to the underlying APIs.
Recently, quite a few different “transfer engines” have been developed for fast weight synchronization between machines using RDMA-capable transports and libraries such as NVLink, ibverbs, EFA, NVMe-oF, and SPDK. These include projects such as Mooncake, NIXL, and Uniflow. As with any large project and abstraction, these projects make trade-offs around specific use cases that might not be optimal for yours. …
Continue reading →Michael Lazos (@mlazos) · June 23, 2026
torchtitandistributedmemoryperformancetorch.compile
TL;DR – We added a graph-based CPU activation-offloading pass to torchtitan’s graph_trainer with agent-tunable knobs and a user-customizable offload policy function. On dense models you can reclaim 10% of peak memory for under 1% throughput loss, scaling to 33% (Llama3) / 38% (Qwen3) with 20% throughput loss. Our implementation achieves SOL PCIe transfer bandwidth of ~300 GB/s.
In model training, the forward pass is run followed by the backward pass to perform updates on the model parameters according to the error gradients. The backward pass computes gradients via the chain rule, and to do so it needs the activations the forward pass produced. Every activation must therefore still be …
Continue reading →