Recent Posts
Edward Z. Yang (@ezyang) · July 25, 2026
compilertorch.compileautogradverificationllm
A reference implementation is a simplified but complete version of a system that trades performance in return for clarity. We might then say a reference “language” is the fabric of APIs and conventions from which these implementations are cut. At first glance, PyTorch obviously is a reference language: it is, after all, commonly called the lingua franca of modern deep learning. But upon a closer look, there is confusion:
Reference implementations usually aren’t deployed to production. But I do my training jobs with PyTorch!
Everyone’s writing kernels with kernel DSLs. What is the role of PyTorch if it’s just gluing kernels together?
AI coding will eventually mean that any stack can be …
Continue reading →Tristan Rice (@d4l3k) · July 21, 2026
distributedrdmanetworkinggpudirectstorage
TL;DR - Python can drive GPUDirect RDMA at line rate without a large transfer-engine abstraction. rdma4py provides lightweight, backend-specific bindings for ibverbs, AWS EFA, and NVMe-oF, reaching about 400 gbps in our benchmarks while preserving direct access to the underlying APIs.
Recently, quite a few different “transfer engines” have been developed for fast weight synchronization between machines using RDMA-capable transports and libraries such as NVLink, ibverbs, EFA, NVMe-oF, and SPDK. These include projects such as Mooncake, NIXL, and Uniflow. As with any large project and abstraction, these projects make trade-offs around specific use cases that might not be optimal for yours. …
Continue reading →Michael Lazos (@mlazos) · June 23, 2026
torchtitandistributedmemoryperformancetorch.compile
TL;DR – We added a graph-based CPU activation-offloading pass to torchtitan’s graph_trainer with agent-tunable knobs and a user-customizable offload policy function. On dense models you can reclaim 10% of peak memory for under 1% throughput loss, scaling to 33% (Llama3) / 38% (Qwen3) with 20% throughput loss. Our implementation achieves SOL PCIe transfer bandwidth of ~300 GB/s.
In model training, the forward pass is run followed by the backward pass to perform updates on the model parameters according to the error gradients. The backward pass computes gradients via the chain rule, and to do so it needs the activations the forward pass produced. Every activation must therefore still be …
Continue reading →Tristan Rice (@d4l3k) · May 14, 2026
distributedtorchcommsncclsymmetric-memorytritonprototyping
TL;DR – Modifying the C++ comms layer is a big barrier when researchers want to prototype new collective features. We’ve added Python bindings to torchcomms (#2080) and built two pure-Python backend prototypes — one wrapping NVIDIA’s new nccl4py bindings (#2515) and one built on SymmetricMemory + Triton (#2521) — both passing the core torchcomms integration test suite. Since they plug into torch.distributed, researchers can fork, tweak, and mix them with existing projects like TorchTitan without touching C++.
We’ve been thinking about how to improve overall research and prototyping speed for comms and collective libraries. LLMs have hugely improved prototyping speed for new ideas and …
Continue reading →Laith Sakka (@laithsakka) · July 10, 2026
dynamic_shapesfxsymintcorrectness
TL;DR – If you’re creating FX graph nodes with raw SymInt arguments, use Graph.materialize_symints or the targeted create_size_node / create_stride_node / create_storage_offset_node helpers instead of passing the raw symbolic value directly. This fixes a common, subtle class of correctness bugs — we’ve already found 3 in PyTorch Inductor and 6 across executorch’s ARM backend passes. Passing raw symbolic values will become a hard error soon.
This is a pattern that’s easy to write and hard to catch. Today it emits a warning; it will become a hard error soon, once we land the executorch fixes:
# `val` is the example/fake tensor stored on a node's meta — under dynamic # shapes its …
Continue reading →Sanket Purandare (@sanketpurandare) · July 10, 2026
dynamic_shapesunbackeddistributeddtensorflex_attentioninductortracing
TL;DR – To capture a whole distributed training step as one FX graph, PT2 has to trace models whose batch and sequence dimensions are unbacked SymInts. When we tried this on the TorchTitan DeepSeek-V3 MoE trainer, every layer of the stack either silently specialized those dims to concrete ints or blew up with a data-dependent error (DDE). This post walks through the 11 fixes — spanning ATen meta kernels, ProxyTensor, the ShapeEnv, Inductor, collective bucketing, DTensor, and FlexAttention — that make each layer keep tensor semantics symbolic while allowing hints for policy decisions only. That contract is what unblocked end-to-end symbolic tracing for the graph trainer and expert-parallel …
Continue reading →Laith Sakka (@laithsakka), Xiao Fu (@fxdawnn) · June 24, 2026
dynamic_shapesunbackedexport
TL;DR – A new dynamic shapes API is available and ready to use! It provides a unified, consistent way for specifying dynamic specs across compile, export and make_fx, brings native unbacked support to torch.export and make_fx, and completes the unbacked story described earlier by providing unified, predictable, declarative control over the shapes of compiled artifacts.
Consider the example below: the user has a function project(x, w) with a fast path for small batches and a general matmul path. The user wants to compile a dynamic-shape artifact that takes the fast path.
The ShapesSpec says x has dynamic shape [B, D] and w has shape [D, D], and the assumption B < 32 commits this artifact to …
Continue reading →Edward Yang (@ezyang) · June 1, 2026
eagercudamemory
Disclosure. This post was drafted by Claude (Anthropic’s coding assistant) with editing from ezyang.
In an ideal world, users of CUDA memory in PyTorch programs should be able to abstract the allocator behavior as: there is a fixed amount of GPU memory, whenever you allocate this available memory goes down, and when you free the available memory goes back up.
Unfortunately, the internal implementation of the CUDA caching allocator means that certain allocation patterns can give rise to fragmentation, where even though there is “technically” enough free space to store a requested allocation, the CUDA caching allocator is unable to actually serve the request.
There are many modern use cases …
Continue reading →Edward Yang (@ezyang) · May 30, 2026
ai-agentscode-reviewossllm
One of the important topics being discussed among the PyTorch team is how the PyTorch codebase should engage with AI coding agents. Today, many PRs to PyTorch are AI-authored, and there have been obvious growing pains as we’ve figured things out. Based on discussions at the most recent PyTorch compiler offsite (May 2026), I’ve assembled this playbook for AI coding in PyTorch. It is half descriptive, half prescriptive: it is trying to codify practices that are being used among some members of the team, and bring everyone else along. Hopefully, this post is just the beginning of our ongoing conversation about how to engage with AI coding agents.
We can think of AI generated code as living in a …
Continue reading →Arsh Zahed (@azahed98) · May 15, 2026
dynamotorch.compilegraph-breaksskills
I’m excited to share debug-graph-breaks, a new skill for debugging Torch Compile graph breaks, now available in the meta-pytorch/skills repository.
Torch Compile graph breaks prevent full graph capture and hurt performance. This skill helps you:
Identify root causes of graph breaks Understand why operations break compilation Get actionable fixes with specific code changes Learn best practices for Torch Compile-friendly code The skill is grounded in the Graph Break Website as its knowledge base—improvements to the website directly improve the skill’s quality.
Evaluated on the OSS Model Graph Break Corpus—a collection of real-world graph break scenarios from open-source models.
Evaluation …
Continue reading →Animesh Jain (@anijain2305) · May 13, 2026
dynamocpythonllm-agentsgraph-breakstp-slots
TL;DR – Dynamo’s ad-hoc CPython support creates fragmented graph breaks that are hard to fix — even for LLM agents. By refactoring Dynamo to mirror CPython’s tp_* slot semantics, we make the system systematically auditable and agent-friendly, already lifting CPython test pass rates from 38% to 45% and proactively eliminating classes of graph breaks in frontier models.
Working with frontier training frameworks has surfaced some fundamental issues in Dynamo. The issues broadly fell into four categories:
CPython language gaps: For example, Dynamo supports calling a functools.partial object but did not support hashing it. Insufficient exception messages: One frontier framework had an unusual …
Continue reading →William Wen (@williamwen42) · May 13, 2026
dynamotorch.compilegraph-breaks
torch._dynamo.config.nested_graph_breaks = True has been enabled on all Dynamo and Inductor unit tests (~250 test files). A sweep of the OSS benchmark models with graph breaks shows 81/82 passing with NGB (the single regression is a pre-existing unstable model), with graph break reductions of up to 67% and graph merging in models with complex nested call structures (GNNs, detection models). Dynamo tracing time is neutral or improved for most models, and models with significant graph merging see up to 15% runtime speedup (8% geomean). The remaining goal is to set nested_graph_breaks to True by default.
The nested graph break problem in torch.compile refers to the Dynamo limitation of only …
Continue reading →Sayak Paul (@sayakpaul), Animesh Jain (@anijain2305), Benjamin Bossan (@BenjaminBossan) · May 11, 2026
torch.compilediffusersregional-compilationdynamic-shapesquantizationlora
TL;DR – torch.compile delivers a ~1.5x speedup on Flux-1-Dev with no quality loss. Use compile_repeated_blocks to cut compile latency 7x (67s → 9.6s) while keeping the speedup, enable dynamic=True to avoid recompiles on shape changes, and combine with CPU offloading, NF4 quantization, and LoRA hot-swap without giving up the compiled kernels.
Diffusion pipelines are heavy: Flux-1-Dev in bf16 weighs ~33 GB and a single image takes 6.7s on an H100. torch.compile can fuse kernels and strip Python overhead, but applying it naively to a real pipeline runs into four practical issues:
Compile latency. First-call JIT cost — 67.4s for the full DiT. Graph breaks. Any unsupported op silently slices the …
Continue reading →Edward Yang (@ezyang) · May 3, 2026
cppcpythonllm
One of the lost arts of PyTorch development is the ability to write idiomatic C++ code that interacts with the CPython API. This was a very important skill in the early days of eager PyTorch, since we spent a lot of time moving large chunks of the framework to C++ for speed reasons, but we don’t touch the C++ code that much these days and many members of the team haven’t written any amount of serious C++.
LLMs seriously lower the barrier for writing C++ and dealing with the minutiae of manual memory management in C. But they’re not perfect. So these devlog is to talk about all of the things that I put into the process. A prompt of sorts. It is based off of the experience driving an LLM to …
Continue reading →Edward Yang (@ezyang) · May 3, 2026
citoolingllmmergedogpytorchbot
Disclosure. This post was drafted by Claude (Anthropic’s coding assistant) with editing from ezyang.
mergedog is an entirely vibe-coded small Python harness that takes one approved pytorch/pytorch PR and shepherds it through CI to the point a human can comment @pytorchbot merge. The idea is to use LLMs to deal with some aspects of the drudgery of landing PRs from external contributors:
Pressing the “Approve CI workflows” button (in a secure way!), Waiting for the CI results to come back, Checking if the CI failures are spurious or real, and Fixing simple CI failures that are just due to brain-os. While each of these tasks is individually not onerous, they take up time in aggregate; and it …
Continue reading →Aaron Orenstein (@aorenste) · April 16, 2026
dispatcherdispatch_keysbackendsautocastfunctionalizationtorch_dispatch
I wanted to write about how PT2 does autograd, but that requires understanding eager autograd, which requires understanding the dispatcher. So let’s start there.
Let’s pretend we’re building Torch. Let’s start from first principles with the problems we encounter and how to solve them.
Problem 1: We want to be able to call operators for each backend.
Solution: Polymorphism! We just define a class where we have every operator defined as a virtual method. Backends just implement every operator.
class Torch: def mm(self, a: Tensor, b: Tensor) -> Tensor: ... def einsum(self, equation: str, *operands: Tensor) -> Tensor: ... ... Now I just need to implement Torch for each “real” backend (CPU, Cuda, …
Continue reading →