Wei Feng (@weifengpy) · September 29, 2026
eagercudastreamsmemoryfsdp
TL;DR – The CUDA caching allocator only knows which stream a tensor was allocated on. When a producer stream allocates a buffer and a consumer stream reads it, keep a Python reference to the buffer and record a CUDA event on the consumer, then make the producer stream wait_event on it before you drop the reference. Don’t reach for Tensor.record_stream: it’s correct, but it hands memory reuse to the allocator’s event polling, which makes peak memory depend on how far the CPU runs ahead.
Overlapping communication with compute comes down to a producer and a consumer on two CUDA streams. In FSDP2’s forward pass, the all-gather stream (the producer) fetches each layer’s full parameters, and the …
Continue reading →Edward Yang (@ezyang) · August 11, 2026
eagercudaperformanceprofiling
Disclosure. This post was drafted by Claude (Anthropic’s coding assistant) with editing from ezyang.
Once you get a bit experienced with writing performant PyTorch code you know to avoid device-to-host syncs: calling .item() is an easy way to become CPU bound in kernel launches afterwards, since we have to wait for the GPU work to finish before we can get the result to the CPU. And with more time, you might find out about a number of API footguns in PyTorch’s API that invisibly cause DtoH syncs, like x[bool_tensor], which implicitly triggers a torch.nonzero sync.
But what you might not be expecting is that host-to-device syncs are bad too! And these syncs can also show up in sneaky ways, …
Continue reading →Edward Yang (@ezyang) · August 9, 2026
eagercudamemorypinned-memorycuda-graphs
Disclosure. This post was drafted by Claude (Anthropic’s coding assistant) with editing from ezyang.
For quite some time, I used to think of pinned memory as something you sprinkled around your code to let you do async transfers to GPU. In fact, the old Caffe2 used to pin every single CPU tensor, even if it never actually participated in GPU compute. Under this regime, you might imagine that you need an allocator for pinned memory that lets you allocate and free pinned memory as necessary.
However, if you think carefully about the implications of async transfers on pinned memory lifetime, as well as the implications for CUDA graphs, it turns out that you don’t… really want to ever free …
Continue reading →Edward Yang (@ezyang) · June 1, 2026
eagercudamemory
Disclosure. This post was drafted by Claude (Anthropic’s coding assistant) with editing from ezyang.
In an ideal world, users of CUDA memory in PyTorch programs should be able to abstract the allocator behavior as: there is a fixed amount of GPU memory, whenever you allocate this available memory goes down, and when you free the available memory goes back up.
Unfortunately, the internal implementation of the CUDA caching allocator means that certain allocation patterns can give rise to fragmentation, where even though there is “technically” enough free space to store a requested allocation, the CUDA caching allocator is unable to actually serve the request.
There are many modern use cases …
Continue reading →