Edward Yang (@ezyang) · August 11, 2026
eagercudaperformanceprofiling
Disclosure. This post was drafted by Claude (Anthropic’s coding assistant) with editing from ezyang.
Once you get a bit experienced with writing performant PyTorch code you know to avoid device-to-host syncs: calling .item() is an easy way to become CPU bound in kernel launches afterwards, since we have to wait for the GPU work to finish before we can get the result to the CPU. And with more time, you might find out about a number of API footguns in PyTorch’s API that invisibly cause DtoH syncs, like x[bool_tensor], which implicitly triggers a torch.nonzero sync.
But what you might not be expecting is that host-to-device syncs are bad too! And these syncs can also show up in sneaky ways, …
Continue reading →Edward Yang (@ezyang) · August 9, 2026
eagercudamemorypinned-memorycuda-graphs
Disclosure. This post was drafted by Claude (Anthropic’s coding assistant) with editing from ezyang.
For quite some time, I used to think of pinned memory as something you sprinkled around your code to let you do async transfers to GPU. In fact, the old Caffe2 used to pin every single CPU tensor, even if it never actually participated in GPU compute. Under this regime, you might imagine that you need an allocator for pinned memory that lets you allocate and free pinned memory as necessary.
However, if you think carefully about the implications of async transfers on pinned memory lifetime, as well as the implications for CUDA graphs, it turns out that you don’t… really want to ever free …
Continue reading →Edward Yang (@ezyang) · June 1, 2026
eagercudamemory
Disclosure. This post was drafted by Claude (Anthropic’s coding assistant) with editing from ezyang.
In an ideal world, users of CUDA memory in PyTorch programs should be able to abstract the allocator behavior as: there is a fixed amount of GPU memory, whenever you allocate this available memory goes down, and when you free the available memory goes back up.
Unfortunately, the internal implementation of the CUDA caching allocator means that certain allocation patterns can give rise to fragmentation, where even though there is “technically” enough free space to store a requested allocation, the CUDA caching allocator is unable to actually serve the request.
There are many modern use cases …
Continue reading →