deep-dive

Run in Google Colab

Colab

Download Notebook

Notebook

View on GitHub

GitHub

Deep Dive

Focused on enhancing model performance, this section includes tutorials on profiling, hyperparameter tuning, quantization, and other techniques to optimize PyTorch models for better efficiency and speed.

---

Profiling PyTorch

Learn how to profile a PyTorch application

Profiling

CUDA Graph Kernel Annotations and Profiling

Learn how to annotate CUDA graph kernels for semantic profiling traces with custom visualization lanes.

Profiling,CUDA,Model-Optimization

Parametrizations Tutorial

Learn how to use torch.nn.utils.parametrize to put constraints on your parameters (e.g. make them orthogonal, symmetric positive definite, low-rank...)

Model-Optimization,Best-Practice

Pruning Tutorial

Learn how to use torch.nn.utils.prune to sparsify your neural networks, and how to extend it to implement your own custom pruning technique.

Model-Optimization,Best-Practice

Inductor CPU Backend Debugging and Profiling

Learn the usage, debugging and performance profiling for ``torch.compile`` with Inductor CPU backend.

Model-Optimization,inductor

(beta) Implementing High-Performance Transformers with SCALED DOT PRODUCT ATTENTION

This tutorial explores the new torch.nn.functional.scaled_dot_product_attention and how it can be used to construct Transformer components.

Model-Optimization,Attention,Transformer

Knowledge Distillation in Convolutional Neural Networks

Learn how to improve the accuracy of lightweight models using more powerful models as teachers.

Model-Optimization,Image/Video

(beta) Channels Last Memory Format in PyTorch

Get an overview of Channels Last memory format and understand how it is used to order NCHW tensors in memory preserving dimensions.

Memory-Format,Best-Practice,Frontend-APIs

Forward-mode Automatic Differentiation

Learn how to use forward-mode automatic differentiation.

Frontend-APIs

Jacobians, Hessians, hvp, vhp, and more

Learn how to compute advanced autodiff quantities using torch.func

Frontend-APIs

Model Ensembling

Learn how to ensemble models using torch.vmap

Frontend-APIs

Per-Sample-Gradients

Learn how to compute per-sample-gradients using torch.func

Frontend-APIs

Neural Tangent Kernels

Learn how to compute neural tangent kernels using torch.func

Frontend-APIs

Using the PyTorch C++ Frontend

Walk through an end-to-end example of training a model with the C++ frontend by training a DCGAN – a kind of generative model – to generate images of MNIST digits.

Frontend-APIs,C++

Autograd in C++ Frontend

The autograd package helps build flexible and dynamic nerural netorks. In this tutorial, exploreseveral examples of doing autograd in PyTorch C++ frontend

Frontend-APIs,C++