:orphan:

.. _torch_tensorrt_distributed_examples:

Distributed Inference
=====================

Examples of multi-GPU distributed inference with Torch-TensorRT, covering
data parallelism (running copies of the same model on multiple GPUs) and
tensor parallelism (splitting a single large model across multiple GPUs).



.. raw:: html

    <div class="sphx-glr-thumbnails">


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This interactive script is intended as a sample of distributed inference using data parallelism...">

.. only:: html

  .. image:: /tutorials/_rendered_examples/distributed_inference/images/thumb/sphx_glr_data_parallel_stable_diffusion_thumb.png
    :alt:

  :ref:`sphx_glr_tutorials__rendered_examples_distributed_inference_data_parallel_stable_diffusion.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Torch-TensorRT Distributed Inference</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This interactive script is intended as a sample of distributed inference using data parallelism...">

.. only:: html

  .. image:: /tutorials/_rendered_examples/distributed_inference/images/thumb/sphx_glr_data_parallel_gpt2_thumb.png
    :alt:

  :ref:`sphx_glr_tutorials__rendered_examples_distributed_inference_data_parallel_gpt2.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Torch-TensorRT Distributed Inference</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Below example shows how to use Torch-TensorRT backend for distributed inference with tensor par...">

.. only:: html

  .. image:: /tutorials/_rendered_examples/distributed_inference/images/thumb/sphx_glr_tensor_parallel_simple_example_thumb.png
    :alt:

  :ref:`sphx_glr_tutorials__rendered_examples_distributed_inference_tensor_parallel_simple_example.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Tensor Parallel Distributed Inference with Torch-TensorRT</div>
    </div>


.. raw:: html

    </div>


.. toctree::
   :hidden:

   /tutorials/_rendered_examples/distributed_inference/data_parallel_stable_diffusion
   /tutorials/_rendered_examples/distributed_inference/data_parallel_gpt2
   /tutorials/_rendered_examples/distributed_inference/tensor_parallel_simple_example


.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-gallery

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download all examples in Python source code: distributed_inference_python.zip </tutorials/_rendered_examples/distributed_inference/distributed_inference_python.zip>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download all examples in Jupyter notebooks: distributed_inference_jupyter.zip </tutorials/_rendered_examples/distributed_inference/distributed_inference_jupyter.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_