.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "generated_examples/basic_example.py" .. LINE NUMBERS ARE GIVEN BELOW. .. rst-class:: sphx-glr-example-title .. _sphx_glr_generated_examples_basic_example.py: ======================================== Decoding a video with SimpleVideoDecoder ======================================== In this example, we'll learn how to decode a video using the :class:`~torchcodec.decoders.SimpleVideoDecoder` class. .. GENERATED FROM PYTHON SOURCE LINES 17-20 First, a bit of boilerplate: we'll download a video from the web, and define a plotting utility. You can ignore that part and jump right below to :ref:`creating_decoder`. .. GENERATED FROM PYTHON SOURCE LINES 20-54 .. code-block:: Python from typing import Optional import torch import requests # Video source: https://www.pexels.com/video/dog-eating-854132/ # License: CC0. Author: Coverr. url = "https://videos.pexels.com/video-files/854132/854132-sd_640_360_25fps.mp4" response = requests.get(url) if response.status_code != 200: raise RuntimeError(f"Failed to download video. {response.status_code = }.") raw_video_bytes = response.content def plot(frames: torch.Tensor, title : Optional[str] = None): try: from torchvision.utils import make_grid from torchvision.transforms.v2.functional import to_pil_image import matplotlib.pyplot as plt except ImportError: print("Cannot plot, please run `pip install torchvision matplotlib`") return plt.rcParams["savefig.bbox"] = 'tight' fig, ax = plt.subplots() ax.imshow(to_pil_image(make_grid(frames))) ax.set(xticklabels=[], yticklabels=[], xticks=[], yticks=[]) if title is not None: ax.set_title(title) plt.tight_layout() .. GENERATED FROM PYTHON SOURCE LINES 55-63 .. _creating_decoder: Creating a decoder ------------------ We can now create a decoder from the raw (encoded) video bytes. You can of course use a local video file and pass the path as input, rather than download a video. .. GENERATED FROM PYTHON SOURCE LINES 63-68 .. code-block:: Python from torchcodec.decoders import SimpleVideoDecoder # You can also pass a path to a local file! decoder = SimpleVideoDecoder(raw_video_bytes) .. GENERATED FROM PYTHON SOURCE LINES 69-72 The has not yet been decoded by the decoder, but we already have access to some metadata via the ``metadata`` attribute which is a :class:`~torchcodec.decoders.VideoStreamMetadata` object. .. GENERATED FROM PYTHON SOURCE LINES 72-74 .. code-block:: Python print(decoder.metadata) .. rst-class:: sphx-glr-script-out .. code-block:: none VideoStreamMetadata: num_frames: 345 duration_seconds: 13.8 average_fps: 25.0 duration_seconds_from_header: 13.8 bit_rate: 505790.0 num_frames_from_header: 345 num_frames_from_content: 345 begin_stream_seconds: 0.0 end_stream_seconds: 13.8 codec: h264 width: 640 height: 360 average_fps_from_header: 25.0 stream_index: 0 .. GENERATED FROM PYTHON SOURCE LINES 75-77 Decoding frames by indexing the decoder --------------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 77-86 .. code-block:: Python first_frame = decoder[0] # using a single int index every_twenty_frame = decoder[0 : -1 : 20] # using slices print(f"{first_frame.shape = }") print(f"{first_frame.dtype = }") print(f"{every_twenty_frame.shape = }") print(f"{every_twenty_frame.dtype = }") .. rst-class:: sphx-glr-script-out .. code-block:: none first_frame.shape = torch.Size([3, 360, 640]) first_frame.dtype = torch.uint8 every_twenty_frame.shape = torch.Size([18, 3, 360, 640]) every_twenty_frame.dtype = torch.uint8 .. GENERATED FROM PYTHON SOURCE LINES 87-96 Indexing the decoder returns the frames as :class:`torch.Tensor` objects. By default, the shape of the frames is ``(N, C, H, W)`` where N is the batch size C the number of channels, H is the height, and W is the width of the frames. The batch dimension N is only present when we're decoding more than one frame. The dimension order can be changed to ``N, H, W, C`` using the ``dimension_order`` parameter of :class:`~torchcodec.decoders.SimpleVideoDecoder`. Frames are always of ``torch.uint8`` dtype. .. GENERATED FROM PYTHON SOURCE LINES 96-99 .. code-block:: Python plot(first_frame, "First frame") .. image-sg:: /generated_examples/images/sphx_glr_basic_example_001.png :alt: First frame :srcset: /generated_examples/images/sphx_glr_basic_example_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 100-102 .. code-block:: Python plot(every_twenty_frame, "Every 20 frame") .. image-sg:: /generated_examples/images/sphx_glr_basic_example_002.png :alt: Every 20 frame :srcset: /generated_examples/images/sphx_glr_basic_example_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 103-107 Iterating over frames --------------------- The decoder is a normal iterable object and can be iterated over like so: .. GENERATED FROM PYTHON SOURCE LINES 107-114 .. code-block:: Python for frame in decoder: assert ( isinstance(frame, torch.Tensor) and frame.shape == (3, decoder.metadata.height, decoder.metadata.width) ) .. GENERATED FROM PYTHON SOURCE LINES 115-126 Retrieving pts and duration of frames ------------------------------------- Indexing the decoder returns pure :class:`torch.Tensor` objects. Sometimes, it can be useful to retrieve additional information about the frames, such as their :term:`pts` (Presentation Time Stamp), and their duration. This can be achieved using the :meth:`~torchcodec.decoders.SimpleVideoDecoder.get_frame_at` and :meth:`~torchcodec.decoders.SimpleVideoDecoder.get_frames_at` methods, which will return a :class:`~torchcodec.decoders.Frame` and :class:`~torchcodec.decoders.FrameBatch` objects respectively. .. GENERATED FROM PYTHON SOURCE LINES 126-131 .. code-block:: Python last_frame = decoder.get_frame_at(len(decoder) - 1) print(f"{type(last_frame) = }") print(last_frame) .. rst-class:: sphx-glr-script-out .. code-block:: none type(last_frame) = Frame: data (shape): torch.Size([3, 360, 640]) pts_seconds: 13.76 duration_seconds: 0.04 .. GENERATED FROM PYTHON SOURCE LINES 132-136 .. code-block:: Python middle_frames = decoder.get_frames_at(start=10, stop=20, step=2) print(f"{type(middle_frames) = }") print(middle_frames) .. rst-class:: sphx-glr-script-out .. code-block:: none type(middle_frames) = FrameBatch: data (shape): torch.Size([5, 3, 360, 640]) pts_seconds: tensor([0.4000, 0.4800, 0.5600, 0.6400, 0.7200], dtype=torch.float64) duration_seconds: tensor([0.0400, 0.0400, 0.0400, 0.0400, 0.0400], dtype=torch.float64) .. GENERATED FROM PYTHON SOURCE LINES 137-140 .. code-block:: Python plot(last_frame.data, "Last frame") plot(middle_frames.data, "Middle frames") .. rst-class:: sphx-glr-horizontal * .. image-sg:: /generated_examples/images/sphx_glr_basic_example_003.png :alt: Last frame :srcset: /generated_examples/images/sphx_glr_basic_example_003.png :class: sphx-glr-multi-img * .. image-sg:: /generated_examples/images/sphx_glr_basic_example_004.png :alt: Middle frames :srcset: /generated_examples/images/sphx_glr_basic_example_004.png :class: sphx-glr-multi-img .. GENERATED FROM PYTHON SOURCE LINES 141-147 Both :class:`~torchcodec.decoders.Frame` and :class:`~torchcodec.decoders.FrameBatch` have a ``data`` field, which contains the decoded tensor data. They also have the ``pts_seconds`` and ``duration_seconds`` fields which are single ints for :class:`~torchcodec.decoders.Frame`, and 1-D :class:`torch.Tensor` for :class:`~torchcodec.decoders.FrameBatch` (one value per frame in the batch). .. GENERATED FROM PYTHON SOURCE LINES 149-158 Using time-based indexing ------------------------- So far, we have retrieved frames based on their index. We can also retrieve frames based on *when* they are displayed with :meth:`~torchcodec.decoders.SimpleVideoDecoder.get_frame_displayed_at` and :meth:`~torchcodec.decoders.SimpleVideoDecoder.get_frames_displayed_at`, which also returns :class:`~torchcodec.decoders.Frame` and :class:`~torchcodec.decoders.FrameBatch` respectively. .. GENERATED FROM PYTHON SOURCE LINES 158-163 .. code-block:: Python frame_at_2_seconds = decoder.get_frame_displayed_at(seconds=2) print(f"{type(frame_at_2_seconds) = }") print(frame_at_2_seconds) .. rst-class:: sphx-glr-script-out .. code-block:: none type(frame_at_2_seconds) = Frame: data (shape): torch.Size([3, 360, 640]) pts_seconds: 2.0 duration_seconds: 0.04 .. GENERATED FROM PYTHON SOURCE LINES 164-171 .. code-block:: Python first_two_seconds = decoder.get_frames_displayed_at( start_seconds=0, stop_seconds=2, ) print(f"{type(first_two_seconds) = }") print(first_two_seconds) .. rst-class:: sphx-glr-script-out .. code-block:: none type(first_two_seconds) = FrameBatch: data (shape): torch.Size([50, 3, 360, 640]) pts_seconds: tensor([0.0000, 0.0400, 0.0800, 0.1200, 0.1600, 0.2000, 0.2400, 0.2800, 0.3200, 0.3600, 0.4000, 0.4400, 0.4800, 0.5200, 0.5600, 0.6000, 0.6400, 0.6800, 0.7200, 0.7600, 0.8000, 0.8400, 0.8800, 0.9200, 0.9600, 1.0000, 1.0400, 1.0800, 1.1200, 1.1200, 1.2000, 1.2400, 1.2800, 1.3200, 1.3600, 1.4000, 1.4400, 1.4800, 1.5200, 1.5600, 1.6000, 1.6400, 1.6800, 1.7200, 1.7600, 1.8000, 1.8400, 1.8800, 1.9200, 1.9600], dtype=torch.float64) duration_seconds: tensor([0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400, 0.0400], dtype=torch.float64) .. GENERATED FROM PYTHON SOURCE LINES 172-174 .. code-block:: Python plot(frame_at_2_seconds.data, "Frame displayed at 2 seconds") plot(first_two_seconds.data, "Frames displayed during [0, 2) seconds") .. rst-class:: sphx-glr-horizontal * .. image-sg:: /generated_examples/images/sphx_glr_basic_example_005.png :alt: Frame displayed at 2 seconds :srcset: /generated_examples/images/sphx_glr_basic_example_005.png :class: sphx-glr-multi-img * .. image-sg:: /generated_examples/images/sphx_glr_basic_example_006.png :alt: Frames displayed during [0, 2) seconds :srcset: /generated_examples/images/sphx_glr_basic_example_006.png :class: sphx-glr-multi-img .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 2.720 seconds) .. _sphx_glr_download_generated_examples_basic_example.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: basic_example.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: basic_example.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: basic_example.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_