.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_examples_pipelines_kfp_dist_pipeline.py>`     to download the full example code
    .. rst-class:: sphx-glr-example-title

    .. _sphx_glr_examples_pipelines_kfp_dist_pipeline.py:


Distributed KubeFlow Pipelines Example
======================================

This is an example KFP pipeline that uses resource_from_app to launch a
distributed operator using the kubernetes/volcano job scheduler. This only works
in Kubernetes KFP clusters with https://volcano.sh/en/docs/ installed on them.


.. code-block:: default


    import kfp
    from torchx import specs
    from torchx.pipelines.kfp.adapter import resource_from_app


    def pipeline() -> None:
        # First we define our AppDef for the component, we set
        echo_app = specs.AppDef(
            name="test-dist",
            roles=[
                specs.Role(
                    name="dist-echo",
                    image="alpine",
                    entrypoint="/bin/echo",
                    args=["hello dist!"],
                    num_replicas=3,
                ),
            ],
        )

        # To convert the TorchX AppDef into a KFP container we use
        # the resource_from_app adapter. This takes generates a KFP Kubernetes
        # resource operator definition from the TorchX app def and instantiates it.
        echo_container: kfp.dsl.BaseOp = resource_from_app(echo_app, queue="default")



To generate the pipeline definition file we need to call into the KFP compiler
with our pipeline function.


.. code-block:: default


    kfp.compiler.Compiler().compile(
        pipeline_func=pipeline,
        package_path="pipeline.yaml",
    )

    with open("pipeline.yaml", "rt") as f:
        print(f.read())


Once this has all run you should have a pipeline file (typically
pipeline.yaml) that you can upload to your KFP cluster via the UI or
a kfp.Client.

See the
`KFP SDK Examples <https://www.kubeflow.org/docs/components/pipelines/legacy-v1/tutorials/sdk-examples/#examples>`_
for more info on launching KFP pipelines.

See the :ref:`examples_pipelines/kfp/advanced_pipeline:Advanced KubeFlow Pipelines Example` for how to chain multiple
components together and use builtin components.


.. code-block:: default



    # sphinx_gallery_thumbnail_path = '_static/img/gallery-kfp.png'


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  0.000 seconds)


.. _sphx_glr_download_examples_pipelines_kfp_dist_pipeline.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: dist_pipeline.py <dist_pipeline.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: dist_pipeline.ipynb <dist_pipeline.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_