[feature] CLI command to run a pipeline from python files

Feature Area

/area sdk

What feature would you like to see?

A kfp CLI command to directly run python pipelines. For example:

kfp run sample_pipeline.py --host <host-name> --params=key1=value1,key2=value2,...

This command directly compiles a python pipeline and submits it to KFP instance with specified hostname.

Expectation of sample_pipeline.py should be the same as dsl-compile command.

Or what can be even better –

Set up KFP instance connection configuration including credentials:

vim ~/.kfp/config.json # this stores all KFP related configuration in a local config file
# or
export KF_PIPELINES_ENDPOINT=<kfp-host>

Run samples you want:

# all host and credentials information are loaded from either a config file or env vars
# so these command do not need to repeat them
kfp run sample_pipeline.py --params=key1=value1,key2=value2,...

What is the use case or pain point?

I have two options for doing the same thing:

• write python script wrappers that calls create_run_from_pipeline_func
• use dsl-compile and then kfp upload and then kfp run

Both options involve some unnecessary steps, I think it’s beneficial providing a CLI command to make this common command easier, because

• this command can be very helpful during pipeline development
• people do not need to write __main__ function for every pipeline, and they do not need to support arguments like host, parameters. All of these interface can be standardized in kfp run CLI.
• it’s convenient for building a KFP component that runs KFP pipelines or hooking up a KFP pipeline in a CI step

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[aronchick]

This is correct 🙂 I’ll do a demo in the coming weeks - as soon as i can find the time!

---

From: Yuan (Bob) Gong @.> Sent: Thursday, July 1, 2021 3:06 AM To: kubeflow/pipelines @.> Cc: David Aronchick @.>; Mention @.> Subject: [EXTERNAL] Re: [kubeflow/pipelines] [feature] CLI command to run a pipeline from python files (#5381)

Cross posting for context, I think @aronchickhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faronchick&data=04|01|david.aronchick%40microsoft.com|a53824335d68462b152508d93c77e32d|72f988bf86f141af91ab2d7cd011db47|1|0|637607307879377677|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=g5GtbOdP0C%2B4lzEbllVRUK4gpTalpybv14x%2BtUxv380%3D&reserved=0 is talking about The SAME Project: A Cloud Native Approach to Reproducible Machine Learning - David Aronchickhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DaQvvGKVX25I&data=04|01|david.aronchick%40microsoft.com|a53824335d68462b152508d93c77e32d|72f988bf86f141af91ab2d7cd011db47|1|0|637607307879377677|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=AdfdEUTYvIwsQZdlCTmiZaMdyOHg42UIY2o%2B%2FUok7AI%3D&reserved=0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubeflow%2Fpipelines%2Fissues%2F5381%23issuecomment-872108450&data=04|01|david.aronchick%40microsoft.com|a53824335d68462b152508d93c77e32d|72f988bf86f141af91ab2d7cd011db47|1|0|637607307879387671|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=zfhFXOgTZgd3QGR5C1NVmGOXvg%2FOxUKBWoDhmxrExXM%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAAMQ5PZVXWXWK5VGHM4UI3TVQ42DANCNFSM4Z4MSSLQ&data=04|01|david.aronchick%40microsoft.com|a53824335d68462b152508d93c77e32d|72f988bf86f141af91ab2d7cd011db47|1|0|637607307879387671|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=p9Vs7JyNkMCEqlH4V%2BiTWwThR9aIgGbAicEwNO%2BvgOs%3D&reserved=0.

[aronchick]

I think i’d like to wait until it’s open, but happy to give as many folks as would like private demos.

---

From: Yuan (Bob) Gong @.> Sent: Tuesday, March 30, 2021 8:14 PM To: kubeflow/pipelines @.> Cc: David Aronchick @.>; Mention @.> Subject: Re: [kubeflow/pipelines] [feature] CLI command to run a pipeline from python files (#5381)

@aronchickhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faronchick&data=04|01|david.aronchick%40microsoft.com|09477d4bb02546f98b2108d8f3f3121b|72f988bf86f141af91ab2d7cd011db47|1|0|637527572602641398|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=2ZxOZP5kPw%2F6148e1mEG1Yx1DrKKyKO5q9wohM2Cpxw%3D&reserved=0 Is it possible to present a demo on KFP community meetings? Or you need to wait until it’s open?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubeflow%2Fpipelines%2Fissues%2F5381%23issuecomment-810726304&data=04|01|david.aronchick%40microsoft.com|09477d4bb02546f98b2108d8f3f3121b|72f988bf86f141af91ab2d7cd011db47|1|0|637527572602651393|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=NK2eX5XZ%2BFHFmqQhoni%2FEV6XwS%2BeUCpQdDyy1MW845E%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAAMQ5KQ37K2TNB5WBTHUWDTGKHQTANCNFSM4Z4MSSLQ&data=04|01|david.aronchick%40microsoft.com|09477d4bb02546f98b2108d8f3f3121b|72f988bf86f141af91ab2d7cd011db47|1|0|637527572602661393|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=WYaV4j3hQPzr6Sw9gBl0DSPbZ7hB0JpmfTT4y%2FasO6I%3D&reserved=0.