Allow custom mounting for kfp local runner
See original GitHub issueFeature Area
Have been using the kfp local runner feature using the docker execution environment. …
run_pipeline_func_locally( pipeline_func, arguments={ “train_output_dir”: TRAINER_OUTPUT_PATH, }, execution_mode=LocalClient.ExecutionMode(“docker”), )
However this always traslated to docker running with mount on '/tmp'.
[‘docker’, ‘run’, ‘-v’, ‘/tmp:/tmp’, '<image>, ‘–training_epochs’, ‘1’, ‘–model_export_dir’, ‘/tmp/model/1629731723/’]
This pollutes the '/tmp' location in local and adds additional steps to components to interact (i.e. move over artifacts and keep them ready in corresponding '/tmp' location.
Alternatively if the component needs to access remote storage it needs to be authenticated and mounting the custom path/configs would be the only relatively safe way to do so.
### What feature would you like to see?
Ability for users to pass parameters to the docker itself like '-v' mount options or '--entrypoint' shell options etc.
### What is the use case or pain point?
There is friction in using kfp local runner as it is through docker vs plain docker due to the feature gap.
### Is there a workaround currently?
Move everything over to '/tmp' to use it for staging.
---
<!-- Don't delete message below to encourage users to support your feature request! -->
Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.
Issue Analytics
- State:
- Created 2 years ago
- Comments:5 (2 by maintainers)
Top Results From Across the Web
Enable local test runner for Kubeflow Pipelines #1138 - GitHub
Today, there is no way to run a KFP pipeline on a local machine (outside of running a mini Kubernetes cluster locally).
Read more >Deploying Kubeflow Pipelines Locally for Elyra
In the example, we will be deploying Kubeflow Pipelines on Kubernetes using Docker Desktop. Note that these instructions will ONLY install the Kubeflow ......
Read more >Local Deployment | Kubeflow
This guide shows how to deploy Kubeflow Pipelines standalone on a local Kubernetes cluster using: ... Deploying Kubeflow Pipelines (KFP). Before ...
Read more >Install Custom Dependency for KFP Op - kubeflow
I'm trying to setup a simple KubeFlow pipeline, and I'm having trouble packaging up dependencies in a way that works for KubeFlow. The...
Read more >Run your first Kubeflow pipeline - DEV Community
Setup local cluster, install KFP and run a pipeline. ... to let Kubeflow Pipelines know to pass this to your component as a...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Just realize
run_pipeline_func_locally
is a wrapper function which does not expose thepipeline_root
parameter.https://github.com/kubeflow/pipelines/pull/6462 should fix the issue.
@chensun Thank you so much for working on it, would there be any way to expose the docker --entrypoint as well? It could be either at pipeline level or at component level to keep it custom.