question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Allow custom mounting for kfp local runner

See original GitHub issue

Feature Area

Have been using the kfp local runner feature using the docker execution environment. …

run_pipeline_func_locally( pipeline_func, arguments={ “train_output_dir”: TRAINER_OUTPUT_PATH, }, execution_mode=LocalClient.ExecutionMode(“docker”), )

However this always traslated to docker running with mount on '/tmp'.  

[‘docker’, ‘run’, ‘-v’, ‘/tmp:/tmp’, '<image>, ‘–training_epochs’, ‘1’, ‘–model_export_dir’, ‘/tmp/model/1629731723/’]

This pollutes the '/tmp' location in local and adds additional steps to components to interact (i.e. move over artifacts and keep them ready in corresponding '/tmp' location.
Alternatively if the component needs to access remote storage it needs to be authenticated and mounting the custom path/configs would be the only relatively safe way to do so.

### What feature would you like to see?
Ability for users to pass parameters to the docker itself like '-v' mount options or '--entrypoint' shell options etc.

### What is the use case or pain point?
There is friction in using kfp local runner as it is through docker vs plain docker due to the feature gap.

### Is there a workaround currently?
Move everything over to '/tmp' to use it for staging.


---

<!-- Don't delete message below to encourage users to support your feature request! -->
Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
chensuncommented, Aug 27, 2021

Just realize run_pipeline_func_locally is a wrapper function which does not expose the pipeline_root parameter.

https://github.com/kubeflow/pipelines/pull/6462 should fix the issue.

0reactions
etsyamitracommented, Aug 30, 2021

@chensun Thank you so much for working on it, would there be any way to expose the docker --entrypoint as well? It could be either at pipeline level or at component level to keep it custom.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Enable local test runner for Kubeflow Pipelines #1138 - GitHub
Today, there is no way to run a KFP pipeline on a local machine (outside of running a mini Kubernetes cluster locally).
Read more >
Deploying Kubeflow Pipelines Locally for Elyra
In the example, we will be deploying Kubeflow Pipelines on Kubernetes using Docker Desktop. Note that these instructions will ONLY install the Kubeflow ......
Read more >
Local Deployment | Kubeflow
This guide shows how to deploy Kubeflow Pipelines standalone on a local Kubernetes cluster using: ... Deploying Kubeflow Pipelines (KFP). Before ...
Read more >
Install Custom Dependency for KFP Op - kubeflow
I'm trying to setup a simple KubeFlow pipeline, and I'm having trouble packaging up dependencies in a way that works for KubeFlow. The...
Read more >
Run your first Kubeflow pipeline - DEV Community ‍ ‍
Setup local cluster, install KFP and run a pipeline. ... to let Kubeflow Pipelines know to pass this to your component as a...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found