Development friendly Kubeflow experience

/kind feature

Why you need this feature: Say I have a local package containing my application logic (i.e cleaning, feature generation, ML model training, etc…). This local package contains modules and functions used in my component.

I want to make changes to the application logic (i.e change a feature scaling method), then run my pipeline and 1) make sure the pipeline works or 2) see an improvement in my offline metrics.

My component image needs to have all the dependencies on the image, so this seems to mean that if I want to run my kubeflow pipeline with new code, I need to re-build and submit an image each time. This is a pretty slow process, and prevents us from wanting to make smaller components (better to develop pipelines in Python and run them as a bigger component via a CLI command).

I’m imagining one solution with a local Kubeflow instance, that has the component images pointing to locally built docker images that have the local application code mounted, you can get a much faster iteration cycle.

Is there a better way to develop faster with Kubeflow? It says it’s experimentation friendly, but I haven’t felt that from working with Kubeflow so far (it is nice that it has experiment management/tracking in the UI though!). I don’t feel like I can swap my current experimentation workflow out for Kubeflow.

Maybe a user guide on developing locally could be a good solution? Something equivalent to pip install -e . for Kubeflow components would be great!

[DavidSpek]

/area pipelines Ping @Bobgy. Seeing as this is related to Pipelines specifically maybe it can be moved to the kubeflow/pipelines repo.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment “/reopen” to reopen it.