question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Allow different stages of a pipeline to run on differently-sized containers

See original GitHub issue

In the pipeline in the covid-notebooks project (https://github.com/CODAIT/covid-notebooks/blob/master/pipelines/us_data.pipeline), one of the notebooks (fit_us_data.ipynb) takes considerably longer than the rest because it performs 10,000 curve-fitting operations serially, one after the other. I would like to make this notebook run faster by performing the curve-fitting operations in parallel.

This change will require having the fit_us_data.ipynb notebook run on a larger container with more cores. I would like for the remaining notebooks in the pipeline to continue running on smaller containers.

Currently there is no way to specify in the pipeline editor UI that different stages of the pipeline should execute on containers with different amounts of CPU and/or memory. Containers inherit whatever is the default container size for the KFP cluster.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
akchinSTCcommented, Jan 15, 2021

scratch that^^. lets stick with our original plan to plumb 3 new fields, CPU, Memory and GPU.

1reaction
akchinSTCcommented, Jan 15, 2021

@marthacryan - we will probably need two fields, one for the container size(sm,med,lg) and one for number of gpus, since not all workloads will require gpus and it doesnt make sense to include gpu resources in with container sizing

Read more comments on GitHub >

github_iconTop Results From Across the Web

Using Multi-Stage Builds to Simplify And Standardize Build ...
Multi-stage builds are a method of organizing a Dockerfile to minimize the size of the final container, improve run time performance, allow ......
Read more >
Comparing Container Pipelines - DZone DevOps
Container pipelines automate each of the stages in the container deployment process, from building the initial image to deploying to production.
Read more >
Pipeline Syntax - Jenkins
When specified, each stage will run in a new container instance on the same node, rather than all stages running in the same...
Read more >
Pipeline Stages - Spinnaker
Run an AWS CodeBuild build by specifying the name of the project. The sources and the image used by the build can be...
Read more >
Define container jobs (YAML) - Azure Pipelines - Microsoft Learn
When you specify a container in your pipeline, the agent will first fetch and start the container. Then, each step of the job...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found