clarify pipeline stages vs experiments
See original GitHub issue- discussion blocked by/depends on https://github.com/iterative/dvc/issues/7866
Some features often underused/misunderstood/unknown could be helped by better docs/messaging/onboarding clarity.
- Should there be a page clearly describing the difference between
stage
s andexp
eriements?
Nothing in use-cases/experiment-tracking nor user-guide/experiment-management seems to tell existing dvc repro
users why they should bother with/what are the use cases of dvc exp
.
It doesn’t seem clear to users what’s the difference between stage/repro
(i.e. pipelines) and exp
(i.e. experiments).
- A feature comparison table would be epic.
Issue Analytics
- State:
- Created a year ago
- Reactions:2
- Comments:7 (6 by maintainers)
Top Results From Across the Web
Pipeline architecture - GitLab Docs
Pipelines are the fundamental building blocks for CI/CD in GitLab. This page documents some of the important concepts related to them.
Read more >pipelines: clarify whether it's possible to have more than one ...
I am assuming that it's not possible to re-use certain stages or that it's possible to use some sort of composition. Let's say...
Read more >Working with Pipelines - MLOps Guide
Let's also consider that we have three stages in this experiment: Preprocessing your data(extract features...) Train the model; Evaluate the model. Here you ......
Read more >Running Experiments | Data Version Control - DVC
These files codify pipelines that specify one or more stagesstages of the experiment workflow (code, dependenciesdependencies, outputsoutputs, etc.). See Get ...
Read more >GitLab: understanding pipelines, stages, jobs and organising ...
Each job belongs to a single stage. Stage can contain zero, one or more jobs to execute. All jobs in a single stage...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
These two clarification points I’ve found in various places (the latter one from @SoyGema) have been very useful for me as a user:
exp
produce a git ref, that is how it stores its state.We do mention
exp run
vs.repro
specifically in several places like https://dvc.org/doc/user-guide/experiment-management/experiments-overview#basic-workflow, https://dvc.org/doc/user-guide/experiment-management/running-experiments#running-the-pipelines, and https://dvc.org/doc/command-reference/exp/run.