proposal: decouple pipeline runner/executor from repro?

We should decouple Pipeline management/runner/scheduler/executor stuff from the Repo which resides on repro. It currently collects stages, builds a DAG, and then schedules the stage/task to run, and then runs them. The running part, which involves removing dependencies and outputs, checking some dependencies and outputs and running the command, and then saving the outputs later is all handled in the Stage.

Honestly, to me, running yourself, as in Stage executing itself does not really make sense. It feels like this could be part of the Pipeline management thing itself.

Why decouple?

1. Events that we are discussing might make it difficult to hook into this, without a separate abstraction.
2. exp run calls dvc repro internally, which is a porcelain command, which feels like a hack. Sharing a common codebase between exp run and dvc repro is better than one using the other, as this will help us change UI output for both easily without affecting one another.
3. We might be able to separate UI output messages from data management commands. This might force us to really separate this with yet another abstraction for data management commands.
4. Decoupling is always a good thing?

This might also open us a few opportunities to:

1. Run stages concurrently?
2. Provide hooks for runners rather than using env variables and signal files? I guess this is a “nice to have” thing.
3. Provide ways to handle outputs from the stage commands intelligently (as in, we might be able to suppress outputs from commands if required or place them in files, show them only when command errors out, etc.) and interactively.
4. Handle runtime contexts easily.

[pmrowla]

@RomanSteinberg this discussion is regarding DVC code organization/modularity. The suggested changes would not affect any existing DVC commands from the user perspective

[karajan1001]

We have a stage executor experiments executor and maybe sometime later a remote executor. Any possibilities to merge them ?