Reconfigurable pipelines

Many issues require reconfiguration of stages and even pipelines:

• In #973 @Hong-Xiang was asking about reusing (reconfigurable) pipelines.
• Discord discussion with vern from 11/27/18: “it’s annoying to write them all (stages) out by hand and then do it again for each color (parameter).”
• #1416 parametrize pipeline \ step - not config file, just parameters.
• #1119 repetitive commands. I see a similarity with parametrizable commands where only a single output is in use and without creating a separate directory for each experiment (./output.p instead of gs1/output.p).

A concept of reconfigurable-stage should be introduced in DVC.

Open questions:

1. how to pass config file (do we need multiple config files)?
2. how to pass params (should we override config by the params or it is a separate concept)?
3. how to specify input files?
4. how to specify the output directory (gs1/)?
5. how to specify an output file without an output directory ( see ./output.p instead of gs1/output.p from the above)?
6. how to make it work for pipelines?
7. Should build cache #1234 be part of the solution? It will allow caching intermediate results of reusable stages (if step1 is the same in a few pipeline instances).

UPDATE: #1214 might be also related to this issue. UPDATE2: Add a quote from vern and open question 7.

[kaleidoescape]

I believe that this issue may also address our use case (but please correct me if I’m wrong or if you have some nice idea for something else that already addresses it better).

Anyways, in our case, we have some large codebase that has various functions in it which perform different steps in our pipeline. We also have multiple customers that we create models for. We also create multiple models for each customer.

1. We want to keep the customer data in repos that are separate from the big codebase, and also separate from each other.
2. We would like to use different steps in our pipeline from our big codebase, depending on the customer, e.g. for customer A we want to use steps (a, b, c, d, e) and for customer B we would like to use steps (b, c, a, e, f). We want to save the output of each step to its own DVC file.
3. In addition, ideally, we would create some kind of configuration, which gives our data file to the first step in our pipeline, and then passes the outputs of each step along the chain to the other pipeline steps (so that we don’t always have to write filepaths everywhere in every single step).
4. And our pipeline steps are of course not necessarily linear (but I think DVC already addresses this aspect quite well).

I think no matter what, we’d have to do quite a bit of custom work to make everything run smoothly, but I think maybe with reconfigurable DVC pipelines, it’d be a little easier.

[dmpetrov]

@pared I’ve just separated two issues: this one and #1472. And thank you for your comments - it made me clarifying the issue and even renaming it.

This issue is just about defining configs/input/params and how to instantiate a pipeline with a new set of params. The instantiation part might and probably should include #1234.

A store of pipeline is related to the new issue. I don’t think any special store is needed. A module can be simply reused from Git repos or just copied.