feature request: allow grid to span multiple notebooks

Grid allows parametrizing a notebook to run multiple experiments with different parameters

  graph LR;
      prepare-->train-model-1;
      prepare-->train-model-2;
      prepare-->train-model-3;

However, in some cases, one may want to grid over many notebooks:

  graph LR;
      prepare-->train-model-1-->evaluate-model-1;
      prepare-->train-model-2-->evaluate-model-2;
      prepare-->train-model-3-->evaluate-model-3;

[mat-ej]

it would be interesting if even the “types” of notebooks connected could be a hyperaparam in the grid. e.g. train model component could be: {scikit-train, pytorch-train, xgb-train}, each with respective hyperparameters. All being part of the parameter space in the grid search.

[edublancas]

Hi,

I’ll start working on this. Requesting some feedback. I’m wondering what’s the cleanest API here, thinking of something like this:

tasks:
   # to create a multi-notebook grid, pass a list
  - source: [model.py, evaluate.py]
    # product must have "model" and "evaluate" as keys - we should validate this
    product:
        model: output/model.html # this can be a dictionary, too
        evaluate: output/evaluate.html
   # this generates 3 * 2 = 6 tasks
    grid:
        n_estimators: [5, 10, 20]
        criterion: [gini, entropy]

The only question on my mind is how do we handle the parameters. For example, for the first task n_estimators=5, and criterion=gini, should we only pass them to the model.py, or also to evaluate.py?

To me, it seems like the grid params are only relevant to the first stage, but there might be circumstances where the other stages might want to know which parameters were used.