Model versioning

I’m curious if the following feature is possible in metaflow. I don’t think so because of some pretty low-level python limitations, but I figured I would as the community and see if I’m missing something.

A pretty wonderful use case for metaflow would be to standardize the validation of models over time. To do that you would need to load in some saved version of the model, and predict on some new data.

However, saving and loading data in metaflow relies on pickle, which would evaluate all dependencies at runtime.

So, am I right that this is an unavoidable problem

train/1/x - model v 1.0.0
train/2/x - model v 1.0.1

... time passes

validate/1/x - model v 1.0.1, so the flow might fail to load v1.0.0 or load it incorrectly

Can metaflow help circumvent this issue in any way?

[romain-intel]

Hi. To provide a bit more detail, as @savingoyal mentioned, we are actually working on a way to access artifacts produced in a conda environment. As you rightly pointed out, an artifact that was saved within a specific environment (model in your TrainFlow) needs to be accessed within that environment again. This is true if you use a Notebook to access it or access it via the client API in another flow (as you do in your ValidateFlow). Currently, for your case, the solution would be to have the validate step run in the same environment as the train step. This is clearly not ideal in large part because we cannot expect the writer of the validate flow to necessarily know what the train flow did.

The solution we are working on would spin up a separate python interpreter in the proper environment (so the conda environment that was used to create the artifact in the first place) and pass all methods on that artifact to that sub-interpreter. There are lots of details and caveats for this but the basic principle has been tested for a different use case within Netflix (it’s basically a form of RPC).

We are hoping to develop something in Q2 which would make our Conda solution more complete (right now, artifacts that are pickled in a specific conda environment are hard to access unless you are in that same conda environment).

[savingoyal]

We are actually working on a feature this quarter (Q2 2020) to essentially enable what you are seeking. run.model.score will execute inside its own environment that was defined when the model was first generated. @romain-intel can share more details on that front.