Tensorflow + metaflow = Can't pickle datasets/models

I’m trying to put some word embeddings exercise into metaflow to avoid running training or downloading/transforming the datasets every time I run the scripts

Specifically:

2020-02-16 13:04:40.669 [1581858157185440/get_dataset_batches/2 (pid 21)] tensorflow.python.framework.errors_impl.InternalError: Tensorflow type 21 not convertible to numpy dtype

Metaflow’s checkpointing is the perfect answer for this instead of me manually coding all the saving/loading logic

However I can’t seem to checkpoint a lot of common tensorflow steps with metaflow, I’m aware you use tensorflow internally, I assume together with metaflow, do you have any solution for this kind of problems?

I’ve produced a repository where the whole issue is documented and reproducible in a dockerised manner, you can check it out here

[sandman21dan]

Is this something like tempfile.SpooledTemporaryFile or is there a Metaflow method/util to go from file into the self scope of a step?

[savingoyal]

@sandman21dan Please reopen this issue if you have more to add.