Relocatable store support

Although zarr supports pickling, currently unpickling fails if the underlying store changes, e.g when moving the directory of a DirectoryStore or the file of a ZipStore. As mentioned in the docs:

The only thing that is pickled is the necessary parameters to allow the store to re-open any underlying files or databases upon being unpickled.

How can I add support for unpickling from a different store than the one used for pickling?

[alimanfoo]

FWIW I thought of using zarr’s hierarchy capability here because it seemed quite close to what you were already building but via some Python data structures, esp. nested dictionaries. But it’s not quite the same because you’re using both dicts and lists, so appreciate it’s not a direct swap-in replacement.

Also FWIW zarr’s hierarchy (group) API is intended to match h5py, so it’s not entirely zarr-specific. I.e., if you use the group API then it will be familiar to anyone who has used HDF5-like data structures before.

Also FWIW, it sounds like you’re putting all the arrays inside a single ZipStore. If you are, then you will be having to decide a path for each array, which will be creating some hierarchical (grouping) structure that I’m guessing mirrors closely the Python nested dicts and lists data structure you’re using. So maybe there is an opportunity to just simplify and leverage the grouping structure within the zarr hierarchy, rather than duplicating grouping information into Python objects as well. But maybe it’s not that straightforward.

In any case, I’m still planning to look at the updated PR and will give it some more thought.

[alimanfoo]

Thanks @gsakkis, I must have been looking at an earlier commit, I’ll take another look asap.