Refactoring stores into a separate library

Over time the Zarr library accumulated more compressors and even some Cython code related to that effort until it made sense for these to go out and live as their own library, Numcodecs. Similarly we seem to be accumulating many stores, perhaps they would be best suited in another library that Zarr depends on.

The stores themselves actually don’t really need to know about compression, Groups, or Arrays (with the exception of some layers on top of the stores). Really stores are just key-value stores that allow associating some blob with some key. As such it may be possible for this library to have fewer requirements (if any) and be accessible to a larger group of people that are just interested in things that provide the MutableMapping interface to their storage format of choice.

By breaking out the stores as a separate library, we gain another benefit. Namely we can release the stores at a different time scale from Zarr itself. So if we get a new store or a bug fix for a store, we can do a new release even while Zarr itself has not changed much.

There are probably more reasons to consider this move. Would be curious to hear what others think about this?

[rabernat]

I really like this idea.

[mbr]

The second is that zarr stores supports some optional methods, including some which are “hierarchy-aware”, i.e., when using keys that include a ‘/’ character and wanting to list keys at a given level, akin to listing the contents of a directory if using a file system.

So far, I have deliberately omittied these kinds of hierarchies; / characters are also not allowed in keys; this is to keep things are portable and simple as possible. There is support for rolling this yourself in a way through the keys method which takes a prefix argument for this reason.

It’s worth pointing out this really helps as we can leverage other methods of MutableMapping like update or clear and they just work off of the handful of methods we define. These are pretty handy when copying or removing data.

I believe that MutableMapping simply predates the development of simplekv, I am not against adding them. Some, like len, might be hard to implement for all backends though and I am not sure if it’s okay to just raise something like NotImplemented there.