Requirements of store data

Raising this issue to get an idea of what our requirements are of stores and what can be placed in them.

For instance in many cases we require Arrays to have an object_codec to allow storing object types and many stores would have difficulty with this data without explicit conversion to some sort of bytes-like object; however, we appear to be placing objects in a store as a test. Also we seem to expect stores to be easily comparable; however, this doesn’t work if the store has NumPy ndarrays in it. ( https://github.com/zarr-developers/zarr/issues/348 )

Should we set some explicit requirements about what stores require? If so, what would those requirements be? Also how would we enforce them?

[jakirkham]

In PR ( https://github.com/zarr-developers/zarr-python/pull/789 ) we added a BaseStore class, which addresses some of these basic needs of Stores

Subsequent discussion around the v3 spec and storing standardized data from libraries handles other concerns raised here

Were there any other things still needing to be addressed here?

cc @joshmoore @grlee77

[alimanfoo]

Thanks Ryan, good points. We certainly could be more explicit about the set of operations that a storage system must support, and make sure we include everything (e.g., listing all keys). We could also state the optional operations, which are not strictly necessary but allow for some optimisations or additional features, like being able to list all the keys that are children of some hierarchy path (the listdir() method in Python implementations).

We could do this in a language-independent way but still make it clear and concrete how this corresponds to specific operations supported by a file system or a cloud object service or whatever.

I think we could also do this as an update to the format spec, without requiring a new spec version, as these would be clarifications of the existing spec.

On Thu, 10 Jan 2019, 10:55 Ryan Abernathey <[e-mail] wrote:

Thanks for the clarification. I see how this thread is specific to python implementations.

I guess I worry that the spec is too vague with regards to the implementation of the key value store, and the methods that can be used to query it:

A Zarr array can be stored in any storage system that provides a key/value interface, where a key is an ASCII string and a value is an arbitrary sequence of bytes, and the supported operations are read (get the sequence of bytes associated with a given key), write (set the sequence of bytes associated with a given key) and delete (remove a key/value pair).

In terms of operations, “Read”, “write”, and “delete” doesn’t seem like enumeration of operations a store must support. When implementing a store, you also need at least some form of “list” operation; otherwise zarr can’t discover what is in the store. (The exception is consolidated metadata stores.) In fact, you have to implement a MutableMapping https://docs.python.org/3/library/collections.abc.html, which has five methods: getitem, setitem, delitem, iter, and len.)

More generally, how do we ensure that DirectoryStore, ZipStore, or any of the myriad cloud stores that have been developed can truly be read from different implementations of zarr? I wonder if it would be worth explicitly defining a spec for certain commonly used stores that gives more detail about the implementation choices that have already been made in the zarr python code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zarr-developers/zarr/issues/349#issuecomment-453054878, or mute the thread https://github.com/notifications/unsubscribe-auth/AAq8QldD8wCmVJiYP8yEB07T3b7V7C8Zks5vBxw5gaJpZM4Y9kXZ .