Nested storage detection in Zarr V2

see: https://github.com/ome/ngff/issues/29 and https://github.com/bcdev/jzarr/pull/17

In order to better handle Zarr arrays created with NestedDirectoryStorage or FSStore(key_separator="/"), SabineEmbacher and I have been working on a “protocol heuristic” that can be used by V2 implementations to detect nested chunking rather than requiring the user to specify it correctly.

tl;dr: This proposes a new key for .zarray which it would be good to have feedback on.

---

Proposal

When creating a zarr array:

• add to .zarray json: {"dimension_separator": "/"}
• always write a 0-position chunk

When opening an array:

• try to read the separator character from the .zarray json
• if not available, try to find a 0-position chunk
• if not available, at every read action try to find chunks with both standard variants until the situation is clarified. (Standard separator list ["/", "."])

---

Points for discussion:

• The name dimension_separator differs from the code implementation key_separator to reduce confusion about whether every separator in the key name is effected.
• There has been some discussion (community call, gitter) about whether or not “/” could become the default.
• Is an addition to the .zarray metadata sufficiently low impact to be rolled into v2?

[joshmoore]

I consider the dimension_separator saga complete! whew

[joshmoore]

I did not make my suggestions regarding the name in order to enforce them.

No worries, @SabineEmbacher. The suggestions were definitely useful. It’s more just a matter of https://martinfowler.com/bliki/TwoHardThings.html …

Since key_separator leads to confusion and dimensionSeparator/dimension_separator is at least used in some other implementation, I’ve updated the description with dimension_separator.

On the community call last night, there were no objections to moving forward with the .zarray addition, so I’ll open a v2 spec PR now.

I’m a bit more hesitant about defining the heuristic as part of the specification (cf. @jbms comment) I’ll leave this open for a discussion of where first-chunk writing falls on the MAY/SHOULD/MUST spectrum.