Nested cloud storage

Following on from gh-177, is it possible to also use the NestedDirectoryStorage with s3fs? I’m currently looking at code roughly equivalent to:

store = s3fs.S3Map(
    root='example.zarr',
    s3=s3, check=True, create=True
)

ds.to_zarr(store=store, mode='w', group=name)

and am missing if there’s a parameter and/or a delegation pattern that I could use to go from . to / notation.

see also:

• http://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_zarr.html
• https://github.com/pydata/xarray/blob/6963164dbdc55f86f34b741073838f083a68a152/xarray/core/dataset.py#L1248
• https://github.com/pydata/xarray/blob/ee44478ed921ae3afdece9d16dc179a197727e8d/xarray/backends/api.py#L892
• https://github.com/pydata/xarray/blob/2667deb74a30dc3bd88752a3ce5da590cf7ddd48/xarray/backends/zarr.py#L222
• https://github.com/dask/s3fs/blob/master/s3fs/mapping.py#L8

[chrisroat]

I’d like to understand what the roadmap is here.

I came across this while assuming everything worked with cloud buckets, but painted myself into a corner in using N5Store (for neuroglancer viz), which does not. My first inclination was that the storage.py layer could be adjusted to replace os calls with fsspec-type code. But in reading this and #410, it looks like something bigger is in the works.

How can I help?

[joshmoore]

With the dimension_separator epic complete (https://github.com/zarr-developers/zarr-python/issues/707), nested cloud storage is now possible with the FSStore implementation. The default value has not been changed, but can be passed on creation. The setting will be stored in the .zarray metadata.