Should the zarr backend support NCZarr conventions?

What is your issue?

As part of the CZI EOSS4 grant, at B-Open we are keen on improving xarray/zarr cross-community conventions. It looks like xarray’s "zarr" backend does not support Unidata NCZarr conventions while NetCDF-c>=4.8.1 supports xarray conventions.

Currently, it is possible to open a NCZarr data source using the "netcdf4" backend only. Here is an example:

import xarray as xr
import numpy as np

ds = xr.Dataset(
    {
        "a": (("y", "x"), np.random.rand(6).reshape(2, 3)),
        "b": (("y", "x"), np.random.rand(6).reshape(2, 3)),
    },
    coords={"y": [0, 1], "x": [10, 20, 30]},
)
ds.to_netcdf("file://test.nczarr#mode=nczarr")

ds_from_nczarr = xr.open_dataset("file://test.nczarr#mode=nczarr", engine="netcdf4")
xr.testing.assert_identical(ds, ds_from_nczarr)

xr.open_dataset("test.nczarr", engine="zarr")
# KeyError: 'Zarr object is missing the attribute `_ARRAY_DIMENSIONS`, which is required for xarray to determine variable dimensions.'

Would the community benefit from a "zarr" backend compatible with NCZarr as well? @rsignell-usgs and @rabernat suggested to discuss this within the xarray community, but I don’t think a dedicated issue has been opened yet.

cc: @alexamici @aurghs @joshmoore

[shoyer]

@DennisHeimbigner I think it would be great to standardize NCZarr as a super-set of the “Xarray-Zarr” standard! I think Xarray should indeed be able to read such files. If you want to read a sub-group, you can read the sub-group in a separate call to xarray.open_zarr().

@rabernat I would not be opposed to adding support inside Xarray for reading NCZarr data, specifically to understand NCZarr’s encoding of dimension names when using Zarr-Python. This wouldn’t give 100% compatibility with NCZarr, but it would be very close (maybe just with incorrect dtypes for attributes) with a minimal amount of work. I don’t think it would be a big deal to look for .nczvar files.

[shoyer]

@wankoelias could you kindly open a new issue for writing GDAL ZARR?