FileNotFoundError when using zarr with dask/s3fs version >= 0.5.0

When using a zarr store from an s3 bucket with not storing physically chunks which are in uninitialized state, as described in the specification of zarr (https://zarr.readthedocs.io/en/stable/spec/v2.html#chunks), a FileNotFoundError occurs. This is new since s3fs version >= 0.5.0.

Minimal, reproducible code sample, a copy-pastable example if possible

import s3fs
import zarr

# AWS S3 path
s3_path = 's3://dcs4cop/bc-cmems-spm-1997-2018_1x704x640.zarr' 
# Initilize the S3 file system
s3 = s3fs.S3FileSystem(anon=True)
store = s3fs.S3Map(root=s3_path, s3=s3, check=False)
# Read Zarr file
ds = zarr.open(store=store, mode='r')

# Get some values:
ds.SPM[5]

This results in FileNotFoundError: The specified key does not exist.

Version and installation information

Please provide the following:

• Value of zarr.__version__ 2.5.0
• Value of s3fs.__version__ 0.5.1
• Value of fsspec.__version__ 0.8.0
• Version of Python interpreter 3.7.6
• Operating system Linux
• Zarr was installed “using conda”

Working as expected with the following versions:

• Value of zarr.__version__ 2.5.0
• Value of s3fs.__version__ 0.4.0.
• Value of fsspec.__version__ 0.6.2

[martindurant]

I mean that in my version of the call to zarr, I’m passing a URL, so this gets routed to the new FSStore, rather than a bare S3Map. There is an optional argument to getitems on what to do with errored/missing keys, and from zarr’s point of view, the argument should be “omit”, as FSStore does, but the default is “raise”.

Note that there is a further problem, in that something is turning the paths lower-case, but I think this must be in zarr. Bear with me.

[forman]

@joshmoore , @rabernat, @martindurant thanks so much for your immediate responses, looks like this will be resolved soon. Don’t hesitate to tell us how we can best support you!