Zarr dataset on requester pays bucket only opening if I list the files first!?

@martindurant , this is pretty bizarre: I have a notebook that used to open a zarr dataset (requester pays bucket), but after upgrading my environment, opening the dataset no longer works unless I list the files first!?

So this doesn’t work:

xr.open_zarr(fsspec.get_mapper('s3://pangeo-data-uswest2/esip/adcirc/adcirc_01d', 
                                    anon=False, requester_pays=True))

but this does:

fs = fsspec.filesystem('s3',anon=False, requester_pays=True)
fs.ls('s3://pangeo-data-uswest2/esip/adcirc/adcirc_01d/')
xr.open_zarr(fsspec.get_mapper('s3://pangeo-data-uswest2/esip/adcirc/adcirc_01d', 
                                    anon=False, requester_pays=True)

Here’s the notebook: https://nbviewer.jupyter.org/github/rsignell-usgs/hurricane-ike-water-levels/blob/master/fsspec_strangeness.ipynb

I don’t have a minimal test, but it should be reproducible on binder using this link: and running the “fsspec_strangeness” notebook.

This is using fsspec=0.8.0 and xarray from master. I guess this could also be an xarray issue?
I have no idea. 😕 But I bet you do. 😄

[martindurant]

See https://github.com/aio-libs/aiobotocore/issues/825

You can try setting the environment variable AWS_DEFAULT_REGION as a workaround, or adding client_kwargs={"default_region": ...} into get_mapper

[martindurant]

Yep - this is something that aiobotocore needs to wort out for us - it would be good to chime in on that issue to give your specifics, to see if it can help them correct the problem. https://github.com/aio-libs/aiobotocore/issues/825