Bug: Using GCS or S3, FSStore doesn't pass `check` and `create` kwargs to FSSPEC's get_mapper()
See original GitHub issueI think this is a bug, if everyone agrees I can fix it and do a PR.
Minimal, reproducible code sample, a copy-pastable example if possible
import zarr
# Works fine with this as expected
url = '~/my_root.zarr'
# Doesn't work with:
url = r'gcs://my_bucket/my_root.zarr'
# or
url = r's3://my_bucket/my_root.zarr'
storage_options = {'key': my_key, 'secret': my_secret, 'token': my_token} # Example for s3
store = zarr.storage.FSStore(url, check=True, create=True, **storage_options)
zarr.open(store, 'w')
Problem description
The above code fails because the FSMap created and used by FSStore did not receive the check and create kwargs (Traceback at the bottom of the post).
The check parameter is used to verify write access to remote stores.
The create function creates the root object/directory if it doesn’t exist, so it can check for write access
The fsspec.mapping.get_mapper used in zarr.storage.FSStore expects explicit kwargs for check and create to send FSMap and passes storage_options aka. other kwargs to ONLY to url_to_fs (here). The FSMap call here expects the check and create kwargs, which explicitly have to be passed to get_mapper.
The issue can be circumvented by dropping the write access checking, but that may not be always the case user wants.
The ideal fix is to make zarr.storage.FSStore to expect check and create kwargs; and default to False, just like FSMap.
Version and installation information
Please provide the following:
zarr.__version__2.8.1numcodecs.__version__0.7.3- Python 3.9.5
- Linux x86_64
- installed via conda
Traceback:
Traceback (most recent call last):
File "/home/my_user/develop/my_library/benchmarks/test.py", line 5, in <module>
store = zarr.storage.FSStore(url, check=True, create=True, **storage_options)
File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/zarr/storage.py", line 1052, in __init__
self.map = fsspec.get_mapper(url, **storage_options)
File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/fsspec/mapping.py", line 224, in get_mapper
return FSMap(root, fs, check, create, missing_exceptions=missing_exceptions)
File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/fsspec/mapping.py", line 52, in __init__
raise ValueError(
ValueError: Path my_bucket/my_root.zarr does not exist. Create with the ``create=True`` keyword
Issue Analytics
- State:
- Created 2 years ago
- Comments:7 (7 by maintainers)

Top Related StackOverflow Question
@rabernat
Let me verify that it still works as expected on a
LocalFileSystemwhen write-permission is not enabled.If it works as expected, I agree, we should probably put a fix in
fsspec.I did a PR to fix this in #814, but it was closed to be handled in
fsspec. I haven’t done a PR there, has anyone else fixed this?