question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Bug: Using GCS or S3, FSStore doesn't pass `check` and `create` kwargs to FSSPEC's get_mapper()

See original GitHub issue

I think this is a bug, if everyone agrees I can fix it and do a PR.

Minimal, reproducible code sample, a copy-pastable example if possible

import zarr

# Works fine with this as expected
url = '~/my_root.zarr'

# Doesn't work with:
url = r'gcs://my_bucket/my_root.zarr'
# or
url = r's3://my_bucket/my_root.zarr'

storage_options = {'key': my_key, 'secret': my_secret, 'token': my_token}  # Example for s3
store = zarr.storage.FSStore(url, check=True, create=True, **storage_options)
zarr.open(store, 'w')

Problem description

The above code fails because the FSMap created and used by FSStore did not receive the check and create kwargs (Traceback at the bottom of the post).

The check parameter is used to verify write access to remote stores. The create function creates the root object/directory if it doesn’t exist, so it can check for write access

The fsspec.mapping.get_mapper used in zarr.storage.FSStore expects explicit kwargs for check and create to send FSMap and passes storage_options aka. other kwargs to ONLY to url_to_fs (here). The FSMap call here expects the check and create kwargs, which explicitly have to be passed to get_mapper.

The issue can be circumvented by dropping the write access checking, but that may not be always the case user wants.

The ideal fix is to make zarr.storage.FSStore to expect check and create kwargs; and default to False, just like FSMap.

Version and installation information

Please provide the following:

  • zarr.__version__ 2.8.1
  • numcodecs.__version__ 0.7.3
  • Python 3.9.5
  • Linux x86_64
  • installed via conda

Traceback:

Traceback (most recent call last):
  File "/home/my_user/develop/my_library/benchmarks/test.py", line 5, in <module>
    store = zarr.storage.FSStore(url, check=True, create=True, **storage_options)
  File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/zarr/storage.py", line 1052, in __init__
    self.map = fsspec.get_mapper(url, **storage_options)
  File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/fsspec/mapping.py", line 224, in get_mapper
    return FSMap(root, fs, check, create, missing_exceptions=missing_exceptions)
  File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/fsspec/mapping.py", line 52, in __init__
    raise ValueError(
ValueError: Path my_bucket/my_root.zarr does not exist. Create  with the ``create=True`` keyword

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:7 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
tasansalcommented, Aug 23, 2021

@rabernat

Let me verify that it still works as expected on a LocalFileSystem when write-permission is not enabled.

If it works as expected, I agree, we should probably put a fix in fsspec.

0reactions
tasansalcommented, Dec 2, 2022

I did a PR to fix this in #814, but it was closed to be handled in fsspec. I haven’t done a PR there, has anyone else fixed this?

Read more comments on GitHub >

github_iconTop Results From Across the Web

How To Use *args and **kwargs in Python 3 - DigitalOcean
In this tutorial, we will cover the syntax of working with *args and **kwargs as parameters within functions to pass a variable number...
Read more >
why doesn't ** unpack kwargs in function calls? - Stack Overflow
To access the target variable in the function, you can either use: def test(target='<default-value>', *args, **kwargs): print target.
Read more >
Python args and kwargs: Demystified
In this step-by-step tutorial, you'll learn how to use args and kwargs in Python to add more flexibility to your functions. You'll also...
Read more >
Understanding *args and *kwargs arguments in Python
These are very simple functions and there are no arguments passed to these functions yet. Now, let us enhance the above greet() function,...
Read more >
A Guide to Args, Kwargs, Packing and Unpacking in Python
When writing functions, *args and **kwargs are often passed directly into a function definition. This function can handle any number of args and ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found