Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Missing keys after upgrade to 0.3.0

See original GitHub issue

The new version of s3fs is not retrieving all bucket keys. E.g.:

In [16]: s3 = s3fs.S3FileSystem(anon=True, client_kwargs=dict(region_name='eu-west-2'))

In [17]: store = s3fs.S3Map(root='zarr-demo/store', s3=s3, check=False) 

In [18]: list(store.keys())                                                           
Out[18]: 
['.zattrs',
 '.zgroup',
 'foo/.zattrs',
 'foo/.zgroup',
 'foo/bar/.zattrs',
 'foo/bar/.zgroup',
 'spam/.zgroup',
 'spam/eggs',
 'spam/eggs/.zarray',
 'spam/eggs/0.0',
 'spam/eggs/0.1',
 'spam/eggs/1.0',
 'spam/eggs/1.1']

In [19]: 'foo/bar/baz/.zarray' in store                                               
Out[19]: True

In [20]: 'foo/bar/baz/.zarray' in list(store.keys())                                  
Out[20]: False

xref https://github.com/zarr-developers/zarr-python/pull/456

Issue Analytics

State:
Created 4 years ago
Comments:6 (5 by maintainers)

Top GitHub Comments

1reaction

martindurantcommented, Jul 18, 2019

Released.

0reactions

jrbourbeaucommented, Jul 18, 2019

Closing as the new fsspec release has fixed the originally posted issue:

In [1]: import s3fs                                                                                                   

In [2]: s3 = s3fs.S3FileSystem(anon=True, client_kwargs=dict(region_name='eu-west-2'))                                

In [3]: store = s3fs.S3Map(root='zarr-demo/store', s3=s3, check=False)                                                

In [4]: 'foo/bar/baz/.zarray' in store                                                                                
Out[4]: True

In [5]: 'foo/bar/baz/.zarray' in list(store.keys())                                                                   
Out[5]: True

Thanks all!