* not working in glob

glob raises an exception when I try to use a simple asterisk on the latest version :

In [5]: storage.glob(‘bucketx/a’) Out[5]: [‘bucketx/a/b’]

In [6]: storage.walk(‘bucketx/’) Out[6]: [‘bucketx/a/b/c/d/sh’]

In [7]: storage.glob(‘bucketx/*/b’)

ClientError Traceback (most recent call last) … ~/.virtualenvs/cassie-backup/lib/python3.6/site-packages/s3fs/core.py in info(self, path, **kwargs) 411 except (ClientError, ParamValidationError) as e: 412 logger.debug(“Failed to head path %s”, path, exc_info=True) –> 413 raise FileNotFoundError(path) 414 415 _metadata_cache = {}

FileNotFoundError: bucketx/*/b

[martindurant]

Would you like to have a go? I recommend by starting with S3FileSystem._walk, adding an optional parameter dictionaries=False, and changing the last line of the function appropriately. The same parameter should then appear in walk, and so when glob calls it, you get both files and directories.

[cscetbon]

I see no blocker to implement it. If we want to work with s3 as we do with files and directories it makes sense to me. I need to find paths that contain a certain folder without having to know exactly what’s stored in deeper levels. A workaround would be to use walk and then apply a regex on it what is done internally, something which can be done with the regular glob.glob.

I don’t think it would be hard to implement. We could also limit the walk when the end of the pattern is reached (avoiding to walk through deeper level when we’re only interested in folders)