glob is inefficient as it's iterating a dir that already scanned

I tried glob method and found it is too slow when there’re millions of files in the directory.

turns out that the glob method will first call list_objects_v2 api first, get all files (every single file including folders and files), identify all files to see if they are folders. and then scan the folders.

The algorighm is corret in traditional fs, while inefficient in s3, s3 will return every object when requesting list_objects_v2 api, iterating subfolders are unneccessary.

Is that possible to fix it in s3path or it can only be fixed in pathlib ?

[four43]

Oh man, python why!? That’s a mega bummer!

[liormizr]

@four43 yes, you are right One of the optimizations that I want to do is remove this list creation in the s3 implementation