Wildcard retrieval of files

I am attempting to access data in our s3 datalake. Since production systems are writing the data I want along with other files that I don’t want, simply using a prefix is insufficient to get the data I need.

Dask allows this sort of wildcarding, for example

import dask.dataframe as dd
dd.read_parquet(f's3://{bucket}/prod-system-*/*/parquet/*.parquet')

Using awswrangler for the above task isn’t viable. I know that boto3 doesn’t allow for wildcard filtering, but surely it must be doable if dask is able to implement that functionality?

[nicholas-miles]

@igorborgest I’m happy to take it on! I’d love any guidance you can give on how to do this though. I’m assuming we want to avoid additional imports/dependencies

[ghost]

Sorry @igorborgest . I am unassigning myself from this because I got a heavy task today from my team. Therefore, I wont be able to contribute for next 3-4 weeks. Please re-assign. I will pick up a new task once I am free. I apologize again for inconvenience caused.