accessing three months of data using fsspec

The way noaa-goes16/ABI-L2 data is stored in s3 has limited access for one day of data for (# of files around 288) using this command:

fs = fsspec.filesystem('s3', anon=True)

urls = ['s3://' + f for f in fs.glob("s3://noaa-goes16/ABI-L2-ACMC/2022/001/*/*.nc")]

or one hour ((# of files around 12) urls = ['s3://' + f for f in fs.glob("s3://noaa-goes16/ABI-L2-ACMC/2022/001/04/*.nc")]

Is there any way to work around this to have access to around three months of data(100 consecutive days)

[lsterzinger]

Check the last few files in the list, in your original code it is just the 1st day of data (2022/001/*) that is being repeated 5 times whereas with the new loop it should get all days from 2022/001 - 2022/004 (since in the first iteration i=0, which does not return any files since there is no 0th day).

[lsterzinger]

The code you shared here is just taking all the file paths in the the first day of 2022 (2022/001/*/*.nc) and appending them to a list 5 times. Did you mean something like this?

urls1=[]
for i in range (5):
    urls = ['s3://' + f for f in fs.glob(f"s3://noaa-goes16/ABI-L2-CODC/2022/{i:03}/*/*.nc")]
    urls1 = urls1+ urls

This takes i, pads preceding zeros to 3 places, and inserts it into the string.

It’s hard to debug why this isn’t working for you from the information I have currently. Would you be willing to share your code/notebook to see if I can replicate your issue?