Add support for opening a dataset memory without writing it to disc

Is it possible to read a Dataset from a Zipfile without writing to disc?

This works like a charm:

with gzip.open('some_file', "rb") as f_in:
     with open("tmp", "wb") as f_out:
           shutil.copyfileobj(f_in, f_out)
grib_files = cfgrib.open_datasets("tmp", backend_kwargs={"indexpath": ""})

I want something like that:

with gzip.GzipFile('some_file', 'rb') as zipfile:
    bytes_content = zipfile.read()
grib_files = cfgrib.open_datasets(zipfile, backend_kwargs={"indexpath": ""})

But i am getting the following error:

UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xff in position 4: invalid start byte

Any ideas?

Kind Regards.

[juhi24]

I’m also looking forward for this feature. In the best case scenario, cfgrib.read_datasets would accept a file-like object. In addition to in-memory objects, this could enable stuff like reading files directly from S3 object storage using boto3.

[Plantain]

This is something that would be useful to me as well - the loading from memory part rather than the zip part. Currently we download GRIB2 files to memory with python, then write them out to disk solely in order to be able to open them with cfgrib. Perhaps this bug should be renamed as I’m not sure the gzip part is relevant. If you give me some pointers I will take a look at getting this implemented.