Add support to open multiple GRIB files as a single Stream / Dataset

At low level we use an explicit file path and file offset in several places.

Note that xr.open_mfdataset handles opening and merging of multiple files without any additional support from the low-level driver so this feature is low priority.

[ShaneMill1]

Hello, I have a quick question regarding this topic. I notice that cfgrib has the following ability:

cfgrib also provides a function that automate the selection of appropriate filter_by_keys and returns a list of all valid xarray.Dataset’s in the GRIB file using the cfgrib.open_datasets().

I wanted to ask if this is only for a single grib file, or if it is possible to supply a path that will create the datasets similar to xarray.open_mfdataset(). I’m looking for the ability to automate the selection of filter_by_keys with the ability to open multiple files to create the datasets.

Thanks!

EDIT: it appears that cfgrib.open_datasets() only handles one grib file at a time. however, if you have a directory of grib files, you can “cat” these files into one grib file and then read with cfgrib.open_datasets()

[alexamici]

@aolt the theory was that everything was lazy already… but in practice I noticed yesterday a really dumb bug that was loading the whole dataset into memory unconditionally at open 🤦‍♂️

The bug is fixed in version 0.9.4, please upgrade and try again.

I’m currently running a mean on 320Gb of GRIB files on 10 dask.distributed nodes, so I’m confident it’s working now 😃