Unable to retrieve all variables with cfgrib

Hello,

I’m trying to handle a Grib2 file from which I can extract all the data using directly the command ‘grib_get_data -s stepUnits=0 out.grb’.

So it seems I don’t need all stepUnits.

In the other hand, when using cfgrib it is impossible to read it fully. Here is the command I tried first :

import cfgrib
cfgrib.open_datasets('out.grb')

And here is the result :

<ipython-input-36-ccea25a1a3be> in <module>
      1 import cfgrib
----> 2 cfgrib.open_datasets('out.grb')

~/.local/lib/python3.6/site-packages/cfgrib/xarray_store.py in open_datasets(path, backend_kwargs, **kwargs)
    100     backend_kwargs = backend_kwargs.copy()
    101     backend_kwargs["squeeze"] = False
--> 102     datasets = open_variable_datasets(path, backend_kwargs=backend_kwargs, **kwargs)
    103 
    104     type_of_level_datasets = {}  # type: T.Dict[str, T.List[xr.Dataset]]

~/.local/lib/python3.6/site-packages/cfgrib/xarray_store.py in open_variable_datasets(path, backend_kwargs, **kwargs)
     88         bk["filter_by_keys"] = backend_kwargs.get("filter_by_keys", {}).copy()
     89         bk["filter_by_keys"]["paramId"] = param_id
---> 90         datasets.extend(raw_open_datasets(path, bk, **kwargs))
     91     return datasets
     92 

~/.local/lib/python3.6/site-packages/cfgrib/xarray_store.py in raw_open_datasets(path, backend_kwargs, **kwargs)
     63     datasets = []
     64     try:
---> 65         datasets.append(open_dataset(path, backend_kwargs=backend_kwargs, **kwargs))
     66     except DatasetBuildError as ex:
     67         fbks.extend(ex.args[2])

~/.local/lib/python3.6/site-packages/cfgrib/xarray_store.py in open_dataset(path, **kwargs)
     36         raise ValueError("only engine=='cfgrib' is supported")
     37     kwargs["engine"] = "cfgrib"
---> 38     return xr.open_dataset(path, **kwargs)  # type: ignore
     39 
     40 

~/.local/lib/python3.6/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta)
    570 
    571         opener = _get_backend_cls(engine)
--> 572         store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
    573 
    574     with close_on_error(store):

~/.local/lib/python3.6/site-packages/xarray/backends/cfgrib_.py in __init__(self, filename, lock, **backend_kwargs)
     43             lock = ECCODES_LOCK
     44         self.lock = ensure_lock(lock)
---> 45         self.ds = cfgrib.open_file(filename, **backend_kwargs)
     46 
     47     def open_store_variable(self, name, var):

~/.local/lib/python3.6/site-packages/cfgrib/dataset.py in open_file(path, grib_errors, indexpath, filter_by_keys, read_keys, time_dims, extra_coords, **kwargs)
    718     return Dataset(
    719         *build_dataset_components(
--> 720             index, read_keys=read_keys, time_dims=time_dims, extra_coords=extra_coords, **kwargs
    721         )
    722     )

~/.local/lib/python3.6/site-packages/cfgrib/dataset.py in build_dataset_components(index, errors, encode_cf, squeeze, log, read_keys, time_dims, extra_coords)
    639                 read_keys=read_keys,
    640                 time_dims=time_dims,
--> 641                 extra_coords=extra_coords,
    642             )
    643         except DatasetBuildError as ex:

~/.local/lib/python3.6/site-packages/cfgrib/dataset.py in build_variable_components(index, encode_cf, filter_by_keys, log, errors, squeeze, read_keys, time_dims, extra_coords)
    499         }
    500         attributes.update(COORD_ATTRS.get(coord_name, {}).copy())
--> 501         data = np.array(sorted(values, reverse=attributes.get("stored_direction") == "decreasing"))
    502         dimensions = (coord_name,)  # type: T.Tuple[str, ...]
    503         if squeeze and len(values) == 1:

TypeError: '<' not supported between instances of 'float' and 'str'

After investigating, I tried to filter like that :

cfgrib.open_datasets('out.grb', backend_kwargs={'filter_by_keys': {'stepUnits': 0}})

It works, BUT I do not retrieve all data. Then I tried with :

cfgrib.open_datasets('out.grb', backend_kwargs={'filter_by_keys': {'stepUnits': 1}})

And it fails with the same above error.

So my question is why I would need to get stepUnits=1 (which seems to generate en error) with cfgrib while with eccodes binary grib_get_data there is no problem ?

And how could I do to use cfgrib without making distinction with stepUnits please?

Thanks for your help, Christophe

[shahramn]

How the templates are chosen by the framework. Would you know ?

Any idea @shahramn, @iainrussell ?

Thanks again, Christophe

@ChristopheLRTE Basically the ecCodes library has a set of definition files (which are ASCII) that define the keys of the GRIB message and some rules. Depending on the content of the message (say a given octet) it loads a given file. So for example if we have a GRIB edition 2 message, the files in the directory “definitions/grib2” are loaded. Also for the various sections there are different files defining the keys for that section, so for GRIB2 section 4 (Product Definition Section) we load either the instantaneous or interval-based files. I am simplifying things a lot here to keep it short. Hope this helps

[iainrussell]

Hi @ChristopheLRTE,

I just wanted to comment on your modification to the code in cfgrib - I was indeed thinking that this could be passed as a backend argument. The thing that will make it complicated is that we are just finalising a big refactoring of the code that makes the GRIB access more abstract, and in the new code it may not be as clear how to get this working nicely in call cases. So in any case, we will not make any changes to the code until the refactoring is done, because merging will be a nightmare! If your fix works for you (and indeed it’s a very reasonable fix just to get that data working) then stick with that, or the solution @shahramn mentioned, whichever you prefer, and we hope to get a proper solution in place in due course.

Cheers, Iain