Missing `v` component in nam.t06z.awip3d00.tm00.grib2 after `filter_by_keys`

I think I may have found a bug or at least something I do not understand in the implementation of the filter_by_keys argument.

Here is some output I receive when trying to open one of those NAM grib files:

>>> xr.open_dataset('nam.t06z.awip3d00.tm00.grib2',
                    engine='cfgrib',
                    backend_kwargs={
                        'filter_by_keys': {'typeOfLevel': 'isobaricInhPa'},
                        'errors': 'ignore'
                    })

skipping variable: paramId==3041 shortName='absv'
Traceback (most recent call last):
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 429, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 407, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='isobaricInhPa' value=39 new_value=7
skipping variable: paramId==1 shortName='strf'
Traceback (most recent call last):
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 430, in build_dataset_components
    dict_merge(variables, vars)
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 407, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='isobaricInhPa' value=Variable(dimensions=('isobaricInhPa',), data=array([1000,  975,  950,  925,  900,  875,  850,  825,  800,  775,  750,
        725,  700,  675,  650,  625,  600,  575,  550,  525,  500,  475,
        450,  425,  400,  375,  350,  325,  300,  275,  250,  225,  200,
        175,  150,  125,  100,   75,   50])) new_value=Variable(dimensions=(), data=250)
skipping variable: paramId==3017 shortName='dpt'
Traceback (most recent call last):
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 429, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 407, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='isobaricInhPa' value=39 new_value=6
skipping variable: paramId==260022 shortName='mconv'
Traceback (most recent call last):
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 429, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/Users/bbonenfant/PycharmProjects/wrf2wxl/venv/lib/python3.6/site-packages/cfgrib/dataset.py", line 407, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='isobaricInhPa' value=39 new_value=2

<xarray.Dataset>
Dimensions:        (isobaricInhPa: 39, x: 185, y: 129)
Coordinates:
    time           datetime64[ns] ...
    step           timedelta64[ns] ...
  * isobaricInhPa  (isobaricInhPa) int64 1000 975 950 925 900 ... 125 100 75 50
    latitude       (y, x) float64 ...
    longitude      (y, x) float64 ...
    valid_time     datetime64[ns] ...
Dimensions without coordinates: x, y
Data variables:
    gh             (isobaricInhPa, y, x) float32 ...
    t              (isobaricInhPa, y, x) float32 ...
    r              (isobaricInhPa, y, x) float32 ...
    q              (isobaricInhPa, y, x) float32 ...
    w              (isobaricInhPa, y, x) float32 ...
    u              (isobaricInhPa, y, x) float32 ...
    tke            (isobaricInhPa, y, x) float32 ...
    clwmr          (isobaricInhPa, y, x) float32 ...
    cice           (isobaricInhPa, y, x) float32 ...
    snmr           (isobaricInhPa, y, x) float32 ...
    strf           (y, x) float32 ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             kwbc
    GRIB_centreDescription:  US National Weather Service - NCEP 
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             US National Weather Service - NCEP 
    history:                 GRIB to CDM+CF via cfgrib-0.9.5.1/ecCodes-2.8.0 ...

You can see that it successfully returns a Dataset, but looking at the variables it returns, there is the U component of the wind but not the V component of the wind. I’m not sure why this is the case, since I’ve inspected the grib and find nothing apparent wrong with the v-winds. I’ve additionally tried this on other NAM gribs with similar results (even in your comment above on Sept. 30 this was the case).

I am unsure if this is an error on my part or if there is a way around this. Thank you.

_Originally posted by @bbonenfant in https://github.com/ecmwf/cfgrib/issues/2#issuecomment-451557811_

[alexamici]

@shahramn identified that the nam GRIB files use MULTI-FIELD messages, a rare feature, that ecCodes can handle just fine but only after being requested to do it by calling codes_grib_multi_support_on.

I’ll need a bit of time to implement the feature correctly, but it is basically solved.

[bbonenfant]

Hello again, I’m just now revisiting this, an I’m still experiencing the same bug where the u winds are present but the v winds are not. This is using cfgrib version 0.9.7.2. Below is the output from opening a NAM awphys** GRIB2 file which can be within this database. I’ve made sure to check all the array.Dataset objects output by the cfgrib.open_datasets method and none contain v winds.

<xarray.Dataset>
Dimensions:        (isobaricInhPa: 39, x: 614, y: 428)
Coordinates:
    time           datetime64[ns] 2019-10-30
    step           timedelta64[ns] 00:00:00
  * isobaricInhPa  (isobaricInhPa) int64 1000 975 950 925 900 ... 125 100 75 50
    latitude       (y, x) float64 12.19 12.22 12.25 12.28 ... 57.39 57.36 57.33
    longitude      (y, x) float64 226.5 226.6 226.8 226.9 ... 310.3 310.4 310.6
    valid_time     datetime64[ns] 2019-10-30
Dimensions without coordinates: x, y
Data variables:
    t              (isobaricInhPa, y, x) float32 ...
    u              (isobaricInhPa, y, x) float32 ...
    w              (isobaricInhPa, y, x) float32 ...
    gh             (isobaricInhPa, y, x) float32 ...
    r              (isobaricInhPa, y, x) float32 ...
    tke            (isobaricInhPa, y, x) float32 ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             kwbc
    GRIB_centreDescription:  US National Weather Service - NCEP
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             US National Weather Service - NCEP