decScale is ignore and is doesn't appear to be available

Look at the attached file. If I use ‘wgrib -V’ and look at a single record info, it shows say:

rec 1460:7671255:date 1977123118 PRES kpds5=1 kpds6=102 kpds7=0 levels=(0,0) grid=220 MSL anl:
  PRES=Pressure [Pa]
  timerange 0 P1 0 P2 0 TimeU 1  nx 63 ny 63 GDS grid 5 num_in_ave 0 missing 0
  center 58 subcenter 0 process 58 Table 1 scan: WE:SN winds(N/S)
  polar stereo: Lat1 -19.115000 Long1 -125.000000 Orient -80.000000
     north pole (63 x 63) Dx 381000 Dy 381000 scan 64 mode 0
  min/max data 97781.2 105112  num bits 11  BDS_Ref 977.812  DecScale -2 BinScale -4

DecScale -2 means to divide the data by 100. If you read using xarray and cfgrib:

grib_file = xr.open_dataset('g1977', engine='cfgrib')
grib_pres = grib_file.pres[1459, :, :]
grib_data = grib_pres.values
grib_data
array([[101387.5 , 101556.25, 101743.75, ..., 101418.75, 101450.  ,
        101481.25],
       [101356.25, 101506.25, 101637.5 , ..., 101387.5 , 101431.25,
        101475.  ],
       [101375.  , 101468.75, 101556.25, ..., 101362.5 , 101425.  ,
        101475.  ],
       ...,
       [101531.25, 101425.  , 101350.  , ..., 101400.  , 101412.5 ,
        101393.75],
       [101556.25, 101456.25, 101400.  , ..., 101475.  , 101512.5 ,
        101537.5 ],
       [101518.75, 101443.75, 101412.5 , ..., 101575.  , 101637.5 ,
        101662.5 ]], dtype=float32)

The data is not scaled as it should be, nor does the DecScale attribute appear to be available in any of the information read in.

g1977.zip

[shahramn]

Looking at the messages of the file “g1977”, we see the following meta-data:

table2Version = 1

indicatorOfParameter = 1

So this is a standard WMO parameter which for GRIB edition 1 translate to:

“Pressure” with units of “Pa”

So the standard says the units of this parameter is “Pa” and that is what ecCodes is reporting.

Looking at the values in the Data section of the 1st message:

min = 97618.8

max = 106619

mean = 101309

---

From: Alessandro Amici [e-mail] Sent: 10 June 2019 10:26:54 To: ecmwf/cfgrib Cc: Shahram Najm; Mention Subject: Re: [ecmwf/cfgrib] decScale is ignore and is doesn’t appear to be available (#88)

Ok, I finally understand the issue. The g1978 file you attached above has either incorrect metadata or its metadata are interpreted incorrectly by ecCodes, as it declares to have values in Pa but clearly contains hPa:

ds = xr.open_dataset(‘g1978’, engine=‘cfgrib’) ds.pres.units ‘Pa’ np.nanmean(ds.pres) 1011.89197

I’m not familiar enough with the GRIB on-disk data structure to know if it is a ecCodes issue or the file is broken, maybe @shahramnhttps://github.com/shahramn can comment.

This is different from the g1977 file where data and metadata are consistent.

So at the moment I’ll do:

1. if this is a bug in ecCodes I’ll move the issue there
2. in any event we need a generic way to access all GRIB keys #89https://github.com/ecmwf/cfgrib/issues/89 so you can fix files with broken metadata

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ecmwf/cfgrib/issues/88?email_source=notifications&email_token=AF4HFUZP2XMKHHSZKZ3YVMTPZYM55A5CNFSM4HVMO2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXJMIVQ#issuecomment-500352086, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AF4HFU2DSTSBSIWJOYNAN2TPZYM55ANCNFSM4HVMO2FA.

[alexamici]

@rmendels if you don’t mind I’ll close this issue as duplicate of #89 that is more specific. Thanks again for reporting it.