If opening with xarray open_mfdataset and parallel=True it will fail unless you have previously opened it with parallel=False

Minimal repro:

import xarray as xr
ds = xr.open_mfdataset('gfs.0p25.201511*00.f0*.grib2', engine='cfgrib', combine='nested', concat_dim=['step'], parallel=True, chunks=24, backend_kwargs={'filter_by_keys': {'typeOfLevel': 'surface'}, 'indexpath': ''})

Expected result: returns xarray Actual result:

ECCODES ERROR   :  grib_handle_create: cannot create handle, no definitions found
ecCodes assertion failed: `h' in /home/conda/feedstock_root/build_artifacts/eccodes_1570714279314/work/src/grib_query.c:529

Note if in the same session/kernel you have previously opened with parallel=False the above will pass. The repro needs to happen in a new session. This was executed on a local dask cluster.

[alexamici]

I confirm this bug report with a different dataset and different error messages.

With parallel=False open_mfdataset always work:

>>> import cfgrib
>>> import xarray as xr
>>> print(xr.__version__, cfgrib.__version__)
0.13.0 0.9.7.4.dev0
>>> ds = xr.open_mfdataset('step*.grib', engine='cfgrib', concat_dim=['step'], combine='nested', parallel=False)
>>> ds
<xarray.Dataset>
Dimensions:     (latitude: 1801, longitude: 3600, step: 3)
Coordinates:
    time        datetime64[ns] 2019-04-01
    number      int64 0
    surface     int64 0
  * latitude    (latitude) float64 90.0 89.9 89.8 89.7 ... -89.8 -89.9 -90.0
  * longitude   (longitude) float64 0.0 0.1 0.2 0.3 ... 359.6 359.7 359.8 359.9
  * step        (step) timedelta64[ns] 01:00:00 02:00:00 03:00:00
    valid_time  (step) datetime64[ns] 2019-04-01T01:00:00 ... 2019-04-01T03:00:00
Data variables:
    t2m         (step, latitude, longitude) float32 dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>
Attributes:
    GRIB_edition:            1
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2019-11-11T19:19:05 GRIB to CDM+CF via cfgrib-0....

Restarting the kernel and running with parallel=True always crashes python inside ecCodes but it returns a few different error messages. I observed at leat:

ECCODES ERROR   :  Unable to find boot.def. Context path=/Users/amici/.conda/envs/ECM/share/eccodes/definitions

Possible causes:
- The software is not correctly installed
- The environment variable ECCODES_DEFINITION_PATH is defined but incorrect

ecCodes assertion failed: `0' in /usr/local/miniconda/conda-bld/eccodes_1566402639979/work/src/grib_context.c:205

ECCODES ERROR   :  grib_handle_create: cannot create handle, no definitions found
ecCodes assertion failed: `h' in /usr/local/miniconda/conda-bld/eccodes_1566402639979/work/src/grib_query.c:458
ECCODES ERROR   :  grib_parser: syntax error at line 34 of /Users/amici/.conda/envs/ECM/share/eccodes/definitions/boot.def
ECCODES ERROR   :  ecCodes Version: 2.13.1

and

ECCODES ERROR   :  ecCodes Version: 2.13.1
ecCodes Version:       2.13.1
Definition files path: /Users/amici/.conda/envs/ECM/share/eccodes/definitions
ECCODES ERROR   :  grib_parser_include: Could not resolve 'ECCODES_USE_' (included in /Users/amici/.conda/envs/ECM/share/eccodes/definitions/boot.def)
ecCodes assertion failed: `0' in /usr/local/miniconda/conda-bld/eccodes_1566402639979/work/src/grib_context.c:205

It looks like a locking/threading problem, @shahramn do you have any hint?

[iainrussell]

Hi @guidocioni , I’m not sure if you’re on macos or Linux, but we’ve managed to update the conda version. Could you do the following: conda search eccodes -c conda-forge and if you see a 2.18.0 version with _1 at the end, install that version please. It takes the conda servers a little while to update their indexes, but it’s appeared now at least on macos.