Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Very slow opening a GRIB file with 100.000 messages

See original GitHub issue

I have download an ERA5 hourly file of ~68 MB (you can download it here). If I use the following command to open it:

ed1 = xr.open_dataset('../data/nl-era5-1981-1993-t2m.grib', engine = 'cfgrib')

Python is not able to open it, after 10 minutes is still loading and then I kill the process. This happens on Jupyter with Python 3.7.6, xarray 0.15.1 and cfgrib 0.9.8.2.

Is this behaviour expected?

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:13

Top GitHub Comments

1reaction

alexamicicommented, Jan 4, 2021

This is actually fixed in master and now cfgrib to_netcdf converts your sample file in 2 and half minute, much faster than ecCodes grib_to_netcdf. Those are 2 minutes to create the index and 30 seconds to actually perform the conversion.

@matteodefelice thanks for reporting the issue with the very best sample file and sorry that it took a bit.

Also, note that this work was sponsored by the CDS.

1reaction

alexamicicommented, May 28, 2020

@shahramn the default install of cfgrib uses the internal bindings (still) in ABI mode so every call to an ecCodes function gets a the CFFI overhead.

So most probably the performance issue with many messages is more on the cfgrib side than on ecCodes.

Top Results From Across the Web

Fast Downloading of GRIB Files - Climate Prediction Center

Downloading meteorological data can be a pain. Servers are under-powered, connections are slow and the "bean counters" figure that 800 GB will store...

python - Read grb2 files is very slow - Stack Overflow

With a 12Mb Grib file, to read a single field, cfgrib takes 19 seconds compared to 0.02 seconds with pynio! more than 900...

ConfigOptions – GDAL - Configuration Options - OSGeo Trac

Starting with GDAL 2.3, the CPL_VSIL_CURL_NON_CACHED configuration option can be set to values like "/vsis3/bucket/foo.tif:/vsis3/another_bucket ...

Read and write grib/grib2 data using Python - Google Sites

1 Read grib/grib2 data with pygrib module. 1.1 Open a grib/grib2 file; 1.2 Print an inventory of file; 1.3 Get a grib message...

A Description of the Nonhydrostatic Regional COSMO-Model Part ...

8.2.5 YUCAUTN — Nudging: Warning Messages on Insufficient Array Sizes . ... With GRIB1, DWD used a Grib file format, where all records...