Hangs while saving netcdf file opened using xr.open_mfdataset with lock=None

I am testing out code that uses xarray to process netcdf files, in particular to join multiple netcdf files into one along shared dimensions. This was working well, except sometimes when saving the netcdf file the process would hang.

I was able to whittle it down to this simple example: https://github.com/jessicaaustin/xarray_netcdf_hanging_issue

This is the code snippet at the core of the example:

 # If you set lock=False then this runs fine every time.
 # Setting lock=None causes it to intermittently hang on mfd.to_netcdf
 with xr.open_mfdataset(['dataset.nc'], combine='by_coords', lock=None) as mfd:
     p = os.path.join('tmp', 'xarray_{}.nc'.format(uuid.uuid4().hex))
     print(f"Writing data to {p}")
     mfd.to_netcdf(p)
     print("complete")

If you run this once, it’s typically fine. But run it over and over again in a loop, and it’ll eventually hang on mfd.to_netcdf. However if I set lock=False then it runs fine every time.

I’ve seen this with the following combos:

• xarray=0.14.1
• dask=2.9.1
• netcdf4=1.5.3

and

• xarray=0.15.1
• dask=2.14.0
• netcdf4=1.5.3

And I’ve tried it with different netcdf files and different computers.

Versions

Output of `xr.show_versions()`

INSTALLED VERSIONS

commit: None python: 3.7.6 | packaged by conda-forge | (default, Mar 23 2020, 23:03:20) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.15.0-20-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.4

xarray: 0.15.1 pandas: 1.0.3 numpy: 1.18.1 scipy: None netCDF4: 1.5.3 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.1.1.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2.14.0 distributed: 2.14.0 matplotlib: None cartopy: None seaborn: None numbagg: None setuptools: 46.1.3.post20200325 pip: 20.0.2 conda: None pytest: None IPython: None sphinx: None

[fmaussion]

Just adding my +1 here, and also mention that (if memory allows), ds.load() also helps. (related: https://github.com/pydata/xarray/issues/4710)

[bekatd]

Please make some dummy tests, I did time.sleep, prior every operation. This was the only workaround that really worked.