File locking error when resuming a h-repex simulation on multiple GPUs

I was trying to resume a single repex simulation (using 4 GPUs) with the following code:

reporter = MultiStateReporter(reporter_file, checkpoint_interval=10)
simulation = HybridRepexSampler.from_storage(reporter)
# Determine how many more iterations are needed
total_iterations = 5000
iterations =  total_iterations - simulation.iteration
# Resume simulation
simulation.extend(n_iterations=iterations)

However, I got the following error:

OSError: [Errno -101] NetCDF: HDF error: b'/data/chodera/zhangi/perses_benchmark/repe
x/31/7/0/0_complex.nc'
CRITICAL:mpiplus.mpiplus:MPI node 1/4 raised an exception and called Abort()! The exc
eption traceback follows

This problem was found (and solved) in Yank as well: https://github.com/choderalab/yank/issues/1165

To fix this, I added the following line to my bash script: export HDF5_USE_FILE_LOCKING=FALSE

@jchodera suggests we fix this in a similar way to how Yank fixes this: https://github.com/choderalab/yank/pull/1168

However, the Yank code containing the fixes above has already been ported to openmmtools (see here), so I’m not sure what the appropriate fix for perses is (or why the lines in openmmtools aren’t sufficient).

[mikemhenry]

The issue is in python 3.3+ IOError was merged into OSError https://docs.python.org/3/library/exceptions.html#OSError IE

>>> try:
...  raise OSError
... except (IOError):
...  print("oh no")
... 
oh no
>>>

So the issue is that instead of just trying a few times, we hit the code path that assumes the file doesn’t exit and exits early. Looking at the source code (and docs) of netcdf https://github.com/Unidata/netcdf4-python/blob/master/src/netCDF4/_netCDF4.pyx#L1939 It it isn’t obvious which error will be thrown if the file doesn’t exit verses some issue with HDF locking, so what I will do is remove the logic that handles the case of the file not exiting since the worse case scenario is instead of failing right away, it will fail in ~ 10 seconds.

[mikemhenry]

Will take a new release but fixed with choderalab/openmmtools#515