Some experiments in multi-experiment MPI SAMS runs hang

I’m still seeing an issue where some experiments in multi-experiment MPI SAMS runs will hang and not make progress, or restart and not make progress. I’ve attached an example logfile where you can see both behaviors.

cmethingesinglehinge5.log.zip

No CRITICAL logs appear in this experiment.

[jchodera]

That set of experiments was generated with

/data/chodera/chodera/2018-08-05/yank-benchmark/c-Met_hinge_binders/run-lsf-neutral-rmsd.sh

using the YAML file

/data/chodera/chodera/2018-08-05/yank-benchmark/c-Met_hinge_binders/neutral-sams-rmsd.yaml

These simulations were run in 6-hour blocks chained together by LSF.

Separately, I tested whether chained 1-hour simulations were likely to show the same behavior more quickly, suspecting that perhaps the second job would start before the first job’s NetCDF file was fully synced by the filesystem. If I add

sync
sleep 10
sync
sleep 10
sync

to the beginning of the bash script (run-lsf-neutral-rmsd-safe.sh and neutral-sams-rmsd-safe.yaml) I seem to get the same sort of behavior where replicas stall out, though I get a bit more information because verbose is on. See

/data/chodera/chodera/2018-08-05/yank-benchmark/c-Met_hinge_binders/neutral-rmsd-safe/experiments/cmethingesinglehinge10

[andrrizzi]

Closing this for now. Let me know if more netcdf corruptions crop up.