Replica propagation times systematically change in cyclic patterns that may slow simulations

In plotting the total replica propagation time per iteration from one of @hannahbrucemacdonald’s experiments (/data/chodera/brucemah/relative_paper/dec/jnk1/lig20to8/complex_0.stderr), it appears that there is a pattern: Connecting successive iterations with lines reveals a clearer pattern: Zooming in shows the pattern more clearly: This is currently just something intriguing, but something we should investigate in the new year since it may lead to speed improvements if we can understand what is going on here.

[jchodera]

Here’s a clue: If I replace the mcmc_moves—which are LangevinDynamicsSplittingMoves—with LangevinDynamicsMoves, the periodic behavior goes away.

I suggest we may want to use LangevinDynamicsMove for now and I can debug the MCMC LangevinDynamicsSplittingMove stuff separately. It may have to do with the LangevinSplittingIntegrator, which @maxentile and I need to refine anyway.

[peastman]

I think I understand part of what’s going on. I don’t know exactly what’s leading to the specific sawtooth shape we’re seeing. I think it relates to the details of the sampling algorithm, which I’m not familiar with. I’ll explain what I’ve found, and then maybe you’ll be able to fill in the missing pieces.

First some background on how OpenMM computes nonbonded interactions. It uses a coarse-grained neighbor list. Instead of finding the neighbors of individual atoms, it finds the neighbors of blocks of 32 contiguous atoms. This is only efficient if those blocks are compact. Atoms whose indices are close also need to be close in space. That’s fine for proteins, since the atoms are in sorted order by residue. But water is a problem. Each molecule can diffuse independently of every other.

To address that, it sorts the water molecules in a way that groups spatially nearby ones close together in order. That makes the neighbor list efficient. But it becomes less efficient with time as the waters move, so it periodically repeats the sorting.

In this script, a single Context is used for 24 replicas. It loops over them, setting the coordinates for each replica in turn and running just a small number of time steps with it. That means it is frequently simulating one replica with a water ordering that was created for a different replica, and hence is inefficient. Within a few hundred steps it will reorder them and become efficient again. But then it promptly switches to yet another replica.

That’s what seems to be going on. When the iteration time goes up, it’s because the neighbor list has become less efficient and it has to compute more interactions. And if I make it reorder atoms more often, the variation gets smaller (though of course it also has to spend more time sorting atoms).