50% performance drop going from 0.8 to 0.9 with the VVVR integrator on GPUs

I recently updated my openmmtools version from 0.8.3. After this I saw a huge (50%) performance drop and narrowed it down to the update from 0.8.3 to 0.9.0. When running the following code on my machine, the average speed on 0.8.3 is 106 ns/day while it is 48.3 ns/day for 0.9.0 and going to 0.10 didn’t fix it. The only difference between those two speeds is the openmmtools version, which I changed via conda. I also tried going to OpenCL and that also didn’t fix the problem.

from simtk.openmm import app                                                     
from simtk import unit                                                           
import simtk.openmm as mm                                                        
import openmmtools as omt                                                        
from sys import stdout                                                           
                                                                                 
system = omt.testsystems.DHFRExplicit(                                           
    nonbondedMethod=app.PME,                                                     
    nonbondedCutoff=1.1*unit.nanometers,                                         
    constraints=app.HBonds,                                                      
    rigidWater=True,                                                             
    ewaldErrorTolerance=0.0005)                                                  
                                                                                 
integrator = omt.integrators.VVVRIntegrator(                                     
    298*unit.kelvin,                                                             
    1.0/unit.picoseconds,                                                        
    2.0*unit.femtoseconds                                                        
)                                                                                
                                                                                 
integrator.setConstraintTolerance(0.00001)                                       
system.system.addForce(                                                          
    mm.MonteCarloBarostat(                                                       
        1*unit.atmospheres,                                                      
        298*unit.kelvin,                                                         
        25                                                                       
    )                                                                            
)                                                                                
                                                                                 
platform = mm.Platform.getPlatformByName('CUDA')                                 
properties = {'CudaPrecision': 'mixed', 'CudaDeviceIndex': '0'}                  
simulation = app.Simulation(system.topology, system.system, integrator, platform,
    properties)                                                                  
simulation.context.setPositions(system.positions)                                
simulation.context.setVelocitiesToTemperature(298*unit.kelvin)                   
simulation.reporters.append(app.StateDataReporter(stdout, 5000, step=True,          
    potentialEnergy=True, temperature=True, progress=True, remainingTime=True,   
    speed=True, totalSteps=2500000, separator='\t'))                             
                                                                                 
print 'Running Production...'                                                    
simulation.step(2500000)                                                         
print 'Done!'  

This is all running python 2.7, with openmm version 7.1.1 on a NVIDIA GeForce GTX TITAN X.

cc: @dwhswenson @arjunwadhawan

[jchodera]

Great!

I wonder if there may be further optimizations @peastman could make in OpenMM that would be able to speed up heat measurement. Will post in the OpenMM issue tracker after we get a simple test case constructed.

We should definitely disable measure_heat=True as the default option, though.

[maxentile]

Glad to hear! The redundant velocity constraint solves shouldn’t affect the statistics of the paths sampled, but are there to make the heat/shadow_work bookkeeping more convenient (by ensuring that, at the end of every substep, we are still on the constraint manifold).

@jchodera : Perhaps measure_heat=False should be default? And when no bookkeeping is done, we can remove redundant velocity-constraint solves?