eval_mode giving inconsistent results

Describe the bug Some networks give different results each time they’re called with eval_mode.

I suspect this might be the cause of https://github.com/Project-MONAI/MONAI/issues/1710.

To Reproduce

import torch
from monai.networks.utils import eval_mode

model = torch.nn.BatchNorm2d(1)
im = torch.randn((1, 1, 2, 2))
for _ in range(5):
    res_train = model(im).flatten().detach()
    with eval_mode(model):
        res_eval = model(im).flatten()
    print(res_train)
    print(res_eval)

Example output, notice how odd rows all match (train), and even rows differ (eval):

tensor([ 0.7991,  0.5358, -1.7123,  0.3774])
tensor([ 0.7824,  0.4933, -1.9753,  0.3193])
tensor([ 0.7991,  0.5358, -1.7123,  0.3774])
tensor([ 0.7689,  0.4882, -1.9086,  0.3193])
tensor([ 0.7991,  0.5358, -1.7123,  0.3774])
tensor([ 0.7579,  0.4842, -1.8531,  0.3195])
tensor([ 0.7991,  0.5358, -1.7123,  0.3774])
tensor([ 0.7489,  0.4810, -1.8064,  0.3198])
tensor([ 0.7991,  0.5358, -1.7123,  0.3774])
tensor([ 0.7413,  0.4784, -1.7667,  0.3201])

[wyli]

I had a quick test it seems we shouldn’t use eval_mode in the vis, we want eval() but without torch.no_grad()

https://github.com/Project-MONAI/MONAI/blob/master/monai/visualize/class_activation_maps.py#L122

[wyli]

anyway eval_mode itself is ok, I’m closing this ticket