Loss automatically detaching inputs breaks some workflows

🐛 Bug description

https://github.com/pytorch/ignite/blob/master/ignite/metrics/loss.py#L60

Before 0.4.3, the y_pred and y were passed without calling detach. Now because of detach a number of usecases are no longer possible.

An example is: https://docs.gpytorch.ai/en/v1.3.1/marginal_log_likelihoods.html#variationalelbo

The output of model(train_x) is an object which does not support detach. This leads to a crash when passing the mll function to Loss as a loss_fn.

Another example is https://github.com/y0ast/deterministic-uncertainty-quantification/blob/master/train_duq_cifar.py#L153

The loss is dependent on the gradient, which means that by calling detach the loss cannot be computed anymore.

I have been trying to work around it, but I can’t figure out a nice way. https://pytorch.org/ignite/metrics.html#ignite.metrics.Average is not input size aware so it cannot correctly compute the average of the loss.

[y0ast]

The duq cifar usecase is a bit special. We compute the gradient penalty of https://arxiv.org/abs/1704.00028 also at test time to monitor it. So we can’t use torch.no_grad in the evaluator there.

So the create_supervised_evaluator is not a good fit for it, but I think that’s ok.

[vfdev-5]

Why not call detach on the output of loss_fn?

Yes, it would work. Thanks for the PR, btw !

For your use-case with duq cifar, I assume it is on purpose that evaluator does not have torch.no_grad ? For example, in basic trainings we have it here : https://github.com/pytorch/ignite/blob/442bd08ea93b76e997bbbd190b8b55a7dc29d216/ignite/engine/__init__.py#L460-L465