Difference between training_outputs with the input of training_epoch_end

🐛 Bug

list[the output of the train_step] is different to the input of the training_epoch_end

This is my code of training step

def training_step(self, batch, batch_idx):
		cfg = self.cfg
		x = batch[cfg.keys[0]]
		y = batch[cfg.keys[1]]
		y_hat = self.net(x)
                loss = self.loss_func(y_hat,y)
                print(loss)  #1
		return {'loss': loss}

def training_epoch_end(self, outputs):
 	print(outputs)  #2

When I use GPU training the model, I have found that the answer printed is very different, and the later is nearly half of the former.

cc @justusschock @awaelchli @akihironitta @rohitgr7 @carmocca @borda @ananthsub @ninginthecloud @jjenniferdai

[carmocca]

It’s likely caused by this https://github.com/PyTorchLightning/pytorch-lightning/blob/184518c2fab188a9679a5b9d73ba95e3a8097280/pytorch_lightning/loops/optimization/optimizer_loop.py#L89 where normalize is accumulate_grad_batches https://github.com/PyTorchLightning/pytorch-lightning/blob/184518c2fab188a9679a5b9d73ba95e3a8097280/pytorch_lightning/loops/optimization/optimizer_loop.py#L436-L438

[rohitgr7]

oh okay… I thought the total length of outputs is half with accumulation. we normalize loss to prepare the effective gradients accordingly.