ValueError: only one element tensors can be converted to Python scalars

🐛 Bug

This happens in the training loop.

ValueError: only one element tensors can be converted to Python scalars

To Reproduce

From my observation, I believe this happens when the batch size can’t be divided by gpu num. For example on the last batch of each epoch, and when you have 4 gpus but set batch size to 2.

Additional context

I think it would be nice to use only some of the gpus the user specified, while printing out a msg telling them that the gpus are not specified correctly. Current implementaion simply throws a not friendly error

[awaelchli]

@Ir1d found the bug in trainer code. It does not reduce the outputs if output_size of training step does not equal num_gpus. I will make a PR to fix it.

[awaelchli]

Hi @Richarizardd I looked at your code and I found that in your validation epoch end, you don’t reduce the outputs properly. PL does not do this for you. This is intentional, right @williamFalcon ? So, in your validation_epoch_end, instead of

for output in outputs:
    metric_total += output[metric_name]
tqdm_dict[metric_name] = metric_total / len(outputs)

you should do the following:

tqdm_dict[metric_name] = torch.stack([output[metric_name] for output in outputs]).mean()

I tested this by adding it to your code and it worked (no error). As far as I can tell, this is not a bug in PL. However, we could print a better error message.

@Ir1d you probably had the same mistake.