Misleading ValueError - Accuracy Metric Multilabel

🐛 Bug description

When using Accuracy.update() with both inputs having the second dimension 1, e.g. in my case torch.Size([256,1]) the raised error message is misleading.

To reproduce

from ignite.metrics import Accuracy
import torch
acc = Accuracy(is_multilabel=True)
acc.update((torch.zeros((256,1)), torch.zeros((256,1))))

ValueError: y and y_pred must have same shape of (batch_size, num_categories, ...).

In this case the y and y_pred do have the same shape but the issue is that it’s not an accepted multilabel input (the and y.shape[1] != 1 in the following code block from _check_shape in _BaseClassification). This should be indicated in the error message (or the if statement changed).

What is the argument to not allow a y.shape[1] of 1?

if self._is_multilabel and not (y.shape == y_pred.shape and y.ndimension() > 1 and y.shape[1] != 1):
            raise ValueError("y and y_pred must have same shape of (batch_size, num_categories, ...).")

Environment

• PyTorch Version (e.g., 1.4):
• Ignite Version (e.g., 0.3.0): 0.3.0
• OS (e.g., Linux): Linux
• How you installed Ignite (conda, pip, source): conda
• Python version:
• Any other relevant information:

[sdesrozis]

We discuss this issue and the behavior of Accuracy with is_multilabel=True in the case of num_categories == 1 won’t change. Actually, the case num_categories == 1 is already covered by binary (i.e. simply is_multilabel=False here). We want to maintain a clear usage and do not have multiple ways (and maybe bad ones wrt performance) to do the same thing. I agree that is_multilabel=True and num_categories == 1 means binary but our implementation does not fit that.

Btw, I can help you about your specific needs. Feel free to share snippets, as you can see, @vfdev-5 is very fast, my challenge is to be faster 😃

[niowniow]

Thanks. It’s fine for me to have another if-condition checking if my input should use binary.