Multi class accuracy metric

I really looked at issues and googled the question but couldn’t find anything I could use so here goes.

I’m porting some code from vanila pytorch to Ignite, and have a CIFAR10 classifier. Every end of Evaluation epoch, I want to report the per class accuracy. In Ignite, I only found total accuracy (which I use) but not a per class one. I wrote this CustomMetric, but I have a few problems with it:

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
class CustomAccuracy(Metric):

    def __init__(self, *args, **kwargs):
        self._num_correct = [0] * len(classes)
        self._num_examples = [0] * len(classes)
        super().__init__(*args, **kwargs)
        self.i = 0

    @reinit__is_reduced
    def reset(self):
        self._num_correct = [0] * len(classes)
        self._num_examples = [0] * len(classes)
        self.i = 0
        super(CustomAccuracy, self).reset()


    @reinit__is_reduced
    def update(self, output):
        y_pred, y = output

        for predtensor,real in zip(y_pred,y):
            pred = torch.argmax(F.softmax(predtensor,0), 0)
            self._num_examples[real] += 1
            if real == pred:
                self._num_correct[real] += 1

    @sync_all_reduce("_num_examples", "_num_correct")
    def compute(self):
        class_accuracy = [0] * len(classes)
        for i, (correct, total) in enumerate(zip(self._num_correct,self._num_examples)):
            class_accuracy[i] = correct / total
        #if self._num_examples == 0:
        #    raise NotComputableError('CustomAccuracy must have at least one example before it can be computed.')
        return class_accuracy

First, It depends on the Loss I chose. I use nn.CrossEntropyLoss() which has softmax, so in my metric I also had to add it. But I assume if I use another Loss without softmax I won’t need it (and my network will handle that). Second, this feels like something that should be out of the box, so I’m wondering if I’m missing something.

Any advice?

[erezalg]

I see, OK makes sense! I’ll open the feature request and let’s see where it leads!

Thanks again!

[vfdev-5]

@erezalg thanks for the feedback !

One last question that I couldn’t figure out myself is, when I have multiclass recall, the name of the class in the TB graph is the label enumeration.

Unfortunately, it is not possible out-of-the-box to add labes. This is something I was also thinking about to add as a feature request (if you’d like to send one, it could be helpful). The limitation is due to a tensor nature of metrics output: we have for example Recall metric that outputs torch.tensor([0.1, 0.2, 0.3, ..., 0.8]) and this is directly used within OutputHandler for the Tensorboard: https://github.com/pytorch/ignite/blob/75e20420a2391ad2e11ee17df65f781a659ae6ec/ignite/contrib/handlers/tensorboard_logger.py#L290-L291

However, there is a workaround to that. Idea is to create N metrics that output scalars instead of a single metric that gives a tensor. Thus we can label the metric as we’d like. We can use metrics arithmetics for that. Something like that should work:

num_classes = 10
cls_name_mapping = ["car", ...]
val_metrics = {}

for i in range(num_classes):
    cls_name = cls_name_mapping[i]
    val_metrics["Recall/{}".format(cls_name)] = Recall(average=False)[i].item()