[Question] Why is true negative represented by 'n' in the classification matrix?

Describe the bug

In the confusion matrix:

Class                           n          tp          fn          fp      recall        prec          f1
Iris-versicolor                16          15           1           0       0.938       1.000       0.968
Iris-virginica                 15          15           0           1       1.000       0.938       0.968
Iris-setosa                    14          14           0           0       1.000       1.000       1.000
Total                          45          44           1           1

The title/label for True negative is shown as n instead of tn

Expected behaviour

Most documentations, on confusion matrix I have seen so far, represent it as tn.

It might lead to doubts by those who may be aware of the standard representations. Especially the dependent metrics like recall, precision, f1, accuracy etc… are made up these base metrics (and True negative is one of them).

[Craigacp]

That’s not the confusion matrix. N is the total number of that class in the test set. It doesn’t show the true negatives because in multiclass settings it would give a misleading looking number. True negatives aren’t found in a confusion matrix anyway, confusion matrices have the possible labels as both axes. True negatives aren’t used by precision, recall, f1 or accuracy.

Thanks for the explanation.

In that case, it would be good to have a legend describing what each of the Classification metrics mean i.e. n = number of classes, tp = true positive, etc…

I think the current layout (without a legend) would confuse some of the others as it has confused me.

This is the output of the toFormattedString method. We could add another method that emits a legend String, or modify the toFormattedString output. I’d prefer the former (or some kind of documentation change) as pretty soon the legend would be irrelevant noise to anyone using Tribuo for any length of time. What kind of other information would you want in it?

How is the accuracy calculations from Tribuo different than those explained here https://en.wikipedia.org/wiki/Precision_and_recall?

That’s the binary classification accuracy. Tribuo treats every classification problem as if it’s multiclass, and in multiclass problems accuracy is the sum of the true positives divided by the total number of test examples. We decided early on to not allow any special casing for binary problems, as it made it difficult for things like moving from two class sentiment (positive/negative) to three class sentiment (positive/negative/neutral) as all the code paths would change.

The methods & lambdas Tribuo uses to calculate the various metrics are here and here.

On an unrelated discussion, false negatives is a useful metric, but I guess the other 3 metrics already provide the data indirectly - whats your thoughts on this?

We provide the false negatives row wise as it’s hard to break down where the misclassifications are without it. However in that case it’s probably best to print the confusion matrix and look at it directly.

This particular formatted string is the output we use as it’s what our data science team wanted to show in their reports, and it’s easy to pull out the relevant information without it taking up too much space. All the metrics we’ve discussed are calculated and can be accessed on the Evaluation object, including the true negatives, so others are welcome to generate their own reporting output. If there are metrics we aren’t calculating we’re happy to take PRs to add them to the LabelEvaluation, but I think it’s a slightly higher bar to get them into the toFormattedString output as every additional metric increases the clutter.

Another question:

Class                           n          tp          fn          fp     
Iris-versicolor                16          16           0           1
Iris-virginica                 15          14           1           0 
Iris-setosa                    14          14           0           0
Total                          45          44           1           1

if N = number of observations per class, and tp, fn, fp are breakdowns of it then should n = tp + fn + fp for each row in the above table?

N = tp + fn. It’s the total number of elements of that class, so it’s the true positives (i.e. the things we correctly predicted as that class) plus the false negatives (i.e. the things we incorrectly predicted as members of another class) which is the denominator of the recall).

[Craigacp]

In the 4.1 release we updated the classification tutorial to discuss the formatted output and generally improved the docs. I’m going to close this issue and make a separate one to track the addition of formatter/printer classes for evaluations.