Macro-averaged MAE

Describe the workflow you want to enable

For ordinal classification, we can use multiple metrics, for example: MAE, MSE… As we would use for regression. But when these classes are imbalanced, one way to deal with imbalance is to use macro-averaged MAE instead, as described here https://stats.stackexchange.com/questions/338904/measures-of-ordinal-classification-error-for-ordinal-regression and here https://www.computer.org/csdl/proceedings-article/2009/isda/3872a283/12OmNybfrbj .

The macro-averaged MAE is like the “classic” MAE, except we compute each MAE for each class and average them, giving equal weights to MAEs. Note that macro-averaged MAE == micro-averaged (or classic) MAE when class are balanced.

To illustrate this, let’s consider:

y_true_balanced = np.array([1, 1, 1, 2, 2, 2])
y_true_imbalanced = np.array([1, 1, 1, 1, 1, 2])
y_pred = np.array([1, 2, 1, 2, 1, 2])

mean_absolute_error(y_true_balanced, y_pred)
>> 0.33
mean_absolute_error(y_true_imbalanced, y_pred)
>> 0.33
macro_averaged_mean_absolute_error(y_true_balanced, y_pred)
>> 0.33
macro_averaged_mean_absolute_error(y_true_imbalanced, y_pred)
>> 0.2

Describe your proposed solution

I coded it, the natural way would be to modify sklearn.metrics.mean_absolute_error directly, by adding a parameter average='macro' for macro-averaged-MAE, and average='micro' for normal (=micro-averaged)MAE… But MAE is 99,9% of the time used for regression, not for ordinal classification, and as macro-averaged MAE is used only for ordinal classification, I am afraid that would be confusing.

That is why I would propose to add it in its own package, sklearn.metrics.macro_averaged_mean_absolute_error.

What do you think, before I propose the associated MR?

[glemaitre]

It might be confusing to introduce the metric in scikit-learn since MAE would refer to both classification and regression in some ways. However, it would be a perfect addition in imbalanced-learn because this the actual scope of the library and one will not use it without knowing its specificity. I would be really keen to get the implementation in the library there.

[glemaitre]

I am closing this issue then and making a review on the other side 😃