Add quantile loss as metric

Describe the workflow you want to enable

I’d like to evaluate and compare the predictive performance of (conditional) quantiles as predicted by GradientBoostingRegressor(loss='quantile', alpha=0.9) for example.

Describe your proposed solution

Implement a new metric quantile_loss(y_true, y_pred, alpha=0.5), Eq. (24) of https://arxiv.org/pdf/0912.0902.pdf.

This is the same loss as in Koenker’s book “Quantile Regression” and in: https://github.com/scikit-learn/scikit-learn/blob/ad3c28818220a4ec9a40b3b674adcdb676a419ea/sklearn/ensemble/_gb_losses.py#L461-L467

[lorentzenchr]

@ogrisel I’m fine with calling it pinball_loss, though I don’t know who has begun to call it so, this is from 2006. The term “asymmetric piecewise linear, strictly consistent scoring function for the quantile” would also nail it down😏

Indeed, there are infinitely many strictly consistent scoring functions of the quantile, all having the form, see paper cited above: S(y, x) = (1_(x ≥ y) − α) (g(x) − g(y)) and g(x) a strictly increasing function.

I know, you only googled, but I’m not that convinced of arxiv:2011.09588.

[GaelVaroquaux]

Note that my comment probably does not rule out the use of the alpha-quantile loss as a metric in GridSearchCV, because surrogate losses can be useful to compare models close enough. I would worry when using it to evaluate a model before putting it in production.