Example with ranking metrices

Describe the issue linked to the documentation

Some of the metrices in #2805 were implemented in #7739.

Suggest a potential alternative/fix

It would be nice to add (to) an example the usage of those ranking metrices with the addition of, e.g., kendall’s tau and spearman’s rho:

from scipy.stats import kendalltau, spearmanr
from sklearn.metrics import make_scorer

kenall_tau_score = make_scorer(kendalltau)
spearman_rho_score = make_scorer(kendalltau)

[sveneschlbeck]

@rth I agree…since having a Recommendation Engine example seemed to be in the interest of multiple people, I got to that first. The other points are also valid but (as you mentioned) not necessarily well-combineable with rec engines

[rth]

I don’t think that we have any example tackling the problem of recommendation. It would be nice to have a full example with a predictive model and the way to evaluate it?

As far as I know ranking is not necessarily a primary evaluation metric in recommendation (see e.g. lightfm.evaluation): one cares more about how many relevant predictions are made in the first N, instead of whether it’s first or third. Though I agree that it would still be be good to have a recommendation example, but maybe more for top_k_accuracy_score metric (which also doesn’t have any examples apparently)?

For DCG and NDCG what comes to mind is more a search or directly a ranking problem. There are a few ranking problems on OpenML maybe we could pick one (or find some other open dataset and put it there)? Though of course we can also illustrate them on a recommendation example.

A side comment that https://www.openml.org/d/40916 looks interesting, but maybe too political. I do wonder how does the partial dependence plot of “Dystopia” wrt “Happines” looks like 😃