[RFC] Registry for scorers

sklearn.metrics.SCORERS acts like registry for getting predefined scorers in cross_val_score, cross_validate, *SearchCV estimators, etc. If someone wants to use his/her custom scoring functions in the aforementioned functions/classes should define the scoring function, make it scorer using the make_scorer function and pass it around. Another option could be to patch the sklearn.metrics.SCORERS.

Instead of patching directly the SCORERS the make_scorer could optionally register the returned scorer. Another, and probably better, solution could be to create a new decorator or to refactor the make_scorer to act as a decorator (it can actually) that accepts arguments. Like this, one could just define a scoring function and it will be available everywhere in scikit-learn as string. Contrib developers could leverage this functionality as well.

What do you think?

[NicolasHug]

Thanks for the details

Registering scorers has a bit of a frameworky flavor, and I think we try to avoid that in general.

I think this would need some convincing (not just from me 😉 ), but I’d be happy to see a PR to get a better idea on the additional complexity.

[NicolasHug]

Just so I understand properly, the goal is to enable passing strings for custom scorers, instead of passing callables? If so, what is the issue with passing callables?