Add option to choose OOB scorer in RandomForest

Describe the workflow you want to enable

Random forests provide for free an estimate of its out-of-sample performance using the concept of out-of-bag (OOB) predictions. In practice, it works well when

1. rows are independent
2. the number of trees is large (500+).

If oob_score=True, currently the following scoring is applied to the OOB predictions:

• RandomForestRegressor() always returns “r2”, even if the trees are built to minimize Poisson deviance or mean absolute error. R-squared is mainly relevant for the MSE. What do you mean, @lorentzenchr ?
• RandomForestClassifier() always returns accuracy, even if it does not enjoy many good properties. For instance, if my criterion is “entropy” and I am modelling a binary target, then the corresponding scoring function would be rather log loss than accuracy.

I would like to enable that the user is able to pick a scoring function to evaluate OOB performance.

Describe your proposed solution

• To RandomForestRegressor(), add option oob_scoring="r2"
• To RandomForestClassifier(), add option oob_scoring="accuracy"

The value could take any meaningful scoring function from the rich list here, in line with cross-validation scoring: https://scikit-learn.org/stable/modules/model_evaluation.html

Describe alternatives you’ve considered, if relevant

My current solution is to use oob_prediction_ attribute of the fitted model and apply the meaningful scorer on those.

Additional context

No response

[adrinjalali]

yes, with a normal callable it’d be tricky, but in another project (fairlearn), I’m also thinking of callables (normal scoring funcs) which also have get_metadata_request. It’d be something like ConsumerCallable(score_func).score_requests(sample_weight=True) which would be passed as a normal score func, but it supports metadata routing and stuff.

[adrinjalali]

we don’t need to pass a scorer, we can simple pass a partial(fbeta_score, beta=.7)