Deprecate normalize parameter in `calibration_curve`

Describe the workflow you want to enable

Similar to the behavior of calibration_curve, I would like to be able to set CalibrationDisplay.from_predictions(normalize=True).

Describe your proposed solution

Add a keyword argument normalize to CalibrationDisplay.from_predictions and pass it to calibration_curve.

Describe alternatives you’ve considered, if relevant

I could manually normalize the values, but given that the functionality is already present in the underlying calibration_curve I would say this is a relatively easy thing to adjust on the scikit-learn side.

Additional context

No response

[glemaitre]

OK, I went into the discussion in the PR and discussed it as well with @ogrisel IRL.

Methodologically, a calibration curve should only be used with a probability. Using calibration_curve with normalize=True is equivalent to a naive linear calibration with additional clipping for value above/under max/min.

So I think that we don’t want people using this type of naive calibration without doing it explicitly.

I try to asses the extent to which a set of given risk scores (that go from 1 to 9) are similarly calibrated for different subgroups, which can be relevant for fairness assessments. I am not interested in actually deriving a calibrated classifier.

I think that in your case it boils down to how you want to relate the risk scores to a probability. If you want a linear mapping then normalize=True was the right move. However, you could as well use a sigmoid transform centred on score 5 to express a probability. Since I don’t have the background on what are the scores, this is difficult to say what is the right choice to transform the score into probability but it shows that using the naive normalization implicitly could be dangerous in general.

By the way, does this mean that normalize in calibration_curve will at some point also be deprecated or will it be kept for backward compatibility?

I would be in favour of deprecating normalize as well with an explicit warning that encourages users to provide probabilities that could be obtained through a calibrated classifier if the output is a decision function.

[lucyleeow]

@glemaitre we had some comments about this: https://github.com/scikit-learn/scikit-learn/pull/17443/files#r670597315

Essentially, we thought that from_estimator should only accept estimators with a predict_proba method, and we are sure predict_proba will output a true probability. We had also decided that we did not want to (/not a good idea to) pass output of decision_function to calibration_curve, which probably boils down to @NicolasHug’s comment here: https://github.com/scikit-learn/scikit-learn/pull/17443#discussion_r440103386, though that whole thread is probably relevant.