Isotonic calibration changes rank-based test metrics values

[As discussed with @ogrisel]

Describe the bug

Using isotonic calibration changes metrics values. This is because it is a non-strictly monotonic calibration. Sigmoid calibration being strictly monotonic doesn’t suffer from this.

Steps/Code to Reproduce

Here is quick example where we split in three train/calibration/test sets and compare ROC AUC on the test set before and after calibrating for both isotonic and sigmoid…

import numpy as np
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import roc_auc_score

X, y = datasets.make_classification(n_samples=100000, n_features=20,
                                    n_informative=18, n_redundant=0,
                                    random_state=42)

X_train, X_, y_train, y_ = train_test_split(X, y, test_size=0.5,
                                                    random_state=42)
X_test, X_calib, y_test, y_calib = train_test_split(X_, y_, test_size = 0.5,
                                                    random_state = 42)

clf = LogisticRegression(C=1.)
clf.fit(X_train, y_train)

y_pred = clf.predict_proba(X_test)
print(roc_auc_score(y_test, y_pred[:,1]))

The ROC AUC is then: 0.88368

isotonic = CalibratedClassifierCV(clf, method='isotonic', cv='prefit')
isotonic.fit(X_calib, y_calib)

y_pred_calib = isotonic.predict_proba(X_test)
print(roc_auc_score(y_test, y_pred_calib[:,1]))

After isotonic calibration, the ROC AUC becomes: 0.88338

isotonic = CalibratedClassifierCV(clf, method='sigmoid', cv='prefit')
isotonic.fit(X_calib, y_calib)

y_pred_calib = isotonic.predict_proba(X_test)
print(roc_auc_score(y_test, y_pred_calib[:,1]))

As expected for sigmoid calibration, the ROC AUC is constant. 0.88368

Versions

System: python: 3.8.1 | packaged by conda-forge | (default, Jan 29 2020, 15:06:10) [Clang 9.0.1 ] executable: /Users/leodreyfusschmidt/opt/miniconda2/envs/isotonic/bin/python machine: macOS-10.13.4-x86_64-i386-64bit

Python dependencies: pip: 20.0.2 setuptools: 45.1.0.post20200119 sklearn: 0.22.1 numpy: 1.16.5 scipy: 1.4.1 Cython: None pandas: None matplotlib: 3.1.2 joblib: 0.14.1

Built with OpenMP: True

[dsleo]

Thanks @lucyleeow for the references and the upcoming additions to the doc !

The solution of the second reference seems a bit expensive. Regarding the first reference, this is not clear to me:

Observe that isotonic regression preserves the ordering of the input model’s scores, although potentially introducing ties i.e. f(ŝ(·)) is not injective. To break ties on training examples, we may simply refer to the corresponding original model’s scores.

Wouldn’t that break monotonicity by substituting calibrated probabilities by their original predicted probabilities ?

We can extend @ogrisel hack with another by adding linear interpolation on all constant sub-arrays of the calibrator._necessary_y_. This generalises to non-constant steps (in the above example the constant sub-arrays were always of size 2).

Here’s a draft code:

def interpolate(min_v, max_v, lenght):
    delta = max_v - min_v
    eps = float(delta)/lenght
    return np.arange(0, delta, eps)


necessary_y = calibrator._necessary_y_

necessary_y_fixed = necessary_y.copy()

sub_ = np.split(necessary_y_fixed,
                np.nonzero(np.diff(necessary_y_fixed))[0] + 1)

n_splits = len(sub_)
for i in range(n_splits - 1):
    sub_length = len(sub_[i])
    if sub_length > 1:
        min_v = sub_[i][0]
        max_v = sub_[i+1][0]
        correction = interpolate(min_v=min_v, max_v=max_v, lenght=sub_length)
        sub_[i] += correction

And we can check as a sanity check, that indeed

calibrator._build_f(calibrator._necessary_X_, necessary_y_fixed)
y_pred_calib = isotonic.predict_proba(X_test)

print(roc_auc_score(y_test, y_pred[:, 1]))
print(roc_auc_score(y_test, y_pred_calib[:, 1]))

gives

0.8836876399954915
0.8836876399954915

Does that seems a reasonable enough strategy ? It’ll be nice to have something of that sort as an optional post-processing of IsotonicRegression .

[NicolasHug]

please ping me on the PR 😃