.predict_proba() for SVC produces incorrect results for binary classification

Description

svm.predict_proba() produces revered results for binary classification

• this seems to be specific to binary classification. For example, it works fine for 3 way classification, which is in the test: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/tests/test_svm.py#L307
• I think this is related to #394, but in this case we are using the same svm object so I think predict and predict_proba should agree
• Acknowledgement: @amennen noticed this error 👍

Steps/Code to Reproduce

here’s the code on colab: https://colab.research.google.com/github/qihongl/random/blob/master/sklearn-svm-predict-proba-bug.ipynb

# train a svm 
X = np.array([[-1, -1], [1, 1]])
y = np.array([0, 1])
svm = SVC(probability=True)
svm.fit(X, y) 

# SVM makes reasonable prediction on the learned examples... 
print(svm.predict(X))

# but it makes the reversed probability estimates... 
print(svm.predict_proba(X))

Expected Results

[0 1]
[[0.66383953 0.33616047]
 [0.33916469 0.66083531]]

# i.e. when the prediction is class 0, prob(class0) should be bigger than prob(class1)

Actual Results

[0 1]
[[0.33616047 0.66383953]
 [0.66083531 0.33916469]]

# i.e. when the prediction is class 0, prob(class0) < prob(class1)

Versions

0.20.3

[amueller]

I came to the same conclusion in my analysis, and could also reproduce the same behavior with CalibratedClassifierCV when not using stratification. This is the evil of LOO in small noisy datasets.

So I agree with the closing this as a separate bug.

[123gkc]

Reopening this as I ran into the same issue. According to my observations, the issue of .predict() and argmax of .predict_proba() not tallying only comes up when class_weights are enabled. Sharing modified code of @rth to recreate the issue :

import numpy as np

from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_digits
from sklearn.metrics import accuracy_score
from sklearn.svm import SVC

X, y = load_digits(return_X_y=True)

mask = (y == 0) | (y == 1)
X = X[mask, :]
y = y[mask]  # make a binary classification problem

X = StandardScaler().fit_transform(X)


class CalibratedSVC(SVC):
    def __init__(self):
        super().__init__(probability=True, class_weight={0:0.9, 1:0.1})

    def predict(self, X):
        # binary classification with targets [0, 1] only
        y_proba = self.predict_proba(X)
        return np.argmax(y_proba, axis=1)


for est in [SVC(probability=False, class_weight={0:0.9, 1:0.1}), CalibratedSVC()]:
    est.fit(X, y)
    y_pred = est.predict(X)
    score = accuracy_score(y, y_pred)
    print(f'Model {est}')
    print(f'  - train accuracy={score:.6f}')

produces

Model SVC(class_weight={0: 0.9, 1: 0.1})
 - train accuracy=0.997222
Model CalibratedSVC()
 - train accuracy=1.000000

My best hypothesis at the moment is that .predict_proba() gives raw probabilities whereas .predict() actually takes into class_weights on the outputs of .predict_proba() to make the final inference.