Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Sklearn metrics roc_auc_score gives the wrong result when using anaconda environment

See original GitHub issue

Describe the bug

Same input, Same machine, but roc_auc_score gives different results.

Steps/Code to Reproduce

import numpy as np
from sklearn.metrics import roc_auc_score

X = np.array([[1., 1., 0., 1., 0., 0., 1., 0., 0., 0., 0.],
 [1., 1. ,0., 1., 0., 0., 0., 0., 1., 0., 0.],
 [1.,1. ,0. ,1. ,0. ,0. ,0. ,0. ,1. ,0. ,0.],
 [1., 1. ,0. ,0. ,0. ,1. ,0., 0. ,1. ,0. ,0.],
 [1., 1. ,0. ,0. ,0. ,1. ,0. ,1. ,0. ,0. ,0.],
 [1., 1. ,1. ,0. ,0. ,0. ,0. ,0. ,1. ,0. ,0.],
 [1., 1. ,0. ,0. ,1. ,0. ,0. ,1. ,0. ,0. ,1.],
 [1., 1. ,0. ,0. ,0. ,1. ,0. ,1. ,0. ,0. ,0.],
 [1., 1. ,0. ,0. ,1. ,0. ,1. ,0. ,0. ,0. ,1.],
 [1., 1. ,0. ,1. ,0. ,0. ,1. ,0. ,0. ,0. ,0.],
 [1., 1. ,0. ,0. ,1. ,0. ,0. ,1. ,0. ,0. ,0.]])

w = np.array([1,1,0.8,0.6,0.6,0.6,0.8,0.8,0.4,0.4,0])

y_pred = X.dot(w)

print('y_pred =', y_pred)

y_binary = np.array([ 1, 0, 0, 0,  1, 0, 0, 0, 0,  1, 0])

score1 = roc_auc_score(y_binary, y_pred)

y_copy = np.array([3.4, 3.,  3.,  3.,  3.4, 3.2, 3.4, 3.4, 3.4, 3.4, 3.4]) #(copy directly the output from y_pred)

score2 = roc_auc_score(y_binary, y_copy)
print('score1 =', score1)
print('score2 =', score2)

Expected Results

Expect score1 and score2 are the same, and equal 0.75

Actual Results

y_pred = [3.4 3. 3. 3. 3.4 3.2 3.4 3.4 3.4 3.4 3.4]

score1 = 0.5833333333333333

score2 = 0.75

Versions

System: python: 3.8.3 (default, Jul 2 2020, 16:21:59) [GCC 7.3.0] executable: /home/hoatrinh/anaconda3/bin/python machine: Linux-4.15.0-128-generic-x86_64-with-glibc2.10

Python dependencies: pip: 20.1.1 setuptools: 49.2.0.post20200714 sklearn: 0.23.1 numpy: 1.18.5 scipy: 1.4.1 Cython: 0.29.21 pandas: 1.0.5 matplotlib: 3.2.2 joblib: 0.16.0 threadpoolctl: 2.1.0

Built with OpenMP: True Linux-4.15.0-128-generic-x86_64-with-glibc2.10 Python 3.8.3 (default, Jul 2 2020, 16:21:59) [GCC 7.3.0] NumPy 1.18.5 SciPy 1.4.1 Scikit-Learn 0.23.1 Imbalanced-Learn 0.7.0

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:6 (2 by maintainers)

Top GitHub Comments

1reaction

glemaitrecommented, Jan 4, 2021

This is indeed an issue with floating point:

In [3]: from sklearn.metrics import roc_curve

In [4]: roc_curve(y_binary, y_pred)
Out[4]: 
(array([0.   , 0.5  , 0.5  , 0.625, 1.   ]),
 array([0.        , 0.33333333, 1.        , 1.        , 1.        ]),
 array([4.4, 3.4, 3.4, 3.2, 3. ]))

In [5]: roc_curve(y_binary, y_copy)
Out[5]: 
(array([0.   , 0.5  , 0.625, 1.   ]),
 array([0., 1., 1., 1.]),
 array([4.4, 3.4, 3.2, 3. ]))

If you store the thresholds value of the former case, you will observe the following:

In [10]: xx, yy, zz = roc_curve(y_binary, y_pred)

In [11]: zz[1]
Out[11]: 3.4000000000000004

In [12]: zz[2]
Out[12]: 3.4

So you have some floating-point error. So the error is introduced by X.dot(w) and not the computation of the ROC AUC. I am not totally sure what could introduce that the floating-point error could be different. One reason could be that NumPy was not built with the same compiler maybe.

0reactions

glemaitrecommented, Jan 4, 2021

Closing since this not related to scikit-learn

Top Results From Across the Web

why does roc auc score give wrong score? - Stack Overflow

I'm using logistic regression for a binary classification problem in Python. I chose ROC AUC score as evaluation metrics because i have ...

sklearn.metrics.roc_auc_score

Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores. Note: this implementation can be used with binary, multiclass ...

Comparing Various ML models(ROC curve comparison)

I am going to apply 6 Supervised machine learning models on the given dataset.The strategy is to apply default model first with no...

Metrics — deepchem 2.6.2.dev documentation

Note this parameter only has value if mode==”classification”. Returns ... A number (but not all) of these metrics are directly sourced from sklearn...

Source code for mlflow.sklearn

The ``mlflow.sklearn`` module provides an API for logging and loading ... :return: The default Conda environment for MLflow Models produced by calls to ......