unable to use wandb.plots.ROC

wandb --version && python --version && uname

• Weights and Biases version: 0.8.35
• Python version: 3.7.6
• Operating System: Linux

Description

I’m trying to use wandb.plots.ROC to plot a ROC curve for a two-class classification problem. I’m executing it just like in this example:

wandb.log({'roc': wandb.plots.ROC(y_true, y_probas, labels)})

What I Did

In my case, y_true has shape (n, ) and y_probas also has shape (n, ).

Each element of y_true is an int (0 or 1), and each element of y_probas is the predicted probability in range (0, 1). I’m also using labels=['class 1', 'class 2']

But I’m getting this error message:

Traceback (most recent call last):
  File "teacher.py", line 407, in <module>
    training_history = train(config, vox, d, predictor, label_out)
  File "teacher.py", line 258, in train
    wandb.log({'ROC': wandb.plots.ROC(y, y_preds, lb)})
  File "/home/mpds/miniconda3/envs/docktdeep/lib/python3.7/site-packages/wandb/plots/roc.py", line 71, in roc
    return roc_table(fpr_dict, tpr_dict, classes, indices_to_plot)
  File "/home/mpds/miniconda3/envs/docktdeep/lib/python3.7/site-packages/wandb/plots/roc.py", line 50, in roc_table
    fpr_dict[i], tpr_dict[i], _ = roc_curve(y_true, probas[:, i],
IndexError: too many indices for array

It seems that what is causing the error is this statement: probas[:, i].

In the docs it’s written (without mention to what shape these should be):

y_true (arr): Test set labels.
y_probas (arr): Test set predicted probabilities.

I’ve looked into the source code and examples, but I wasn’t able to understand what shape is expected of probas for the binary classification case.

Thanks.

[lavanyashukla]

Hi, sorry the docs aren’t clear! This is the expected format –

y_test:  [1 1 0 1 1 1 1 0 0 1 1 1 1 1 1]
y_probas:  [[0. 1.]
 [0. 1.]
 [1. 0.]
 [0. 1.]
 [0. 1.]
 [0. 1.]
 [0. 1.]
 [1. 0.]
 [1. 0.]
 [0. 1.]
 [0. 1.]
 [0. 1.]
 [0. 1.]
 [0. 1.]
 [0. 1.]]
labels:  ['setosa' 'versicolor' 'virginica']

I also added a complete working example for you.

[ramit-wandb]

Hey @sngyo,

You are right. wandb.plot.roc_curve expects the probabilities of both classes as input, so y_score will need to be transformed to [[0.6, 0.4], [0.3, 0.7], [0.2, 0.8]].

Thanks, Ramit