Confusion Matrix is of wrong shape

Describe the bug

Having 100 percent true positive rate gives a 1x1 matrix. This can cause bugs if you are using the confusion matrix in a function to calculate evaluation parameters such as sensitivity, auc, specifity, accuracy.

from sklearn.metrics import confusion_matrix, roc_auc_score

ground_truth, predictions = np.ones((2,1))
cf_matrix = confusion_matrix(ground_truth, predictions)
assert cf_matrix.shape == (2,2), ("The confusion matrix is of shape"+str(cf_matrix.shape))

Expected Results

No error is thrown.

Actual Results

AssertionError                            Traceback (most recent call last)
<ipython-input-23-9c79b721fccb> in <module>
      3 ground_truth, predictions = np.ones((2,1))
      4 cf_matrix = confusion_matrix(ground_truth, predictions)
----> 5 assert cf_matrix.shape == (2,2), ("The confusion matrix is of shape"+str(cf_matrix.shape))

AssertionError: The confusion matrix is of shape(1, 1). 

Versions

System:
    python: 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 15:59:12)  [Clang 11.0.1 ]
executable: /Users/julian/opt/anaconda3/envs/cadd-course/bin/python
   machine: Darwin-20.3.0-x86_64-i386-64bit

Python dependencies:
          pip: 21.0.1
   setuptools: 49.6.0.post20210108
      sklearn: 0.24.1
        numpy: 1.20.1
        scipy: 1.6.1
       Cython: None
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True

[glemaitre]

I agree with @jnothman that it is not as easy as it seems since we don’t have any knowledge of the classification problem to be solved (binary or multiclass). Without #12385, issuing a warning as proposed by @NicolasHug might the most sensible thing to do, mentioning that providing labels would silence the warning and provide the confusion matrix with the expected shape.

[NicolasHug]

Maybe we should raise an error or a warning when only one label is found in the input?