EpochMetric signature check caused error

🐛 Bug description

I tried to re-implement the ROC_AUC like the one in contrib.metrics.roc_auc, and passed some extra kwarg options to support for multiclass and multilabel classification. However, when I run my code, an Error "Number of classes in y_true not equal to the number of columns in ‘y_score’ " was raised by scikit-learn. I checked the y_true of the whole epoch, and the number of unique values was equal to the number of columns of y_score. Next, I checked the source code of EpochMetric, and found that in update function, it tried to execute the compute_fn at the first iteration. I supposed this was the cause: the y_true of a single batch might not contain all the classes of the dataset, therefore scikit-learn would throws the error mentioned above. The same problem occured for AveragePrecision.

Environment

• PyTorch Version (e.g., 1.5): 1.5
• Ignite Version (e.g., 0.3.0): 0.3.0
• OS (e.g., Linux): Ubuntu 18.04
• How you installed Ignite (conda, pip, source): pip
• Python version: 3.7.6
• Any other relevant information: None

[vfdev-5]

@sandylaker sorry for delay with the issue. We just merged a PR that should introduce the argument as discussed here. This functionality will be available in nightly in 12h and in stable 0.4.0 in 1.5-2 weeks.

[vfdev-5]

OK, I could reproduce the warning. Anyway, let’s introduce an argument to disable this check. By default, it is True (run checking). Such that if user knows what he/she is doing, can disable it.