[Metrics] Sklearn metrics

Do we want to include metrics of sklearn?

The following constraints would apply if we had an sklearn metric interface:

• All imports would be within the class space, so sklearn would only remain an optional dependency (due to it’s size)
• All metrics will be provided by sklearn interface at first, but will be removed later on (once we got a native pytorch implementation)

The Advantages:

• We have a pretty impressive set of metrics directly in the beginning
• the maintenance effort is minimal
• they’re well tested

The disadvantages

• For every call to a metric, we need to convert tensors to numpy
• We will have to synchronise GPU(s) whenever we call a metric with inputs lying on GPU, which may be a bottleneck the users aren’t aware of

My proposal

• Add this layer in the beginning and try to replace as many of the metrics used there by our own 😃
• Explicitly state the downfalls in docs (and maybe also raise a warning on that) to ensure users are aware of this problem.

Thoughts?

cc @Borda @williamFalcon @ethanwharris @PyTorchLightning/core-contributors

[rzepinskip]

The AllenNLP also started with CPU-based metrics to move some of them to GPU later - PR 3851

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.