Feature: standardize inputs/outputs of metrics

Currently there are several different inputs/output formats possible in Metrics. We should standardize them as much as possible and respecting the following principle:

• inputs/outputs are easy to understand and use
• outputs are compatible with other frameworks

For the output standardization: probably a dictionary structure, even if nested would be ok. Also a dedicated output class could be considered like in transformer models but this is probably not necessary here. To make it compatible with e.g. keras we could add a postprocess function at initialization similar a transform in datasets.

There are three options we could implement:

load_metric(..., postprocess="metric_key") # equivalent result to `metric.compute()["metric_key"]`
load_metric(..., postprocess="flatten") # equivalent flattening the output dict: `flatten(metric.compute())`
load_metric(..., postprocess=func) # equivalent result to `func(metric.compute())`

[sashavor]

As per our meeting today, we proposed to have standardized structure for inputs, in dictionary form.

An initial proposal of that structure can be:

     {
        "references":  ,
        "predictions": ,
     }

With references and predictions being of any format (strings, images, numbers, vectors, etc.). I was looking at examples of computer vision metrics and this should work for those as well.

Edge cases:

• COMET, WikiSplit and SARI – take an additional input, sources
• F1, Precision and Recall – require average parameter for multiclass labels, but we could define a default if needed
• Perplexity – needs an input string and a model

I think we could have additional, optional, source and average inputs, but I don’t really know what to so for perplexity 😅 (I mean, in any case, the metrics will not function without these arguments, but I guess waiting for them to crash isn’t the best solution)

CC @lvwerra @lhoestq @apsdehal

[lvwerra]

I have been thinking about the scalar vs. dict question. Having a dict across all metrics at least internally is nice as it allows to treat them all the same way and we can also combine metrics by merging dicts. At the same time we could check if the return is just a dict with one value and if that’s the case just return its. value.

metric = evaluate.load("accuracy") 
>>> 0.6

metric = evaluate.load("accuracy",  force_dict=True)
>>> {"accuracy": 0.6}

What do you think @lhoestq?

Regarding Keras, I’ll think a bit more about how to do that smoothly.