Feature: add community metrics

We would like a way for community members to easily add custom or new metrics. These custom metrics could be hosted on the hub either in a dedicated repository type (like datasets or models) or on Spaces.

Spaces comes with a few advantages, especially in short term:

1. No dedicated repo type needed
2. Infrastructure already exists
3. We can easily setup a demo without needing a custom widget (e.g. with Gradio)

@osanseviero made a PoC here: https://huggingface.co/spaces/osanseviero/accuracy_metric

A few things that we should add to make this as user friendly and useful as possible:

• create a template to add a metric space
  • template README.md (like model/dataset card) with basic structure
  • template my_metric.py file (similar to datasets loading scripts)
  • gradio app.py that loads all information from README.md and my_metric.py to create a default widget. This does probably not need to be changed.
• debugging tools
  • instructions on how to clone the repository directly on spaces
  • maybe an evaluate-cli command to test if everything is fine evaluate-cli test-metric PATH_TO_REPO
• dedicated tag: if we ever want to migrate the metrics to a dedicated place like datasets and models it is good if we had a tag to find them all. Similarly, we could then easily do a list_metrics() function in the evaluate without needing to look at the content of each repo.

The goal is to enable behaviour (example with the space above):

accuracy_metric = load_metric("osanseviero/accuracy_metric")

[julien-c]

Yes, you can just add a tags: - metric to the README.md

To clarify: I think there is a small thing left to implement in moon-landing (cc @cbensimon) but then you will be able to query https://hf.co/api/spaces?filter=metric i.e. same as for other repo types

But in the meantime you can already add tags to README.md “space cards”

[lvwerra]

Thanks, that’s a good idea. Maybe we keep metric.py and app.py separated in case we want to switch the widget or metric logic at some point we don’t need to touch/disentangle both.