RFC on schema for the `metrics` field of weights meta-data

Problem

We currently store the metrics as a simple dictionary metric -> value. For classification weights this looks like this:

https://github.com/pytorch/vision/blob/7d0d7fd7a940df94a12e9c1853eec7e42ab6ee80/torchvision/models/resnet.py#L320-L323

and the underlying assumption is that the dataset is ImageNet.

But in the future, we might want to provide metrics for more than one dataset, not just ImageNet. Our current schema doesn’t allow us to do that nicely. In fact we already have such an problem with the Optical Flow weights, which typically report EPE on different datasets, and keys of the metrics dict currently look like:

sintel_train_cleanpass_epe                                                     
sintel_train_finalpass_epe                                                                                                                
sintel_test_cleanpass_epe                                                      
sintel_test_finalpass_epe                   

which is pretty ugly.

Proposed schema

There’s a million ways to structure this. I guess the simplest one is to have a metrics dict with such structure:

"metrics": { 
    "ImageNet": {
        "acc@1": 69.758, 
         "acc@5": 89.078, 
    },
    "ImageNetV2": {
        "acc@1": 99.999, 
         "acc@5": 0.0001, 
    },
 },

This makes it slightly less convenient to access values programatically, but I’m wondering whether this is actually something we should encourage?

I’m almost tempted to rename the "metrics" key of the meta dict into "_metrics" to strongly discourage users to use it, and to allow us to change the schema in the future if we find a need for it. Happy to hear your thoughts

[NicolasHug]

I don’t have a strong opinion.

At this point my preferred solution would actually to not use - or _, but instead write purely human-readable dataset names like ImageNet 1K or Sintel (test), clean pass. It would make it a bit more awkward to access programmatically by users, which IMHO is a good thing.

But again, no strong opinion. Using - vs _ feels the same to me, so if you submit a PR I’ll approve it.

[datumbox]

I’m OK adopting @NicolasHug proposal. I don’t think it’s necessary to specify a default dataset or a default value. I would be more inclined to keep the metrics public thought as it would be useful for downstream projects (Papers with code repo, D2go Zoo, etc) to programmatically read the values but I’m OK if we want to release it first as private until we are certain about the schema.

@NicolasHug are you unblocked to implement your proposal?