MRR score uses a "mean" instead of "min"

If I’m not mistaken, MRR score is a mean of reciprocal rank scores over a set of examples, where reciprocal rank is given as 1 / (rank of the first positive item).

Currently implementation in spotlight takes a mean over ranks of positive items. I think this is not correct. It is also inconsistent with lightfm. It also makes it impossible to have mrr score of 1 even with perfect model (even if positive items are ranked [1, 2, 3], taking the mean gives result lower than 1).

Can I send a PR to fix this?

[maciejkula]

It’s not necessarily incorrect, but I can see why you might want to do it differently.

Could you make a PR that keeps the current code as default, but allows the alternative as an option? You could pass a mode argument that’s either mean or min.

[cmacdonald]

bump