[ASK] Discovered a strange behavior on ranking metrics

Description

I tried to evaluate some MF models (and UserKNN) with ranking metrics on the MovieLens dataset (100k). The results do not look as expected (first table), so I implemented the evaluation step with sklearn.metrics and the output looks much more realistic (second table). The implementation can be found here: https://gist.github.com/tpoerschke/1d823e1b9dbc0f290c763854e9fa2a52.

The implementation of my evaluation should be similar to that in Cornac. The metrics are evaluated per user and then averaged over all users.

Am I missing something here? Or is this a bug?

TEST:
...
        |  F1@-1 | Precision@-1 | Recall@-1 | Train (s) | Test (s)
------- + ------ + ------------ + --------- + --------- + --------
PMF     | 0.0143 |       0.0073 |    1.0000 |    6.4883 |   0.3277
NMF     | 0.0143 |       0.0073 |    1.0000 |    1.4425 |   0.4025
SVD     | 0.0143 |       0.0073 |    1.0000 |    0.4447 |   0.4037
UserKNN | 0.0143 |       0.0073 |    1.0000 |    0.1430 |   5.7097


CUSTOM EVALUATION
Model      F1 score    Precision    Recall    Train (s)
-------  ----------  -----------  --------  -----------
PMF          0.3498       0.3226    0.4506       6.8282
NMF          0.3243       0.3334    0.3707       1.4856
SVD          0.3423       0.3158    0.4398       0.3843
UserKNN      0.2628       0.2688    0.3319       0.0957

System

• OS: macOS Catalina (10.15.7)
• Python: 3.8.3 [Clang 10.0.0 ] :: Anaconda, Inc. on darwin

Other comments

[tqtg]

@tpoerschke precision is capped by the number of ground-truth items. More recommendations will increase chance of getting correct items. However, the number of total predictions/recommendations (normalization) by the model also increased.

First of all, I think you should be clear about how those ranking metrics are used to evaluate top-k recommendations. The book I mentioned in the previous comment is one of the good references.

[tpoerschke]

@tqtg This will be the next step. If it works for the entire set, I’m going to limit it to a number of K. But shouldn’t the precision also be quite high when evaluating over the entire set? The value near zero just feels wrong.