LightFM always produces same precision@6 result metric

I’m running parameter optimization on a LightFM implicit factorization model using “warp” loss. When I run np.mean(precision_at_k(…)) on the test data, it is virtually always the same result, 0.316850870848, out to 12 digits. Is this expected for some reason? If not, any ideas how to figure out what might be wrong?

Here is sample output showing the param combinations and the output:

2017-01-20 21:10:02,363 INFO __main__: Starting training iteration 1, params:
{
    "alpha": 0.0041233076328981919,
    "epochs": 45,
    "learning_rate": 0.50174314850490254,
    "no_components": 184
}
2017-01-20 21:10:02,363 INFO lightfm_uma: Training model...
2017-01-20 21:25:20,518 INFO lightfm_uma: Finished precision@6 = 0.316850870848
2017-01-20 21:25:21,453 INFO __main__: Starting training iteration 2, params:
{
    "alpha": 0.0064873564172718886,
    "epochs": 63,
    "learning_rate": 0.50406151543722921,
    "no_components": 180
}
2017-01-20 21:25:21,453 INFO lightfm_uma: Training model...
2017-01-20 21:44:36,565 INFO lightfm_uma: Finished precision@6 = 0.316850870848
2017-01-20 21:44:37,495 INFO __main__: Starting training iteration 3, params:
{
    "alpha": 0.020473212242717205,
    "epochs": 62,
    "learning_rate": 0.74135691825946459,
    "no_components": 156
}
2017-01-20 21:44:37,496 INFO lightfm_uma: Training model...
2017-01-20 22:00:45,661 INFO lightfm_uma: Finished precision@6 = 0.316850870848

[maciejkula]

All SGD algorithms are sensitive to hyperparameters. NaNs are most often the result of excessive learning rates. Arguably I should warn when this happens, it can be easy to miss.

Can I suggest you pull the branch I linked above and use it for your optimization? It will have the advantage of giving you terrible precision scores when you algorithm has actually diverged or regularized itself to zero.

[maciejkula]

I think this is fixed in 1.12 — thanks for your help!