Generating Predictions

This may be a stupid question corresponding to basic functionality, but…

I have n users (O(millions)) and m items (O(100k)), with very sparse interactions. Building a csr matrix and training (in a pure CF case) is pretty straight forward. I’m interested in generating the top k (O(100)) recommendations for each user. Is there an example of how to do this? Is there an efficient batch operation? I can generate recs for each person individually, but, even with multithreading, generating recs for everyone will take days.

[maciejkula]

The easiest way is probably to use the get_item_representations and get_user_representations methods.

They return matrices of embeddings and arrays of biases, allowing you to perform fast batch multiplications via standard numpy dot operations. Combined with pre-allocating the receiver (using the out argument in dot), this should give you pretty decent speed for predictions themselves.

For sorting, np.argpartition may help for efficient partial sort up to 100, followed by a full argsort on the top 100.

Together, and parallelized using joblib or similar, this should take you close enough to C speed. If that is still too slow, you may have to distribute this across a cluster.

[jattenberg]

This is a good lead, thanks. i think i should be able to do some manipulation and use nmslib too.