Interpreting prediction scores
See original GitHub issueI built a recommendation model on a user-item transactional dataset where each transaction is represented by 1.
model = LightFM(learning_rate=0.05, loss='warp')
model.fit(train, epochs=NUM_EPOCHS, num_threads=NUM_THREADS)
Here are the results from the prediction experiment
Train auc score: 0.978294
Test auc score : 0.810757
Train precision at k=3: 0.115301
Test precision at k=3: 0.0209936
Train recall at k=3: 0.238312330233
Test recall at k=3: 0.0621618086561
Can anyone help me interpret this result? How is it that I am getting such good auc score and such bad precision/recall? The precision/recall gets even worse for ‘bpr’ Bayesian personalized ranking.
Prediction task
users = [0]
items = np.array([13433, 13434, 13435])
model.predict(users, item)
Result
array([-1.45337546, -1.39952552, -1.44265926])
How do I interpret the prediction scores?
Another problem is if I split the data into unequal train test datasets then I get the error
print(train.shape)
print(test.shape)
(88741, 18109) (29581, 18109)
test_auc = auc_score(model, test, train_interactions=train, num_threads=NUM_THREADS).mean()
Error
/usr/local/lib/python3.5/site-packages/lightfm/lightfm.py in predict_rank(self, test_interactions, train_interactions, item_features, user_features, num_threads)
658
659 if not user_features.shape[1] == self.user_embeddings.shape[0]:
--> 660 raise ValueError('Incorrect number of features in user_features')
661
662 test_interactions = test_interactions.tocsr()
ValueError: Incorrect number of features in user_features
Thanks
Issue Analytics
- State:
- Created 7 years ago
- Comments:12
Top Results From Across the Web
4. Regression and Prediction - Practical Statistics for Data ...
Universities use regression to predict students' GPA based on their SAT scores. A regression model that fits the data well is set up...
Read more >Providing a Context for Interpreting Predictions of Job ...
Test scores are often used to predict levels of a criterion or outcome variable.
Read more >Making Predictions with Regression Analysis - Statistics By Jim
Learn how to use regression analysis to make predictions and determine whether they are both unbiased and precise.
Read more >Chapter 38 Predicting Criterion Scores Based on Selection ...
In this chapter, we will learn how to estimate a simple linear regression model and apply the model's equation to predict future scores...
Read more >Assessing the performance of prediction models - NCBI
The performance of prediction models can be assessed using a variety of different methods and metrics. Traditional measures for binary and survival outcomes ......
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
It gives you the position of item X in a list where you ordered all items by how strongly they are recommended (so yes, 42 is ‘goodness’, and smaller numbers are better, because they are closer to the head of the list). Of course, if you are actually serving recommendations, you should use the
predict
method (which will give you a score where higher is better).Recommendations are a ranking problem, not a classification problem, so things like thresholds don’t really make sense. The idea is that you have, say, 4 slots to fill in your interface, and you sort all products by their recommendation score and pick the top 4; you never set a threshold.
Reading the references is a good start. The BPR paper is a classic. There is also this but I can’t say anything about how good it is.