[Question]: Using other metrics such as `AUC`.

In much of the literature / guides outside of this project AUC seems to be a popular metric for recommender systems. TF and Keras specifically has an implementation here.

However I am not sure on:

• is this a valid metric to use for retrieval when using this project?
• what hyperparams would be important to consider?
• would the use of this as batch_metric make sense?

Thanks in advance!

[patrickorlando]

Just the positives for that batch @rlcauvin. In practice sampling a negative that is a positive in another batch doesn’t affect performance and provides some mild regularisation.

You have candidate_ids of shape (batch_size, 1), and scores of shape (batch_size, batch_size). Essentially we are just creating a mask where tf.cast(candidate_ids == tf.transpose(candidate_ids), dtype=tf.float32) - tf.identity(batch_size). e.g.,

candidate_ids = [[0,1,2,0,3,4,3,5]]
mask = [
  [0, 0, 0, 1, 0, 0, 0, 0], # 0
  [0, 0, 0, 0, 0, 0, 0, 0], # 1
  [0, 0, 0, 0, 0, 0, 0, 0], # 2
  [1, 0, 0, 0, 0, 0, 0, 0], # 0 
  [0, 0, 0, 0, 0, 0, 1, 0], # 3
  [0, 0, 0, 0, 0, 0, 0, 0], # 4
  [0, 0, 0, 0, 1, 0, 0, 0], # 3
  [0, 0, 0, 0, 0, 0, 0, 0]  # 5
]

[patrickorlando]

Hey @rlcauvin, I would start with

just using user IDs and item IDs

The model should learn based on just this.

When you experimented with the ranking model, is everything else in your code kept the same? Same lookup layers, embedding layers, tf.data pipelines?

I would:

1. Ensure that the lookups are working as expected, take a few examples and manually pass them through. Are any items being mapped to the [UNK] token 1? Is the shape correct, they should be only 1 dimensional, (batch_size,).
2. Pass them through the embedding layers. is each row different? Is the shape correct (batch_size, n_dim)
3. Do the matrix multiplication, are the scores different? do you get a shape that is (batch_size, batch_size)?

The shape is important, because if the query and candidate tensors have an extra dimension, the matrix multiplication will produce an incorrect result. Your loss will decrease but your model will be junk.