Questions about Topk REINFORCE

Hello, thanks for sharing! I have some questions about pi_beta_sample in models.py, you use this function in _select_action_with_TopK_correction, but it seems only sample one item each time? I am also confused by Equation 6 in the original paper, as we want to sample a set of top k item, shouldn’t it be ? a_{t, i} represent the ith item at time t. I appreciate any comments for my question since it’s been bothering me for a long time

[almajo]

In case someone else might come back to this at some point: I was wondering the same thing and I implemented it in the scenario where per slate only one action can/will be clicked anyway, hence when receiving feedback we know which item that feedback responds to.

I guess the authors did the same thing because this sounds like it:

(2) While the main policy head π θ is trained using only items on the trajectory with non-zero reward^3 , the behavior policy β θ ′ is trained using all of the items on the trajectory to avoid introducing bias in the β estimate.

with footnote 3 saying:

We ignore them in the user state update as users are unlikely to notice them and as a result, we assume the user state are not influenced by these actions

[wwwangzhch]

Ok, I will keep watching this repository, please let me know if you have any new thought, and thanks for your sharing too.