Questions about MCTS

Hi, in update_branch function, you update the parent node value with the same total_reward with the current node. Why? I think they have different values, because action from parent node to current node will get a reward which should be added in parent total_reward.

    def update_branch(self, total_reward):
        self.update(total_reward)
        if self.parent:
            self.parent.update_branch(total_reward)

[sritee]

I think the two formulations are equivalent (ie when choosing the next action to take, comparing past + future1 with past + future 2 is the same as just comparing future1 with future2. But I agree that your proposal is more intuitive, maybe I should change it. I don’t really remember why I implemented it this way in the first place

That makes sense, I suppose the shift invariance of the UCB makes it so that the magnitude of the value function doesn’t quite matter, but I suppose for other non-argmax or other policies it might make a difference.

[eleurent]

should the weight for the first term be 1 - regularization?

Yes it should! thank you for catching this, I’ll fix it.

but I suppose for other non-argmax or other policies it might make a difference

Indeed, it would matter for a Boltzmann policy for instance. In which case, it is probably best to ignore the past.