Discount not applied in evaluate_policy?

Maybe I am missing something here but I feel the line to calculate return: https://github.com/keiohta/tf2rl/blob/82d9eecda78e22021efa0821bf02429ac7827f4d/tf2rl/experiments/trainer.py#L207

should be updated to include the discount factor

            for j in range(total_steps):
                action = self._policy.get_action(obs, test=True)
                next_obs, reward, done, _ = self._test_env.step(action)
                avg_test_steps += 1
                if self._save_test_path:
                    replay_buffer.add(obs=obs, act=action,
                                      next_obs=next_obs, rew=reward, done=done)

                if self._save_test_movie:
                    element = self._test_env.render(mode='rgb_array')
                    frames.append(element)
                elif self._show_test_progress:
                    self._test_env.render()
                episode_return += reward * np.power(self._policy.discount, j)

[keiohta]

I agree with @ymd-h that the evaluation score does not include the discount factor. I think the reason why the DDQN paper reports the discounted return is to evaluate the overestimation phenomenon: since the Q-network produces the estimated discounted cumulative rewards, the “true” return should be computed with the discount factor. I don’t think other paper reports discounted return.

[ymd-h]

@naji-s

Although the paper (maybe) doesn’t describe the definition, I think the plots show non-discounted rewards by using models trained with discounted rewards.

As long as discount factor (gamma) is fixed (and n-step is fixed), you can use the discounted reward for model comparison, but it is not universal metric. In order to improve model performance, we try to tune discount factor, so that the metric itself should be independent of discount factor.