Clarification for "training steps" in habitat paper and the implementation

In Figure 3 of habitat paper, the number of steps taken in training and corresponding SPL scores are reported.

I’m basically running habitat_baselines/train_ppo.py with the following default params

--num-processes 4
--num-steps 128
--num-mini-batch 4
--num-updates 100000

Under this setting, how many steps does the agent experience?

If I understand the paper and the implementation correctly, the number of steps this can experience should be 128 * 100000 * 4 = 51.2M.

But when I evaluated the trained agent by using habitat_baselines/evaluate_ppo.py, the resulting SPL score was way lower than expected. ( I got 0.25 on RGB, whereas Fig3 says it should be above 0.4)

[erikwijmans]

The number of steps of experience a model is trained for can be determined by the number of processes (--num-processes), the number of steps per rollout (--num-steps), and the number of updates/rollouts (--num-updates). You have the correct math for number of steps of experience, i.e. 128 * 100000 * 4 = 51.2M, and that is what you should refer to for comparing to results in the paper 😃

For recreating val numbers, I you’ll need --count-test-episodes 994 and --num-processes 1 as we need to fix the splitting logic (#89). The numbers may not be exactly the same, but they should be close.

[erikwijmans]

--count-test-episodes 100 will only evaluate 100 episodes. There are 994 episodes in val, looks like you got unlucky with the ones it randomly selected 😃