Reproduction of 80K/sec throughput

Hi, I tried to reproduce the 80K/sec throughput reported in the paper, but only got around 22K/sec.

I ran the single learner on a GPU machine (the GPU is P40):

python experiment.py --job_name=learner --task=0 --num_actors=150 \
    --level_name=rooms_keys_doors_puzzle --batch_size=32 \
    --entropy_cost=0.0033391318945337044 \
    --learning_rate=0.00031866995608948655 \
    --total_environment_frames=10000000000 --reward_clipping=soft_asymmetric 

and ran 150 actors each on a CPU machine (each one is actually a docker machine in remote allocated by a cloud service):

python experiment.py --job_name=actor --task=$i \
      --num_actors=150 --level_name=rooms_keys_doors_puzzle

where i denotes the i-th actor.

Could you give some hints on how to reproduce the throughput? Did you require a proprietary intra net connection?

[pengsun]

Thanks!

How does the number 2-3GB/sec come (e.g., batch_size * width * height * rollout_len * BytesOfFloat, etc.)? I’m still reading the tf.FIFOQueue code (with capacity=1) and struggling to understand the sync mechanism. I guess answering this question my help me (and others) to understand how the Actor code works 😃

Also, I just asked around and found I was unable to access a P100, the best GPU in hand is only P40… So please feel free to close the issue.

[lespeholt]

Yes, we used 1 CPU per actor. Can you try 150 actors with 1 CPU each?

It’s a bit hard to interpret the timelines without interacting with them. Since dequeuemany is taking that much time on the learner, it looks like they are bottlenecked by actors or the bandwidth to them. Not sure why there is a gap between the actor steps. If they wait on enqueuing, then it suggest a bottleneck in the learner or the bandwidth. In this case it would then be the network.

Can you try and create new variables for each actor? i.e. no sharing of variables. If that is significantly faster, it’s network bandwidth.