Cannot reproduce the benchmark results of DQN on Breakout

I use the instruction below to train DQN on breakout environment, here is my instruction:

python3 -m baselines.run --alg=deepq --env=BreakoutNoFrameskip-v4 --num_timesteps=10000000

And at the end of training, I could only get 14-15 for 100 episode reward mean, I want to know how could I reproduce the results?

[DanielTakeshi]

@bywbilly Did you use the exploration schedule I had earlier? I probably should have made it clear, the exploration schedule is shown in the lower left plot in my figure above. Here it is in my actual code:

    exploration = PiecewiseSchedule([
            (0,        1.0),
            (int(1e6), 0.1),
            (int(1e7), 0.01)
    ], outside_value=0.01)

[DanielTakeshi]

@bywbilly

To get DQN to work you need to adjust hyperparameters that are different from what’s the default in Baselines.

I got Breakout to work several times for different random seeds, all within the past week from master. Here is one example of a training curve I have with a code base I’m testing with (the top left is probably what you want, past 100 episode reward):

Off the top of my head:

• Use the exploration schedule above or something closer to what OpenAI was using before they changed DQN around Oct 2017
• Replay buffer size: 1e6
• Learning starts: 80k
• Update target net: 40k
• Adam lr 1e-4, adam epsilon 1e-4

edit: this is PDD-DQN, just to be clear. I ran for 2.5e7 steps.