[rllib] IMPALA can't converge on cluster with Ray 0.6.4

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• Ray installed from (source or binary): Binary
• Ray version: 0.6.4
• Python version: 3.6.8

Describe the problem

I have a cluster consist of a p2 head and a c4 worker on AWS. IMPALA can’t converge when I train on the cluster with Ray==0.6.4. I originally run on a smaller computer with Ray==0.6.3, so I tried the old Ray version on the cluster, IMPALA converge again. This was tested on BreakoutNoFrameskip-v4, PongNoFrameskip-v4 and AtlantisNoFrameskip-v4.

Source code / logs

import ray
from ray import tune
from ray.tune.registry import register_env
from ray.rllib.env.atari_wrappers import wrap_deepmind
from ray.rllib.agents.impala import ImpalaAgent
from ray.rllib.agents.ppo import PPOAgent
from ray.rllib.agents.dqn import DQNAgent

ray.init()

''' Breakout  Experiment '''

trials = tune.run_experiments({
    "breakout": {
        "run": "IMPALA",
        "env": "BreakoutNoFrameskip-v4",
        "checkpoint_freq": 5, # model checkpoint
        "stop": {
            "timesteps_total": 10000000
        },
        "config": {
            "num_gpus": 1,
            "num_workers": 32,
            "num_envs_per_worker": 10,
            "clip_rewards": True
        }
    },
}, resume=False)```

[bjg2]

Hey guys, sorry about the issue and you did the good thing by reverting the commit. We did train PongDeterministic-v4 with our changes, not sure if we changed the convergence rate on that one. @stefanpantic and I will take a look at why IMPALA behaves differently with our changes on BreakoutNoFrameskip-v4.

[jincongho]

Sorry, false alert. It was my problem, I created a class wrapper around gym.make object, but then atari_wrapper cannot properly preprocess the frames.