FetchPickAndPlace not training using DDPG+HER

I am trying to train FetchPickAndPlace as per https://arxiv.org/pdf/1802.09464.pdf using DDPG+HER, however, regardless of how long I train, agent fails to learn anything. I saw that #198 mentioned that OpenAI used a number of tricks to get it to work. Has anyone had any luck doing so in stable baselines? Thanks!

FetchReach and FetchPush both train fine.

My current hyperparameters:

FetchPickAndPlace-v1:
  n_timesteps: !!float 5e6
  policy: 'MlpPolicy'
  model_class: 'ddpg'
  n_sampled_goal: 4
  goal_selection_strategy: 'future'
  buffer_size: 1000000
  batch_size: 256
  gamma: 1.0
  critic_l2_reg: 1.0
  observation_range: [-200.0, 200.0]
  random_exploration: 0.3
  actor_lr: !!float 1e-3
  critic_lr: !!float 1e-3
  noise_type: 'normal'
  noise_std: 0.2
  normalize_observations: true
  normalize_returns: false
  policy_kwargs: "dict(layers=[256, 256, 256])"

[araffin]

During testing, all the exploration noise is removed, we use a deterministic policy, hence the difference.

[fisherxue]

Will do, I’ll get back to you in a few weeks (hardware I would run it on is currently occupied) 😃