Reproducing Figure 11 and reporting success rate

Hi all and @avnishn,

I’ve been trying to reproduce results from Figure 11 in https://arxiv.org/pdf/1910.10897.pdf using https://github.com/rlworkgroup/garage/blob/08492007d6e2d9ead9beb83a8a4247e52019ac7d/metaworld_examples/sac_metaworld.py and hyper-parameters reported in Table 3. Should I use Table 3 for hyper-parameters?

One thing which is not clear to me is how the success rate is reported. I notice the env.step returns ‘success’ but want to verify here that is what reported in the paper. Here is the code the I use to report results ( random action is used for simplicity):

from metaworld.envs import ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE
env_cls = ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE['hammer-v2-goal-observable']
eval_env= env_cls(seed=0)
eval_env.seed(0)
avg_reward = 0 
success_rate = 0 
num_evals = 2

for _ in range(num_evals):
    obs = eval_env.reset()
    done = False
    stp = 0
    while not done and stp < eval_env.max_path_length:
        obs, reward, done, info = eval_env.step(eval_env.action_space.sample())
        avg_reward += reward
        stp += 1
        if 'success' in info:
            success_rate += info['success']
avg_reward /= num_evals
success_rate /= num_evals

Is this the right way to report the success rate like Figure 11? Thanks for your help. Rasool

[avnishn]

1. whoops sorry, I edited my answer. Gave you the wrong answer the first time.
2. yes, and they should be the same as the hparams in the launcher that you linked. 3)yes.

[krzentner]

To be clear, since the above conversation was unclear to me: In MetaWorld, an episode is considered successful if the info['success'] ever becomes 1.0 during that episode. SuccessRate therefore needs to be computed across many episodes to be meaningful.