[Enhancement] Support Different VecEnvWrapper Depths

Is there a reason that this while loop assumes env and eval_env are wrapped in the same number of VecEnvWrappers? https://github.com/DLR-RM/stable-baselines3/blob/2ada2dd0b2b10f71d0be38ce032d49391b875c4e/stable_baselines3/common/vec_env/__init__.py#L66 This assumption is causing a bug for me in one of my use cases. The following code relaxes this assumption and fixes my use case:

while isinstance(env_tmp, VecEnvWrapper) or isinstance(eval_env_tmp, VecEnvWrapper)
    if isinstance(env_tmp, VecNormalize):
        eval_env_tmp.obs_rms = deepcopy(env_tmp.obs_rms)
        eval_env_tmp.ret_rms = deepcopy(env_tmp.ret_rms)
    try:
        env_tmp = env_tmp.venv
    except AttributeError:
        pass
    try:
        eval_env_tmp = eval_env_tmp.venv
    except AttributeError:
        pass

[araffin]

Is it okay to apply VecMonitor to a VecEnv where each individual env already has Monitor applied? (The VecEnv was created with stable_baselines3.common.env_util.make_vec_env.)

It is unless you have some low level wrappers that changes the reward: https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/vec_env/vec_monitor.py#L43

[scottemmons]

Great - these solutions should work for me!