[question] VecNormalize with hyper parameter tuning

Hi, I’m trying to do some hyper parameter tuning, but am getting this error: ValueError("Trying to set venv of already initialized VecNormalize wrapper.") How am I supposed to use the normalization statistics from the training phase during validation without saving it to file during every trial?

def objective_fn(trial):
    model_params = optimize_ppo(trial)

    train_env, validation_env = initialize_envs()
    norm_env = VecNormalize(train_env, norm_obs=True, norm_reward=True, training=True)

    model = PPO(policy, 
                norm_env,
                device=device,
                **model_params)

    train_maxlen = len(train_env.get_attr('df')[0].index) - 1
    try:
        model.learn(train_maxlen)
    except Exception as error:
        print(error)
        raise optuna.structs.TrialPruned()

    norm_env.set_venv(validation_env)
    norm_env.training = False
    norm_env.norm_reward = False

    mean_reward, _ = evaluate_policy(model, norm_env, n_eval_episodes=5)

    if mean_reward == 0:
        raise optuna.structs.TrialPruned()

    return -mean_reward

[araffin]

How am I supposed to use the normalization statistics from the training phase during validation without saving it to file during every trial?

You will find your answer here 😉: https://github.com/DLR-RM/stable-baselines3/issues/473 (we have a sync_env_normalization for that.

https://github.com/DLR-RM/stable-baselines3/blob/75b6f3b3b0f207456d9dcac2c6e86e8e2a22115f/stable_baselines3/common/vec_env/__init__.py#L59-L72

[aleksanderhan]

Thank you both very much!