[question] Definition of observation space without knowing low and high values

I would like to ask what is the proper way to define the observation space if we do not know the values that the different observations could take during training.

I tried to define them by using some large values, in which they will for sure be in, but when I try to check the environment with check_env, I get the following warning:

warnings.warn("We recommend you to use a symmetric and normalized Box action space (range=[-1, 1]) 

Even if I define them to be in [-1, 1], then I will have to normalize them before pass the values to the agent, but again I need their maximum and minimum values.

Do I miss something?

[Miffyli]

Thanks, this is great! I could not find information on how to do this. It works now! I have not used VecEnvs before but noticed, the step method returns a list which contains the the observation np.arrays. Thus, when I want to pass this observation to the model, should I call agent.predict(obs) in a loop or is there a defined way for doing this?

Yes, if you are manually running the environments (e.g. manual evaluation), you might need to do that.

PS: I recommend migrating to stable-baselines3 if possible, as it is more actively supported.

[Miffyli]

Compared to the example code this seems to check out (note: I have not used normalize wrappers myself). I would double check that the observation statistics do not change wildly between creating different envs.

What do the training stats tell you (verbose=2 in creation of agent)? With PPO, this gives you a good idea of the agent’s performance even before separate evaluation.

Also, looking at the code of the VecNormalize wrapper I see, that the parameters clip_obs and clip_reward are only used to clip the values after they were normalized. Therefore it does not clip anything if they are larger than 1, right? When I look at the values in obs, some are larger than 1 or smaller than -1.

Those >1 and <-1 values would be clipped if you set it to clip at 1, yes. But by default it clips to [-10, 10], so no clipping probably happens in your case.

Note that we do not have time to offer per-user tech support, so my answers are short. If you continue having bugs, I still recommend using SB3 if possible. If you spot a bug or confusing behaviour, feel free to open an issue about fixing it 😃