Add support for MultiDiscrete/MultiBinary observation spaces

• preprocessing.py and policy.py need to be updated (cf Stable-Baselines for the preprocessing)
• tests using envs from identity_env.py

@rolandgvc is working that one

[araffin]

Playing devil’s advocate here, but since tuple observation (and action spaces) is on the roadmap, wouldn’t that make this feature a redundancy?

So, yes and no. Yes, because what you describe is equivalent to MultiDiscrete space. No, because supporting MultiDiscrete separately is cleaner and would be anyway a building block, as done in the flatdim() method:

https://github.com/openai/gym/blob/b2727d6fd396f7fe2b19a541ef3a6ea47559cdb3/gym/spaces/utils.py#L24

Also, MultiDiscrete is a very special case when all the sub-spaces are of the same type. Usually, in that case, you don’t need Dict or Tuple spaces (e.g. multiples Box spaces can be re-combined into one).

[araffin]

but from the two existent implementations I see it return integers.

so, this method is only used in the buffer, good catch, this should return a tuple 😕 (so 1, instead of 1 normally) I have to check again, otherwise, we will need to change slightly the type hint.