[RLlib] Support for nested container action spaces

Gym Platform is an environment used as a benchmark in many academic papers for RL algorithms supporting hybrid action spaces (discrete and continuous).

The action space generated by this environment is a nested gym Tuple, this is common in many environments that have hybrid action space:

Tuple(Discrete(3), Tuple(Box(1,), Box(1,), Box(1,)))

When Rllib tries to initialise itself, it fails when creating the action placeholders in ray/rllib/models/catalog.py when calling get_action_shape and there is no way to customise this without re-patching the library.

You could reproduce the behaviour with the code below:

import ray
from ray import tune
import gym
import gym_platform
from ray.tune.registry import register_env

class Platform(gym.Env):
    def __init__(self, env_config):
        self.env = gym.make("gym_platform:Platform-v0")        
        self.action_space = self.env.action_space
        self.observation_space = self.env.observation_space

    def reset(self):
        return self.env.reset()

    def step(self, action):
        return self.env.step(action)

register_env("platform", lambda config: Platform(config))

ray.init()

tune.run(
    "A3C",
    stop={"training_iteration": 10},
    config={
        "env": "platform",
        "num_workers": 1,
    },
)

[ericl]

Yeah this would be great to have, since non-nested ones work already. Might be good to also support Dict action spaces (which is basically the same as Tuple but with names for the indices).

[sven1977]

PR (https://github.com/ray-project/ray/pull/8019) is out and will be merged in the next few days. I’m closing this issue. It contains an example learning (PPO) script in rllib/examples/nested_action_spaces.py.