[Question] Flatten Discrete box potentially problematic

Question

Flattening Discrete space to Box space may be problematic. The flatten wrapper converts Discrete to Box as a one-hot encoding. Suppose the original space is Discrete(3), then:

0 maps to [1, 0, 0]
1 maps to [0, 1, 0]
3 maps to [0, 0, 1]

When we sample the action space for random actions, it samples the Box, which can produce any of the eight combination of 0s and 1s in a three-element array, namely:

[0, 0, 0],
[0, 0, 1], *
[0, 1, 0], *
[0, 1, 1],
[1, 0, 0], *
[1, 0, 1],
[1, 1, 0],
[1, 1, 1]

Only three of these eight that I’ve starred are useable in the strict sense of the mapping. The unflatten function for a Discrete space uses np.nonzero(x)[0][0], and here’s at table of what the above arrays map to:

+ ------------------ + ---------------- + --------------------------------------------- +
| In Flattened Space | np.nonzero(x)[0] | np.nonzero(x)[0][0] (aka discrete equivalent) |
+ ------------------ + ---------------- + --------------------------------------------- +
| 0, 0, 0            | Error            | Error                                         |
| 0, 0, 1            | [2]              | 2                                             |
| 0, 1, 0            | [1]              | 1                                             |
| 0, 1, 1            | [1, 2]           | 1                                             |
| 1, 0, 0            | [0]              | 0                                             |
| 1, 0, 1            | [0, 2]           | 0                                             |
| 1, 1, 0            | [0, 1]           | 0                                             |
| 1, 1, 1            | [0, 1, 2]        | 0                                             |
+ ------------------ + ---------------- + --------------------------------------------- +

Implications

Obviously, [0, 0, 0] will fail because there is no nonzero. Importantly, only one eighth of the random samples will map to 2. One fourth will map to 1, and one half will map to 0. This has some important implications on exploration, especially if action 2 is the “correct action” throughout much of the simulation. I’m very curious why I have not seen this come up before. This type of skewing in the random sampling can have major implications in the way the algorithm explores and learns, and the problem is exacerbated when Discrete(n), n is large. Am I missing something here?

Solution

This is unique to Discrete spaces. Instead of mapping to a one-hot encoding we could just map to a box of a single element with the appropriate range. Discrete(n) maps to Box(0, n-1, (1,), int) instead of Box(0, 1, (n,), int).

[rusu24edward]

@pseudo-rnd-thoughts Yup, just wanted to provide an example script in case anyone else comes by this post.

Sure, I can add that

[RedTachyon]

I’m not yet completely sure what we want to do here. I’d definitely be opposed to adding an extra space just for the right flattening, unless we completely change our philosophy with regards to the spaces.

If this one change makes it so that we can always ensure the reversibility of flatten -> unflatten, then it’d be tempting.

At the same time, the whole idea of flatten and unflatten is not necessarily well though-out, since it was introduced in some OpenAI code, and then received various updates which might or might not have followed the original intentions.

My interpretation of this functionality is more or less “I don’t care what this fancy space is, I want a fixed sized float32 vector for my NN”. In which case any reversibility fails due to the dtypes. But we don’t actually do any automatic type conversion.

I also feel like flattening into one-hot is much better on the algorithmic side. If you want to “flatten” a discrete action, you definitely don’t want to keep the ordinal structure – i.e. action 1 isn’t between action 0 and action 2 in any meaningful sense, so a one-hot embedding is likely to work much better.