Action values become a negative float value after 100 total timestep

🐛 Bug

Action values become a negative float value after 100 total timestep

To Reproduce

My env file

class NQEnv(gym.Env):
    metadata = {'render.modes': ['console']}
    def __init__(self):
        self.action_space = spaces.Discrete(6)
        self.observation_space = spaces.Box(low=0, high=255, shape=(70, 70, 1), dtype=np.uint8)
...
actions = ['left_arrow', 'right_arrow', 'up_arrow', 'down_arrow', 'spacebar', 'nothing']
        try:
            print("action: " + str(actions[action]) + "  which stage: " + str(self._stage))
...

main.py

from stable_baselines3 import SAC
from stable_baselines3.common.cmd_util import DummyVecEnv
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.sac.policies import CnnPolicy
..

env = DummyVecEnv([lambda: NQEnv()])
model = SAC(CnnPolicy, env, verbose=2).learn(total_timesteps=5000, log_interval=1)

I run main.py and it works fine until the total timesteps hit 100 (3 minutes?). Action values become a negative float value afterwards as shown on the following screenshots.

Expected behavior

Action values continue to be integer

### System Info

Describe the characteristic of your environment: -macos big sur -python 3.8.6 -tf-nightly: 2.5.0.dev20201204 -numpy : 1.19.4 -stable-baselines3: 0.10.0 -gym 0.17.3

pip check doesn’t show any broken dependencies.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have read the documentation (required)
• I have provided a minimal working example to reproduce the bug (required)

[Miffyli]

Thanks for the kind words! These help us stay motivated to contribute projects like this 😃

Lets keep this issue open until @araffin comments on my comment above about possible enhancement.

[cosmir17]

@Miffyli, I am super excited to use this library!