Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Bug] Different outputs with and without DummyVecEnv

See original GitHub issue

Important Note: We do not do technical support, nor consulting and don’t answer personal questions per email. Please post your question on the RL Discord, Reddit or Stack Overflow in that case.

If your issue is related to a custom gym environment, please use the custom gym env template.

🐛 Bug

A clear and concise description of what the bug is.

To Reproduce

Steps to reproduce the behavior.

Please try to provide a minimal example to reproduce the bug. Error messages and stack traces are also helpful.

Please use the markdown code blocks for both code and stack traces.

import gym
import random
import numpy as np

from stable_baselines3.common.vec_env import DummyVecEnv

seed = 0
random.seed(seed)
np.random.seed(seed)

env = gym.make('Hopper-v2')
env.seed(seed)

action = env.action_space.sample()
print('Sampled action:', action)

print('---No DummyVecEnv---')
obs = env.reset()
print('Starting state:', obs)
obs, reward, done, info = env.step(action)
print('After step:', obs, reward)

print('---With DummyVecEnv---')
env2 = gym.make('Hopper-v2')
env2.seed(seed)
env2 = DummyVecEnv([lambda: env2])
obs = env2.reset()
print('Starting state:', obs)
obs, reward, done, info = env2.step(action)
print('After step:', obs, reward)

Sampled action: [-0.19805549  0.2225707   0.20607542]
---No DummyVecEnv---
Starting state: [ 1.24769787e+00 -4.59026476e-03 -4.83472364e-03  3.13270239e-03
  4.12755577e-03  1.06635776e-03  2.29496561e-03  4.36249915e-04
  4.35072424e-03  3.15853554e-03 -4.97261500e-03]
After step: [ 1.24738165 -0.00560824 -0.00596334  0.00300481  0.00530661 -0.01191321
 -0.08141437 -0.25273907 -0.28655767 -0.0302732   0.29911847] 0.9942287844396452
---With DummyVecEnv---
hopper.xml
Starting state: [[ 1.24769787e+00 -4.59026476e-03 -4.83472364e-03  3.13270239e-03
   4.12755577e-03  1.06635776e-03  2.29496561e-03  4.36249915e-04
   4.35072424e-03  3.15853554e-03 -4.97261500e-03]]
After step: [[ 1.24743864 -0.00595406 -0.00586327  0.00222761  0.00300255 -0.04029498
  -0.06715943 -0.34103212 -0.26118558 -0.22920861 -0.27597505]] [0.9803258]

Expected behavior

A clear and concise description of what you expected to happen.

### System Info

Describe the characteristic of your environment:

Describe how the library was installed (pip, docker, source, …)
GPU models and configuration: A100-SXM4-40GB NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.6
Python version: Python 3.8.12
PyTorch version: torch 1.11.0a0+bfe5ad2
Gym version: gym 0.25.2
Versions of any other relevant libraries

You can use sb3.get_system_info() to print relevant packages info: OS: Linux-5.4.0-58-generic-x86_64-with-glibc2.10 #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 Python: 3.8.12 Stable-Baselines3: 1.6.0 PyTorch: 1.11.0a0+bfe5ad2 GPU Enabled: True Numpy: 1.22.0 Gym: 0.25.2

({‘OS’: ‘Linux-5.4.0-58-generic-x86_64-with-glibc2.10 #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020’, ‘Python’: ‘3.8.12’, ‘Stable-Baselines3’: ‘1.6.0’, ‘PyTorch’: ‘1.11.0a0+bfe5ad2’, ‘GPU Enabled’: ‘True’, ‘Numpy’: ‘1.22.0’, ‘Gym’: ‘0.25.2’}, ‘OS: Linux-5.4.0-58-generic-x86_64-with-glibc2.10 #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020\nPython: 3.8.12\nStable-Baselines3: 1.6.0\nPyTorch: 1.11.0a0+bfe5ad2\nGPU Enabled: True\nNumpy: 1.22.0\nGym: 0.25.2\n’)

Additional context

Add any other context about the problem here.

Checklist

[Yes ] I have checked that there is no similar issue in the repo (required)
[Yes ] I have read the documentation (required)
[Yes] I have provided a minimal working example to reproduce the bug (required)