[Bug] Model on saving: "PicklingError: Could not pickle object as excessively deep recursion required."

🐛 Bug

I was going through stable baselines 3’s first example code and after I had downloaded the whole code into my notebook and run it I got this:

---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
d:\python\python37\lib\site-packages\cloudpickle\cloudpickle_fast.py in dump(self, obj)
    601         try:
--> 602             return Pickler.dump(self, obj)
    603         except RuntimeError as e:

d:\python\python37\lib\site-packages\pickle5\pickle.py in dump(self, obj)
    484             self.framer.start_framing()
--> 485         self.save(obj)
    486         self.write(STOP)

d:\python\python37\lib\site-packages\pickle5\pickle.py in save(self, obj, save_persistent_id)
    569                 if issubclass(t, type):
--> 570                     self.save_global(obj)
    571                     return

d:\python\python37\lib\site-packages\pickle5\pickle.py in save_global(self, obj, name)
   1092         if self.proto >= 4:
-> 1093             self.save(module_name)
   1094             self.save(name)

d:\python\python37\lib\site-packages\pickle5\pickle.py in save(self, obj, save_persistent_id)
    600         # Save the reduce() output and finally memoize the object
--> 601         self.save_reduce(obj=obj, *rv)
    602 

d:\python\python37\lib\site-packages\pickle5\pickle.py in save_reduce(self, func, args, state, listitems, dictitems, state_setter, obj)
    684             args = args[1:]
--> 685             save(cls)
    686             save(args)

... last 4 frames repeated, from the frame below ...

d:\python\python37\lib\site-packages\pickle5\pickle.py in save(self, obj, save_persistent_id)
    569                 if issubclass(t, type):
--> 570                     self.save_global(obj)
    571                     return

RecursionError: maximum recursion depth exceeded in comparison

The above exception was the direct cause of the following exception:

PicklingError                             Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_5096\2811699402.py in <module>
     13 model.learn(total_timesteps=int(10))
     14 # Save the agent
---> 15 model.save("dqn_lunar")
     16 del model  # delete trained model to demonstrate loading
     17 

d:\python\python37\lib\site-packages\stable_baselines3\common\base_class.py in save(self, path, exclude, include)
    816         params_to_save = self.get_parameters()
    817 
--> 818         save_to_zip_file(path, data=data, params=params_to_save, pytorch_variables=pytorch_variables)

d:\python\python37\lib\site-packages\stable_baselines3\common\save_util.py in save_to_zip_file(save_path, data, params, pytorch_variables, verbose)
    307     # try to serialize them blindly
    308     if data is not None:
--> 309         serialized_data = data_to_json(data)
    310 
    311     # Create a zip-archive and write our objects there.

d:\python\python37\lib\site-packages\stable_baselines3\common\save_util.py in data_to_json(data)
     97             # from other languages/humans, so we have an
     98             # idea what was being stored.
---> 99             base64_encoded = base64.b64encode(cloudpickle.dumps(data_item)).decode()
    100 
    101             # Use ":" to make sure we do

d:\python\python37\lib\site-packages\cloudpickle\cloudpickle_fast.py in dumps(obj, protocol, buffer_callback)
     71                 file, protocol=protocol, buffer_callback=buffer_callback
     72             )
---> 73             cp.dump(obj)
     74             return file.getvalue()
     75 

d:\python\python37\lib\site-packages\cloudpickle\cloudpickle_fast.py in dump(self, obj)
    607                     "required."
    608                 )
--> 609                 raise pickle.PicklingError(msg) from e
    610             else:
    611                 raise

PicklingError: Could not pickle object as excessively deep recursion required.

Also, I had run the same code in my vscode ide before and I got basically the same error but that error was larger so here I pasted the smaller (jupyter) error.

To Reproduce

Have this piece of code:

import gym

from stable_baselines3 import DQN
from stable_baselines3.common.evaluation import evaluate_policy


# Create environment
env = gym.make('LunarLander-v2')

# Instantiate the agent
model = DQN('MlpPolicy', env, verbose=1)
# Train the agent
model.learn(total_timesteps=int(2e5))
# Save the agent
model.save("dqn_lunar")
del model  # delete trained model to demonstrate loading

# Load the trained agent
# NOTE: if you have loading issue, you can pass `print_system_info=True`
# to compare the system on which the model was trained vs the current one
# model = DQN.load("dqn_lunar", env=env, print_system_info=True)
model = DQN.load("dqn_lunar", env=env)

# Evaluate the agent
# NOTE: If you use wrappers with your environment that modify rewards,
#       this will be reflected here. To evaluate with original rewards,
#       wrap environment in a "Monitor" wrapper before other wrappers.
mean_reward, std_reward = evaluate_policy(model, model.get_env(), n_eval_episodes=10)

# Enjoy trained agent
obs = env.reset()
for i in range(1000):
    action, _states = model.predict(obs, deterministic=True)
    obs, rewards, dones, info = env.step(action)
    env.render()

And then run it.

Expected behavior

I expected the model to save and then run the environment, but it does not.

### System Info

Describe the characteristic of your environment:

• Library installed by: pip install stable_baselines3
• GPU: Nvidia Gtx 1650
• Python version: 3.7
• PyTorch version: no pytorch
• Gym version: 0.21.0

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have read the documentation (required)
• I have provided a minimal working example to reproduce the bug (required)

[araffin]

Hello,

PyTorch version: no pytorch

This is not possible, please follow the issue template and give the output of

import stable_baselines3 as sb3
sb3.get_system_info()

I was going through stable baselines 3’s first example code and after I had downloaded the whole code into my notebook and run it I got this:

Have you tried in a colab notebook instead? (see links in our doc)

It sounds like a problem from your python env.

[Miffyli]

Hey. Could you also include full list of the library versions you have with pip freeze? Could you also try this code without notebooks or VSCode IDE (just in case they are interfering somehow)? The tests pass in CI so the code should work, but maybe some library version is off.