Python 3.8: "an integer is required (got type bytes)" when loading models saved under older Python versions

Describe the bug

There is an issue loading trained agent files in Python 3.8.

Code example

I checked this on the rl-baselines3-zoo repository.

(base) manifold$ python enjoy.py --algo sac --env Pendulum-v0
Loading latest experiment, id=1
Traceback (most recent call last):
  File "enjoy.py", line 225, in <module>
    main()
  File "enjoy.py", line 141, in main
    model = ALGOS[algo].load(model_path, env=env, **kwargs)
  File "/home/manifold/stable-baselines3/stable_baselines3/common/base_class.py", line 561, in load
    data, params, pytorch_variables = load_from_zip_file(path, device=device)
  File "/home/manifold/stable-baselines3/stable_baselines3/common/save_util.py", line 390, in load_from_zip_file
    data = json_to_data(json_data)
  File "/home/manifold/stable-baselines3/stable_baselines3/common/save_util.py", line 164, in json_to_data
    deserialized_object = cloudpickle.loads(base64_object)
TypeError: an integer is required (got type bytes)

System Info Describe the characteristic of your environment:

• SB3 downloaded from git trunk
• Python version 3.8.5
• PyTorch version 1.6.0
• Gym version: latest from git
• Cloudpickle 1.6.0

Additional context

I printed out info and saw that it occurs when loading the data key learning_rate on the SAC example above. For policies trained using recent versions of SB3 it happens for lr_schedule.

[pablogsal]

Seeing this I agree having a nice fallback would be more elegant until things are sorted out upstream.

✋ Hi, CPython maintainer here. Unfortunately, there is nothing to be done in our side, as the signature of code objects is not considered stable between versions. The constructor is purpously not even documented and we have this advice here:

If you instantiate any of these types, note that signatures may vary between Python versions.

(https://docs.python.org/3/library/types.html#standard-interpreter-types)

You can also use this new convenience method that we added to make the transition easier for people using code objects:

https://docs.python.org/3/library/types.html#types.CodeType.replace

[araffin]

closing this in favor of #172 (and as it is not an issue coming from SB3)