[Enhancement] Support for fallbacks when loading cloudpickle items fail
See original GitHub issueRelated issue #171
As seen in #171, cloudpickle/pickle can easily fail when transferring models between Python versions (and in case some other shenanigans). There is currently no way to address this issue as loading of model tries to un-pickle all pickle objects and does not catch errors.
Two suggestions to deal with this, both of which would be nice to have:
- Provide
custom_objects
dictionary (similar to what we had in SB2), which will override any objects read from file. This could be used to even prevent loading the item from the file, in which case all JSON-ed items can be used as before. - Update
set_parameters
function to only load parameters from the file but not thedata
objects (which include pickled items). This was included in SB2.
@ManifoldFR If you had other suggestions please feel free to throw them here 😃
Issue Analytics
- State:
- Created 3 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
prefect.serializers - Coordinating the world's dataflows
Prefect Python API for JSON and cloudpickle serializers. ... These are stored on the instance so the same settings can be used to...
Read more >9. Release Notes — Numba 0.23.1-py2.7-macosx ... - PyData |
Version 0.23.1¶. This is a bug-fix release to address several regressions introduced in the 0.23.0 release, and a couple other issues.
Read more >Databricks Runtime 8.0 (Unsupported) - Azure - Microsoft Learn
Support char/varchar data type (SPARK-33480); ANSI mode: runtime errors ... Support fallback storage during decommission (SPARK-33545).
Read more >Rasa Open Source Change Log
#11773: Add support for Python 3.10 version. ... #10447: Update rasa data validate to not fail when active_loop is null.
Read more >Release notes — Anaconda documentation
Qt support for macOS M1 (osx-arm64), enabling full use of GUI ... The installer for linux-64 will now error out if a glibc...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
From what I saw the problem I encountered is with the learning rate and clip ratio schedulers (for PPO), in the particular case where a constant scheduler is created using
constant_fn
(I don’t know about other cases though) because pickling/unpickling functions is brittle especially when they arelocals
of a higher-order functon. I think we can look at how PyTorch supports saving state dicts for schedulers. From the source I see they use classes and the__dict__
attribute to save/load state dicts.In general I think the ideas for other fallbacks are good because it allows the user more flexibility.
as a follow-up, I added support for custom objects here: https://github.com/DLR-RM/stable-baselines3/pull/336 and included that in the rl zoo: https://github.com/DLR-RM/rl-baselines3-zoo/pull/69 (so models trained with python 3.6/3.7 can be loaded with python 3.8+, but not retrained yet)