Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Feature discussion] Improving model save format

See original GitHub issue

Continuation on the discussion in #312 related to improving save file format.

Issue with current way of storing/loading models

Cloudpickle can have issues with different version of Python (mentioned here)
Cloudpickle itself mentions that it shouldn’t be used for long-term storage (something RL people might want).
Since everything is stored into one serialized file, a single corrupted part prevents easily loading rest of the things. Example: My saved model had Tensorflow code that did not work on different Python version, thus resulting error upon loading. I only needed the model parameters, but I couldn’t load them because one of the class items broke deserialization.
A minor point, but cloudpickle is Python-specific, so accessing anything inside these files requires going through Python+ cloudpickle.

Suggestion Using Python-native Zipfile to manually construct a zipped archive. Each entry in the zip file would be a separately serialized object. E.g. A2C would have one entry for “gamma”, another for “n_steps” and so on. Finally the zip file would include one more entry for the model parameters, serialized with numpy’s savez or just with pickle as is done now.

Since many learning algorithms store policies, obs/action spaces, functions and other custom Python objects, it might make sense to store these using pickle. However now if these become non-deserializable (e.g. new Python version), we can still load other items. This works especially well with load_parameters, which could directly access the parameters in zip file rather than trying to deserialize the whole file.

An alternative would be to use JSON to serialize class parameters, but this may not play well with custom functions and the likes. JSON would offer very human-readable and long-lasting files, though. I also took a look at jsonpickle, but I am not sure if relying on another library is a better idea than sticking with pickle for serializing things.

Issue Analytics

State:
Created 4 years ago
Reactions:1
Comments:10

Top GitHub Comments

1reaction

Miffylicommented, Aug 29, 2019

@AdamGleave

I agree on both points. Hence I would just cloudpickle/pickle the non-JSON-able objects and leave it at that.

One open point is the storing of model parameters: Should this be Numpy savez object or something more universal? I figure savez (and save) format is long-lasting and easy enough for people to use the values somewhere else-

I did quick experiments with storing objects with this new format, and here is the JSON (class parameters) part of the saved model. Objects that are serialized with cloudpickle (:serialized:) also include first-level members of the object so human reader can get some idea what the serialization contains. These are not used in any way when reading the file (only :serialized: is read and deserialized).

Example JSON file of class parameters

{
    "gamma": 0.99,
    "timesteps_per_batch": 1024,
    "max_kl": 0.01,
    "cg_iters": 10,
    "lam": 0.98,
    "entcoeff": 0.0,
    "cg_damping": 0.01,
    "vf_stepsize": 0.0003,
    "vf_iters": 3,
    "hidden_size_adversary": 100,
    "adversary_entcoeff": 0.001,
    "expert_dataset": null,
    "g_step": 1,
    "d_step": 1,
    "d_stepsize": 0.0003,
    "using_gail": false,
    "verbose": 0,
    "policy": {
        ":type:": "<class 'abc.ABCMeta'>",
        ":serialized:": "gASVMgAAAAAAAACMIHN0YWJsZV9iYXNlbGluZXMuY29tbW9uLnBvbGljaWVzlIwJTWxwUG9saWN5lJOULg==",
        "__module__": "stable_baselines.common.policies",
        "__doc__": "\n    Policy object that implements actor critic, using a MLP (2 layers of 64)\n\n    :param sess: (TensorFlow session) The current TensorFlow session\n    :param ob_space: (Gym Space) The observation space of the environment\n    :param ac_space: (Gym Space) The action space of the environment\n    :param n_env: (int) The number of environments to run\n    :param n_steps: (int) The number of steps to run for each environment\n    :param n_batch: (int) The number of batch to run (n_envs * n_steps)\n    :param reuse: (bool) If the policy is reusable or not\n    :param _kwargs: (dict) Extra keyword arguments for the nature CNN feature extraction\n    ",
        "__init__": "<function MlpPolicy.__init__ at 0x7feb24b73268>",
        "__abstractmethods__": "frozenset()",
        "_abc_registry": "<_weakrefset.WeakSet object at 0x7feb24c340f0>",
        "_abc_cache": "<_weakrefset.WeakSet object at 0x7feb24c34128>",
        "_abc_negative_cache": "<_weakrefset.WeakSet object at 0x7feb24c34198>",
        "_abc_negative_cache_version": 52
    },
    "observation_space": {
        ":type:": "<class 'gym.spaces.discrete.Discrete'>",
        ":serialized:": "gASVTgsAAAAAAACME2d5bS5zcGFjZXMuZGlzY3JldGWUjAhEaXNjcmV0ZZSTlCmBlH2UKIwBbpRLCowFc2hhcGWUKYwFZHR5cGWUjAVudW1weZSMBWR0eXBllJOUjAJpOJRLAEsBh5RSlChLA4wBPJROTk5K/////0r/////SwB0lGKMCW5wX3JhbmRvbZSMFG51bXB5LnJhbmRvbS5fcGlja2xllIwSX19yYW5kb21zdGF0ZV9jdG9ylJOUjAdNVDE5OTM3lIWUUpR9lCiMDWJpdF9nZW5lcmF0b3KUaBSMBXN0YXRllH2UKIwDa2V5lIwVbnVtcHkuY29yZS5tdWx0aWFycmF5lIwMX3JlY29uc3RydWN0lJOUaAiMB25kYXJyYXmUk5RLAIWUQwFilIeUUpQoSwFNcAKFlGgKjAJ1NJRLAEsBh5RSlChLA2gOTk5OSv////9K/////0sAdJRiiULACQAAW6DUsAzWsNOFcUrEouywwBF3V4lPOLX1Xz38oPU+Ft9cmSdH+UdKVn9ZtIUnq6RP6Mn1mqpv9eamqbx1OvrRab59kHcR+kFSd8mlTnwjWpsL797ktjG68OFSTa52PM4iqJpJOdzQFK7Ya4YlvS238Xav/nWTBJgNoxEQ82f0pka7olRWNoR54gIVszlaFWckVM0PArDWId+aJ+TWDqHpOkHom217lPxCqiHJvG/jZ3CyOLA5ID7nwjQRQ2x/6FLJnW2hAPwje7eDUmmSxhm3J2OAuzqZPI30/o9P7OiFAN6aM4wb8Mg3fVua1xIxsgI/88AkcSMhpYXgHnxhf6UkXb/c3pwQ6HQhCWrPEaMgonDwGDMazfV2oSaUmhzWxVP27GpNKRX2OryVeJoKAcCvsXRXkrigucUQiESyksdsTE1HQYmxZeAM2PhNL3hG8jwaUZqsIADZk1vUGMQXX5C9ZMUkHg/ESKviQwacFSKZtt/7RF8iSFB/0zFF/JwQ+xtwueDgwWRB2MnS8q/JvkC7uriGXG1qDEfjaNAM751mZy/X4PQv0AH+YE4o5pYplmO/+hwix/xec3whdl4+KAinMUpLGJMGKERO/i174h8YAmhMepSEN6yQPuTszOCwDJOZGdfZ+ap3SYzC6PIG3ELgXf69Ek++j0zvpRkLP1rCLprf76pd9hgSLrX2kaT1jAbUgnJhdJdEhyJy+4JbXSxc9P6hPLiHZeicFHzXSWTE0hJ0H8PuP8RNEEJe+oL9E4xgSJLLNXFhQd63zVkg6v9bxR0X1pzag8kC1UtF6oH9xhBUoAavFA2J6J+PsHzTsZE5YOGw1YhKv8camBtB+InSXLeL56wwXYc/m+D+fWJeL1P8/hHDB5VuqIajBaEHppbTtH3osXyd9ewomecdRmLYaI6GP+Vy2wlu8/AlCp7fcZChr4JOwcuIqfhU+Ij063ALAQWwglAuHzgO/BAL6Wqw36LCkiARi4WbSROuj3CRYLqhk4AizvA4zDEz7cUCjWm4tS73WkDVeN0U4sa3PDY2uNpwF7Nj9jl6+IwSCv2HDqHvPkOcjwORfQMaJx7i990yst8p+naj13pZCQB9semWWGxbi3LY21UVG8lg5lvmizJBHFpCYTBV4AlaHnk9BBXnmnsZftuFtHvDDXAV3nvvuouP2qQhy3VE3dbmH9xazRQRy4dENoB4ydClla5yAzv6G3fxMs1uxOf5Vo+MZrlmWxCUsitH6A0p2qq1ulMixe73W51u/bcb6JquDCgtFF8xIm8ZGzP+jyvQSi0oPCWH+dIKipQlc7thg8fJ3zxQ0GvNAzmXqoOXr5+OkClJnGA+WWGYEl4Jy5y5UKOKtTmv91BZbciXByuGnGEDk4z/aWrdHBRzLrIqRpo8rvu/XRDpSt1Crh/woK4l1CtohhHfgybR0+dZzHoew5m9Dd0slOdkUFB0YJDFAouVDhbaKfmmxZO14ZS3Lt43yBEiv67U5cHwNdudZP7bnQV8bP2JF0a9atXC+h/iHBZgMJSYvByflIEFMkZUAUyeq3mDeudXTj6f6EIOduOuqGLNgZui9Sy303WUpyD+IHZQt+PwbFyae+gaGQ4wFN7zp0B2fnNYwb7K0RQImf4/4gkABgQp/J5h4JVQXxt+f06dJLHroswT2MhUR2b/7go0O/D7GewQaB4SWLmRpzmCFsOHp7akEiRPFLQrOXvg0jgiUUptf0cW+LuyeLo3bYbs9k8bkvOX/VgbdSujE6SvGA6oSe6ztUT11dvw8+aft0IWMa9EnJzILXa1NShJVM6nTu2r2cZRG7vaqQE4OJTr8XQZw8qMgBbStI/NRLfgbSA+XJIVV/L6ixpRYQCC5lHlbHcnSMoJi0pSYZ5fr71eWjDijhULcH6csQnIO/YNZKKJF3Q0xbxaqP6aq7C3RsIRoQsls2sSv9/uZ4IQ4D/ioe8m397X/Zs6QJZ/hYK0nXZn1c6dDSOaDJibVtkKBUTdWhbo6e93y1bwMWC+PYkUkxAmEx2Y1SnntwoUmiJYH179BCQLignL1omY+HagH5ZP5odULYNdU/A4oJHoSEgPxYDtnta4VRsu21fN5a9Tx/+0rA08cghouw32yH4A7qjboJdpX3b7YhKinhhzjFtNLmjy79k6o5KDLM9vb6oswJFAp82SEFkLCiycVvUTTE7pwOB3PpRzcHUdHTBfnPF0FuPp//hjNv7srG5XjzuiH9wvmK1oYxsnvm2vqcomlBE0gjiW/nsqXybwJtqpGxYu8ipbqsYpqDN+QzRtu+S+jRfnwdkcAazG9cpasrAoUDhWTYzVbgNJBdraafYa6WxJNXdlB26/CkgXQEAJ28qZDLccx7h20V+0NH/Mj9fMd2+jOq9Ksd50k5MQqzWosk4dynbUhbGM/atehHVf10fxTWftyXBL1aI+FvaGKJj/NxcxnhWhNwFn5zd0tBgI6BpXZup/vdsMrc5S+ZrRafbQi5Rhhjr88gZHmOcBMljLe7cOLtmXhqk/HrI+47ftamQ8Oy6MGY+F6RcxSOWlQpU4w3MjhfnKqw9NEc18+Bjg9O/+60oj9SNNteMW8xL/d2D6+PS0k8R18oFQl/zOk9yBbgeqYkYTZ39pFygk1KdbbgybFkj/VMwkWEKGy+qIFWP2NWbVHf0nqd/U8rVfvo9PHcDF92HphtkYGqqMgbwfZhQWJfXO8Ijw7f+vvWZzgpt2khJSwYaBiJH9czE2rg5ps4CDlQ64nKnGWE/USQCAfyicP7MesC2+wE1gmLMR4WCDLHq7YhaRHTpUPmYqWx9xuB9AVIIL147OZPeYTfqTHDI53kbVnw4nqds4pAa64MvQtsZT/r46alHXeWJDvIwF3S2E1hHoGsXSt62RPtZpbcZ1hl4wZqaM3tkuX6ZIWsFJ7+Q8pGrPrvWdaFRMg/g/QDE0SmWu706c3q0yPYoEdqxQldbeEveY81V0sR6rRftLnWWGJ637ujgDNYujxwCZJpJ52hy8gdhTgli78HKd9nhoqx2wo3BS403A9CCAcSOCTFR8zCde6foFoPCp0pUp6DAtNLwvpmdNBTRlfF4cLh38FUvfM8r6gwNs3WFQ63pyy9fIHFPI6ApUFcgZ1VbmnA9jUOiJ6KKhQygLFZ/q91ZCqCYTgDfUUj8UqWXfpa4Ucmljc6KX2JDgDhKFa5wvYUiIHO6LdMClSU6sc5FxVpMJOx+jFgD2O7HII9kueQM9QjKjFo7H4ZxX1uX88a+oV04JkJDIDSd3iTFqqfOn//1KrCutmxdLoyHJ0PeizgDmcQbFsrGi0qya1LsW2PdIuwXscMRJ7ZRJlHSUYowDcG9zlEsDdYwJaGFzX2dhdXNzlEsAjAVnYXVzc5RHAAAAAAAAAAB1YnViLg==",
        "n": 10,
        "shape": [],
        "dtype": "int64",
        "np_random": "RandomState(MT19937)"
    },
    "action_space": {
        ":type:": "<class 'gym.spaces.discrete.Discrete'>",
        ":serialized:": "gASVTgsAAAAAAACME2d5bS5zcGFjZXMuZGlzY3JldGWUjAhEaXNjcmV0ZZSTlCmBlH2UKIwBbpRLCowFc2hhcGWUKYwFZHR5cGWUjAVudW1weZSMBWR0eXBllJOUjAJpOJRLAEsBh5RSlChLA4wBPJROTk5K/////0r/////SwB0lGKMCW5wX3JhbmRvbZSMFG51bXB5LnJhbmRvbS5fcGlja2xllIwSX19yYW5kb21zdGF0ZV9jdG9ylJOUjAdNVDE5OTM3lIWUUpR9lCiMDWJpdF9nZW5lcmF0b3KUaBSMBXN0YXRllH2UKIwDa2V5lIwVbnVtcHkuY29yZS5tdWx0aWFycmF5lIwMX3JlY29uc3RydWN0lJOUaAiMB25kYXJyYXmUk5RLAIWUQwFilIeUUpQoSwFNcAKFlGgKjAJ1NJRLAEsBh5RSlChLA2gOTk5OSv////9K/////0sAdJRiiULACQAAW6DUsAzWsNOFcUrEouywwBF3V4lPOLX1Xz38oPU+Ft9cmSdH+UdKVn9ZtIUnq6RP6Mn1mqpv9eamqbx1OvrRab59kHcR+kFSd8mlTnwjWpsL797ktjG68OFSTa52PM4iqJpJOdzQFK7Ya4YlvS238Xav/nWTBJgNoxEQ82f0pka7olRWNoR54gIVszlaFWckVM0PArDWId+aJ+TWDqHpOkHom217lPxCqiHJvG/jZ3CyOLA5ID7nwjQRQ2x/6FLJnW2hAPwje7eDUmmSxhm3J2OAuzqZPI30/o9P7OiFAN6aM4wb8Mg3fVua1xIxsgI/88AkcSMhpYXgHnxhf6UkXb/c3pwQ6HQhCWrPEaMgonDwGDMazfV2oSaUmhzWxVP27GpNKRX2OryVeJoKAcCvsXRXkrigucUQiESyksdsTE1HQYmxZeAM2PhNL3hG8jwaUZqsIADZk1vUGMQXX5C9ZMUkHg/ESKviQwacFSKZtt/7RF8iSFB/0zFF/JwQ+xtwueDgwWRB2MnS8q/JvkC7uriGXG1qDEfjaNAM751mZy/X4PQv0AH+YE4o5pYplmO/+hwix/xec3whdl4+KAinMUpLGJMGKERO/i174h8YAmhMepSEN6yQPuTszOCwDJOZGdfZ+ap3SYzC6PIG3ELgXf69Ek++j0zvpRkLP1rCLprf76pd9hgSLrX2kaT1jAbUgnJhdJdEhyJy+4JbXSxc9P6hPLiHZeicFHzXSWTE0hJ0H8PuP8RNEEJe+oL9E4xgSJLLNXFhQd63zVkg6v9bxR0X1pzag8kC1UtF6oH9xhBUoAavFA2J6J+PsHzTsZE5YOGw1YhKv8camBtB+InSXLeL56wwXYc/m+D+fWJeL1P8/hHDB5VuqIajBaEHppbTtH3osXyd9ewomecdRmLYaI6GP+Vy2wlu8/AlCp7fcZChr4JOwcuIqfhU+Ij063ALAQWwglAuHzgO/BAL6Wqw36LCkiARi4WbSROuj3CRYLqhk4AizvA4zDEz7cUCjWm4tS73WkDVeN0U4sa3PDY2uNpwF7Nj9jl6+IwSCv2HDqHvPkOcjwORfQMaJx7i990yst8p+naj13pZCQB9semWWGxbi3LY21UVG8lg5lvmizJBHFpCYTBV4AlaHnk9BBXnmnsZftuFtHvDDXAV3nvvuouP2qQhy3VE3dbmH9xazRQRy4dENoB4ydClla5yAzv6G3fxMs1uxOf5Vo+MZrlmWxCUsitH6A0p2qq1ulMixe73W51u/bcb6JquDCgtFF8xIm8ZGzP+jyvQSi0oPCWH+dIKipQlc7thg8fJ3zxQ0GvNAzmXqoOXr5+OkClJnGA+WWGYEl4Jy5y5UKOKtTmv91BZbciXByuGnGEDk4z/aWrdHBRzLrIqRpo8rvu/XRDpSt1Crh/woK4l1CtohhHfgybR0+dZzHoew5m9Dd0slOdkUFB0YJDFAouVDhbaKfmmxZO14ZS3Lt43yBEiv67U5cHwNdudZP7bnQV8bP2JF0a9atXC+h/iHBZgMJSYvByflIEFMkZUAUyeq3mDeudXTj6f6EIOduOuqGLNgZui9Sy303WUpyD+IHZQt+PwbFyae+gaGQ4wFN7zp0B2fnNYwb7K0RQImf4/4gkABgQp/J5h4JVQXxt+f06dJLHroswT2MhUR2b/7go0O/D7GewQaB4SWLmRpzmCFsOHp7akEiRPFLQrOXvg0jgiUUptf0cW+LuyeLo3bYbs9k8bkvOX/VgbdSujE6SvGA6oSe6ztUT11dvw8+aft0IWMa9EnJzILXa1NShJVM6nTu2r2cZRG7vaqQE4OJTr8XQZw8qMgBbStI/NRLfgbSA+XJIVV/L6ixpRYQCC5lHlbHcnSMoJi0pSYZ5fr71eWjDijhULcH6csQnIO/YNZKKJF3Q0xbxaqP6aq7C3RsIRoQsls2sSv9/uZ4IQ4D/ioe8m397X/Zs6QJZ/hYK0nXZn1c6dDSOaDJibVtkKBUTdWhbo6e93y1bwMWC+PYkUkxAmEx2Y1SnntwoUmiJYH179BCQLignL1omY+HagH5ZP5odULYNdU/A4oJHoSEgPxYDtnta4VRsu21fN5a9Tx/+0rA08cghouw32yH4A7qjboJdpX3b7YhKinhhzjFtNLmjy79k6o5KDLM9vb6oswJFAp82SEFkLCiycVvUTTE7pwOB3PpRzcHUdHTBfnPF0FuPp//hjNv7srG5XjzuiH9wvmK1oYxsnvm2vqcomlBE0gjiW/nsqXybwJtqpGxYu8ipbqsYpqDN+QzRtu+S+jRfnwdkcAazG9cpasrAoUDhWTYzVbgNJBdraafYa6WxJNXdlB26/CkgXQEAJ28qZDLccx7h20V+0NH/Mj9fMd2+jOq9Ksd50k5MQqzWosk4dynbUhbGM/atehHVf10fxTWftyXBL1aI+FvaGKJj/NxcxnhWhNwFn5zd0tBgI6BpXZup/vdsMrc5S+ZrRafbQi5Rhhjr88gZHmOcBMljLe7cOLtmXhqk/HrI+47ftamQ8Oy6MGY+F6RcxSOWlQpU4w3MjhfnKqw9NEc18+Bjg9O/+60oj9SNNteMW8xL/d2D6+PS0k8R18oFQl/zOk9yBbgeqYkYTZ39pFygk1KdbbgybFkj/VMwkWEKGy+qIFWP2NWbVHf0nqd/U8rVfvo9PHcDF92HphtkYGqqMgbwfZhQWJfXO8Ijw7f+vvWZzgpt2khJSwYaBiJH9czE2rg5ps4CDlQ64nKnGWE/USQCAfyicP7MesC2+wE1gmLMR4WCDLHq7YhaRHTpUPmYqWx9xuB9AVIIL147OZPeYTfqTHDI53kbVnw4nqds4pAa64MvQtsZT/r46alHXeWJDvIwF3S2E1hHoGsXSt62RPtZpbcZ1hl4wZqaM3tkuX6ZIWsFJ7+Q8pGrPrvWdaFRMg/g/QDE0SmWu706c3q0yPYoEdqxQldbeEveY81V0sR6rRftLnWWGJ637ujgDNYujxwCZJpJ52hy8gdhTgli78HKd9nhoqx2wo3BS403A9CCAcSOCTFR8zCde6foFoPCp0pUp6DAtNLwvpmdNBTRlfF4cLh38FUvfM8r6gwNs3WFQ63pyy9fIHFPI6ApUFcgZ1VbmnA9jUOiJ6KKhQygLFZ/q91ZCqCYTgDfUUj8UqWXfpa4Ucmljc6KX2JDgDhKFa5wvYUiIHO6LdMClSU6sc5FxVpMJOx+jFgD2O7HII9kueQM9QjKjFo7H4ZxX1uX88a+oV04JkJDIDSd3iTFqqfOn//1KrCutmxdLoyHJ0PeizgDmcQbFsrGi0qya1LsW2PdIuwXscMRJ7ZRJlHSUYowDcG9zlEsDdYwJaGFzX2dhdXNzlEsAjAVnYXVzc5RHAAAAAAAAAAB1YnViLg==",
        "n": 10,
        "shape": [],
        "dtype": "int64",
        "np_random": "RandomState(MT19937)"
    },
    "n_envs": 1,
    "_vectorize_action": true,
    "policy_kwargs": {}
}

1reaction

Miffylicommented, Aug 28, 2019

If we encode the bytes of non-JSON-able objects into a string and save it in the JSON, then other languages can still read the JSON file all nice but can’t really do much with these byte-strings (they do not have any meaning outside Python/Cloudpickle/Pickle).

Did a quick run of what can be JSONiable and what needs pickling. As expected, spaces and policy are the main issue as they contain classes and functions. However, in my opinion, these can stay byte-serialized as they only have proper meaning in Python. These are also the parts that will likely break something when changing Python or library versions.

One thing we can do with this much like Keras does: When loading model, we can replace some of the items from a file with something else. E.g. A2C.load(..., custom_objects={'policy': other_policy}) would skip reading policy from the file and instead uses the provided one other_policy. This is already in load function but I would move it bit deeper so that it could prevent loading invalid objects (and thus crashing).

<class 'stable_baselines.a2c.a2c.A2C'>
        gamma                     <class 'float'>                          JSON      
        n_steps                   <class 'int'>                            JSON      
        vf_coef                   <class 'float'>                          JSON      
        ent_coef                  <class 'float'>                          JSON      
        max_grad_norm             <class 'float'>                          JSON      
        learning_rate             <class 'float'>                          JSON      
        alpha                     <class 'float'>                          JSON      
        epsilon                   <class 'float'>                          JSON      
        lr_schedule               <class 'str'>                            JSON      
        verbose                   <class 'int'>                            JSON      
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON      
<class 'stable_baselines.acer.acer_simple.ACER'>
        gamma                     <class 'float'>                          JSON      
        n_steps                   <class 'int'>                            JSON      
        q_coef                    <class 'float'>                          JSON      
        ent_coef                  <class 'float'>                          JSON      
        max_grad_norm             <class 'int'>                            JSON      
        learning_rate             <class 'float'>                          JSON      
        lr_schedule               <class 'str'>                            JSON      
        rprop_alpha               <class 'float'>                          JSON      
        rprop_epsilon             <class 'float'>                          JSON      
        replay_ratio              <class 'int'>                            JSON      
        replay_start              <class 'int'>                            JSON      
        verbose                   <class 'int'>                            JSON      
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON      
<class 'stable_baselines.acktr.acktr_disc.ACKTR'>
        gamma                     <class 'float'>                          JSON      
        nprocs                    <class 'int'>                            JSON      
        n_steps                   <class 'int'>                            JSON      
        vf_coef                   <class 'float'>                          JSON      
        ent_coef                  <class 'float'>                          JSON      
        vf_fisher_coef            <class 'float'>                          JSON      
        max_grad_norm             <class 'float'>                          JSON      
        learning_rate             <class 'float'>                          JSON      
        kfac_clip                 <class 'float'>                          JSON      
        lr_schedule               <class 'str'>                            JSON      
        verbose                   <class 'int'>                            JSON      
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON      
<class 'stable_baselines.deepq.dqn.DQN'>
        checkpoint_path           <class 'NoneType'>                       JSON      
        param_noise               <class 'bool'>                           JSON      
        learning_starts           <class 'int'>                            JSON      
        train_freq                <class 'int'>                            JSON      
        prioritized_replay        <class 'bool'>                           JSON      
        prioritized_replay_eps    <class 'float'>                          JSON      
        batch_size                <class 'int'>                            JSON      
        target_network_update_freq <class 'int'>                            JSON      
        checkpoint_freq           <class 'int'>                            JSON      
        prioritized_replay_alpha  <class 'float'>                          JSON      
        prioritized_replay_beta0  <class 'float'>                          JSON      
        prioritized_replay_beta_iters <class 'NoneType'>                       JSON      
        exploration_final_eps     <class 'float'>                          JSON      
        exploration_fraction      <class 'float'>                          JSON      
        learning_rate             <class 'float'>                          JSON      
        gamma                     <class 'float'>                          JSON      
        verbose                   <class 'int'>                            JSON      
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON      
<class 'stable_baselines.ppo2.ppo2.PPO2'>
        gamma                     <class 'float'>                          JSON      
        n_steps                   <class 'int'>                            JSON      
        vf_coef                   <class 'float'>                          JSON      
        ent_coef                  <class 'float'>                          JSON      
        max_grad_norm             <class 'float'>                          JSON      
        learning_rate             <class 'float'>                          JSON      
        lam                       <class 'float'>                          JSON      
        nminibatches              <class 'int'>                            JSON      
        noptepochs                <class 'int'>                            JSON      
        cliprange                 <class 'float'>                          JSON      
        cliprange_vf              <class 'float'>                          JSON      
        verbose                   <class 'int'>                            JSON      
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON      
<class 'stable_baselines.trpo_mpi.trpo_mpi.TRPO'>
        gamma                     <class 'float'>                          JSON      
        timesteps_per_batch       <class 'int'>                            JSON      
        max_kl                    <class 'float'>                          JSON      
        cg_iters                  <class 'int'>                            JSON      
        lam                       <class 'float'>                          JSON      
        entcoeff                  <class 'float'>                          JSON      
        cg_damping                <class 'float'>                          JSON      
        vf_stepsize               <class 'float'>                          JSON      
        vf_iters                  <class 'int'>                            JSON      
        hidden_size_adversary     <class 'int'>                            JSON      
        adversary_entcoeff        <class 'float'>                          JSON      
        expert_dataset            <class 'NoneType'>                       JSON      
        g_step                    <class 'int'>                            JSON      
        d_step                    <class 'int'>                            JSON      
        d_stepsize                <class 'float'>                          JSON      
        using_gail                <class 'bool'>                           JSON      
        verbose                   <class 'int'>                            JSON      
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON