[Feature Proposal] `load(...)` but only for model parameters

Problem Currently ActorCriticRLModel implements load(...) so that it creates the network/model from zero and then loads stored parameters. This creates hefty amount of overhead, and bit of a memory leak if you are not careful, when you only want to change to different set of model parameters, e.g. when evaluating bunch of stored model.

Suggestion

• New function for ActorCriticRLModel named load_parameters(str:filename, ...) which takes in path to a saved model (from save(...)) and only loads the model parameters for the agent. Most of this can be done by re-using code already in load(...), with some re-structuring.
• Required TF assign operations would be created during creation of the graph, after setup_model.

We have been using hacky-ish setup like this for our work with success, and thought it could be useful for others as well.

[araffin]

My idea was more to keep the pickle file in the general case but:

1. save parameters (trained weights) in a dict rather than a list (which is currently the case)
2. allow the use to load/save the parameters only from/in a numpy archive, the use case is you have one pickle file that contains all the hyperparameters for an experiments and several numpy archives for different results of the optimization with the same hyperparams

[araffin]

Hello,

I think this is a good feature to have. Especially when you want to deploy your model for instance (where you just need the weights of the policy) .

I have some additional remarks, questions:

If we implement a load_parameters (that takes either a list of numpy arrays or a path to a file) we should also implement a save_parameters (save weight to a file) and get_parameters (returns list/dict of numpy arrays representing the weights)

As I understand, you were suggesting to still use the same .pkl file that contains both parameters and data (e.g. hyperparams) for loading the parameters? I think in that case have a different format for saving the parameters only (e.g. numpy dict) would make more sense.

New function for ActorCriticRLModel

I would do that for all models.

Required TF assign operations would be created during creation of the graph, after setup_model.

Why not when calling the load_parameters?

Note that if we change the parameters format to be a dict of numpy arrays instead of a list, this would solve that issue too: https://github.com/hill-a/stable-baselines/issues/113