[Feature] [RLLib] Remove RNNSAC

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

RNNSAC is badly broken. I lost a lot of time trying to use it. It would be an improvement to RLLib to remove it as it is not maintained. And obviously no one has ever used it since there are no bug reports.

I found many bugs and I’m sure there are many more.

RNNSACTrainer.validate_config() modifies the input config["replay_sequence_length"]. But config validation is called twice in the grand scheme of things and it crashes the second time it’s called.

In rnn_torch_sac_policy.py:action_distribution_fn() calls model.get_q_values() (twice if twin_q) but without adding the final actions argument input_dict["actions"]. Somehow get_q_values() accepts a missing actions and uses None instead. But that crashed in my use case. I had to pass the actions within action_distribution_fn(), which are anyway available as input_dict["actions"].

RNNSACTorchModel._get_q_value() is called twice when twin_q is used. But the input argument model_out is modified by the function and it crashes the second time it’s called with a now modified model_out input.

Also I had to replace RNNSACTorchModel.get_policy_output() with SACTorchModel.get_policy_output(). The SAC version does a lot of preprocessing to the argument model_out which was certainly needed for my use case.

Something else which may be completely irrelevant is that ComplexInputNetwork.__init__() calls TorchModelV2.__init__(..., self.original_space, ...), but I changed it to obs_space instead of original_space. I honestly can’t remember why I did that; I’m not sure if I needed to change that to get RNNSAC working. But it might break RLLib elsewhere to do that. I’m just mentioning this so you can take a look and double check if the current TorchModelV2.__init__(..., self.original_space, ...) is correct.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[max333]

My state was multiple levels of dicts of dicts.

A slightly related bug I reported related to flattening a complex space: #22659.

[max333]

No. You can close the issue. Thanks for fixing the other issues.