[Feature Request] To change the way to deal with the logarithm of standard deviation for SAC

🚀 Feature

To change the way to deal with the logarithm of standard deviation for SAC.

Motivation

in sac/policies.py line 169-171, we use

        log_std = self.log_std(latent_pi)
        # Original Implementation to cap the standard deviation
        log_std = th.clamp(log_std, LOG_STD_MIN, LOG_STD_MAX)

However, this may lead to zero gradients when log_std is out of range due to torch.clamp.

Alternatives

Replace code above with

        log_std = torch.tanh(log_std)
        log_std = LOG_STD_MIN + 0.5 * (
            LOG_STD_MAX  - LOG_STD_MIN 
        ) * (log_std + 1)

as in rad line 81-84.

[araffin]

Hello, I guess the piece of code you are showing was inspired by spinning up: https://github.com/openai/spinningup/blob/2ce0ee91826128497078fba7f25ba1d1bd9c3789/spinup/algos/sac/core.py#L66

They had that in the original release with TF, but then changed it when releasing PyTorch version: https://github.com/openai/spinningup/commit/20921137141b154454c0a2698709d9f9a0302101#diff-a626546e80a598fcbd10dd94a870a5049b1e3346234aebaa71256158f2147113

Is there formal comparison somewhere between these two?

if you can show that there is actually a benefit to it, I would agree for that change, otherwise, I would keep it as is, matching the original implementation and avoiding an unnecessary breaking change.

[lzhyu]

Hello,

Comparing training curve on one env with only three random seeds is far from enough… Especially as the two curves overlap greatly.

The minimum would be testing at least with 10 seeds on 4 envs (for instance the PyBullet ones) comparing deterministic controller (see EvalCallback included in the zoo) and using proper evaluation like rliable. The good news for you is that I just included rliable plot in the rl zoo. And obviously, please use the rl zoo for training and comparison.

I know that 80 runs of 1M steps sounds like a lot but that’s what needed to avoid hasty conclusions.

Thank you for telling me that. Considering my own time and energy, I would abandon the proposition.