[SAC] Potential bug in temperature learning

https://github.com/DLR-RM/stable-baselines3/blob/78e8d405d7bf6186c8529ed26967cb17ccbe420c/stable_baselines3/sac/sac.py#L184

In the paper they compute the loss as -alpha * (logp + entropy_target) but here you implemented -log(alpha) * (logp + entorpy_target) if I am not mistaken.

I guess it should look like:

            ent_coef_loss = None
            if self.ent_coef_optimizer is not None:
                # Important: detach the variable from the graph
                # so we don't change it with other losses
                # see https://github.com/rail-berkeley/softlearning/issues/60
                ent_coef = th.exp(self.log_ent_coef) # No need to detach here anyways
                ent_coef_loss = -(ent_coef * (log_prob.detach() + self.target_entropy)).mean()
                ent_coef_losses.append(ent_coef_loss.item())
            else:
                ent_coef = self.ent_coef_tensor

Of course you could detach ent_coef afterwards in both cases, to maybe save time when computing actor_loss and doing the respective .backward().

[araffin]

Hello, Thanks for pointing that out. There are several issues about that:

• https://github.com/rail-berkeley/softlearning/issues/37
• https://github.com/hill-a/stable-baselines/issues/801
• https://github.com/hill-a/stable-baselines/issues/828

The short answer is yes, you are right, in practice, it does not change the results (the original implementation was doing that before).

EDIT: it may change something https://github.com/rail-berkeley/softlearning/issues/136#issuecomment-620533436

[johannespitz]

All right assuming that we constrain alpha >= 0 in the last line but not in the first two lines then the min is the same. However, you go on and use alpha to compute the other losses. Meaning we want to know if the argmin is the same.

min E[-alpha * log_pi - alpha * H]
= min E[-alpha * (log_pi + H)]
= min E[-log alpha * (log_pi + H)]

I believe that what’s really going on here is that the gradient of alpha or some alpha_logit has the same sign depending on whether log_pi + H is positive or negative and then Adam does its usual magic. Anyways, personally I would prefer to use the originally derived formula if it doesn’t degrade the performance…