[question] SAC target q nets may be updated many times in each round?

This line of code: https://github.com/hill-a/stable-baselines/blob/d288b6d911094268b17abb61dbe9ac3ca05e1ce3/stable_baselines/sac/sac.py#L464

since step is in range(total_timesteps) and grad_step is nested in range(self.gradient_steps), then if self.gradient_steps is not 1, the target networks are updated repeatedly?

[hartikainen]

Ah I see, yeah. The target update in my opinion should be dependent of the gradient updates so probably preferable to do what TD3 does.

[araffin]

Thanks @hartikainen for your answer.

to apply x/y gradient steps after each collected sample or Nx gradient steps after every Ny samples

yes, I have also experienced that. My question was more about the target network update. In DQN, it is independent of the gradient update. In TD3, there is one soft update per gradient step. For SAC, it is the same as TD3 for now, but the current code in soft q learning seems to be closer to DQN… (currently it does not change anything though because target_update_interval=1.