Huge actor loss in `TD3`

Question

I am using a TD3 agent to solve a MountainCarContinuous-v0 environment. Is it normal that the actor’s loss gets huge at some point? Isn’t this dangerous for the weights of the model? The weird part is that the agent is actually solving the environment, since the accumulated reward is gradually increasing.

Additional context

This is my code (as suggested by @qgallouedec in https://github.com/DLR-RM/stable-baselines3/issues/936):

import gym
import numpy as np
from stable_baselines3 import TD3
from stable_baselines3.common.noise import OrnsteinUhlenbeckActionNoise

env = gym.make("MountainCarContinuous-v0")

action_noise = OrnsteinUhlenbeckActionNoise(np.zeros(1), 0.5 * np.ones(1))
model = TD3("MlpPolicy", env, action_noise=action_noise, verbose=1)

model.learn(total_timesteps=1_000_000)

This is the logger’s output after some iterations.

---------------------------------
| rollout/           |          |
|    ep_len_mean     | 361      |
|    ep_rew_mean     | 60.8     |
| time/              |          |
|    episodes        | 68       |
|    fps             | 111      |
|    time_elapsed    | 219      |
|    total_timesteps | 24547    |
| train/             |          |
|    actor_loss      | -14.4    |
|    critic_loss     | 1.08     |
|    learning_rate   | 0.001    |
|    n_updates       | 24471    |
---------------------------------

[araffin]

it depends on cumulative reward in your environment…

yes, that’s how the actor loss is defined: https://github.com/DLR-RM/stable-baselines3/blob/a6f5049a99a4c21a6f0bcce458ca3306cef310e0/stable_baselines3/td3/td3.py#L182

It depends on the magnitude of the reward and on the discount factor.

[AndreasKaratzas]

Yeah, correct. Well, I think that covers my question. Thank you all 😃