[Feature Request] give users a way to provide their own exploration noise function?

🚀 Feature

Let the users provide, if they want, a function that is used to sample the exploration noise. Possibly an API something like the following (it may be very naive though), so that it can collect data from the agent and its policies, or take any arguments the users want:

def user_noise(agent, *args, **kwargs)
    """this user_noise function will be called at each step to generate the exploration noise"""
    return exploration_noise

Motivation

The gSDE paper is extremely interesting, however, in some cases, using the before last policy layer may be a suboptimal choice; offering an API way to set other more specific exploration noise functions that depend on the agent internals exactly in the way the user wants would be great.

[qgallouedec]

I understand your need. Have you considered the following workaround:

from stable_baselines3 import DDPG
from stable_baselines3.common.noise import Noise

class MyNoise(Noise):
    def set_model(self, model):
        self.model = model

    ...

action_noise = MyNoise()
model = DDPG("MlpPolicy", "Pendulum-v1", action_noise=action_noise)
action_noise.set_model(model)

[qgallouedec]

The intuition behind not using action noise for on-policy agents is that an on-policy algorithm learns from the actions it has taken. However, if you add noise to the action, it is no longer “taken by the agent” but by the agent augmented with noise. So you introduce a bias.