[Question] Bias only, linear NN with PPO

Important Note: We do not do technical support, nor consulting and don’t answer personal questions per email. Please post your question on the RL Discord, Reddit or Stack Overflow in that case.

Question

Hi, I have in the past used a 2 hidden layer NN with 32 neurons each and non-linear activation function. I want to try something new and I am having issues implementing it.

policy_kwargs = dict(activation_fn=torch.nn.LeakyReLU, net_arch=[32, 32])
model = PPO(MlpPolicy, env, verbose=1, n_steps=10, batch_size=10, seed=my_seed, policy_kwargs=policy_kwargs)

I want to use the exact same simulation as before but modify policy_kwargs in a way that I will have only bias taken into account and a linear activation function and no actual inputs. I know that this is not a regular RL problem and there are other methods that will do as good or even better but I want to use PPO for it.

I want to define no inputs and only bias, and set the activation function to be a linear activation function. I know how to define the linear regression net in pytorch but I havent found any information of how to do it using the policy_kwargs

Could anyone share an idea how this can be done? If there is no way to define that type of network then is there a way to define it as a simple linear regression type of net?

I attach image to illustrate the network I am trying to build.

Additional context

Add any other context about the question here.

Checklist

• I have read the documentation (required)
• I have checked that there is no similar issue in the repo (required)

[Miffyli]

@danielstankw Ah, sorry about the little typo: the dict should be wrapped in a list:

policy_kwargs = dict(activation_fn=torch.nn.Identity, net_arch=[dict(pi=[], vf=[])])

That should work 😃. Please re-close issue if so.

[danielstankw]

policy_kwargs = dict(activation_fn=torch.nn.Identity, net_arch=dict(pi=[], vf=[]))

I used what you suggested and get error: AssertionError: Error: the net_arch list can only contain ints and dicts