Unusual NaN occurring in action output

My action space is defined as:

self.action_space = gym.spaces.Box(
            -1, 1, shape=(len(self.src.asset_names) + 1,), dtype=np.float32)

When I run:

model = PPO1(MlpPolicy, env, verbose=0, tensorboard_log='tensorboard/')   
model.learn(total_timesteps=10000)

print(action) gives this: First it is ok then gives a warning and then all actions are nan.

...
[ 1.          0.8173821  -0.30915284 -0.4841192   1.         -0.8486665
  0.999535   -1.        ]
[ 1.          0.17815794 -1.          0.00990602 -0.15018277  1.
  0.786909    0.40793106]
[ 1.         -1.         -0.22874616  0.8188446   0.0233863  -0.06452941
  0.255739    1.        ]
[ 1.         -0.14166626  1.          0.16046658  0.9575441  -0.67399544
 -1.          1.        ]
[ 0.41213733 -0.04675869  0.7445776   1.          0.46755818 -0.35608172
 -1.          0.5503573 ]
...\stable-baselines\stable_baselines\common\console_util.py:39: RuntimeWarning: invalid value encountered in less
  if (value < 1e-4 or value > 1e+4) and value > 0:
...\stable-baselines\stable_baselines\common\console_util.py:39: RuntimeWarning: invalid value encountered in greater
  if (value < 1e-4 or value > 1e+4) and value > 0:
[-1.         -0.49604103 -0.54152864  0.07228398  0.6079284  -0.00336414
  1.          1.        ]
[nan nan nan nan nan nan nan nan]
[nan nan nan nan nan nan nan nan]
[nan nan nan nan nan nan nan nan]
[nan nan nan nan nan nan nan nan]
[nan nan nan nan nan nan nan nan]
[nan nan nan nan nan nan nan nan]
[nan nan nan nan nan nan nan nan]
...

[hn2]

I will try to add your code for invalid reward. I did not change the defaults for ppo1.

[hill-a]

ah, you are loading data from a file. Have you checked the validity of the file? It may have NaNs or infs in it. Add this to the __init__() function after self.history, self.abbreviation = read_stock_history(filepath=datafile):

print("has NaNs: {}, has infs: {}".format(np.any(np.isnan(self.history)), np.any(np.isinf(self.history))))

Furthermore, make sure that your epsilon value EPS is not too low (as in higher than 1e-20 for example), as a low epsilon value that is too low can cause NaNs.

Finally, add this after importing numpy

np.seterr(invalid='raise')

It will crash when NaNs occure, allowing you to see exaclty where is issue is comming from. And whether or not it is from stable baselines, or your data, or a simple bug.