NaN values in acktr

Hi everyone, I’m trying to use continuous acktr to learn to reach a target with a mujoco simulation of the jaco arm. I use exactly the same hyperparameters as for the reacher env and acktr definitely learns something meaningful, the reward goes up and I can also see it when I render the frames.

The problem is, that after some 2000-3000 iterations, the algorithm starts to produce nan values.

The log at the time when it starts to happen looks as follows:

Iteration 3025
kl just right!

| EVAfter   | 0.984      |
| EVBefore  | 0.976      |
| EpLenMean | 200        |
| EpRewMean | -8.5       |
| EpRewSEM  | 0.82       |
| KL        | 0.00148061 |

Iteration 3026 
kl too low

| EVAfter   | 0.984       |
| EVBefore  | 0.98        |
| EpLenMean | 200         |
| EpRewMean | -7.31       |
| EpRewSEM  | 0.613       |
| KL        | 0.000913428 |

Iteration 3027
kl just right!

| EVAfter   | 0.98     |
| EVBefore  | 0.976    |
| EpLenMean | 200      |
| EpRewMean | -8.92    |
| EpRewSEM  | 0.937    |
| KL        | nan      |

Then of course the nans start to spread and everything becomes nan. Does anyone have an idea what could cause such behaviour and what to do against it?

[mansimov]

Ok I found a small detail in adjusting stepsize that wasn’t in baselines code that fixes the NaN issue in @lukashermann Jaco environment and roboschool humanoid @Breakend

Change lines 121-129 in https://github.com/openai/baselines/blob/master/baselines/acktr/acktr_cont.py to

        min_stepsize = np.float32(1e-8)
        max_stepsize = np.float32(1e0)
        # Adjust stepsize
        kl = policy.compute_kl(ob_no, oldac_dist)
        if kl > desired_kl * 2:
            logger.log("kl too high")
            U.eval(tf.assign(stepsize, tf.maximum(min_stepsize, stepsize / 1.5)))
        elif kl < desired_kl / 2:
            logger.log("kl too low")
            U.eval(tf.assign(stepsize, tf.minimum(max_stepsize, stepsize * 1.5)))
        else:
            logger.log("kl just right!")

I will create pull request with this fix and other misc small tweaks soon. Thanks for your patience !

[jirenu]

I’m still getting Nans for a custom environment. This was remedied by scaling the reward down as suggested.