[Question] Callback for GAIL

Hello!

I am training GAIL on Pendulum-v0. The expert data for Pendulum-v0 is available in expert_path. I am trying to save the model using the Callback documentation as reference. Here’s the code:

import gym
import tensorflow as tf
from stable_baselines import GAIL, SAC, TRPO, PPO2
from stable_baselines.gail import ExpertDataset, generate_expert_traj
from stable_baselines.common.callbacks import CheckpointCallback, EvalCallback

tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)

if __name__=='__main__':
    expert_path = 'stable_baselines/gail/dataset/expert_pendulum.npz'
    dataset = ExpertDataset(expert_path=expert_path, traj_limitation=10, verbose=0)
    model = GAIL('MlpPolicy', 'Pendulum-v0', dataset, verbose=0)

    checkpoint_callback = CheckpointCallback(save_freq=1000, save_path='./logs/', name_prefix='gail')
    model.learn(total_timesteps=2000, callback=checkpoint_callback)

Here’s the traceback:

Traceback (most recent call last):
  File "checking_callback.py", line 18, in <module>
    main()
  File "checking_callback.py", line 15, in main
    model.learn(total_timesteps=2000, callback=checkpoint_callback)
  File "C:\IL_UAV\GAIL-TF-SB\stable_baselines\gail\model.py", line 54, in learn
    return super().learn(total_timesteps, callback, log_interval, tb_log_name, reset_num_timesteps)
  File "C:\IL_UAV\GAIL-TF-SB\stable_baselines\trpo_mpi\trpo_mpi.py", line 455, in learn
    ob_expert, ac_expert = self.expert_dataset.get_next_batch()
  File "C:\IL_UAV\GAIL-TF-SB\stable_baselines\gail\dataset\dataset.py", line 151, in get_next_batch
    if dataloader.process is None:
AttributeError: 'NoneType' object has no attribute 'process'

Some comments and queries:

1. The callback runs without errors if GAIL is replaced with an RL algorithm (i.e. removing the dataset parameter & replacing GAIL with TRPO, SAC, PPO2, etc.) I believe the error has to do with the way CheckpointCallback’s save_path parameter works specifically for GAIL.
2. I ran the same code with EvalCallback. The result is the same, and it turns out the error has to do with best_model_save_path (remove this parameter and the code runs fine).
3. Should I be using CustomCallback and rollouts instead? If yes, could you give me a specific example code? Thanks!

PS: I am new to both Stable Baselines and RL

System Info

• Windows 10
• Python 3.7.0
• Tensorflow 1.14.0

[araffin]

Ahh. Is there any workaround for this?

You need to customize SB a bit and pass the callback in https://github.com/hill-a/stable-baselines/blob/c4c31cb5687800cf102054f0935aa09e7545159f/stable_baselines/common/runners.py#L58

We don’t do that because this would break the code if users use MPI with more than 1 node.

You can also change timesteps_per_batch but this not recommended if it deteriorates the performance.

https://github.com/hill-a/stable-baselines/blob/c4c31cb5687800cf102054f0935aa09e7545159f/stable_baselines/trpo_mpi/trpo_mpi.py#L50

[araffin]

Thank you! I would like to use EvalCallback to save the best model. The complete model (oops).

#847 solves both issues as they were related to the same problem (continuing training after saving).