Models not saved during training

Question I tried asteroid/egs/wham/DPRNN/run.sh but the error was occurred at the end of the training process. The messages are below:

~~~
sep_clean_8kmin_7101f1a8/checkpoints/_ckpt_epoch_4.ckpt as top 5
Epoch 5: 100%|██████████| 4022/4022 [28:05<00:00,  2.39it/s, loss=-11.728, v_num=0, val_loss=-11.5]
Traceback (most recent call last):
  File "train.py", line 121, in <module>
    main(arg_dic)
  File "train.py", line 92, in main
    best_path = [b for b, v in best_k.items() if v == min(best_k.values())][0]
IndexError: list index out of range
~~~

I have tried to add some codes at train.py and confirmed the length of checkpoint.best_k_models.items() is zero. And best_k_models.json contains only {}.

Does anyone have any idea to fix it? Let me know if you have any comments.

Environment

• Python 3.7.7

• torch 1.5.1 (I’ve tried 1.3.0 but same result)

• pytorch-lightning 0.7.6

• Ubuntu 18.04 on GCP

[mpariente]

Let’s keep this open until it’s merged, thanks!

[mpariente]

This should be fixed in master