have trouble about resume from ckpt

I try to train DPRNN and DPTNet on wsj0-3mix from a checkpoint. and val_loss look like this (blue line is train from a ckpt). I think it should be a curve much smoother.

Scripts were copied from wham folder. to resume from ckpt I add some code in train.py like this:

state_dict = torch.load(os.path.join(exp_dir, ckpt_name))   
system.load_state_dict(state_dict['state_dict'])
...
trainer = pl.Trainer(..., resume_from_checkpoint=os.path.join(exp_dir, ckpt_name))

And to train DPRNN & DPTNet on wsj0-3mix, I replace wham dataset & dataloader code in train.py to this:

train_loader, val_loader = make_dataloaders(n_src=3, **conf["data"], **conf["training"])

no change in conf.yml. I also tried deepclustering on wsj0-3mix, dprnn & dptnet on wham and my own dataset, did not find this problem.

what did I miss? Need help plz.

[popcornell]

I have a pull request which should solve this issue. Need feedback: https://github.com/mpariente/asteroid/pull/326

What about this difference:

definitely one of the issues but not the main one changed it to the torch one

[mpariente]

Cool ! The other question is: is the scheduler’s resumed when using resume_from_checkpoint?

What about this difference:

# In torch
def state_dict(self):
    return {key: value for key, value in self.__dict__.items() if key != 'optimizer'}
# In asteroid
def state_dict(self):
    return {key: value for key, value in self.__dict__.items()}

I’m not sure the optimizer should be in the state_dict…