"Step must only increase in log calls" when adding W&B logger after some training

Describe the bug

First time W&B user here. I added the WandbLogger to my PyTorch-Lightning project using the default settings, ie. Trainer(logger=WandbLogger()). Note that I’m resuming training from a checkpoint (didn’t use W&B logger before).

I’m getting lots of warnings like wandb: WARNING Step must only increase in log calls. Step 47 < 270952; dropping {'loss': -2.309272527694702} and no metrics are reported. Other data like GPU utilisation is collected correctly.

[jonashaag]

encountered the same issue. @jonashaag did you figure out how to deal with this?

I’m dealing with it by not using W&B anymore ^^

[vanpelt]

Hey @jonashaag instead of passing the “step” argument, you can log separate x-axis as separate metrics. I.E.

self.logger.experiment.log({
            "val_epoch": wandb.Image(test_img, caption='caption'),
            "val_step": self.val_step}, step=self.current_epoch)

Then in the UI you can choose a custom x-axis if you need to plot metrics against “val_step” for instance. You also shouldn’t be adding the “epoch” number to the key as this will create an unbounded number of charts. You can remove step=self.current_epoch in the above but it’s an optimization to ensure all metrics for a given epoch are logged together.