WandB-Logger drops all the logged values in training step for PyTorch Lightning

train-step:

def training_step(self, batch, batch_idx):
            x, y = batch
            logits, masks = self(x)
            loss = self.loss_func(logits, y)
            self.log(f"train-loss", loss, prog_bar=True)
            return {
                "loss": loss,
                "pred": torch.argmax(logits, -1),
                "labels": y,
            }

and on epoch end it logs the Precision and recall. after the validation step. I get a wandb: WARNING Step must only increase in log calls. Step 379 < 380; dropping {train-loss: 0.494, 'train-precision': 0.9, 'train-recall': 1, 'epoch': 1}

WandB version: 0.10.10 PyTorch-Lightning: 1.0.6

[isaacrob]

Fixed it! I was calling trainer.logger.experiment.log in a callback to log Matplotlib images and apparently this increases the step unless commit is set to False, in which case it gets group with the next logging message sent via PyTorch Lightning. Subtle bug! Everything’s ok now and working as expected 😃 thanks for your help @borisdayma. Hope @IsCoelacanth was able to resolve their issue too!

[borisdayma]

This was fixed with https://github.com/PyTorchLightning/pytorch-lightning/pull/5931. You can now just log at any time with self.log('my_metric', my_value) and you won’t have any dropped value. Just choose your x-axis appropriately in the UI (whether global_step or just the auto-incremented step).