mlflow training loss not reported until end of run

I think I’m logging correctly, this is my training_step

    result = pl.TrainResult(loss)
    result.log('loss/train', loss)
    return result

and validation_step

    result = pl.EvalResult(loss)
    result.log('loss/validation', loss)
    return result

The validation loss is updated in mlflow each epoch, however the training loss isn’t displayed until training has finished. Then it’s available for every step. This may be a mlflow rather than pytorch-lighting issue - somewhere along the line it seems to be buffered?

Versions:

pytorch-lightning==0.9.0 mlflow==1.11.0

Edit: logging TrainResult with on_epoch=True results in the metric appearing in mlflow during training, it’s only the default train logging which gets delayed. i.e.

    result.log('accuracy/train', acc, on_epoch=True)

is fine

[david-waterworth]

Thanks for the assistance, no nothing unresolved.

[patrickorlando]

So I think this is because of the default behaviour of the TrainResult and the way row_log_interval works. And it only appears if the number of batches per epoch is less than row_log_interval

By default TrainResult logs on step and not on epoch. https://github.com/PyTorchLightning/pytorch-lightning/blob/aaf26d70c4658e961192ba4c408558f1cf39bb18/pytorch_lightning/core/step_result.py#L510-L517

When logging only per step, the logger connector only logs when the batch_idx is a multiple of row_log_interval. However if you don’t have more than row_log_interval batches, the metrics are not logged. https://github.com/PyTorchLightning/pytorch-lightning/blob/aaf26d70c4658e961192ba4c408558f1cf39bb18/pytorch_lightning/trainer/logger_connector.py#L229-L237

@david-waterworth Do you have less than 50 batches per epoch in your model? can you try setting row_log_interval to be less than the number of train batches to confirm whether the issue is caused by this?