wandb.log delayed

I am running wandb.log({"loss":loss}, step=epoch) at the end of every epoch. However, the logs are delayed 1-2 epochs in each iteration. This means that I am seeing the logs for the first epoch only on the platform when the second or third epoch is finished. Unfortunately, after stopping the code the last epochs are then not in the logs and are missing.

Any ideas?

[vanpelt]

Hey @psinger @vadapalliravikumar the long term fix is a bit more involved but we better understand the underlying issue. The problem is we use the step argument to decide when to push metrics to the server (only when it increases). Unfortunately in jupyter we don’t handle the final step properly and this also creates a delay in metrics being pushed until the step increases. The quick fix is to not specify step and instead use the commit argument if you need to rollup metrics in a single step. i.e.

wandb.log({"acc": 0.5}, commit=False)
# Some other code
wandb.log({"val_acc": 0.6, "custom_step": 32})

Every time wandb.log is called without a step argument or with commit=False we increment an internal step counter and push the metrics to the server. If you want to keep track of a different step counter you can pass it into log just like any other metric. We then allow you to change the x-axis in the UI to any metrics you’ve logged that are monotonically increasing.

I know this is unfortunate. We’re working on a big overhaul of our backend that should address these issues and make for a cleaner api. And again, apologies for not getting back to you sooner on this one.

[vanpelt]

Please try this again with 0.8.35 we made some big changes to the Jupyter integration that *should fix your issues.