In Multi GPU DDP, pytorch-lightning creates several tfevents files

Describe the bug

Right now pytorch-lightning seems to create several tfevent files in the multi-gpu ddp way: e.g. for 2 GPUs:

-rw-rw-r--. 1 sam sam   40 Sep 19 08:11 events.out.tfevents.1568880714.google2-compute82.3156.0
-rw-rw-r--. 1 sam sam 165K Sep 19 08:22 events.out.tfevents.1568880716.google2-compute82.3186.0
-rw-rw-r--. 1 sam sam   40 Sep 19 08:11 events.out.tfevents.1568880718.google2-compute82.3199.0

I suppose the first one is created by the main process and the next 2 are created by the 2 DDP processes (one per GPU). Unfortunately, the actual events are not logged in the last created one, and that confuses tensorboard, cf https://github.com/tensorflow/tensorboard/issues/1011

I have to restart tensorboard if I want to see the new data.

A clear and concise description of what the bug is.

To Reproduce Launch any training on multi GPU DDP.

Expected behavior Only one tfevent file is created, from the master GPU.

[s-rog]

I removed that call and I’m still getting multiple tfevents, no other calls to logging besides metrics returned by train and val steps. Currently using the experimental --reload_multifile=true in tensorboard to get around the issue.

[awaelchli]

if you manually log things, then do this:

if self.trainer.is_global_zero:
    # your custom non-Lightning logging