ClearrMLLogger freezes logging in distributed env

🐛 Bug description

Instantiating a ClearMLLoggerinstance in a distributed training freezes the logging of the subprocesses. The bug can be reproduced by executing the following script: https://github.com/H4dr1en/trains/commit/642c1130ad1f76db10ed9b8e1a4ff0fd7e45b3cc

I will open a PR to fix it.

Environment

• PyTorch Version (e.g., 1.7.1):
• Ignite Version (e.g., 0.5.0):
• OS (e.g., Linux): linux
• How you installed Ignite (conda, pip, source): pip/source
• Python version: 3.8.5
• Any other relevant information:

[bmartinn]

@H4dr1en you are correct 😄 The first “main process” creates the actual Task and sets an OS environment variable specifying the newly created Task. When the child processes call the “same” Task.init they inherit the environment variable from the main process, so it knows to connect with the main Task and log everything back to it, instead of creating a new Task, specifically here, I think the issue is that the second Task.init call should have the same project/experiment name, which it does not have, hence the issue.

I can verify I was able to reproduce the issue, and I can also verify the PR solved it 🎉 🎊

[H4dr1en]

@vfdev-5 To the best of my knowledge, it happens like that:

• The main process creates the task, the first time it calls that Task.init function
• The subprocesses, when calling Task.init, will contact the main process to retrieve the task. They don’t recreate a new task, but get a copy of the task created by the main process.
• Any logging done on the subprocesses will be tunnelled to the main process, that will take care of logging it to the clearml-server

@bmartinn please correct me if I am wrong 👍