Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
summarywriter on each process?
See original GitHub issueDescription
It looks like figures are being drawn to TensorBoard on every process. I could be wrong, but would it be better in the multi-gpu/multi-node case to only draw on the rank zero process using pytorch_lightning.utilities.distributed.rank_zero_only?
Steps to reproduce
I was looking into why validation was slow and noticed the plotting portion. Though this probably wouldn’t speed anything up, I wonder if each process is trying to overwrite eachother’s TB logs. Could be missing something though!
Version
0.4.0.dev0
Issue Analytics
- State:
- Created a year ago
- Reactions:1
- Comments:5
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
I think because of this: https://github.com/Lightning-AI/lightning/blob/4eb7766f3c222b95216f5e2831894a5143f70882/src/pytorch_lightning/loggers/tensorboard.py#L157-L158 And https://github.com/Lightning-AI/lightning/blob/4eb7766f3c222b95216f5e2831894a5143f70882/src/pytorch_lightning/loggers/logger.py#L34 returning a DummyExperiment
Turns out I was wrong about this!
Adrian Wälchli: “It’s taken care of by Lightning already. If you use self.log or like you show above, tainer.logger.experiment, it won’t log on rank > 0. We consider this boilerplate, and thus handle it for the user.”