Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Support for max/min ops for `sync_all_reduce`

See original GitHub issue

Running the distributed tests while world_size > 1 makes the tests fail. I was trying to make the metric compatible with DDP with doing the same thing we have done with the other metrics but the results are different! I tried to print out some things for a better understanding, everything seemed to be working fine (calculating the metric is very simple) but after doing metric.compute() returns a strange result! (even printing the values computed in _update() function seems fine!).

Code snippet

Issue Analytics

State:
Created 3 years ago
Comments:6 (6 by maintainers)

Top GitHub Comments

1reaction

sdesroziscommented, Mar 17, 2021

Otherwise, you can create a zero tensor of size world_size. Each process writes in the rank position. You reduce this tensor and compute the max. It should do the job but it is a hack and that does not scale.

1reaction

sdesroziscommented, Mar 17, 2021

@KickItLikeShika Thank you.