How does the `synchronize` function work?

❓ Questions and Help

I found that in your codes you use synchronize helper function to synchronize gpus. I am not familiar with the usage of torch.distributed.deprecated, and I am trying to understand how does the following codes actually work.

def synchronize():
    """
    Helper function to synchronize between multiple processes when
    using distributed training
    """
    if not torch.distributed.deprecated.is_initialized():
        return
    world_size = torch.distributed.deprecated.get_world_size()
    rank = torch.distributed.deprecated.get_rank()
    if world_size == 1:
        return

    def _send_and_wait(r):
        if rank == r:
            tensor = torch.tensor(0, device="cuda")
        else:
            tensor = torch.tensor(1, device="cuda")
        torch.distributed.deprecated.broadcast(tensor, r)
        while tensor.item() == 1:
            time.sleep(1)

    _send_and_wait(0)
    # now sync on the main process
    _send_and_wait(1)

I get some questions here.

1. Will broadcast block the main process until all other processes receive the tensor? What are the behaviors of this function in different type of processes?
2. Why use _send_and_wait(1) at the end of this function? I know that rank 0 is the master, but what is special for rank 1?

[yelantf]

Oh, yes, I didn’t use coco and error did not always happen. It happens more frequently when I run multiple copy of inference codes on a single machine at the same time. All 8 GPUs are the same. I use 0-3 to run a copy of codes and 4-7 to run another. I think maybe the cause is the high workload of CPU.

[pietern]

Yes, this happens because of the lazy initialization in the NCCL backend. The faster process will try to create a new NCCL communicator and is waiting for the slower process to do the same. This times out after 5 minutes. This timeout is set on the k/v store (be it a file backed store or TCP store where a single process acts as server) and is currently not configurable.

This is a dup of pytorch/pytorch#16225 so this one can be closed and we can continue discussion there.