Be able to Set Timeout When DDP Strategy is Used

🚀 Feature

Motivation

When I use Pytorch Lightning in cluster of our company to train a big model which needs a lot of gpu cards, it normally means that I have to wait more than 30 minutes before all the cards I need get all prepared. Therefore, It will be great if I can set the timeout parameter of init_process_group rather than just use the default value.

And I think this can be accomplished by adding some parameters at the __init__ function of DDPStrategy, which will then be passed to the init_dist_connection fucntion residing in the DDPStrategy.setup_distributed. If you guys believe this is ok, I can create a PR.

cc @borda @justusschock @kaushikb11 @awaelchli @akihironitta @rohitgr7

[carmocca]

There’s already checks for this inside PyTorch: https://github.com/pytorch/pytorch/blob/a8b098859688a3f1993821eecc036be973a15605/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L555-L572

I suggest we don’t go into these internal details in PL

[lsy643]

@carmocca OK. A new PR https://github.com/Lightning-AI/lightning/pull/13383 is here