Update examples/documentation for torch.distributed.run

Most examples now read local rank from a CLI argument --local_rank. This ensures for it to work when using the torch launch utility torch.distributed.launch. However, since recent torch versions, launch is deprecated. The new suggested way to run distributed code from CLI is with torch.distributed.run. The difference from before is that instead of automatically passing CLI arguments, it sets the rank as an environment variable. The examples therefore need to be updated to read the ENV variable instead of the CLI argument.

See https://pytorch.org/docs/stable/elastic/run.html#launcher-api for more.

[BramVanroy]

Definitely. I think it’s a good idea to keep bumping this to open, so that others can also find this issue easily, and it can serve as a reminder that as soon as launch is obsolete, the docs need an update. It’s a tiny thing and easy to forget, so an open issue may help to remember as time goes on.

[sgugger]

I wouldn’t update the documentation as it won’t work for older versions of PyTorch and I would prefer to only put one command vs If your PyTorch version is this then xxx else yyy. Let’s wait a bit more 😃