Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Support more backends in distributed operations

See original GitHub issue

The current version of MONAI utilizes torch.distributed for all_gather, all_reduce etc., e.g. in line 409 below. However, this will raise an Error when I launch the parallel programs in ignite.distributed.Parallel context with horovod or xla backend: RuntimeError: Default process group has not been initialized, please make sure to call init_process_group. Is it better to replace all the torch.distributed operations with ignite.distributed ones? They have naive support for all the backends.

https://github.com/Project-MONAI/MONAI/blob/1a707f0226bfe10c000cc54ac87b35d50973e221/monai/utils/misc.py#L359-L423

Issue Analytics

State:
Created 2 years ago
Comments:11 (10 by maintainers)

Top GitHub Comments

1reaction

vfdev-5commented, Jul 1, 2021

@sandylaker for XLA quick checking we can use Google Colab and specify TPU as accelerator

1reaction

Nic-Macommented, Jul 1, 2021

Hi @sandylaker ,

Cool! Really appreciate your help here!

Thanks.

Top Results From Across the Web

The Case for Custom Storage Backends in Distributed ...

This is a preferred choice for most distributed file sy... ... Third, supporting emerging storage hardware is painstakingly slow.

The Case for Custom Storage Backends in Distributed

This is a preferred choice for most distributed file systems today, ... An emerging requirement for storage backends is support for novel ...

The Case for Custom Storage Backends in ... - ResearchGate

This is a preferred choice for most distributed file systems today, because it allows ... Third, supporting emerging storage hardware is painstakingly slow....

Backend-to-Backend Communication - Level Up Coding

Any distributed system resolves around sharing information between its distinct components, frontend or backend alike.

[RFE] Add support for distributed CPU-backend mode #11182

Unless I am mistaken, it is only possible to use the distributed backend (initialised with jax.distributed.initialize ) with the GPU and TPU ...