Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

AttributeError: module 'horovod.torch' has no attribute 'nccl_built'

See original GitHub issue

The PL.pytorch-lightning (GPUs) CI job is failing with AttributeError: module 'horovod.torch' has no attribute 'nccl_built'

https://dev.azure.com/PytorchLightning/pytorch-lightning/_build/results?buildId=61024&view=logs&j=3afc50db-e620-5b81-6016-870a6976ad29&t=059bb337-5de1-5e6a-4c56-42f55a328c0e

cc @tchaton @rohitgr7 @akihironitta @carmocca @borda

Issue Analytics

State:
Created 2 years ago
Reactions:2
Comments:6 (6 by maintainers)

Top GitHub Comments

2reactions

akihironittacommented, Mar 17, 2022

@awaelchli @Borda I’ll try to fix this in the next hour(s). If you’re already finishing up your fix (or you’re working on it now), let me now here or in the chat to avoid conflict in pushing fixed docker image to hub 😃

2reactions

awaelchlicommented, Mar 15, 2022

There is this message:

Extension horovod.torch has not been built: /usr/local/lib/python3.7/dist-packages/horovod/torch/mpi_lib/_mpi_lib.cpython-37m-x86_64-linux-gnu.so not found If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error. Warning! MPI libs are missing, but python applications are still available.

There seems to be a missing MPI package. The GPU uses the docker container from this file. We can try to add it there. Horrible

Top Results From Across the Web

module 'horovod.torch' has no attribute 'Average' · Issue #1590

Environment: Framework: PyTorch Framework version: 1.3.1 Horovod version: 0.18.2 MPI version: 4.0.2 CUDA version: 10.2 NCCL version: 2.5.6 ...

horovod.torch

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet. Navigation. Overview · Concepts · Horovod Installation Guide · API · Horovod ......

AttributeError: module 'torch' has no attribute 'permute'

I tried to run the code below for training a sequence tagging model (didn't list all of the code because it works fine)....

How to use the horovod.torch.rank function in horovod - Snyk

Horovod based try: import horovod.torch as hvd except ImportError: raise SystemError("horovod is not working, try to set using_horovod=False.

module 'torch.distributed' has no attribute 'is_initialized' in ...

In order to solve this problem. Actually Window and Mac doesn't support distributed training facility. so this issue is occuring.