Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unexpected endless waiting

See original GitHub issue

when I use 4 gpus to train a model, the training process always wait something suddenly. Do nothing, and take up the gpu resources. When I forcibly stop the process, It shows:

File "/opt/conda/lib/python3.6/runpy.py" , line 193, in _run_module_as_main
  "__main__", mod_spec)
File "/opt/conda/lib/python3.6/runpy.py" , line 85, in _run_code
  exec(code, run_globals)
File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py" , line 235, in <module>
  main()
File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py" , line 228, in main
  process.wait()
File "/opt/conda/lib/python3.6/subprocess.py" , line 1477, in wait
  (pid, sts) = self._try_wait(0)
File "/opt/conda/lib/python3.6/subprocess.py", line 1424, in _try_wait
  (pid, sts) = os.waitpid(self.pid, wait_flags)

Even after half a day, the process do nothing but wait. Can you explain this? How to continue the training process or aviod the endless waiting?

Issue Analytics

State:
Created 3 years ago
Reactions:5
Comments:8 (2 by maintainers)

Top GitHub Comments

3reactions

idontlikelongnamecommented, Jun 22, 2020

I didn‘t. Maybe it’s jupyter notebook‘s fault which was used to train a model

fixed it, maybe some bugs with pytorch 1.4 and nccl, fixed it by add params “–nnodes=1 --node_rank=0” to train.sh

python -m torch.distributed.launch \
    --nproc_per_node=10 \
    --nnodes=1 \
    --node_rank=0 \
    ./tools/train.py \

1reaction

Chi-Zaozaocommented, Jul 2, 2020

It works! Thank you very much!

Top Results From Across the Web

The Frustration Of Waiting - Mental Help Net

One of the factors that can make a wait feel endless is awareness of time. A minute can feel like a second or...

Node JS with async unexpected infinite loop - Stack Overflow

The problem is that you are calling the same HTTP server again and again, without returning a response. The HTTP server is listening...

When Waiting on God: How to Embrace the Delay Well

Waiting is inevitable and sometimes painful. Discover ten approaches today to embrace the delay while waiting on God.

wait until bundle finished, stopped at 99% [webpack 5] #14405

when I run start, the progress bar is stuck at 99%. and then my local web page opened but just keep loading. I...

Busy waiting - Wikipedia

In computer science and software engineering, busy-waiting, busy-looping or spinning is a technique in which a process repeatedly checks to see if a ......