Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Creates extra-processes

See original GitHub issue

I’ve configured accelerate to handle 4,6,8,10 processes(different setups for testing). And it creates 7, 11, 15 and 19 processes respectively instead. The number is N-1 +N, because it creates N-1 extra processes on the 4th card. I.e.

|    0   N/A  N/A     58825      C   python3    28175MiB |
|    1   N/A  N/A     58826      C   python3    28195MiB |
|    2   N/A  N/A     58827      C   python3    28199MiB |
|    3   N/A  N/A     58825      C   python3     1405MiB |
|    3   N/A  N/A     58826      C   python3     1405MiB |
|    3   N/A  N/A     58827      C   python3     1405MiB |
|    3   N/A  N/A     58828      C   python3    28179MiB |

And it doesn’t matter which card will be 4th. I changed the order in CUDA_VISIBLE_DEVICES and it still creates on the 4th in order.

Is this a bug or I’m doing something wrong?

Issue Analytics

State:
Created 2 years ago
Comments:6 (2 by maintainers)

Top GitHub Comments

1reaction

gleb-shnshncommented, Sep 14, 2021

Well, I fixed It with adding device in loading torch.load(self.continue_from, self.accelerator.device) It created extra objects on inappropriate gpu

0reactions

gleb-shnshncommented, Sep 14, 2021

compute_environment: LOCAL_MACHINE
deepspeed_config: {}
distributed_type: MULTI_GPU
fp16: false
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
num_machines: 1
num_processes: 4

CUDA_VISIBLE_DEVICES=0,1,2,3,5,7,13,15 accelerate launch train.py

Top Results From Across the Web

Why so many multiple processes in Windows 10?

Hi, I noticed in the task manager that several processes occure more than one time, and some processes occure multiple times, ...

Google Chrome Creates Multiple Windows Processes

You can view the Chrome.exe processes in the Windows Task Manager. You may wonder why multiple processes are needed. What does each of...

Creating multiple processes in C - Stack Overflow

I am writing a program that needs to create multiple processes. Let's say this number is 3. I want each of these processes...

Can an application create multiple processes? - Super User

Yes. The Chrome web browser is a good example of this. Open Chrome and you'll find dozens of chrome.exe processes.

Create Multiple Processes - Synchro Arts

Processes are created on successive output tracks, starting with the one selected, and numbered sequentially. NOTE: Each process created in this step will ......