How to kill distributed processes

Hi, I am running on one 8-GPU machine in nvidia docker with pytorch 1.0, cuda 10.

I follow the script as here to run the program.

However, the distributed processes do not terminate after I Ctrl+C. Some of the process still running on background and killing does not terminate it either. Please help.

Hence, how to properly terminate a ongoing process with distributed running?

Thank you,

[frankang]

I use this script to kill zombie processes. kill $(ps aux | grep "train.py" | grep -v grep | awk '{print $2}')

[XiwuChen]

I use this script to kill zombie processes. kill $(ps aux | grep "train.py" | grep -v grep | awk '{print $2}')

Run the following command if you use python train.py, i.e. spawn processes from the main function: kill $(ps aux | grep multiprocessing.spawn | grep -v grep | awk '{print $2}')

Hello. Is there any better way to kill these children processes in the training code? We do NOT want to kill these processes manually. Also, if there are two training tasks for one user, we have to figure out one task before killing it.