Tracking Child Processes (tree of child processes)

I’ve noticed that when using subprocess.run on my dask tasks, the workers still report low CPU usage, when the child process is in fact maxing out my CPU usage. Does Dask track the forked/child process CPU usage under the worker CPU usage? If not, I think it should since child processes can occur when interfacing dask workers against programs written in non-python.

[CMCDragonkai]

This is natively supported in the psutil package: https://unix.stackexchange.com/a/339071/56970

pid=2235; python3 -c "import psutil
for c in psutil.Process($pid).children(True):
  print(c.pid)"

The True ensures that it gets it recursively.

https://psutil.readthedocs.io/en/latest/#psutil.Process.children

Another way is to realise that process group on Linux will always encapsulate the entire process tree when launched by a shell. See: https://en.wikipedia.org/wiki/Process_group This means that on Linux, if you start each worker on its own process group. Then you can just query the PGID and get all processes part of that group. This is also useful for distribution of signals. However this may not be portable to Windows or other operating systems, whereas process tree via direct descendants may be more standard among all OS.

I wrote a gist about this process groups: https://gist.github.com/CMCDragonkai/f58afb7e39fcc422097849b853caa140

[fbriol]

I tried to find a solution to follow the workers’ child processes.

To do this, the class handling the system monitoring must track the children; otherwise, the CPU usage is always 0 for all children (no previous time exists to calculate the usage). Initially, the children’s set is empty. Then we have to add new children and remove the old ones. In short, it adds a little bit of calculation. In my test case, I didn’t see any problem with the monitoring performance.

--- 8,14 ----
  class SystemMonitor:
      def __init__(self, n=10000):
          self.proc = psutil.Process()
+         self.children = set()
  
          self.time = deque(maxlen=n)
          self.cpu = deque(maxlen=n)
*************** class SystemMonitor:
*** 45,50 ****
--- 46,60 ----
          with self.proc.oneshot():
              cpu = self.proc.cpu_percent()
              memory = self.proc.memory_info().rss
+             children = set(self.proc.children(True)) - self.children
+             if children:
+                 self.children.update(children)
+             self.children = set(item for item in self.children
+                                 if item.is_running())
+             for item in self.children:
+                 with item.oneshot():
+                     cpu += item.cpu_percent()
+                     memory += item.memory_info().rss
          now = time()
  
          self.cpu.append(cpu)

I can open a PR to insert this change.