Leaking semaphores tracking issue.

The purpose of this issue is to track cases of semaphore leaks with view of hopefully getting to the root cause and fixing it (assuming it is a problem!). This issue is to also hold any useful notes whilst narrowing down the problems.

Things that would be really helpful:

• Anyone being able to supply a concrete reproducer of a semaphore leak coming from dask/distributed.
• If the leak is not (easily) reproducible, anecdotal evidence about what was running at the time including things like:
  • The setup of dask-scheduler and dask-workers.
  • Roughly what the application code was doing at the time.
  • If any signals were sent to the code (e.g. SIGINT).
  • The reported message.

Many thanks for your help.

[pitrou]

@ijstokes, please wrap your code in if __name__ == '__main__' and retry.

[pitrou]

@2gotgrossman because of how multiprocessing works under the hood. This is explained here: https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods