LocalCluster CPU usage climbs to 100% over time without submitting computations, connecting workers

I noticed when I was working locally in a Jupyter Lab session on my laptop that computations would gradually slow down, down, down, until there was nothing to do but restart the notebook kernel (and therefore scheduler/workers). I started to pare down a minimal reproducing example, and got all the way down to launching a LocalCluster with zero workers. After five minutes, CPU use is hovering around 100%, and a few minutes later I start getting the message distributed.utils_perf - WARNING - full garbage collections took 19% CPU time recently repeated every ~1s. This is all without connecting a worker or trying to do any computation.

What happened: Requesting a LocalCluster results in a process that grows to use 100% of the CPU even when no workers are launched and no computations are requested.

What you expected to happen: CPU use should remain roughly static when not under load

Minimal Complete Verifiable Example:

from dask.distributed import Client, LocalCluster
cluster = LocalCluster(n_workers=0)
client = Client(cluster.scheduler.address)
# wait 5 minutes and check CPU use

Anything else we need to know?:

I also tried setting the loop implementation to uvloop in ~/.config/dask/dask.yaml and the issue persisted. Above screenshots are from running with the default config except for:

logging:
  distributed: debug
  distributed.client: warning
  distributed.scheduler: debug
  bokeh: error

Environment:

• Dask version: 2021.5.0
• Python version: 3.9.4
• Operating System: macOS Big Sur 11.3.1
• Install method (conda, pip, source): conda

[joseph-long]

Tried the new release and it worked for me, but forgot to comment. Thanks!

[jrbourbeau]

Thanks for reporting this issue @joseph-long! I tried this out locally and observed similar behavior.

cc @ian-r-rose due to the Jupyter connection