A lot of objects are being created

What happened: Whenever I use the extension after a while I start getting: distributed.utils_perf - WARNING - full garbage collections took x% CPU time recently (threshold: 10%), and the percentage just keeps increasing.

What you expected to happen: Not getting these warnings.

Minimal Complete Verifiable Example:

import gc
from collections import Counter

from dask.distributed import Client

def add(x, y):
    return x + y

def obj_types(n=10):
    return Counter(str(type(obj)) for obj in gc.get_objects()).most_common(n)

client = Client()

Link the extension to the created client.

future = client.submit(add, 1, 2)

By just running the code above and leaving the notebook idle I can see that the counts for bokeh objects start to rise. Run 1:

client.run_on_scheduler(obj_types)
# [("<class 'dict'>", 143178),
#  ("<class 'tuple'>", 60317),
#  ("<class 'list'>", 42544),
#  ("<class 'set'>", 41334),
#  ("<class 'function'>", 35093),
#  ("<class 'bokeh.core.property.wrappers.PropertyValueDict'>", 23340),
#  ("<class 'parso.python.tree.Operator'>", 17625),
#  ("<class 'bokeh.core.property.wrappers.PropertyValueList'>", 13784),
#  ("<class 'parso.python.tree.PythonNode'>", 13126),
#  ("<class 'parso.python.tree.Name'>", 12038)]

Run 2 (after like 10-15 minutes):

client.run_on_scheduler(obj_types)
# [("<class 'dict'>", 510971),
#  ("<class 'tuple'>", 212323),
#  ("<class 'set'>", 178042),
#  ("<class 'bokeh.core.property.wrappers.PropertyValueDict'>", 106980),
#  ("<class 'bokeh.core.property.wrappers.PropertyValueList'>", 62999),
#  ("<class 'list'>", 59689),
#  ("<class 'function'>", 41300),
#  ("<class 'cell'>", 21047),
#  ("<class 'parso.python.tree.Operator'>", 17625),
#  ("<class 'parso.python.tree.PythonNode'>", 13126)]

Anything else we need to know?: Please let me know if I can provide some more insights. I don’t have much experience with the gc so I’m not sure if that’s enough information.

Environment:

• Dask version: 2021.2.0
• Python version: 3.8.8
• Operating System: Ubuntu 20.10
• Install method (conda, pip, source): conda

[ian-r-rose]

Fixed in 5.0.2

[jose-moralez]

Thanks for the fix! Happy to help.