WARNING - Memory use is high but worker has no data to store to disk

I end up getting a lot of error messages like this:

distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 6.15 GB -- Worker memory limit: 8.45 GB

I’m relatively sure that this warning is actually true. Also, the workers hitting this warning end up in idling all the time.

Is there a way to automatically retire + restart the workers that hit this error message?

Anything else we need to know?: I’m using Dask v2.30

Environment:

• Dask version: v2.30
• Python version: 3.7.8
• Operating System: Centos 7
• Install method (conda, pip, source): conda

[Hoeze]

@KrishanBhasin I am currently running with a fork of d7f532caa1564ef09d456d60125c03200fa60fef (see #4221). So, yes, I am already using the fixes from 2.30.1, but they do not seem to solve the issue.

The workers still keep freezing and I cannot get my calculations to finish.

[mrocklin]

The testing infrastructure is pretty good for this, especially if you can write async code: https://distributed.dask.org/en/latest/develop.html#writing-tests

Dask can run itself entirely in one thread, which really helps with debugging.

On Sat, Oct 31, 2020 at 11:59 AM Florian R. Hölzlwimmer < [e-mail]> wrote:

OK sure. I’m happy to debug 😃 Nonetheless if you have tips how to get dask workers into debug mode (with pycharm) I’d be even more happy 😄

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/4193#issuecomment-719973852, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTHXTTLIXQJJJP63MGLSNRM7XANCNFSM4TBKFTIQ .