Idle memory use increasing over time

In the process of debugging some memory issues I noticed that memory usage of a scheduler+worker with no client connection was steadily increasing over time.

Command: dask-scheduler --no-bokeh & dask-worker localhost:8786 --nthreads 1 --nprocs 120 --memory-limit 3.2GB --no-bokeh & (default config.yaml) Result: after a few idle hours, total memory usage went from about 1GB at startup to 4GB (as reported by the Google Cloud dashboard).

I’m aware that there are a lot of subtleties around measuring memory usage on Linux so I’m not sure if this is a real issue or maybe an artifact of the measurement process, but it seemed like a lot of memory for totally inactive processes. Curious if anyone has any thoughts about what might be happening.

[cpaulik]

We are still running into this issue. And I was not yet able to find a minimal example.

There are two things that I noticed in our processing chain that might cause issues:

1. We are running in docker containers on Google Cloud. Our idle workers always have 10% memory consumption. In https://github.com/dask/distributed/issues/2079 I understood that this should not be the case so it might be related to some container settings.

2. Our processing uses compiled extensions that do not release the GIL. Could that be a cause for a memory increase?

I will try to get a minimal example still but I thought this information might help narrow it down a little.

[ameetshah1983]

We are running dask scheduler on windows VM and memory utilization gradually increases till system memory usage reaches 98%. We then have to restart the scheduler as else we receive timeouts from workers trying to connect. This does take a few days and our allocated memory for VM is 16GB.

We are currently on distributed ‘1.21.3’ and dask ‘0.17.1’

Sorry one thing to add, in our case the grid is not completely idle but do have jobs running from time to time. Please let me know if this should be listed as a separate issue in that case.