Memory doesn't get freed up after changing reference

df = dd.read_csv('s3://xx/*.csv')
e.persist(df) #first result persists in memory

df = dd.read_csv('s3://yy/*.csv')
e.persist(df)

the memory on the nodes grows and df now references to the new data.However, it appears that original df doesnt get cleared in the worker memory. Is this expected behavior?

[hussainsultan]

I disconnected all my Clients but the memory doesnt get freed up all the way. I looked at the scheduler logs and it registers my client disconnect.

[mrocklin]

If you’re interested in doing even more work, distributed/http/scheduler.py is easy to extend. The relevant state here is Scheduler.who_wants and Scheduler.wants_what.