dask not respecting worker resources

I believe this issue is related to this SO post.

I’m finding that, despite resource restrictions, the dask-scheduler will assign keys to worker nodes even when I’m specifying otherwise.

For example:

from dask.distributed import LocalCluster, Client
cluster = LocalCluster(n_workers=0)
cluster.start_worker(ncores=1)  #  resources={'CPU': 1}, no resources to compute

client = Client(cluster.scheduler.address)
fut = client.submit(lambda x=1: x+1, resources={'CPU':1})
client.who_has()  # here we expect to see nothing as no workers have resources

fut2 = client.submit(lambda x=1: x+1)
client.who_has() # yep, assigned because it doesn't need resources

import dask.dataframe as dd
import pandas.util.testing as tm
import os 
files = dd.from_pandas(tm.makeTimeSeries(1000, freq='10ms'), npartitions=4).to_csv('example*.csv')

# simulate some pipeline where data is read and transformed 
fut3 = client.compute(dd.read_csv('example*.csv').to_delayed(), resources={'CPU':1})
client.who_has()  
# now we see that the scheduler has placed the delayed keys 
# onto the worker even though it has no resources to compute.
#  "('from-delayed-238ec9c6404d8e52399becf66971834c', 2)": ('tcp://127.0.0.1:44429',),
#  "('pandas_read_text-read-block-from-delayed-238ec9c6404d8e52399becf66971834c', 2)": (),
[os.remove(f) for f in files]

Likewise, I’m finding similar behavior with methods like read_parquet.

dask.__version__
'1.2.2'

[nuKs]

I did checked with htop and resources are respected in my case, even though worker’s “Processing” tab displays multiples “ongoing task”, only the top one task in the worker task list is actually being processed.

I have Consumed > Total as well. I think it is normal, as consumed tasks amounts for all tasks that have moved through the worker:

• Some may have moved to other worker, so “Consumed” > “Finished task”
• There can be more “Processing” tasks than there are “Consumed”.
• Total == “Most at a single time”. Still a bit confusing to me though, this is my interpretation.

[bw4sz]

My experience here is the same as above. Consumed >total, but if you look at the call stacks, only the top task is actually being computed, which is good news.

My conclusion is that resources behavior is acting properly, but that the ‘consumed’ label is just a bit confusing. My intuition was that dask would label the other tasks as ‘waiting’ and not ‘processing’, because all resources are occupied. Hope this helps others.