question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

client.compute() not properly working with "workers" and dask.array

See original GitHub issue

Hi there,

client.compute() properly works with a delayed object and “workers” specified, but not with a dask array.

For an existing (distributed) client

from dask import delayed
from operator import add
x = delayed(add)(1, 2)
xx = client.compute([x],workers=['tcp://130.183.183.160:34655'])

is fine, but

import dask.array as da
y = da.random.random(10)
y = client.compute([y],workers=['tcp://130.183.183.160:34655'])

fails with:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/freya/u/cbyrohl/repos/voroILTIS/production_LAE_SBr/haloprofiles.py in <module>
      1 from dask import delayed
      2 y = da.random.random(10)
----> 3 y = client.compute([y],workers=['tcp://130.183.183.160:34655'])

~/anaconda3/envs/lya/lib/python3.7/site-packages/distributed/client.py in compute(self, collections, sync, optimize_graph, workers, allow_other_workers, resources, retries, priority, fifo_timeout, actors, traverse, **kwargs)
   2767 
   2768         restrictions, loose_restrictions = self.get_restrictions(
-> 2769             collections, workers, allow_other_workers
   2770         )
   2771 

~/anaconda3/envs/lya/lib/python3.7/site-packages/distributed/client.py in get_restrictions(cls, collections, workers, allow_other_workers)
   3814         """ Get restrictions from inputs to compute/persist """
   3815         if isinstance(workers, (str, tuple, list)):
-> 3816             workers = {tuple(collections): workers}
   3817         if isinstance(workers, dict):
   3818             restrictions = {}

TypeError: unhashable type: 'Array'

Easy workaround is to create a delayed object from the array, but I think client.compute() is supposed to work with all dask collections?

Thanks, Chris

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:6 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
mrocklincommented, Sep 26, 2020

Great. Thanks for checking @andrewfulton9 . Closing.

0reactions
jrbourbeaucommented, Mar 20, 2021

Closing as today one can specify which workers to compute a collection on using Dask’s annotation macheinery

xref https://github.com/dask/distributed/pull/4406

Read more comments on GitHub >

github_iconTop Results From Across the Web

Best Practices - Dask documentation
compute() within the loop would await the result of the computation every time, and so inhibit parallelism. Don't mutate inputs¶. Your functions should...
Read more >
Dask Best Practices - Dask documentation
This is a short overview of Dask best practices. This document specifically focuses on best practices that are shared among all of the...
Read more >
Futures - Dask documentation
Dask futures reimplements the Python futures API so you can scale your Python futures workflow across a Dask cluster.
Read more >
Debug - Dask documentation
This allows you to use all of your normal Python debugging tricks in Dask computations, as long as you don't need parallelism. The...
Read more >
Managing Memory — Dask.distributed 2022.12.1 documentation
Calls to Client.compute or Client.persist submit task graphs to the cluster and return ... Construct dataframe; no work happens >>> df = dd.read_csv(....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found