client.compute() not properly working with "workers" and dask.array
See original GitHub issueHi there,
client.compute() properly works with a delayed object and “workers” specified, but not with a dask array.
For an existing (distributed) client
from dask import delayed
from operator import add
x = delayed(add)(1, 2)
xx = client.compute([x],workers=['tcp://130.183.183.160:34655'])
is fine, but
import dask.array as da
y = da.random.random(10)
y = client.compute([y],workers=['tcp://130.183.183.160:34655'])
fails with:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/freya/u/cbyrohl/repos/voroILTIS/production_LAE_SBr/haloprofiles.py in <module>
1 from dask import delayed
2 y = da.random.random(10)
----> 3 y = client.compute([y],workers=['tcp://130.183.183.160:34655'])
~/anaconda3/envs/lya/lib/python3.7/site-packages/distributed/client.py in compute(self, collections, sync, optimize_graph, workers, allow_other_workers, resources, retries, priority, fifo_timeout, actors, traverse, **kwargs)
2767
2768 restrictions, loose_restrictions = self.get_restrictions(
-> 2769 collections, workers, allow_other_workers
2770 )
2771
~/anaconda3/envs/lya/lib/python3.7/site-packages/distributed/client.py in get_restrictions(cls, collections, workers, allow_other_workers)
3814 """ Get restrictions from inputs to compute/persist """
3815 if isinstance(workers, (str, tuple, list)):
-> 3816 workers = {tuple(collections): workers}
3817 if isinstance(workers, dict):
3818 restrictions = {}
TypeError: unhashable type: 'Array'
Easy workaround is to create a delayed object from the array, but I think client.compute() is supposed to work with all dask collections?
Thanks, Chris
Issue Analytics
- State:
- Created 4 years ago
- Comments:6 (6 by maintainers)
Top Results From Across the Web
Best Practices - Dask documentation
compute() within the loop would await the result of the computation every time, and so inhibit parallelism. Don't mutate inputs¶. Your functions should...
Read more >Dask Best Practices - Dask documentation
This is a short overview of Dask best practices. This document specifically focuses on best practices that are shared among all of the...
Read more >Futures - Dask documentation
Dask futures reimplements the Python futures API so you can scale your Python futures workflow across a Dask cluster.
Read more >Debug - Dask documentation
This allows you to use all of your normal Python debugging tricks in Dask computations, as long as you don't need parallelism. The...
Read more >Managing Memory — Dask.distributed 2022.12.1 documentation
Calls to Client.compute or Client.persist submit task graphs to the cluster and return ... Construct dataframe; no work happens >>> df = dd.read_csv(....
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Great. Thanks for checking @andrewfulton9 . Closing.
Closing as today one can specify which workers to compute a collection on using Dask’s annotation macheinery
xref https://github.com/dask/distributed/pull/4406