Fail in cluster for load my lib

Describe the issue:

I tried using LocalCluster to process my data, using the code below. I had no problem with the empty Client() object, but when I put the cluster in the client. I will get some errors.

The task for do_prediction is:

1. use URL to read file in S3 and do feature engineering
2. use a pretrained model which is loaded from a pickle file to predict the data
3. upload prediction result to S3

The feature engineering part in do_prediction used some function in my utl.py

all_meter_lst = load('all_meter_lst.pkl')
for cks in chunks(all_meter_lst, 999):
    lazy_results = []
    for meter_url in cks:
        lazy_result = dask.delayed(do_prediction)(meter_url)
        lazy_results.append(lazy_result)
    dask.compute(*lazy_results)

# Not work!!!!!
# cluster = LocalCluster(n_workers=8, processes=True)
# client = Client(cluster)
# Work!!!!!
client = Client()
client

The error I got:

2022-11-25 03:09:13,886 - distributed.worker - ERROR - Could not deserialize task do_prediction-3a99f190-0611-4bc9-bc7f-1764cb6fe83b
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/lake/lib/python3.10/site-packages/distributed/worker.py", line 2219, in execute
    function, args, kwargs = await self._maybe_deserialize_task(ts)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/lake/lib/python3.10/site-packages/distributed/worker.py", line 2192, in _maybe_deserialize_task
    function, args, kwargs = _deserialize(*ts.run_spec)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/lake/lib/python3.10/site-packages/distributed/worker.py", line 2863, in _deserialize
    function = loads_function(function)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/lake/lib/python3.10/site-packages/distributed/worker.py", line 2857, in loads_function
    return pickle.loads(bytes_object)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/lake/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x)
ModuleNotFoundError: No module named 'utl'

• Dask version: latest
• Python version: 3.10
• Operating System: macos
• Install method (conda, pip, source):

[phobson]

I’m going to close this for now. @b-y-f, ping me on the Discourse channel when you have something going on there.

[phobson]

@b-y-f I’m trying to figure out what’s going on here. I think this would be better discussed in out Discourse Group. But I am curious if you get better results defining your cluster and client before asking dask to do a computation. By that I mean:

cluster = LocalCluster(n_workers=8, processes=True)
client = Client(cluster)
all_meter_lst = load('all_meter_lst.pkl')
for cks in chunks(all_meter_lst, 999):
    lazy_results = []
    for meter_url in cks:
        lazy_result = dask.delayed(do_prediction)(meter_url)
        lazy_results.append(lazy_result)
    dask.compute(*lazy_results)

You’d probably be better off with a dask.bag instead of using chunks, e.g.,

cluster = LocalCluster(n_workers=8, processes=True)
client = Client(cluster)
all_meter_bag = dask.bag.from_sequence(load('all_meter_lst.pkl'))
results = all_meter_bag.map(do_prediction).compute()