Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

thread.lock serialization error when passing a custom function to a local cluster

See original GitHub issue

I’m using dask 0.14.0, distributed 1.16.0, and python 2.7.11 (all installed via conda) on MacOS 10.11.6. When I try to run certain custom Python functions that call various other packages with binary extensions on a local cluster via the following code,

from distributed import Client, LocalCluster

def func(x):
    # my custom function

args = # some list of args

lc = LocalCluster(10)
c = Client(lc)
results = c.gather(c.map(func, args))
lc.close()

I observe the following exception:

Traceback (most recent call last):
  File "process.py", line 151, in <module>
    results = c.gather(c.map(func, args))
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/distributed/client.py", line 885, in map
    loose_restrictions, priority=priority, resources=resources)
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/distributed/client.py", line 1455, in _graph_to_futures
    'tasks': valmap(dumps_task, dsk3),
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/toolz/dicttoolz.py", line 84, in valmap
    rv.update(zip(iterkeys(d), map(func, itervalues(d))))
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/distributed/worker.py", line 601, in dumps_task
    return {'function': dumps_function(task[0]),
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/distributed/worker.py", line 568, in dumps_function
    b = dumps(func)
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/distributed/protocol/pickle.py", line 54, in dumps
    return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 706, in dumps
    cp.dump(obj)
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 146, in dump
    return Pickler.dump(self, obj)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 264, in save_function
    self.save_function_tuple(obj)
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 317, in save_function_tuple
    save(f_globals)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 264, in save_function
    self.save_function_tuple(obj)
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 317, in save_function_tuple
    save(f_globals)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 604, in save_reduce
    save(state)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 604, in save_reduce
    save(state)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 642, in _batch_appends
    save(tmp[0])
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 604, in save_reduce
    save(state)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/lebedov/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 604, in save_reduce
    save(state)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File "/Users/lebedov/anaconda2/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
TypeError: can't pickle thread.lock objects

There seems to be a way to address this when using dask’s own interfaces; is there some way to use the serializable lock with a custom function?

The package I’m calling from within my function, incidentally, is SimpleElastix.

Issue Analytics

State:
Created 7 years ago
Comments:10 (5 by maintainers)

Top GitHub Comments

2reactions

lebedovcommented, Mar 17, 2017

Seems like the problem was a global logger instance that I was accessing in my function.

0reactions

lebedovcommented, Sep 7, 2018

@annvakulchyk I was able to log by creating a logger instance inside the function I passed to Client.map and using that to emit log messages.

Top Results From Across the Web

Task not serializable: java.io.NotSerializableException when ...

I modify custom RDD transformations that I encounter or personally write by using explicit broadcast variables and utilizing the new inbuilt twitter-chill api, ......

Threading in C# - Part 2 - Basic Synchronization

Exclusive locking constructs are most common — these allow just one thread in at a time, and allow competing threads to access common...

Serialization — Ray 2.2.0 - the Ray documentation

When data is put into the object store, it does not get automatically broadcasted to other nodes. Data remains local to the writer...

Transaction locking and row versioning guide - SQL Server

If a statement completes successfully, it is committed; if it encounters any error, it is rolled back. A connection to an instance of...

Embarrassingly parallel for loops - Joblib - Read the Docs

To hint that your code can efficiently use threads, just pass ... For instance, it cannot serialize functions which are defined interactively or...