pickle function call uses kwargs added in python 3.8

What happened: With python 3.6.9, my job got stuck at 998/1000 and left the following log message on the worker. At this point the job didn’t continue for >10 minutes. I had to stop it manually.

distributed.worker - ERROR - 'buffers' is an invalid keyword argument for this function Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/distributed/worker.py", line 2511, in execute data[k] = self.data[k] File "/usr/local/lib/python3.6/site-packages/zict/buffer.py", line 78, in __getitem__ return self.slow_to_fast(key) File "/usr/local/lib/python3.6/site-packages/zict/buffer.py", line 65, in slow_to_fast value = self.slow[key] File "/usr/local/lib/python3.6/site-packages/zict/func.py", line 38, in __getitem__ return self.load(self.d[key]) File "/usr/local/lib/python3.6/site-packages/distributed/protocol/serialize.py", line 502, in deserialize_bytes return deserialize(header, frames) File "/usr/local/lib/python3.6/site-packages/distributed/protocol/serialize.py", line 302, in deserialize return loads(header, frames) File "/usr/local/lib/python3.6/site-packages/distributed/protocol/serialize.py", line 64, in pickle_loads return pickle.loads(x, buffers=buffers) File "/usr/local/lib/python3.6/site-packages/distributed/protocol/pickle.py", line 64, in loads return pickle.loads(x, buffers=buffers) TypeError: 'buffers' is an invalid keyword argument for this function

What you expected to happen: My job finishes

Minimal Complete Verifiable Example: On python 3.6, run

>>> from distributed.protocol import pickle
>>> pickle.loads(b"", buffers=(1))
...
TypeError: 'buffers' is an invalid keyword argument for this function

Anything else we need to know?: There is a backport of pickle5 to 3.6 on https://pypi.org/project/pickle5/

I am trying to work around that problem as follows. Thx @samaust for the remark in #3843 .

import pickle5
from distributed.protocol import pickle
pickle.pickle = pickle5

def pickle_dumps(x):
    header = {"serializer": "pickle"}
    frames = [None]
    buffer_callback = lambda f: frames.append(memoryview(f))
    frames[0] = pickle.dumps(x, buffer_callback=buffer_callback)
    return header, frames

def pickle_loads(header, frames):
    x, buffers = frames[0], frames[1:]
    return pickle.loads(x, buffers=buffers)

from distributed.protocol.serialize import register_serialization_family
register_serialization_family('pickle', pickle_dumps, pickle_loads)

Environment:

• Python version: 3.6.9
• Operating System: ubuntu
• Install method (conda, pip, source): pip

[jakirkham]

The fix is in distributed version 2.18.0. So please upgrade to get the fix 🙂

[jakirkham]

Great, thanks for the feedback! 😄