`No dispatch for <class 'numpy.ndarray'>`

This deserialization error then caused https://github.com/dask/distributed/issues/7060:

2022-09-23 19:01:53,760 - distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/distributed/protocol/core.py", line 158, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/usr/local/lib/python3.9/site-packages/distributed/protocol/core.py", line 138, in _decode_default
    return merge_and_deserialize(
  File "/usr/local/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 497, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/usr/local/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 426, in deserialize
    return loads(header, frames)
  File "/usr/local/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 58, in dask_loads
    loads = dask_deserialize.dispatch(typ)
  File "/usr/local/lib/python3.9/site-packages/dask/utils.py", line 632, in dispatch
    raise TypeError(f"No dispatch for {cls}")
TypeError: No dispatch for <class 'numpy.ndarray'>

AFAICT, NumPy was installed on the worker. There are plenty of other NumPy keys in memory on the same worker. So I’m not sure why this happened.

Unfortunately, I don’t have more information here. Not sure how easily reproducible it is.

[quasiben]

cc @jakirkham

[zklaus]

Just to comment that I have put a PR up at dask/dask#9544 that addresses the problem for me locally. It is a very rough draft, but I’d be interested to know if you think that general approach is worth pursuing.