'distributed.protocol.core - CRITICAL - Failed to deserialize' -- With Cupy

Here is the code up to the point I get this error message:

import cupy as cp
import numpy as np
import dask.array as da
from dask.bag import read_text
from dask.distributed import Client

def row_to_numpy(row):
    return np.array([np.float(x) for x in row.split()])

client = Client()

S = read_text(my text file).str.strip().map(row_to_numpy)
S = da.stack(S, axis=0).map_blocks(cp.array)

u_old = da.random.random(T) 
u_old = v.map_blocks(cp.array)

_V_  = client.scatter(v, broadcast=True)
v = da.dot(_U_.result(),S).compute()

Basically I am reading in a txt file into a dask cupy array. I get this error message everytime I map as cupy, (numpy results in no errors). I am broadcasing this 1D array to all of the workers before doing this matrix multiplication. I looked at most other tickets and I couldn’t quite see a similar issue. All of my workers are updated and I have read the Serialization page in the docs but wasn’t sure how that would help me.

I am getting this error message:

distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/home/myid/crumleyc/anaconda3/lib/python3.7/site-packages/distributed/protocol/core.py", line 124, in loads
    value = _deserialize(head, fs, deserializers=deserializers)
  File "/home/myid/crumleyc/anaconda3/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 268, in deserialize
    return loads(header, frames)
  File "/home/myid/crumleyc/anaconda3/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 54, in dask_loads
    return loads(header, frames)
  File "/home/myid/crumleyc/anaconda3/lib/python3.7/site-packages/distributed/protocol/cupy.py", line 81, in dask_deserialize_cupy_ndarray
    arr = cuda_deserialize_cupy_ndarray(header, frames)
  File "/home/myid/crumleyc/anaconda3/lib/python3.7/site-packages/distributed/protocol/cupy.py", line 65, in cuda_deserialize_cupy_ndarray
    memptr=cupy.asarray(frame).data,
  File "/home/myid/crumleyc/anaconda3/lib/python3.7/site-packages/cupy/creation/from_data.py", line 66, in asarray
    return core.array(a, dtype, False, order)
  File "cupy/core/core.pyx", line 1692, in cupy.core.core.array
  File "cupy/core/core.pyx", line 1718, in cupy.core.core.array
  File "cupy/core/core.pyx", line 2498, in cupy.core.core._convert_object_with_cuda_array_interface
TypeError: zip argument #2 must support iteration
distributed.core - ERROR - zip argument #2 must support iteration

[jakirkham]

Sorry for the delay. Sounds like the last exception is related to issue ( https://github.com/cupy/cupy/issues/3178 ).

[crumleyc]

I went ahead and added some bad doc strings and fixed the indents!