Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Array > 2GB hitting `msgpack` limit

See original GitHub issue

I know that there’s supposed to be alternate protocols used for larger arrays but I’m not sure what needs to done to use them (or if they don’t play nice with map_blocks?

MRE:

import numpy
import dask.array as da
from distributed import Client

client = Client('127.0.0.1:8786')
def increment_by_one(my_array):
    return my_array + 1

data = numpy.random.random(300000000)
chunks = (10000,)
data = da.from_array(data, chunks=chunks)
output = da.map_blocks(increment_by_one, data)
output.compute()

Traceback:

distributed.utils - ERROR - 2400000161 exceeds max_bin_len(2147483647)
Traceback (most recent call last):
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/distributed/utils.py", line 207, in log_errors
    yield
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/distributed/client.py", line 460, in _handle_report
    six.reraise(*clean_exception(**msg))
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/tornado/gen.py", line 1024, in run
    yielded = self.gen.send(value)
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/distributed/core.py", line 258, in read
    msg = protocol.loads(frames)
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/distributed/protocol.py", line 152, in loads
    msg = loads_msgpack(small_header, small_payload)
  File "/home/gil/anaconda/envs/dasknumbagpu/lib/python3.5/site-packages/distributed/protocol.py", line 256, in loads_msgpack
    return msgpack.loads(payload, encoding='utf8')
  File "pandas/msgpack/_unpacker.pyx", line 138, in pandas.msgpack._unpacker.unpackb (pandas/msgpack/_unpacker.cpp:2059)
ValueError: 2400000161 exceeds max_bin_len(2147483647)