Series.values.compute() leads to "TypeError: can't concat buffer to bytearray"
See original GitHub issueRunning a local dask-scheduler
+ dask-worker
pair, the following code leads to a crash of the worker:
import dask.dataframe as dd
from dask.distributed import Executor
e = Executor('127.0.0.1:8786', set_as_default=True)
df = pd.DataFrame({"A": [1, 2, 3] * 10})
ddf = dd.from_pandas(df, npartitions=3)
ddf["A"].values.compute()
The worker crashes with:
Traceback (most recent call last):
File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/core.py", line 259, in handle_comm
result = yield result
File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/worker.py", line 439, in get_data
compressed = yield comm.write(msg)
File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 292, in wrapper
result = func(*args, **kwargs)
File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/comm/tcp.py", line 196, in write
stream.write(frame)
File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/iostream.py", line 395, in write
self._write_buffer += data
TypeError: can't concat buffer to bytearray
Other computations like ddf["A"].compute()
or even ddf.values.compute()
work fine though.
Issue Analytics
- State:
- Created 6 years ago
- Comments:23 (22 by maintainers)
Top Results From Across the Web
python - Can't concat bytes to str - Stack Overflow
subprocess.check_output() returns a bytestring. In Python 3, there's no implicit conversion between unicode ( str ) objects and bytes objects.
Read more >ArrayBuffer, binary arrays - The Modern JavaScript Tutorial
Uint8Array – treats each byte in ArrayBuffer as a separate number, with possible values from 0 to 255 (a byte is 8-bit, so...
Read more >How to Convert Int to Bytes in Python? - GeeksforGeeks
An int value can be converted into bytes by using the method int.to_bytes(). The method is invoked on an int value, is not...
Read more >Buffer Protocol — Python 3.11.1 documentation
Certain objects available in Python wrap access to an underlying memory array or buffer. Such objects include the built-in bytes and bytearray ,...
Read more >TypeError when Buffer.concat · Issue #52588 · microsoft/vscode
Issue Type: Bug It is workable in js: function getBuffer(readable) { let bytes = new Buffer(0); let length = 0; return new Promise((resolve, ......
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
For the record:
Lets wait for @pitrou to take a look at the error. He is more familiar with the networking stack than I am (I think he wrote the bytearray code in tornado) and may have thoughts. Unfortunately I think he’s on vacation until Monday.