More problems with msgpack v1

This hanged nvim completely on hitting backspace in a denite buffer (I’m on the old Shougo/denite.nvim@67475c7, before the v3 rewrite).

Here’s a traceback:

2020-04-06 22:09:23,683 [DEBUG @ session.py:_yielding_request:163] 6621 - yielding from greenlet <greenlet.greenlet object at 0x7fbc9aaf0400> to wait for response
2020-04-06 22:09:23,683 [DEBUG @ msgpack_stream.py:_on_data:58] 6621 - waiting for message...
2020-04-06 22:09:23,683 [DEBUG @ msgpack_stream.py:_on_data:63] 6621 - unpacker needs more data...
2020-04-06 22:09:23,684 [DEBUG @ msgpack_stream.py:_on_data:58] 6621 - waiting for message...
2020-04-06 22:09:23,684 [DEBUG @ msgpack_stream.py:_on_data:60] 6621 - received message: [1, 1555, None, None]
2020-04-06 22:09:23,685 [DEBUG @ async_session.py:_on_response:100] 6621 - received response: None, None
2020-04-06 22:09:23,685 [DEBUG @ session.py:response_cb:159] 6621 - response is available for greenlet <greenlet.greenlet object at 0x7fbc9aaf0400>, switching back
2020-04-06 22:09:23,685 [DEBUG @ msgpack_stream.py:send:33] 6621 - sent [0, 1556, 'nvim_call_function', ('denite#util#getchar', [False])]
2020-04-06 22:09:23,685 [DEBUG @ base.py:send:118] 6621 - Sending 'b'\x94\x00\xcd\x06\x14\xb2nvim_call_function\x92\xb3denite#util#getchar\x91\xc2''
2020-04-06 22:09:23,685 [DEBUG @ session.py:_yielding_request:163] 6621 - yielding from greenlet <greenlet.greenlet object at 0x7fbc9aaf0400> to wait for response
2020-04-06 22:09:23,685 [DEBUG @ msgpack_stream.py:_on_data:58] 6621 - waiting for message...
2020-04-06 22:09:23,686 [DEBUG @ msgpack_stream.py:_on_data:63] 6621 - unpacker needs more data...
2020-04-06 22:09:23,686 [DEBUG @ msgpack_stream.py:_on_data:58] 6621 - waiting for message...
2020-04-06 22:09:23,686 [ERROR @ base_events.py:default_exception_handler:1707] 6621 - Exception in callback _UnixReadPipeTransport._read_ready()
handle: <Handle _UnixReadPipeTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.8/asyncio/unix_events.py", line 500, in _read_ready
    self._protocol.data_received(data)
  File "/usr/lib/python3.8/site-packages/pynvim/msgpack_rpc/event_loop/asyncio.py", line 60, in data_received
    self._on_data(data)
  File "/usr/lib/python3.8/site-packages/pynvim/msgpack_rpc/msgpack_stream.py", line 59, in _on_data
    msg = next(self._unpacker)
  File "msgpack/_unpacker.pyx", line 528, in msgpack._cmsgpack.Unpacker.__next__
  File "msgpack/_unpacker.pyx", line 459, in msgpack._cmsgpack.Unpacker._unpack
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

I have the latest v0.4.1 on the latest Arch Linux:

• python 3.8.2-1
• python-pynvim 0.4.1-1
• python-msgpack 1.0.0
• neovim 0.4.3-3

Downgrading to python-msgpack v0.6.2 resolves it.

See #436

---

From https://pypi.org/project/msgpack/#description and https://github.com/msgpack/msgpack-python/commit/5534d0c7af0114db3d27f7b96c82a7fe22ce1e40#diff-88b99bb28683bd5b7e3a204826ead112R40, I’m guessing the Unpacker needs the raw_as_bytes option to keep the same behaviour as before?

[jamessan]

Thanks for the ping. I’ve released a new version of pynvim.

[justinmk]

release steps are here: https://github.com/neovim/pynvim#release

Let’s see if @bfredl replies, else I will think about what to do in a few days