vim.Function serialisation in `to_client`

My knowledge about vim internals is almost nothing, let alone my neovim knowledge. That being said…

I am using pynvim in combination with deoplete. I just filed https://github.com/Shougo/deoplete.nvim/issues/884, but it seems like it’s not deoplete’s fault. At a certain point, deoplete requests vim_eval of the global variable scope g:, and apparently I have a few function pointers in there.

This is an excerpt of the RPC server log:

2018-11-30 21:34:21,484 [INFO @ neovim_rpc_server.py:process_pending_requests:372] 15119 - get msg from channel [1]: [0, 39, 'nvim_eval', ['g:']]
2018-11-30 21:34:21,484 [INFO @ neovim_rpc_server.py:process_pending_requests:400] 15119 - sending result: [1, 39, None, <vim.dictionary object at 0x7f9b3cbc0b40>]
2018-11-30 21:34:21,484 [INFO @ neovim_rpc_protocol.py:handler:66] 15119 - serialising obj Function <vim.Function '2'>
2018-11-30 21:34:21,484 [INFO @ neovim_rpc_protocol.py:handler:66] 15119 - serialising obj Function <vim.Function '1'>

This is after I patched neovim_rpc_protocol.py to print whenever it’s serialising an Function-typed variable (after line 65). If, instead of return obj.name I return an empty string '', deoplete works again 😃

So, I think the serialisation of Functions is incorrect, but I have no clue how it should be done.

For completeness sake, this is what happens normally:

Traceback (most recent call last):
  File "/usr/share/vim/vimfiles/pythonx/neovim_rpc_server.py", line 400, in process_pending_requests
    packed = msgpack.packb(neovim_rpc_protocol.to_client(result))
  File "/usr/share/vim/vimfiles/pythonx/neovim_rpc_protocol.py", line 66, in to_client
    return walk(handler, msg)
  File "/usr/share/vim/vimfiles/pythonx/neovim_rpc_protocol.py", line 20, in walk
    return list(walk(fn, o) for o in obj)
  File "/usr/share/vim/vimfiles/pythonx/neovim_rpc_protocol.py", line 20, in <genexpr>
    return list(walk(fn, o) for o in obj)
  File "/usr/share/vim/vimfiles/pythonx/neovim_rpc_protocol.py", line 23, in walk
    obj.items())
  File "/usr/share/vim/vimfiles/pythonx/neovim_rpc_protocol.py", line 22, in <genexpr>
    return dict((walk(fn, k), walk(fn, v)) for k, v in
  File "/usr/share/vim/vimfiles/pythonx/neovim_rpc_protocol.py", line 20, in walk
    return list(walk(fn, o) for o in obj)
  File "/usr/share/vim/vimfiles/pythonx/neovim_rpc_protocol.py", line 20, in <genexpr>
    return list(walk(fn, o) for o in obj)
  File "/usr/share/vim/vimfiles/pythonx/neovim_rpc_protocol.py", line 24, in walk
    return fn(obj)
  File "/usr/share/vim/vimfiles/pythonx/neovim_rpc_protocol.py", line 64, in handler
    return obj.name
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

Apparently, the function name contains a non-UTF-8 character?

EDIT: When I access the obj.name field in the errornous condition, even by trying to print type(obj.name), I get the UTF-8 error. It makes it impossible to access the field 😂

Please let me know if more information is required.

[bfredl]

@rubdos This repo’s (pynvim) original purpose was to implement the python support for neovim only, and not vim 8. Then after vim 8 gained async support, https://github.com/roxma/vim-hug-neovim-rpc/ was implemented as a bridge to use some neovim async plugins also in vim 8. This bridge relies on the builtin if_python3 support of vim 8 itself, which unfortunately has some limitations that neovim/pynvim doesn’t have (neovim is 8-bit clean for all supported python versions, vim 8 is not 8-bit clean for python3).

The issue is happening in this bridge https://github.com/roxma/vim-hug-neovim-rpc/blob/master/pythonx/neovim_rpc_protocol.py which is before pynvim sees the data. You could file this issue at https://github.com/roxma/vim-hug-neovim-rpc/ which might add a workaround, such as catching the exception and ignoring these names that cause trouble.

[rubdos]

The issue is happening in this bridge https://github.com/roxma/vim-hug-neovim-rpc/blob/master/pythonx/neovim_rpc_protocol.py which is before pynvim sees the data. You could file this issue at https://github.com/roxma/vim-hug-neovim-rpc/ which might add a workaround, such as catching the exception and ignoring these names that cause trouble.

Seems like someone else discovered it there indeed: https://github.com/roxma/vim-hug-neovim-rpc/pull/45

Excuse me for misidentifying the repository of neovim_rpc_protocol.py. I should take more time reading these things.

Thank you for your help.