Cannot execute a vim command from a new asyncio task

To reproduce: Start Neovim with NVIM_PYTHON_LOG_FILE set, and run these commands:

py3 async def foo(): vim.out_write('testing')
py3 import asyncio, vim; asyncio.ensure_future(foo())

Then look in the python log file (_script suffix) and see:

2018-06-30 20:20:29,362 [ERROR @ base_events.py:default_exception_handler:1266] 31555 - Task exception was never retrieved
future: <Task finished coro=<foo() done, defined at <string>:1> exception=AttributeError("'NoneType' object has no attribute 'switch'",)>
Traceback (most recent call last):
  File "<string>", line 1, in foo
  File "/usr/local/lib/python3.6/site-packages/neovim/api/nvim.py", line 378, in out_write
    return self.request('nvim_out_write', msg, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/neovim/api/nvim.py", line 170, in request
    res = self._session.request(name, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/neovim/msgpack_rpc/session.py", line 91, in request
    v = self._yielding_request(method, args)
  File "/usr/local/lib/python3.6/site-packages/neovim/msgpack_rpc/session.py", line 160, in _yielding_request
    return parent.switch()
AttributeError: 'NoneType' object has no attribute 'switch'

[bfredl]

nvim-managed greenlets and asyncio coroutines are like water and oil, they are incompatible event/async primitives that simply don’t mix: any attempt to “seamlessly” integrate them might fail in the first request stack/event handing situation that wasn’t explicitly considered.

What we could add is (1) a greenlet-callable function, that runs an asyncio coro/future until completion and then resumes the greenlet and (2) an awaitable function that suspends a coroutine and runs a closure as a greenlet until it completes and then resumes the coroutine. Though this will more or less be callback hell and thus have none of the benefits of either the stackful greenlet approach or the explicitly yieldable approach of asyncio coroutines.

Rather what I would imagine useful is to reimplement the python client using only asyncio and native coroutines and no greenlets, I suspect that will be a lot simpler than the exisiting multi-layer client: it will more or less be a library providing a nvim asyncio protocol for its native transports, and request awaitables for coroutines which asyncio otherwise manages by itself.

[bfredl]

though wouldn’t there be a backwards compatibility problem?

Yes. It would need be a separate project from this. I wouldn’t even think “fork”, rather new codebase which can happen to borrow code from this where it makes sense. Starting from scratch with no regards of vim compatibility (which necessitates a sync-like interface backed by greenlets or similar) would give a lot more freedom for both simplicity and expressivity. For instance call_atomic could be exposed as a function converting multiple request awaitables to a single awaitable, while with this codebase to use call_atomic you must throw all existing abstractions above the raw API out through the window.