Ready queue mutation, IndexError: pop from an empty deque

Hey,

First, thanks for sharing the patch and publishing this library. I’m trying to workaround re-entrancy while converting a codebase incrementally to asyncio, where there are call chains of async > loop.run_until_complete(async).

I’ve hit an issue with the patch with handles from the _ready queue seemingly being popped from elsewhere. I.e. from python3.6/asyncio/base_events.py popping from the ready queue throws, because it iterates too far.

        ntodo = len(self._ready)
        for i in range(ntodo):
            handle = self._ready.popleft() # throws.
            if handle._cancelled:
                continue

The full exception stack trace looks like this:

return aio_batcher.run_until_complete(aio_batcher.evaluate_coroutines(json_data))
  File "/Users/cory/src/nd/util/aio_batcher/decorators.py", line 50, in run_until_complete
    return loop.run_until_complete(awaitable)
  File "/var/lib/conda/envs/nextdoor3/lib/python3.6/site-packages/nest_asyncio.py", line 63, in run_until_complete
    return self._run_until_complete_orig(future)
  File "/var/lib/conda/envs/nextdoor3/lib/python3.6/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/var/lib/conda/envs/nextdoor3/lib/python3.6/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/var/lib/conda/envs/nextdoor3/lib/python3.6/asyncio/base_events.py", line 1411, in _run_once
    handle = self._ready.popleft()
IndexError: pop from an empty deque

My hypothesis is that one of the enqueued items triggers the nest_async, or else an enqueued item is processing other elements from the queue.

It is the worst kind of problem where it is reproducible, but I have yet to isolate a simple test case to make it easy to understand. Do you have a hunch of what might be happening? Perhaps something you ran into while writing the patch?

Thanks for your help, Cory

[erdewit]

The repo has been updated to follow Cory’s approach of always using the patched loop. It also patches run_forever now instead of run_until_complete and with this it can stay closer to the base asyncio code.

I believe this should fix the issues, if it doesn’t let me know. I’ll keep this bug open for a while and when it works okay a 1.1 version will be released.

[ostruk]

Happy to help!