'IndexError: pop from empty list' Exception in Queue (V3) due to uncleared _evput state

This is using the temporary V3 Queue. Not sure if Im wasting my breath if there is already a new version somewhere else (at least I dont know of it…)? Anyway, I used this one and ran into a tad bit of an annoyance 😃

Due to some “unfortunate” scheduling of routines, one can get greeted by a lovely “IndexError: pop from empty list” in the get() method of queue, where one would expect it to wait if the queue is empty - sadly doesn’t always work…

I believe I found the reason (and fix) but first a minimal example and my explanation.

import uasyncio as asyncio
import Queue


async def putter(q, sleep_ms=50):
    # put some item, then sleep
    while True:
        await q.put(1)
        await asyncio.sleep_ms(sleep_ms)


async def getter(q):
   # checks for new items, and relies on the "blocking" of the get method
    while True:
        print(await q.get())


async def task():
    q = Queue()

    await asyncio.gather(
        putter(q),
        getter(q)
    )

asyncio.run(task())

The error appears only under some (scheduling) circumstances. One instance is:

• if the putter routine is called first, meaning some item is placed in the queue, making it not empty and the _evput will be set (https://github.com/peterhinch/micropython-async/blob/master/v3/primitives/queue.py#L47).
• if now the getter method is executed, it will return the item placed in queue as the ‘if self.empty():’ check (https://github.com/peterhinch/micropython-async/blob/master/v3/primitives/queue.py#L34) will evaluate to False.
• if the getter methods now goes for the next iteration, BEFORE another item is placed (for example due to the putter method sleeping or some scheduling happenstance), the L34 check will now be True as the queue is empty, however, the L36 call ‘await self._evput.wait()’ will immediately return as the _evput is still set, since we never cleared it. Clearing only happens after an empty queue was awaited and a new item was put.

So, a proposal solution is:

    async def get(self):  #  Usage: item = await queue.get()
        if self.empty():
            # Queue is empty, put the calling Task on the waiting queue
            await self._evput.wait()
        self._evput.clear()
        return self._get()

Not sure if causes other problems, that I didnt consider due to 1.30 am tiredness, but ye. But I guessed, regardles of whether self.empty() was True or False, if the ‘self._evput.clear()’ is reached, some item was placed by now (and it will be set, so I guess we dont need any check about it…) I can also make a PR tomorrow after work if you agree that is a bug after all.

PS: In https://github.com/peterhinch/micropython-async/blob/master/v3/primitives/queue.py#L51 and https://github.com/peterhinch/micropython-async/blob/master/v3/primitives/queue.py#L59 it should be preferable to have

if self.maxsize and self.qsize() >= self.maxsize:

instead of the current

if self.qsize() >= self.maxsize and self.maxsize:

?

I believe Python evaluates boolean conditions lazily, meaning ‘if self.maxsize’ evaluates to false, we save ourself the second evaluation. Albeit we should probably check for ‘if self.maxsize > 0’, as we do not check or otherwise validate negative inputs. In the cpython/official version, negative numbers are treated like 0 that way and imply infinite size.

[dbadrian]

Understood, thanks. I hope to get back to my project soon and then I’ll give it a review/testing and close it after.

[peterhinch]

Closing as I believe this is fixed.