queue length for SQLiteQueue is incorrect when running in multiple processes

Possibly expected behavior, but I think it’s worth reporting, because the queue looks usable otherwise.

The queue size is set only once on queue creation. self.total = self._count(), so if we have a producer in 1 process and a consumer in another process, we end up with size in the negatives.

To reproduce, we need producer and a consumer that’s faster than the producer.

# producer process
import persistqueue as Q; q = Q.SQLiteQueue('queue', multithreading=True)
while True: q.put('hi'); time.sleep(0.01)

# consumer process
import persistqueue as Q; q = Q.SQLiteQueue('queue', auto_commit=False, multithreading=True)


while True:
    try:
        q.qsize(), q.get(block=False); q.task_done()
    except persistqueue.exceptions.Empty:
        pass

Calling q._count() returns the correct size, because it hits the DB, of course.

[decatur]

Yes, this seems to be a bug. So instead of

    while not queue.empty():
        item = queue.get()
        pprint(item)
        queue.task_done()

use the workaround

    while True:
        try:
            item = queue.get(block=False)
        except persistqueue.exceptions.Empty:
            break
        pprint(item)
        queue.task_done()

[imidoriya]

Recent updates have added max(0, count) to remove a negative qsize(). That doesn’t change the overall issue, but prevents impossible size results. On the Ack Queues, a new active_size() was added which includes the nack cache. It may be anecdotal, but I believe this has produced a more accurate return in my multi-threaded environment as it’s calculating when an item is put/ack/ack_failed, and not on put/get/nack. But that’s more of a decision on when you think the queue size should be decremented, on get or on completion.