Kombu 4.1.0 - Memory usage increase (leak?) on a worker when using kombu queues

Hi,

I have implemented a worker using Kombu’s SimpleQueue. The implementation is as given below. When I run this worker for a few hours on a Ubuntu 16.04 system with redis as the backend, I notice a gradual memory build up on the process. When I run this worker for over a day, it ends up consuming all memory on the system and the system ends up being unusable, until the worker is killed.

On redis server, I have it configured with a timeout set to 5 seconds and a tcp_keepalive set to 60 seconds.

Worker Code:

from kombu import Connection

myqueue_name = 'test_queue'
backendURL = 'redis://127.0.0.1:6379/'

def GetConnection():
    conn = Connection(backendURL)

    return conn

def dequeue():
    conn = GetConnection()
    with conn:
        myqueue = conn.SimpleQueue(myqueue_name)

        item = None

        try:
            qItem = myqueue.get(block=True, timeout=2)
            item = qItem.payload
            qItem.ack()
        except Exception as e:
            qItem = None

        myqueue.close()

    conn.close()
    conn.release()

    return item

if __name__ == '__main__':
    try:
        i = 1
        while True:
            print 'Iteration %s: %s' % (i, dequeue())
            i = i + 1
    except (KeyboardInterrupt, SystemExit):
        print 'Terminating'

Here’s a plot of free memory on the system:

What is going wrong here? Did I miss anything in the implementation?

Any help here will be greatly appreciated.

[pawl]

@auvipy Nice find, it seems like this could definitely be related to: https://github.com/celery/celery/issues/4843#issuecomment-999168967

[sradhakrishna]

I’ve tried the same with rabbitmq as the backend. Saw the same behavior in that scenario too. Seems that the issue might not be in the backend specific implementation.