question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Memory leaks when using redis as a broker

See original GitHub issue

Using SimpleQueue and Redis as a broker causes memory leaks, at rate from 3 and up to 8 Mbytes/s.

Tested on two machines running: Python 2.7.12, OSX 10.11.6 and one with Debian 8.6 x86_64.

Pip freeze:

amqp==1.4.9 anyjson==0.3.3 kombu==3.0.35 redis==2.10.5

Script to reproduce:

#!/usr/bin/env python

import json
from kombu import Connection

class D(object):
    broker_uri = 'redis://'
    def put_message(self, message):
        with Connection(self.broker_uri) as conn:
            simple_queue = conn.SimpleQueue('simple_queue')
            simple_queue.put(json.dumps(message))
            print('Sent: %s' % message)
            simple_queue.close()

    def fill(self, stop=1800000):
        for i in xrange(1, stop + 1):
            self.put_message(i)


d = D()
d.fill()

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
tb0hdancommented, Sep 29, 2016

Thanks 👍

0reactions
askcommented, Sep 28, 2016

Able to reproduce using just:

for i in range(1800000):
    with Connection('redis://') as conn:
        assert conn.default_channel
        print('Opened #{0!r}'.format(i))
Read more comments on GitHub >

github_iconTop Results From Across the Web

[celery-users] memory leak using redis for broker and backend
Hi All, I'm experiencing some serious memory leak problems with workers in a django/celery combo using redis for the broker and the backend....
Read more >
Fixing Memory Leaks In Popular Python Libraries
The memory leak would happen on the main Celery worker process ... fix it in py-amqp (a celery dependency when using RabbitMQ as...
Read more >
And do not use Redis as a broker under a high load. It will ...
And do not use Redis as a broker under a high load. It will random crash or refuse to set workers to work,...
Read more >
[Solved]-Celery and Redis keep running out of memory-django
Is that a solution? in addition to _kombu.bindings.celeryev set there will be e.g. celeryev.i-am-alive. keys with TTL set (e.g. 30sec);; celeryev process ...
Read more >
How to determine Redis memory leak? - Stack Overflow
If you use commands retrieving very large data from Redis (in one shot), it can be an explanation as well. For instance, a...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found