Memory leaks when using redis as a broker
See original GitHub issueUsing SimpleQueue and Redis as a broker causes memory leaks, at rate from 3 and up to 8 Mbytes/s.
Tested on two machines running: Python 2.7.12, OSX 10.11.6 and one with Debian 8.6 x86_64.
Pip freeze:
amqp==1.4.9
anyjson==0.3.3
kombu==3.0.35
redis==2.10.5
Script to reproduce:
#!/usr/bin/env python
import json
from kombu import Connection
class D(object):
broker_uri = 'redis://'
def put_message(self, message):
with Connection(self.broker_uri) as conn:
simple_queue = conn.SimpleQueue('simple_queue')
simple_queue.put(json.dumps(message))
print('Sent: %s' % message)
simple_queue.close()
def fill(self, stop=1800000):
for i in xrange(1, stop + 1):
self.put_message(i)
d = D()
d.fill()
Issue Analytics
- State:
- Created 7 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
[celery-users] memory leak using redis for broker and backend
Hi All, I'm experiencing some serious memory leak problems with workers in a django/celery combo using redis for the broker and the backend....
Read more >Fixing Memory Leaks In Popular Python Libraries
The memory leak would happen on the main Celery worker process ... fix it in py-amqp (a celery dependency when using RabbitMQ as...
Read more >And do not use Redis as a broker under a high load. It will ...
And do not use Redis as a broker under a high load. It will random crash or refuse to set workers to work,...
Read more >[Solved]-Celery and Redis keep running out of memory-django
Is that a solution? in addition to _kombu.bindings.celeryev set there will be e.g. celeryev.i-am-alive. keys with TTL set (e.g. 30sec);; celeryev process ...
Read more >How to determine Redis memory leak? - Stack Overflow
If you use commands retrieving very large data from Redis (in one shot), it can be an explanation as well. For instance, a...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Thanks 👍
Able to reproduce using just: