Memory leak
See original GitHub issueWe have been noticing a memory leak with watchtower for long running processes.
Logs are being delivered to Cloudwatch but memory usage keeps going up.
Python version 2.7.10 and watchtower 0.3.3
Snippet of code to reproduce
import watchtower, logging, time
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(str(time.time()))
wt = watchtower.CloudWatchLogHandler(use_queues=True, send_interval=60, max_batch_count=10)
logger.addHandler(wt)
logger.propagate = False
for _ in xrange(50000):
logger.info('Junk data | {}'.format("0" * 10000))
time.sleep(0.05)
Issue Analytics
- State:
- Created 7 years ago
- Reactions:1
- Comments:6 (1 by maintainers)
Top Results From Across the Web
Memory leak - Wikipedia
In computer science, a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations in...
Read more >What is Memory Leak? How can we avoid? - GeeksforGeeks
Memory leak occurs when programmers create a memory in heap and forget to delete it. The consequences of memory leak is that it...
Read more >Definition of memory leak - PCMag
When memory is allocated, but not deallocated, a memory leak occurs (the memory has leaked out of the computer). If too many memory...
Read more >Memory Leaks and Garbage Collection | Computerworld
DEFINITION A memory leak is the gradual deterioration of system performance that occurs over time as the result of the fragmentation of a...
Read more >Find a memory leak - Windows drivers - Microsoft Learn
A memory leak occurs when a process allocates memory from the paged or nonpaged pools, but doesn't free the memory.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I checked the snippet above with Python 2.7.6 and latest
watchtower
master by using the memory_profiler.When using queues (
use_queues=True
):Without using queues (
use_queues=False
):Upon investigation, it seems that the issue is related to
botocore
, and might be further amplified with the way threads are used inwatchtower
: https://github.com/boto/botocore/issues/805This seems to be confirmed by doing manual garbage collection after pushing logs to CloudWatch:
When using queues
(use_queues=True)
and gc:We use watchtower in production and do not observe any memory leaks. Also, watchtower does not share boto3 clients across threads unless explicitly passed a configuration that forces it to do so.
I am going to close this issue for now. When I run the reproduction script above, it stabilizes at a steady state memory consumption of 60MB. Network outages and associated retries may temporarily increase this footprint. If somebody has a specific concern, please post a complete reproduction and an explanation of why it is different from what you expect.