Memory leak

We have been noticing a memory leak with watchtower for long running processes.
Logs are being delivered to Cloudwatch but memory usage keeps going up. Python version 2.7.10 and watchtower 0.3.3

Snippet of code to reproduce

import watchtower, logging, time

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(str(time.time()))
wt = watchtower.CloudWatchLogHandler(use_queues=True, send_interval=60, max_batch_count=10)
logger.addHandler(wt)
logger.propagate = False

for _ in xrange(50000):
    logger.info('Junk data | {}'.format("0" * 10000))
    time.sleep(0.05)

[jcerjak]

I checked the snippet above with Python 2.7.6 and latest watchtower master by using the memory_profiler.

When using queues (use_queues=True):

Without using queues (use_queues=False):

Upon investigation, it seems that the issue is related to botocore, and might be further amplified with the way threads are used in watchtower: https://github.com/boto/botocore/issues/805

This seems to be confirmed by doing manual garbage collection after pushing logs to CloudWatch:

import gc

...
response = self.cwl_client.put_log_events(**kwargs)
gc.collect()

When using queues (use_queues=True) and gc:

[kislyuk]

We use watchtower in production and do not observe any memory leaks. Also, watchtower does not share boto3 clients across threads unless explicitly passed a configuration that forces it to do so.

I am going to close this issue for now. When I run the reproduction script above, it stabilizes at a steady state memory consumption of 60MB. Network outages and associated retries may temporarily increase this footprint. If somebody has a specific concern, please post a complete reproduction and an explanation of why it is different from what you expect.