Memory leak in consumer when using lz4 compression

I’ve noticed a problem with unbounded memory usage in the consumer when handling lz4-compressed payloads. The symptoms point to a memory leak of some description. I’ve created a test case which should allow easy reproduction of the issue.

I’ve tried experimenting with various consumer settings such as fetch_max_bytes , max_partition_fetch_bytes and max_in_flight_requests_per_connection in an attempt to lessen the buffering requirements of each individual request to Kafka. In all cases, memory usage of consumer processes has continued to rise until such a point that those processes are killed by the host system.

I can additionally confirm that this issue is present in the latest PyPI release (1.3.2) as well as master and that the issue only manifests when using lz4 compression specifically. gzip appears to be working fine, for example.

Any help is greatly appreciated. Thanks!

[dpkp]

Given the complexity here w/ both options, I think I’m going to defer this until after the next release (which I’m hoping to push out in the next few days).

[dpkp]

We might consider switching to https://github.com/python-lz4/python-lz4 for primary lz4 support going forward. It seems to be more active.