Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Kafka consumers are using high CPU with latest 2.0.2 release

See original GitHub issue

While using the latest version 2.0.2, we have observed heavy CPU usage that fluctuates between 100% to 20% during the consumer poll()/heartbeat operations. This is observed especially when the there are zero events polled from the topic (events are caught up). The issue could be due to this [fix] (https://github.com/dpkp/kafka-python/issues/1985) as we were able to confirm that 2.0.1 is working fine. It appears this issue is happening with the gevent based consumers.

cc: @dpkp @huangcuiyang

Issue Analytics

State:
Created 3 years ago
Comments:11

Top GitHub Comments

2reactions

cooolinxcommented, Sep 16, 2021

same issue here, any idea?

0reactions

berrfredcommented, Sep 9, 2022

Same issue on HPE NonStop running Python 3.6.15. 40% of a cpu just to be alive without reading any message …

In my case the issue seems to be linked to using poll(xxx) where xxx is a positive number of ms. If instead I use poll(), therefore with a 0 timeout and then call time.sleep(xxx/1000) , then I do not observe any longer a high cpu usage.

Top Results From Across the Web

Documentation - Apache Kafka

The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems.

Tail Latency at Scale with Apache Kafka - Confluent

The figure below shows how latencies observed by Kafka clients, usually called producer latency and consumer latency, relate to end-to-end ...

Changelog — kafka-python 2.0.2-dev documentation

This release includes breaking changes for any application code that has not migrated from older Simple-style classes to newer Kafka-style classes. Deprecation¶.

CPU Usage | Kafka Documentation - Instaclustr

If your CPU Usage is consistently high you may need to increase the capacity of your cluster. This can be achieved by either...

What is new in Apache Kafka 2.0 - Cloudera Documentation

Kafka clients are now notified of throttling before any throttling is applied when quotas are enabled. This enables clients to distinguish between network ......