Producer high CPU utilization

Description

We have some producers on production that are causing high CPU usage (60% of cpu usage on the service). Our current throughput is about 200k messages/300MB per second .

• Confluent kafka nuget version

Confluent.Kafka Version=“1.0.1.1” librdkafka.redist Version=“1.0.1”

• Client configuration

var bootstrapServers = "x";
var clientConfig = new ClientConfig
{
  BootstrapServers = bootstrapServers
};
var producerConfig = new ProducerConfig(clientConfig);

• Operating system

We are running on a Windows Machine (Windows Server 2016 Datacenter) with Dotnet Core 3.0.

How to reproduce

We are using one of our abstractions of queue of our library https://github.com/takenet/elephant/blob/master/src/Take.Elephant.Kafka/KafkaSenderQueue.cs

And we are using for producing this EnqueueAsync method.

Checklist

Please provide the following information:

• A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• Confluent.Kafka nuget version.
• Apache Kafka version.
• Client configuration.
• Operating system.
• Provide logs (with “debug” : “…” as necessary in configuration).
• Provide broker log excerpts.
• Critical issue.

[mhowlett]

poll blocks until librdkafka is ready to inform the application of a new event. it doesn’t busy-wait, so won’t result in high-CPU if there are no events. you don’t need to call poll for every produce call, just periodically. when it’s called, callbacks are executed corresponding to every currently outstanding event (if a corresponding callback is registered, else the event is dropped). events include delivery callback notifications, or error or log events. this all happens behind the scenes in the .net library.

[j2jensen]

I have an interesting update on this.

I identified an issue in the application that accounted for the second-highest hotspot on the hotspot image above (TimerQueueTimer.Fire: 8.9%). Here’s what happened after releasing the fix:

The entire issue I reported got resolved by addressing something which Dynatrace thought was a tiny part of the overall problem.

Now the same service is showing 95% of its time in the same StartPollTask hotspot. Looking at other services that use Kafka, I’m seeing similarly high percentages as well. But if you do the math, it doesn’t add up. Before Kafka accounted for 75% of 20% total CPU usage, so supposedly 15% of total CPU. Now, it accounts for 95% of 5% total usage, so ~4.8%? And the only change made had nothing to do with Kafka. That doesn’t make sense.

My hypothesis: Dynatrace does its hotspot evaluation, not by knowing what’s actually using CPU time, but by evaluating stack traces and seeing where the code appears to be sitting. Since Kafka has a single, persistent background thread that’s almost always sitting (effectively blocked) on this line of code at the boundary between .NET and the unmanaged librdkakfa library, it gives Dynatrace the impression that this line of code is a hotspot, when in reality it’s hardly accounting for any of the actual CPU usage.