[Question] High CPU utilization in Consumer App

We implemented Consumer apps as a set of Azure Service Fabric services. Each app consumes messages only from one topic. And typical Kafka Consumer app has an infinite loop as its heart. Here is the pseudo-code for it:

public static void Main()
{
    [Initialize]
    consumer.Subscribe("my-topic");
    while (true)
    {
      var result = consumer.Consume();
      [Transform Result]
      producer.Produce(transformedResult); // Publish to another topic
    }
}

And the application above is trying to consume up to 50% of CPU.

Issue: When we deploy such service as a part of Azure Service Fabric Application then there could be a few consumer services running on the same node. And CPU usage would be ~100%.

We re-wrote this infinite loop

public static async Task Main()
{
    [Initialize]
    consumer.Subscribe("my-topic");
    while (true)
    {
      var result = consumer.Consume();
      [Transform Result]
      producer.Produce(transformedResult); // Publish to another topic
      
      //  Give chance to run other threads
      await Taks.Delay(1ms);  // We probably should use Task.Yield() or Task.Delay(0)
    }
}

And the app consumes less CPU, but a number of messages per second suffer.

Questions: Is there a deployment recommendation for Kafka consumer applications? One physical node - one consumer? Or is there a way to make a consumer application asynchronous without sacrificing performance?

[mhowlett]

one thread per broker (3), plus an additional thread, plus a .net managed thread in the case of the producer.

[saherahwal]

Thank you!

On Thu, May 13, 2021 at 2:43 PM Matt Howlett @.***> wrote:

one thread per broker (3), plus an additional thread, plus a .net managed thread in the case of the producer.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/confluentinc/confluent-kafka-dotnet/issues/1261#issuecomment-840852526, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEGN257CGBXQ62DQMQNA5LTNRBZTANCNFSM4MQZS37A .

– Saher Ahwal

Massachusetts Institute of Technology '13, '14 Department of Electrical Engineering and Computer Science @.*** @.**> | 617 680 4877