High memory consumption of the unmanaged memory of librdkafka

Description

Hello, We are facing an issue when using the lib that may be blocking for us.

I’m in an application that starts 12 consumers consuming messages on different topics. These topics are filled with protobuf JSON messages.

So the 12 consumers are consuming at max speed and for the sake of this test, I do nothing in the consume, just logging a message to let me know that it consumed the message.

The consumer code is as follow:

// the method starting the consuming thread
public void Start()
   {
     _readerWriterLockSlim.EnterWriteLock();

     try
     {
       if (IsStarted)
       {
         _logger.SmartLogDebug("Kafka Consumer {0} @ {1} > Already started @ {2}", ConsumerId, Topic, BrokerList);
         return;
       }

       _logger.SmartLogDebug("Kafka Consumer {0} @ {1} > Starting consumer @ {2}", ConsumerId, Topic, BrokerList);

       _kafkaConsumer = CreateConsumer();
       _adminClient.WaitTopic(Topic, _logger);
       _kafkaConsumer.Subscribe(Topic);

       _cancellationTokenSource = new CancellationTokenSource();
       Task.Factory.StartNew(Consume, TaskCreationOptions.LongRunning);
       IsStarted = true;
     }
     finally
     {
       _readerWriterLockSlim.ExitWriteLock();
     }
   }

// the method creating the consumer
private IConsumer<string, TEvent> CreateConsumer()
   {
     var kafkaConsumer = new ConsumerBuilder<string, TEvent>(_consumerConfig)
       .SetErrorHandler(ErrorHandler(_consumerConfig.GroupId))
       .SetValueDeserializer(_deserializer)
       .SetLogHandler(LogHandler)
       .SetKeyDeserializer(new StringDeserializer(Encoding.UTF8))
       .SetPartitionsAssignedHandler(PartitionsAssignedHandler(_consumerConfig.GroupId))
       .SetPartitionsRevokedHandler(PartitionsRevokedHandler(_consumerConfig.GroupId))
       .Build();

     return kafkaConsumer;
   }

// the method consuming the messages
public void Consume()
   {
     while (!_cancellationTokenSource.IsCancellationRequested)
     {
         var consumeResult = _kafkaConsumer.Consume(_cancellationTokenSource.Token);
         _logger.LogInformation("Event consumed!");
     }
   }

So we are 12 consumers looking like that consuming events at the max speed they can and the memory allocated to the .NET process is more than 1GB and in this the most part is the unmanaged memory used by librdkafka.

This memory seems to be “hold” by the SafeKafkaHandle class of this project.

How to reproduce

Create an app that start more than 10 consumers at the same time then consumes events at max speed and check the memory.

So my question is : Is this memory consumption normal for you? If so, can we mitigate that by doing something?

Checklist

Please provide the following information:

• A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• Confluent.Kafka nuget version. tried again 1.7.0, 1.6.3, 1.4.4, 1.5.3, always the same result.
• Apache Kafka version. 2.2.0
• Client configuration.
• Operating system. Windows 10 64 bits. (but Kafka is running inside a docker container)
• Provide logs (with “debug” : “…” as necessary in configuration).
• Provide broker log excerpts.
• Critical issue.

[edenhill]

Since >=v1.5.0 a consumer will pre-fetch and buffer up to roughly 64MB of messages (plus some overhead), so that could account for slightly more than half of the memory usage you’re seeing. You could try reducing this by setting QueuedMaxMessageKbytes to something like 10MB and see if there’s a difference in memory usage.

[edenhill]

Generally I’d advice to use fewer consumers, if possible. E.g., have fewer consumers consume the same set of topics and partitions.