Detect max.poll.interval.ms expiration

Description

Some time my consumer throws ConsumeException:

Confluent.Kafka.ConsumeException: Local: Maximum application poll interval (max.poll.interval.ms) exceeded

at Confluent.Kafka.Consumer2.ConsumeImpl[K,V](Int32 millisecondsTimeout, IDeserializer1 keyDeserializer, IDeserializer1 valueDeserializer) at Confluent.Kafka.Consumer2.Consume(TimeSpan timeout)

The log informs:

Application maximum poll interval (300000ms) exceeded by 2134298747ms (adjust max.poll.interval.ms for long-running message processing): leaving group

and the consumer stops receiving new messages (consume() returns null).

Is there are a way of detect when max.poll.interval.ms occours?

Is there another way besides try to check the exception message?

I was searching and found 931.

2316 has been released?

Lib Version: 1.2.2.0

[AndyPook]

Hi,

This is usually caused because Consume is not being called often enough (there can be some house-keeping interaction with the broker at this point). Typically there’s a consume/handle loop (ie consume to get a message; process the message; repeat). If your handle method/processing takes longer than the interval then you will see this exception.

I have seen this where a circuit-breaker was added so that the consumer could be “paused”. but that meant that the consumer wasn’t able to do its house-keeping and the interval expires and the exception is thrown. More simply, it can be seen if you leave the debugger on a break point for too long.

Usually, when the “loop” starts again, the underlying client is smart enough to restart the connection. Consume()==null can happen legitimately. Publish another message and you should see the consumer resume.

A massively simplified message loop…

while(!cancellationToken.IsCancellationRequested)
{
  var msg = consumer.Consume( 100ms or cancellationToken );
  if (msg == null || msg.IsPartitionEOF || cancellationToken.IsCancellationRequested)
    continue;

  Handle(msg);
}

If Handle takes longer than 300000ms (5 minutes) then you have other problems ! (note: if you pass a cancellationToken, internally it’s just a loop of Consume(100) waiting for the ct to be cancelled. Depending on what you’re trying to do, you take you choice…)

I hope this makes some sense? Happy to help further if you can share some more on what code pattern you are using.

[edenhill]

You should not call Assign() outside of OnPartitionsAssigned/Revoked