Consume read messages already commited

Hi,

I face a problem since few weeks. Sometimes the consumer function give me the message with offset 0, even i consumed him long time ago And it makes me read again all messages. I tried on a local kafka and another kafka instance and both have the same problem.

Here is my code :

        private Dictionary<string, object> _constructConfig(string brokerList, bool enableAutoCommit) =>
            new Dictionary<string, object>
            {
                {"group.id", "jokerGroup"},
                {"enable.auto.commit", enableAutoCommit},
                {"auto.commit.interval.ms", 5000},
                {"statistics.interval.ms", 60000},
                {"bootstrap.servers", brokerList},
                {
                    "default.topic.config", new Dictionary<string, object>
                    {
                        {"auto.offset.reset", "smallest"}
                    }
                }
            };

                while (foundMessage == false && nbRetry < 20)
                {
                    while (consumer.Consume(out msg, TimeSpan.FromMilliseconds(1000)))
                    {
                        foundMessage = true;
                        ....
                   }
                }

        public async Task<CommittedOffsets> CommitAsync(Consumer<Null, string> consumer, Message<Null, string> msg)
        {
            CommittedOffsets offsets = await consumer.CommitAsync(msg);
            _logger.LogInformation(LogEvent.NeedCreateCode, $"COMMIT : {offsets.Offsets.FirstOrDefault()}");

            return offsets;
        }

This problem happend randomly, i have no more informations (Maybe every 24h ?) And i also read kafka on my windows using basics scripts and sometimes i got this warning (Maybe it can help us to find the issue) :

[2018-03-09 13:23:34,135] WARN Auto-commit of offsets {JokerTopic-0=OffsetAndMetadata{offset=10, metadata=‘’}} failed for group console-consumer-82368: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

Thank you for your precious help !

Confluent.Kafka nuget version: v0.11.3-ci-255 Operating system: W10

[Oipo]

One of the kafka instances in the cluster was broken, leading to different offsets in the leader and the follower. Thanks for the help.

[Oipo]

Is there a way to log this debug to a file? The service is running under an account without rights to login to a console.

Edit: Got it working with stderr redirection.