Offset loosing

Description

Hello,

I have faced with periodical offsets resetting on broker configuration change. Attempts to understand what was going on did not lead me to anything.

Offsets are loosing for one group and keeping for others. I can stable reproduce this behaviour for a single consumer group.

How to reproduce

I have one producer, one consumer, 3 brokers. Producer is constantly writes messages with performance=300 msgs / sec. Consumer is reading messages from topic and commiting offsets of read messages automatically. I monitor consumer latency and lag.

Consumer configuration:

    config["group.id"] = "KafkaTests"; 
    config["statistics.interval.ms"] = 10000;
    config["fetch.message.max.bytes"] = 128000;
    config["queued.max.messages.kbytes"] = 1024;
    config["queued.min.messages"] = 200;
    config["fetch.wait.max.ms"] = 1000;

    config["session.timeout.ms"] = 30000;
    config["offset.store.method"] = "broker";
    config["default.topic.config"] = new Dictionary<string, object>() { { "auto.offset.reset", "smallest" } };
    config["enable.auto.commit"] = "true;

When I shut a single kafka broker down, consumer latency (and lag) is growing up and consumer starts to consume already processed messages second time.

When I turn broker back, lag is usually decreases but not to original value. It might look like this

I don’t understand what’s going on, it looks like broker misconfiguration, but can’t find any relevant information in internet.

You can see log of fetched offsets below:

7|2018-01-29 14:26:22.943|rdkafka#consumer-1|OFFSETFETCH| [thrd:main]:  HttpLogMessage [0] offset INVALID
7|2018-01-29 14:26:22.944|rdkafka#consumer-1|OFFSETFETCH| [thrd:main]: kafka1:9092/1: OffsetFetchResponse: HttpLogMessage [0] offset 2449840
7|2018-01-29 14:27:44.861|rdkafka#consumer-1|OFFSETFETCH| [thrd:main]:  HttpLogMessage [0] offset INVALID
7|2018-01-29 14:27:44.861|rdkafka#consumer-1|OFFSETFETCH| [thrd:main]: kafka2:9092/3: OffsetFetchResponse: HttpLogMessage [0] offset 2346535
7|2018-01-29 14:28:44.347|rdkafka#consumer-1|OFFSETFETCH| [thrd:main]:  HttpLogMessage [0] offset INVALID
7|2018-01-29 14:28:44.347|rdkafka#consumer-1|OFFSETFETCH| [thrd:main]: kafka1:9092/1: OffsetFetchResponse: HttpLogMessage [0] offset 2373064

Please let me know if I should provide more info / logs

Checklist

Please provide the following information:

• Confluent.Kafka nuget version: 0.11.2
• Apache Kafka version: 0.10.2
• Client configuration:
• Operating system: windows
• Provide logs (with “debug” : “…” as necessary in configuration)
• Provide broker log excerpts
• Critical issue

[TonyWang666]

Check the broker offsets.retention.minutes, it could be that your committed offsets are deleted due to retention policy.

@edenhill Bro, you just saved my life. I have spent 2 days on it without any clues. It turns out that I set retention time to 1 min. Thanks again for the @edenhill and also to question contributor @divan-9.

[edenhill]

Check the broker offsets.retention.minutes, it could be that your committed offsets are deleted due to retention policy.