Consumer misses messages when upgrading from v0.9.5 to v0.11.*

Description

We have a topic with a single partition that contains (reasonably) static data. We have no issues connecting to the topic and reading all messages (65,223) held in the partition using the following version of the library:

Nuget package: Confluent.Kafka.StrongName Nuget package version: 0.9.5-ci-43

However, when we upgrade our nugget package to a newer version (0.11.0-RC, 0.11.0, 0.11.1-RC, 0.11.1 or 0.11.2) we find that some messages are missing – 11 messages missing from 65,223 message in total. It’s the same messages on each run. We can immediately see the messages again if we downgrade our application back to using version 0.9.5-ci-43.

The configuration we supply to the consumer regardless of the library version is:

{ "group.id", $"TestApp.{Environment.MachineName}.1234"}, { "enable.auto.commit", false }, { "auto.offset.reset", "earliest" }, { "fetch.message.max.bytes", 5000000 },

Since we noticed the issue we have had another team in our organisation that also consumes the same topic check their message count. They are using the Java library (through Maven):

<groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>0.10.2.0</version>

With the Java library they are able to consume all message without losing any.

We are subscribed to OnError, OnConsumeError and OnLog, but never have an error raised and the log does not show anything that looks suspicious, well to our untrained eye.

Since discovering the issue we have added the following to our consumer’s configuration, but these have not helped:

{"debug", "all" }, { "api.version.request" , true}, { "broker.version.fallback" , "0.10.2.0"}

The Kafka version running on the server is 0.10.2.0 and the message format used is JSON.

Any help in diagnosing the cause of the missing messages would be greatly appreciated.

How to reproduce

Upgrade from 0.9.5-ci-43 to 0.11.* of the library.

Checklist

Please provide the following information:

• Confluent.Kafka nuget version:
• Apache Kafka version:
• Client configuration:
• Operating system: Consumer runs on Windows 7 Enterprise
• Provide logs (with “debug” : “…” as necessary in configuration)
• Provide broker log excerpts
• Critical issue

[crann]

Hi @edenhill I believe this was sorted a while ago - at least from our perspective.

I emailed various logs to you in November (2017). You responded that you thought the issue was related to how message sets were being compacted. You asked me to try the following build on 19th Feb (2018):

https://ci.appveyor.com/project/edenhill/librdkafka/build/0.11.1-R-post405/job/mdi75i4w7q66m4pa/artifacts

I emailed you on 19th Feb to let you know that I could not reproduce the issue using this build. I don’t believe anything changed in our infrastructure at this time, so it’s likely that changes in the build resolved the issue.

I hope this helps close this issue. Apologies I should have left a note here too at the time.

[edenhill]

Thank you, I’d forgotten about that!