Not consuming the first message on the topic
See original GitHub issueOk, this is a really strange bug: I’m trying to use parallel-consumer with Clojure. I’m using a local Kafka cluster (only one broker) to test things on my machine.
If I send a single message, and fire up the consumer, it consumes that message. So far, so good. If I immediately stop the consumer, send another message, and fire up the broker… nothing happens. The message is not consumed at all.
BUT, if I stop the consumer, send ANOTHER message, and fire up the consumer again, it consumes ONLY the new message… not the old one.
If I send a batch of messages, the same problem happens: it ignores the first message, and consumes the rest. Here’s a video for reference:
https://user-images.githubusercontent.com/138037/103380836-cd1bd500-4ac8-11eb-9b94-00092befe9ca.mp4
The code that causes this error is the following: please notice that there’s nothing special about it - just instantiates a single consumer that prints the received message:
(defn -main [& args]
(let [consumer (KafkaConsumer. (doto (java.util.Properties.)
(.put "group.id" "example-1")
(.put "bootstrap.servers" "localhost:9092")
(.put "auto.offset.reset" "earliest")
(.put "enable.auto.commit" false)
(.put "key.deserializer" "org.apache.kafka.common.serialization.StringDeserializer")
(.put "value.deserializer" "org.apache.kafka.common.serialization.StringDeserializer")))
options (.. (ParallelConsumerOptions/builder)
(ordering ParallelConsumerOptions$ProcessingOrder/UNORDERED)
(defaultMessageRetryDelay (java.time.Duration/ofMillis 500))
(maxConcurrency 1000)
(commitMode ParallelConsumerOptions$CommitMode/PERIODIC_CONSUMER_ASYNCHRONOUS)
(consumer consumer)
build)
poller (doto (ParallelStreamProcessor/createEosStreamProcessor options)
(.subscribe ["example"]))]
(.poll poller (as-consumer (fn [record]
(locking poller
(prn :HANDLING record)))))
(Thread/sleep 3000)
(.close poller)))
Issue Analytics
- State:
- Created 3 years ago
- Comments:12 (10 by maintainers)
Released. https://repo1.maven.org/maven2/io/confluent/parallelconsumer/parallel-consumer-parent/0.3.0.1/
Tested and verified 👍