Not consuming the first message on the topicSee original GitHub issue
Ok, this is a really strange bug: I’m trying to use parallel-consumer with Clojure. I’m using a local Kafka cluster (only one broker) to test things on my machine.
If I send a single message, and fire up the consumer, it consumes that message. So far, so good. If I immediately stop the consumer, send another message, and fire up the broker… nothing happens. The message is not consumed at all.
BUT, if I stop the consumer, send ANOTHER message, and fire up the consumer again, it consumes ONLY the new message… not the old one.
If I send a batch of messages, the same problem happens: it ignores the first message, and consumes the rest. Here’s a video for reference:
The code that causes this error is the following: please notice that there’s nothing special about it - just instantiates a single consumer that prints the received message:
(defn -main [& args] (let [consumer (KafkaConsumer. (doto (java.util.Properties.) (.put "group.id" "example-1") (.put "bootstrap.servers" "localhost:9092") (.put "auto.offset.reset" "earliest") (.put "enable.auto.commit" false) (.put "key.deserializer" "org.apache.kafka.common.serialization.StringDeserializer") (.put "value.deserializer" "org.apache.kafka.common.serialization.StringDeserializer"))) options (.. (ParallelConsumerOptions/builder) (ordering ParallelConsumerOptions$ProcessingOrder/UNORDERED) (defaultMessageRetryDelay (java.time.Duration/ofMillis 500)) (maxConcurrency 1000) (commitMode ParallelConsumerOptions$CommitMode/PERIODIC_CONSUMER_ASYNCHRONOUS) (consumer consumer) build) poller (doto (ParallelStreamProcessor/createEosStreamProcessor options) (.subscribe ["example"]))] (.poll poller (as-consumer (fn [record] (locking poller (prn :HANDLING record))))) (Thread/sleep 3000) (.close poller)))
- Created 2 years ago
- Comments:12 (10 by maintainers)
Top GitHub Comments
Tested and verified 👍