Not consuming the first message on the topic

Ok, this is a really strange bug: I’m trying to use parallel-consumer with Clojure. I’m using a local Kafka cluster (only one broker) to test things on my machine.

If I send a single message, and fire up the consumer, it consumes that message. So far, so good. If I immediately stop the consumer, send another message, and fire up the broker… nothing happens. The message is not consumed at all.

BUT, if I stop the consumer, send ANOTHER message, and fire up the consumer again, it consumes ONLY the new message… not the old one.

If I send a batch of messages, the same problem happens: it ignores the first message, and consumes the rest. Here’s a video for reference:

https://user-images.githubusercontent.com/138037/103380836-cd1bd500-4ac8-11eb-9b94-00092befe9ca.mp4

The code that causes this error is the following: please notice that there’s nothing special about it - just instantiates a single consumer that prints the received message:

(defn -main  [& args]
  (let [consumer (KafkaConsumer. (doto (java.util.Properties.)
                                       (.put "group.id" "example-1")
                                       (.put "bootstrap.servers" "localhost:9092")
                                       (.put "auto.offset.reset" "earliest")
                                       (.put "enable.auto.commit" false)
                                       (.put "key.deserializer" "org.apache.kafka.common.serialization.StringDeserializer")
                                       (.put "value.deserializer" "org.apache.kafka.common.serialization.StringDeserializer")))
        options (.. (ParallelConsumerOptions/builder)
                    (ordering ParallelConsumerOptions$ProcessingOrder/UNORDERED)
                    (defaultMessageRetryDelay (java.time.Duration/ofMillis 500))
                    (maxConcurrency 1000)
                    (commitMode ParallelConsumerOptions$CommitMode/PERIODIC_CONSUMER_ASYNCHRONOUS)
                    (consumer consumer)
                    build)
        poller (doto (ParallelStreamProcessor/createEosStreamProcessor options)
                     (.subscribe ["example"]))]
    (.poll poller (as-consumer (fn [record]
                                 (locking poller
                                   (prn :HANDLING record)))))
    (Thread/sleep 3000)
    (.close poller)))

[astubbs]

Released. https://repo1.maven.org/maven2/io/confluent/parallelconsumer/parallel-consumer-parent/0.3.0.1/

[JorgenRingen]

Tested and verified 👍