Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Not consuming the first message on the topic

See original GitHub issue

Issue Description

Ok, this is a really strange bug: I’m trying to use parallel-consumer with Clojure. I’m using a local Kafka cluster (only one broker) to test things on my machine.

If I send a single message, and fire up the consumer, it consumes that message. So far, so good. If I immediately stop the consumer, send another message, and fire up the broker… nothing happens. The message is not consumed at all.

BUT, if I stop the consumer, send ANOTHER message, and fire up the consumer again, it consumes ONLY the new message… not the old one.

If I send a batch of messages, the same problem happens: it ignores the first message, and consumes the rest. Here’s a video for reference:

The code that causes this error is the following: please notice that there’s nothing special about it - just instantiates a single consumer that prints the received message:

(defn -main  [& args]
  (let [consumer (KafkaConsumer. (doto (java.util.Properties.)
                                       (.put "" "example-1")
                                       (.put "bootstrap.servers" "localhost:9092")
                                       (.put "auto.offset.reset" "earliest")
                                       (.put "" false)
                                       (.put "key.deserializer" "org.apache.kafka.common.serialization.StringDeserializer")
                                       (.put "value.deserializer" "org.apache.kafka.common.serialization.StringDeserializer")))
        options (.. (ParallelConsumerOptions/builder)
                    (ordering ParallelConsumerOptions$ProcessingOrder/UNORDERED)
                    (defaultMessageRetryDelay (java.time.Duration/ofMillis 500))
                    (maxConcurrency 1000)
                    (commitMode ParallelConsumerOptions$CommitMode/PERIODIC_CONSUMER_ASYNCHRONOUS)
                    (consumer consumer)
        poller (doto (ParallelStreamProcessor/createEosStreamProcessor options)
                     (.subscribe ["example"]))]
    (.poll poller (as-consumer (fn [record]
                                 (locking poller
                                   (prn :HANDLING record)))))
    (Thread/sleep 3000)
    (.close poller)))

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:12 (10 by maintainers)

github_iconTop GitHub Comments

JorgenRingencommented, Jan 21, 2021

Tested and verified 👍

Read more comments on GitHub >

github_iconTop Results From Across the Web

Kafka consumer not consuming from beginning - Stack Overflow
I have Kafka setup on my local machine and have started the zookeeper and a single broker server. Now i have a single...
Read more >
Message Queue for Apache RocketMQ:Ordered message
Ordered message,Message Queue for Apache RocketMQ:All messages in a topic are published and consumed in a FIFO order.
Read more >
Documentation - Apache Kafka
The Kafka cluster retains all published messages—whether or not they have been consumed—for a configurable period of time. For example if the log...
Read more >
Reliability Guide - RabbitMQ
The following guides discuss data safety and resilience topics in more detail: ... A consuming application should not acknowledge messages until it has...
Read more >
Pull subscriptions | Cloud Pub/Sub Documentation
... the first client maintains a lease on the messages that it receives. ... those messages continue to consume memory, CPU, and bandwidth...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found