Cannot re-process old Kafka messages because of Quarkus overriding the group.id
See original GitHub issueI’m running a simple integration in Camel K:
from('kafka:bitcoin-kafka-topic?brokers=my-cluster-kafka-bootstrap.kafka-demo.svc.cluster.local:9092&groupId=myowngroupid&autoOffsetReset=earliest')
.to('log:info')
I’m using myowngroupid
to reprocess old messages using a different consumer group. But when I run this integration, I notice that the Kafka group Id is always a fixed string camel-k-integration
.
The reason seems to be due to this part of the code: https://github.com/apache/camel-quarkus/blob/9344d527b2979a859e3678746bb55e54fa1abcc5/extensions/kafka/runtime/src/main/java/org/apache/camel/quarkus/component/kafka/QuarkusKafkaClientFactory.java#L63-L67
Which seems correct at first glance, but unfortunately the Quarkus config unexpectedly contains wrong values at runtime:
Issue Analytics
- State:
- Created 2 years ago
- Comments:6 (6 by maintainers)
Top Results From Across the Web
Apache Kafka Reference Guide - Quarkus
Configuring Smallrye Kafka Connector. Because Smallrye Reactive Messaging framework supports different messaging backends like Apache Kafka, AMQP, Apache Camel, ...
Read more >Apache Kafka - SmallRye Reactive Messaging
Setting the consumer rebalance listener's name takes precedence over using the group id. Writing Kafka Records. The Kafka Connector can write Reactive Messaging ......
Read more >7 mistakes when using Apache Kafka | by Michał Matłoka
Sending a message to non-existing Kafka topic, by default results in its ... the maximum number of consumers from a single consumer group....
Read more >Kafka Streams with Quarkus - Piotr's TechBlog
If you would like to figure out what is a streaming platform and how it differs from a traditional message broker this article...
Read more >Testing Quarkus Web Applications: Reactive Messaging ...
Let's see how we can write tests for these use cases. The application. Let's consider a small use case of a checkout process...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
For the record, we merged a fix in Quarkus.
I created a PR that should fix this https://github.com/apache/camel-quarkus/pull/2913. To my knowledge, none of the other Camel projects dependent on CQ is explicitly forcing usage of
quarkus-kubernetes-service-binding
, so having a check for whether the capability is present and not setting up the customKafkaClientFactory
should be good enough.I need to do some follow up research for changing the map
put
toputIfAbsent
. Turns out it’s a bit tricky to have this because of the defaults that the Camel component sets for various options. However, I introduced a new config option to disable the config merging if required.