question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cannot re-process old Kafka messages because of Quarkus overriding the group.id

See original GitHub issue

I’m running a simple integration in Camel K:

from('kafka:bitcoin-kafka-topic?brokers=my-cluster-kafka-bootstrap.kafka-demo.svc.cluster.local:9092&groupId=myowngroupid&autoOffsetReset=earliest')
    .to('log:info')

I’m using myowngroupid to reprocess old messages using a different consumer group. But when I run this integration, I notice that the Kafka group Id is always a fixed string camel-k-integration.

The reason seems to be due to this part of the code: https://github.com/apache/camel-quarkus/blob/9344d527b2979a859e3678746bb55e54fa1abcc5/extensions/kafka/runtime/src/main/java/org/apache/camel/quarkus/component/kafka/QuarkusKafkaClientFactory.java#L63-L67

Which seems correct at first glance, but unfortunately the Quarkus config unexpectedly contains wrong values at runtime:

Screenshot from 2021-07-13 16-09-11

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (6 by maintainers)

github_iconTop GitHub Comments

2reactions
cescoffiercommented, Jul 20, 2021

For the record, we merged a fix in Quarkus.

0reactions
jamesnethertoncommented, Jul 19, 2021

I created a PR that should fix this https://github.com/apache/camel-quarkus/pull/2913. To my knowledge, none of the other Camel projects dependent on CQ is explicitly forcing usage of quarkus-kubernetes-service-binding, so having a check for whether the capability is present and not setting up the custom KafkaClientFactory should be good enough.

I need to do some follow up research for changing the map put to putIfAbsent. Turns out it’s a bit tricky to have this because of the defaults that the Camel component sets for various options. However, I introduced a new config option to disable the config merging if required.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Apache Kafka Reference Guide - Quarkus
Configuring Smallrye Kafka Connector. Because Smallrye Reactive Messaging framework supports different messaging backends like Apache Kafka, AMQP, Apache Camel, ...
Read more >
Apache Kafka - SmallRye Reactive Messaging
Setting the consumer rebalance listener's name takes precedence over using the group id. Writing Kafka Records. The Kafka Connector can write Reactive Messaging ......
Read more >
7 mistakes when using Apache Kafka | by Michał Matłoka
Sending a message to non-existing Kafka topic, by default results in its ... the maximum number of consumers from a single consumer group....
Read more >
Kafka Streams with Quarkus - Piotr's TechBlog
If you would like to figure out what is a streaming platform and how it differs from a traditional message broker this article...
Read more >
Testing Quarkus Web Applications: Reactive Messaging ...
Let's see how we can write tests for these use cases. The application. Let's consider a small use case of a checkout process...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found