question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

2 Consumers of same topic

See original GitHub issue

Friends,

Since using the latest version of spring cloud and spring-cloud-stream-binder-kafka-streams:3.2.1

I lost the ability to have 2 consumers against the same topic. The spring code actually creates the binder proxy against the topic name. Since another already exists it throws a duplicate bean exception. Another project I am using spring-cloud-stream-binder-kafka-streams:3.0.9.RELEASE and I don’t have this problem. Another use posted this issue but then closed it so perhaps its a config thing.

My config is quite basic.

spring:
  cloud:
    stream:
      function:
        definition: func1,func2
        bindings:
          func1-in-0:
            destination: topic1
          func2-in-0:
            destination: topic1

The root issue is that a bean is put as the topic name.

KafkaStreamsBindableProxyFactory.bindInput

            private void bindInput(ResolvableType arg0, String inputName) {
		            if (arg0.getRawClass() != null) {
			            KafkaStreamsBindableProxyFactory.this.inputHolders.put(inputName,
					            new BoundTargetHolder(getBindingTargetFactory(arg0.getRawClass())
							            .createInput(inputName), true));
		            }
		            BeanDefinitionRegistry registry = (BeanDefinitionRegistry) beanFactory;
		            RootBeanDefinition rootBeanDefinition = new RootBeanDefinition();
		            rootBeanDefinition.setInstanceSupplier(() -> inputHolders.get(inputName).getBoundTarget());
		            registry.registerBeanDefinition(inputName, rootBeanDefinition);
            }

In older versions and my other project kafka streams uses the binding name: func1-in-0 in the code above but the latest version use topic name. The documentation still defines the configuration the same in both versions.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
chrismpalmercommented, Jan 27, 2022

Quick response thanks and I am glad its not a bug.

0reactions
chrismpalmercommented, Jan 27, 2022

Thank you.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Can multiple Kafka consumers read same message from the ...
If you have 2 Kafka consumers with the same Group Id, they will both read 6 partitions, meaning they will read different set...
Read more >
Can multiple Kafka consumers read the same message from a ...
[…] A consumer can be assigned to consume multiple partitions. So the rule in Kafka is only one consumer in a consumer group...
Read more >
Chapter 4. Kafka Consumers: Reading Data from Kafka
Kafka consumers are typically part of a consumer group . When multiple consumers are subscribed to a topic and belong to the same...
Read more >
Can two consumers consume from same partition at ... - Quora
Yes if a topic has more than 1 partition then they will be distributed across the nodes in a cluster. That's one reason...
Read more >
How to parallelise Kafka consumers | by Jhansi Karee - Medium
A. Kafka can't assign the same partition to two consumers within the same group. What about different consumer groups then? Partitions are only ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found