Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

destinationIsPattern: true using @StreamListener not picking up new topics with matching pattern (KafkaStreamsBinder)

See original GitHub issue

I am struggling to get the destinationIsPattern: true using @StreamListener in Spring Boot application. @KafkaListener(topicPattern = “.\.topicName.”) works fine but having issue with @StreamListener.

Following is my current configuration.

  cloud:
    stream:
      bindings:
        topic-in:
            destination: .*\..*topicName
            contentType: application/json
      kafka:
        streams:
          binder:
            brokers: ${KAFKA_SERVERS:localhost:9092}
            configuration:
              metadata.max.age.ms: 6000
              auto.offset.reset: earliest
          bindings:
            topic-in:
              consumer:
                  destinationIsPattern: true

Appreciate any help in getting through this problem.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:22 (15 by maintainers)

github_iconTop GitHub Comments

1reaction
sobychackocommented, Oct 22, 2020

@jitendradpatel Created a sample demonstrating this feature here. Please take a look. I am closing this issue. Feel free to re-open if you still encounter issues or if you can reproduce the issues using the sample I mentioned.

1reaction
garyrussellcommented, Sep 25, 2020

@sobychacko Looks like a problem with the binder. Maybe the max.metadata.age.ms property is not being set?

This works fine for me…

@SpringBootApplication
@EnableBinding(Sink.class)
@EnableKafkaStreams
public class Kbgh961Application {

	public static void main(String[] args) {
		SpringApplication.run(Kbgh961Application.class, args);
	}

	@StreamListener(Sink.INPUT)
	public void listen(String in) {
		System.out.println(in);
	}

	@Bean
	KStream<String, String> stream(StreamsBuilder builder) {
		KStream<String, String> stream = builder.stream(Pattern.compile(".*foo"));
		stream.foreach((k, v) -> System.out.println(v));
		return stream;
	}

}

spring:
  application:
    name: kbgh961s
  kafka:
    streams:
      properties:
        metadata.max.age.ms: 30000
  cloud:
    stream:
      bindings:
        input:
          destination: .*foo
      kafka:
        binder:
          configuration:
            metadata:
              max:
                age:
                  ms: 30000

Started with no matching topics, then added my.foo…

2020-09-25 09:40:50.582  INFO 59692 --- [           main] com.example.demo.Kbgh961Application      : Started Kbgh961Application in 1.927 seconds (JVM running for 2.412)
2020-09-25 09:40:50.619  INFO 59692 --- [-StreamThread-1] o.a.k.s.p.i.StreamsPartitionAssignor     : stream-thread [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer] Assigned tasks to clients as 
4ab62aa9-7293-4f7f-b666-1b133c906326=[activeTasks: ([]) standbyTasks: ([]) assignedTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) prevAssignedTasks: ([]) prevOwnedPartitionsByConsumerId: ([]) capacity: 1].
2020-09-25 09:40:50.620  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Finished assignment for group at generation 1: {kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer-f78f9119-e92b-4fc3-b65f-f8375f5047da=Assignment(partitions=[], userDataSize=32)}
2020-09-25 09:40:50.624  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Successfully joined group with generation 1
2020-09-25 09:40:50.625  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Updating assignment with
now assigned partitions: 
compare with previously owned partitions: 
newly added partitions: 
revoked partitions: 

2020-09-25 09:40:50.626  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Adding newly assigned partitions: 
2020-09-25 09:40:50.626  INFO 59692 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread         : stream-thread [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1] State transition from STARTING to PARTITIONS_ASSIGNED
2020-09-25 09:40:50.626  INFO 59692 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread         : stream-thread [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1] partition assignment took 0 ms.
	currently assigned active tasks: []
	currently assigned standby tasks: []
	revoked active tasks: []
	revoked standby tasks: []

2020-09-25 09:40:50.680  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Finished assignment for group at generation 1: {consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1-51ede99d-d68c-4234-9652-b265788c5d24=Assignment(partitions=[])}
2020-09-25 09:40:50.684  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Successfully joined group with generation 1
2020-09-25 09:40:50.685  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Adding newly assigned partitions: 
2020-09-25 09:40:50.686  INFO 59692 --- [container-0-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1  : anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2: partitions assigned: []
2020-09-25 09:40:50.713  INFO 59692 --- [-StreamThread-1] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions
2020-09-25 09:40:50.713  INFO 59692 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread         : stream-thread [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING
2020-09-25 09:40:50.713  INFO 59692 --- [-StreamThread-1] org.apache.kafka.streams.KafkaStreams    : stream-client [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326] State transition from REBALANCING to RUNNING
2020-09-25 09:41:20.618  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] (Re-)joining group
2020-09-25 09:41:20.622  INFO 59692 --- [-StreamThread-1] o.a.k.s.p.i.StreamsPartitionAssignor     : stream-thread [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer] Assigned tasks to clients as 
4ab62aa9-7293-4f7f-b666-1b133c906326=[activeTasks: ([0_0, 0_1, 0_2]) standbyTasks: ([]) assignedTasks: ([0_0, 0_1, 0_2]) prevActiveTasks: ([]) prevStandbyTasks: ([]) prevAssignedTasks: ([]) prevOwnedPartitionsByConsumerId: ([]) capacity: 1].
2020-09-25 09:41:20.623  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Finished assignment for group at generation 2: {kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer-f78f9119-e92b-4fc3-b65f-f8375f5047da=Assignment(partitions=[my.foo-0, my.foo-1, my.foo-2], userDataSize=56)}
2020-09-25 09:41:20.624  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Successfully joined group with generation 2
2020-09-25 09:41:20.625  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Updating assignment with
now assigned partitions: my.foo-0, my.foo-1, my.foo-2
compare with previously owned partitions: 
newly added partitions: my.foo-0, my.foo-1, my.foo-2
revoked partitions: 

2020-09-25 09:41:20.627  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Adding newly assigned partitions: my.foo-0, my.foo-1, my.foo-2
2020-09-25 09:41:20.627  INFO 59692 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread         : stream-thread [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED
2020-09-25 09:41:20.627  INFO 59692 --- [-StreamThread-1] org.apache.kafka.streams.KafkaStreams    : stream-client [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326] State transition from RUNNING to REBALANCING
2020-09-25 09:41:20.641  INFO 59692 --- [-StreamThread-1] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions
2020-09-25 09:41:20.641  INFO 59692 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread         : stream-thread [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1] partition assignment took 14 ms.
	currently assigned active tasks: [0_0, 0_1, 0_2]
	currently assigned standby tasks: []
	revoked active tasks: []
	revoked standby tasks: []

2020-09-25 09:41:20.645  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Found no committed offset for partition my.foo-0
2020-09-25 09:41:20.645  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Found no committed offset for partition my.foo-1
2020-09-25 09:41:20.646  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Found no committed offset for partition my.foo-2
2020-09-25 09:41:20.651  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Resetting offset for partition my.foo-0 to offset 0.
2020-09-25 09:41:20.652  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Resetting offset for partition my.foo-1 to offset 0.
2020-09-25 09:41:20.652  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Resetting offset for partition my.foo-2 to offset 0.
2020-09-25 09:41:20.691  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Revoke previously assigned partitions 
2020-09-25 09:41:20.691  INFO 59692 --- [container-0-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1  : anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2: partitions revoked: []
2020-09-25 09:41:20.691  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] (Re-)joining group
2020-09-25 09:41:20.693  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Finished assignment for group at generation 2: {consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1-51ede99d-d68c-4234-9652-b265788c5d24=Assignment(partitions=[my.foo-0, my.foo-1, my.foo-2])}
2020-09-25 09:41:20.699  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Successfully joined group with generation 2
2020-09-25 09:41:20.699  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Adding newly assigned partitions: my.foo-0, my.foo-1, my.foo-2
2020-09-25 09:41:20.700  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Found no committed offset for partition my.foo-0
2020-09-25 09:41:20.700  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Found no committed offset for partition my.foo-1
2020-09-25 09:41:20.700  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Found no committed offset for partition my.foo-2
2020-09-25 09:41:20.701  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Found no committed offset for partition my.foo-0
2020-09-25 09:41:20.701  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Found no committed offset for partition my.foo-1
2020-09-25 09:41:20.701  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Found no committed offset for partition my.foo-2
2020-09-25 09:41:20.702  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Resetting offset for partition my.foo-0 to offset 0.
2020-09-25 09:41:20.702  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Resetting offset for partition my.foo-1 to offset 0.
2020-09-25 09:41:20.702  INFO 59692 --- [container-0-C-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=consumer-anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2-1, groupId=anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2] Resetting offset for partition my.foo-2 to offset 0.
2020-09-25 09:41:20.706  INFO 59692 --- [container-0-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1  : anonymous.ce44ebca-c1d5-443c-adcf-8747046daba2: partitions assigned: [my.foo-0, my.foo-1, my.foo-2]
2020-09-25 09:41:20.714  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Found no committed offset for partition my.foo-0
2020-09-25 09:41:20.718  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Found no committed offset for partition my.foo-1
2020-09-25 09:41:20.719  INFO 59692 --- [-StreamThread-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-consumer, groupId=kbgh961s] Found no committed offset for partition my.foo-2
2020-09-25 09:41:20.720  INFO 59692 --- [-StreamThread-1] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions
2020-09-25 09:41:20.720  INFO 59692 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread         : stream-thread [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING
2020-09-25 09:41:20.720  INFO 59692 --- [-StreamThread-1] org.apache.kafka.streams.KafkaStreams    : stream-client [kbgh961s-4ab62aa9-7293-4f7f-b666-1b133c906326] State transition from REBALANCING to RUNNING
Read more comments on GitHub >

github_iconTop Results From Across the Web

Spring Cloud Stream Kafka Binder Reference Guide
When true, the destination is treated as a regular expression Pattern used to match topic names by the broker. When true, topics are...
Read more >
spring-cloud/spring-cloud-stream - Gitter
Hello, I'm using spring cloud stream Kafka binder 3.1.0, and I'm facing a behaviour that I don't know if it's normal or not....
Read more >
Spring Boot application using the Spring Cloud Stream Kafka ...
Now I can't see any exception but the messages don't get to the topic. In another application that only uses the kafka binder...
Read more >
39. Apache Kafka Binder - Spring Cloud Reference
When true, the destination is treated as a regular expression Pattern used to match topic names by the broker. When true, topics are...
Read more >
Streaming with Spring Cloud Stream and Apache Kafka
We'll also venture into seeing how the new functional programming model in Spring Cloud Stream makes it easier to write Kafka- and Kafka ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

spring.kafka.bootstrap-servers overrides brokers configuration of every binder.

Next page

Update Topic configuration from topic.properties on topic provisioning