Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Error in KafkaTopicProvisioner

See original GitHub issue

I am getting the following error from the KafkaTopicProvisioner:

2020-05-01 11:35:56 - Failed to create producer binding; retrying in 30 seconds
org.springframework.cloud.stream.provisioning.ProvisioningException: Problems encountered with partitions finding; nested exception is java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
        at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.provisionProducerDestination(KafkaTopicProvisioner.java:161) ~[spring-cloud-stream-binder-kafka-core-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        at org.springframework.cloud.stream.binder.kafka.streams.KStreamBinder.doBindProducer(KStreamBinder.java:118) ~[spring-cloud-stream-binder-kafka-streams-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        at org.springframework.cloud.stream.binder.kafka.streams.KStreamBinder.doBindProducer(KStreamBinder.java:52) ~[spring-cloud-stream-binder-kafka-streams-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        at org.springframework.cloud.stream.binder.AbstractBinder.bindProducer(AbstractBinder.java:152) ~[spring-cloud-stream-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        at org.springframework.cloud.stream.binding.BindingService.doBindProducer(BindingService.java:296) ~[spring-cloud-stream-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        at org.springframework.cloud.stream.binding.BindingService.bindProducer(BindingService.java:271) ~[spring-cloud-stream-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        at org.springframework.cloud.stream.binding.AbstractBindableProxyFactory.createAndBindOutputs(AbstractBindableProxyFactory.java:136) ~[spring-cloud-stream-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        at org.springframework.cloud.stream.binding.OutputBindingLifecycle.doStartWithBindable(OutputBindingLifecycle.java:58) ~[spring-cloud-stream-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        at java.base/java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608) ~[na:na]
        at org.springframework.cloud.stream.binding.AbstractBindingLifecycle.start(AbstractBindingLifecycle.java:57) ~[spring-cloud-stream-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        at org.springframework.cloud.stream.binding.OutputBindingLifecycle.start(OutputBindingLifecycle.java:34) ~[spring-cloud-stream-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
        at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
        at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
        at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
        at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:122) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:894) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:162) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
        at com.someclass.Main.main(Main.java:90) ~[classes!/:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) ~[baseline-pc-pattern-reconciliation-0.1.0-SNAPSHOT.jar:na]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) ~[baseline-pc-pattern-reconciliation-0.1.0-SNAPSHOT.jar:na]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) ~[baseline-pc-pattern-reconciliation-0.1.0-SNAPSHOT.jar:na]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52) ~[baseline-pc-pattern-reconciliation-0.1.0-SNAPSHOT.jar:na]

Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) ~[kafka-clients-2.4.1.jar!/:na]
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) ~[kafka-clients-2.4.1.jar!/:na]
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104) ~[kafka-clients-2.4.1.jar!/:na]
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272) ~[kafka-clients-2.4.1.jar!/:na]
        at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.provisionProducerDestination(KafkaTopicProvisioner.java:158) ~[spring-cloud-stream-binder-kafka-core-3.0.3.RELEASE.jar!/:3.0.3.RELEASE]
        ... 33 common frames omitted
Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.

From what I can tell the warning “This server does not host this topic-partition” is a normal message from the kafka broker when the metadata is stale. Waiting some time gives the broker some time to clean this up.

The issue is that the KafkaTopicProvisioner doesn’t wait or try again but chooses to throw an error. Worse, it doesn’t even put the streams application in a bad state where it could be restarted by k8s but just continues like everything is fine.

Would it be possible to add some retry options in the KafkaTopicProvisioner? Currently the only way to work around this issue is to either turn off automatic topic creation or check that the topic describe command works before starting up the streams app.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
tkaszubacommented, May 6, 2020

You can turn off the admin client and create the topics before hand and then the topic describe will not be called. The other option is to create the topics beforehand and wait for the describe command to not return errors before starting up your service/app.

0reactions
sobychackocommented, May 27, 2020
Read more comments on GitHub >

github_iconTop Results From Across the Web

Spring Cloud Kafka Stream Unable to create Producer Config ...
Strange: you show config for consumer, but error is about producer. Also the client.id is different... – Artem Bilan. Mar 22, 2018 at...
Read more >
Handling Errors when Topics dont exist · Issue #513 - GitHub
KafkaTopicProvisioner is throwing a NPE when topics don't exist. I do believe it can handle the scenario and log a meaningful message.
Read more >
org.springframework.cloud.stream.binder.kafka ... - Tabnine
Best Java code snippets using org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner (Showing top 20 results out of 315).
Read more >
spring-cloud/spring-cloud-stream - Gitter
There is no error or problem with working of the app. It is about a case, where scs logs exception, when it doesn't...
Read more >
KafkaMessageChannelBinder (spring-cloud-stream-docs ...
KafkaTopicProvisioner>; Parameters: destination - the name of the target destination: producerProperties - the producer properties: errorChannel - the error ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

batch-mode doesn't work when using 3.x

Next page

Overriding "(consumer|producer)Properties.bootstrap.servers" should be prohibited