question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

batch-mode doesn't work when using 3.x

See original GitHub issue

as mentioned in the README,

Starting with version 3.0, when spring.cloud.stream.binding.<name>.consumer.batch-mode is set to true, all of the records received by polling the Kafka Consumer will be presented as a List<?> to the listener method. Otherwise, the method will be called with one record at a time. The size of the batch is controlled by Kafka consumer properties max.poll.records, min.fetch.bytes, fetch.max.wait.ms; refer to the Kafka documentation for more information.`

but when I use org.springframework.cloud:spring-cloud-stream-binder-kafka:3.0.4.RELEASE, it still doesn’t work. When I try to use List<T> in listener, I got the below error messages:

org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.cloud.stream.binding.StreamListenerMessageHandler@656f62dc]; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot parse payload ; nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.util.ArrayList<java.lang.Object>` out of START_OBJECT token
 at [Source: (String)"{"id":3205795182502117669,"location":"ShangHai","ownerName":"Tina","publishedTime":"2020-05-13T07:52:31.960Z"}"; line: 1, column: 1], failedMessage=GenericMessage [payload={"id":3205795182502117669,"location":"ShangHai","ownerName":"Tina","publishedTime":"2020-05-13T07:52:31.960Z"}, headers={kafka_offset=8, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@35b7e9aa, deliveryAttempt=3, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=1, kafka_receivedTopic=org.example.alvin.spring.cloud.kafka.input, kafka_receivedTimestamp=1589356373714, contentType=application/json, kafka_groupId=receive-in-group-1}]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:1790) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeErrorHandler(KafkaMessageListenerContainer.java:1779) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1692) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:1618) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:1521) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1262) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1009) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:929) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_211]
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) [na:1.8.0_211]
	at java.util.concurrent.FutureTask.run(FutureTask.java) [na:1.8.0_211]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_211]

the below is part of my configuration in application.yml

spring:
    cloud:
    stream:
      kafka:
        binder:
          brokers: 192.168.20.10:9092
          zkNodes: 192.168.20.10:2181
          autoCreateTopics: false
          configuration:
            security.protocol: PLAINTEXT
            sasl.mechanism: PLAIN
        bindings:
          kafka-in-0:
            consumer:
              batch-mode: true
              configuration:
                auto.offset.reset: earliest
                group.id: receive-in-group-1
                key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
                value.deserializer: org.apache.kafka.common.serialization.StringDeserializer
          kafka-out-0:
            producer:
              configuration:
                key.serializer: org.apache.kafka.common.serialization.StringSerializer
                value.serializer: org.apache.kafka.common.serialization.StringSerializer
      bindings:
        kafka-in-0:
          binder: kafka
          destination: org.example.alvin.spring.cloud.kafka.input
          contentType: application/json
        kafka-out-0:
          binder: kafka
          destination: org.example.alvin.spring.cloud.kafka.output
          contentType: application/json

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
AlvinYueChaocommented, May 16, 2020

@garyrussell thanks for your demo. it works. I got another way to get the same result is I still use @StreamListener and use ThreadLocal to maintains batch of messages then process them.

0reactions
sobychackocommented, Dec 14, 2020

@Bryce-huang There was no bug associated with this issue. There was only a docs issue that got addressed through the commit mentioned above.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Can't get accelerator to work in batch mode - Unity Forum
I'm trying to set up a build machine with accelerator and I can't get any cacheing in batch mode. It's a Windows VM...
Read more >
[BUG] Batch mode doesn't work w/o changes being applied
Just the batch execution doesn't. The command I use is cd C:\Program Files\KNIME knime.exe -nosplash -reset -nosave --launcher.suppressErrors - ...
Read more >
RE: st: Windows batch mode not working-solved - Stata
RE: st: Windows batch mode not working-solved ... but my hand slipped and I hit the send button before this was fully edited...
Read more >
Macro doesn't run when using it in Batch mode - Image.sc Forum
Hi all. This is my first time writing a macro for Imagej so I'm still getting familiar with the language. With the record...
Read more >
Performing Android and iOS batch-mode builds causes error ...
2. Open the log file 3. Notice that this error is logged constantly: "RenderTexture.Create with shadow sampling failed: Hardware does not ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found