batch-mode doesn't work when using 3.x
See original GitHub issueas mentioned in the README,
Starting with version 3.0, when spring.cloud.stream.binding.<name>.consumer.batch-mode is set to true, all of the records received by polling the Kafka Consumer will be presented as a List<?> to the listener method. Otherwise, the method will be called with one record at a time. The size of the batch is controlled by Kafka consumer properties max.poll.records, min.fetch.bytes, fetch.max.wait.ms; refer to the Kafka documentation for more information.`
but when I use org.springframework.cloud:spring-cloud-stream-binder-kafka:3.0.4.RELEASE
, it still doesn’t work.
When I try to use List<T>
in listener, I got the below error messages:
org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.cloud.stream.binding.StreamListenerMessageHandler@656f62dc]; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot parse payload ; nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.util.ArrayList<java.lang.Object>` out of START_OBJECT token
at [Source: (String)"{"id":3205795182502117669,"location":"ShangHai","ownerName":"Tina","publishedTime":"2020-05-13T07:52:31.960Z"}"; line: 1, column: 1], failedMessage=GenericMessage [payload={"id":3205795182502117669,"location":"ShangHai","ownerName":"Tina","publishedTime":"2020-05-13T07:52:31.960Z"}, headers={kafka_offset=8, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@35b7e9aa, deliveryAttempt=3, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=1, kafka_receivedTopic=org.example.alvin.spring.cloud.kafka.input, kafka_receivedTimestamp=1589356373714, contentType=application/json, kafka_groupId=receive-in-group-1}]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:1790) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeErrorHandler(KafkaMessageListenerContainer.java:1779) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1692) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:1618) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:1521) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1262) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1009) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:929) [spring-kafka-2.3.7.RELEASE.jar:2.3.7.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_211]
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) [na:1.8.0_211]
at java.util.concurrent.FutureTask.run(FutureTask.java) [na:1.8.0_211]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_211]
the below is part of my configuration in application.yml
spring:
cloud:
stream:
kafka:
binder:
brokers: 192.168.20.10:9092
zkNodes: 192.168.20.10:2181
autoCreateTopics: false
configuration:
security.protocol: PLAINTEXT
sasl.mechanism: PLAIN
bindings:
kafka-in-0:
consumer:
batch-mode: true
configuration:
auto.offset.reset: earliest
group.id: receive-in-group-1
key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.StringDeserializer
kafka-out-0:
producer:
configuration:
key.serializer: org.apache.kafka.common.serialization.StringSerializer
value.serializer: org.apache.kafka.common.serialization.StringSerializer
bindings:
kafka-in-0:
binder: kafka
destination: org.example.alvin.spring.cloud.kafka.input
contentType: application/json
kafka-out-0:
binder: kafka
destination: org.example.alvin.spring.cloud.kafka.output
contentType: application/json
Issue Analytics
- State:
- Created 3 years ago
- Comments:8 (4 by maintainers)
Top Results From Across the Web
Can't get accelerator to work in batch mode - Unity Forum
I'm trying to set up a build machine with accelerator and I can't get any cacheing in batch mode. It's a Windows VM...
Read more >[BUG] Batch mode doesn't work w/o changes being applied
Just the batch execution doesn't. The command I use is cd C:\Program Files\KNIME knime.exe -nosplash -reset -nosave --launcher.suppressErrors - ...
Read more >RE: st: Windows batch mode not working-solved - Stata
RE: st: Windows batch mode not working-solved ... but my hand slipped and I hit the send button before this was fully edited...
Read more >Macro doesn't run when using it in Batch mode - Image.sc Forum
Hi all. This is my first time writing a macro for Imagej so I'm still getting familiar with the language. With the record...
Read more >Performing Android and iOS batch-mode builds causes error ...
2. Open the log file 3. Notice that this error is logged constantly: "RenderTexture.Create with shadow sampling failed: Hardware does not ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@garyrussell thanks for your demo. it works. I got another way to get the same result is I still use @StreamListener and use ThreadLocal to maintains batch of messages then process them.
@Bryce-huang There was no bug associated with this issue. There was only a docs issue that got addressed through the commit mentioned above.