question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Serde with custom aggregater

See original GitHub issue

Can any please let me know what is the best way to set the serde for the materialized stores with custom aggregater class Following is my aggregation

groupByKey() .windowedBy(TimeWindows.of(1000).until(1000)) .aggregate(DataAggregate::new, (key, value, aggregate) -> { aggregate.add(1); return aggregate; }, Materialized.as("aggstore"))

DataAggregate is the class that holds the aggregation - I need to save that to aggstore

but the default serde I am using is the spring.cloud.stream.kafka.streams.binder.configuration.commit.interval.ms=1000 spring.cloud.stream.kafka.streams.binder.configuration.default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde

exception I am getting is Exception in thread "eventsin-00aacb65-5224-4b2f-858b-fe98bd134b7c-StreamThread-1" java.lang.ClassCastException: sample.streamprocessor.dataaggregator.DataAggregate cannot be cast to java.lang.String at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:28) at org.apache.kafka.streams.state.StateSerdes.rawValue(StateSerdes.java:178) at org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:96) at org.apache.kafka.streams.kstream.internals.KStreamWindowAggregate$KStreamWindowAggregateProcessor.process(KStreamWindowAggregate.java:122)

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
sobychackocommented, May 22, 2018

@vinu - You cannot directly use the JsonSerde class like that. However, you can extend from this in your project and use it in properties. For instance,

FooSerde extends JsonSerde<Foo> {}

and then spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde=FooSerde

1reaction
sobychackocommented, May 21, 2018

@vinu what @olegz and @orchesio said above. You need to pass the proper key and value Serde when you materialize the state store. Since you are not passing any Serdes for that, the defaults are kicking in and running into that exception. You need to use the Materialized.with... methods for that. If your DataAggregate is json friendly, you can use the JsonSerde provided as part of spring-kafka and pass a reference of this to your aggregation. Hope this helps.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Error while serializing aggregate state store with custom serde ...
When you see an error like: serializer (org.apache.kafka.common.serialization.ByteArraySerializer) is not compatible to the actual value ...
Read more >
Custom serialization - Serde
For unusual needs, Serde allows full customization of the serialization behavior by manually implementing Serialize and Deserialize traits for your type.
Read more >
SerDe reference - Amazon Athena
Athena supports several SerDe libraries for parsing data from different data formats, such as CSV, JSON, Parquet, and ORC. Athena does not support...
Read more >
KafkaStreams : Custom Serdes - Sacha's Blog - WordPress.com
Every Kafka Streams application must provide SerDes (Serializer/Deserializer) for the data types of record keys and record values (e.g. java.
Read more >
Compute an average aggregation using Kafka Streams
In this tutorial, learn how to compute an average aggregation like count or sum using Kafka ... implementation "io.confluent:kafka-streams-avro-serde:7.3.0" ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found