Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ZStream#groupByKey performance

See original GitHub issue

I experienced that when using ZStream#groupByKey, stream throughput suffers a lot. I suspect it’s because distributedWithDynamic breaks the chunk structure.

It might be worth to have a dedicated implementation that tries to preserve chunks, like so or perhaps distributedWithDynamic can be improved.

Issue Analytics

State:
Created 3 years ago
Comments:9 (7 by maintainers)

Top GitHub Comments

1reaction

svroonlandcommented, Apr 13, 2022

Was about to post an update on this issue about this one of these days. I noticed a similar performance difference with ZIO 2’s groupByKey. I’m using a custom version of the method here: https://github.com/svroonland/zio-kinesis/blob/77c92cbd71d8c99167e4d62229137d74aaf49496/core/src/main/scala/nl/vroste/zio/kinesis/client/Util.scala#L13

0reactions

miroslavmachuracommented, May 18, 2022

I see the fix was merged into zio:series/2.x branch. Is the fix going to also be applied to zio1?

Top Results From Across the Web

KStream (kafka 2.2.0 API)

KStream is an abstraction of a record stream of KeyValue pairs, i.e., each record is an independent entity/event in the real world.

Kafka Streams - groupByKey returns unexpected result

I have a Kafka topic t1 than contains 100 messages. After a flatMapValues() operation, the result is written to a second topic t2....

Stream Aggregation In Kafka - Narayan Kumar - Medium

Introducing the aggregation in Kafka and explained this in easy way to implement the Aggregation on real time streaming. groupBy or groupByKey uses...

Optimizing Kafka Streams Applications | Confluent

Along with it, we will demonstrate a few known issues that impact efficiency of the generated processor topology. Then we will describe how ......

Let mortal combat begin! Apache Beam's GroupByKey vs ...

GroupByKey groups all elements with the same key and produces ... We have to create a new accumulator being the result of merging...