question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Parallel consumption of one stream

See original GitHub issue

How can I have multiple consumers receiving data from one object stream in parallel? parallel only works on streams of streams.

Issue Analytics

  • State:closed
  • Created 10 years ago
  • Comments:26 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
svozzacommented, Sep 5, 2016

The answer is above there, use map to convert your stream into a stream of streams and then use mergeWithLimit.

_(['www.example.com/foo', 'www.example.com/bar', 'www.example.com/baz'])
    .map(makeHtttpRequest) // makeHtttpRequest must return a Highland stream
   .mergeWithLimit(noOfWorkers);
2reactions
caolancommented, Mar 9, 2014

This definitely sounds like something we should add. Also cc @theefer - I think this is what we talked about recently.

Essentially, we want a pool of consumers, each reading from the source as soon as one is ready instead of blocking until all are ready. The results are then pushed out on a single output stream (like merge()). Perhaps it could be called balance(workers), where workers is an Array or Stream of writable worker streams? Other name suggestions most welcome!

Read more comments on GitHub >

github_iconTop Results From Across the Web

When to Use a Parallel Stream in Java - Baeldung
Any stream in Java can easily be transformed from sequential to parallel. ... Parallel streams enable us to execute code in parallel on...
Read more >
Parallel streams in Java: Benchmarking and performance ...
A parallel stream is split into multiple substreams that are processed in parallel by multiple instances of the stream pipeline being executed ...
Read more >
Introducing the Confluent Parallel Consumer
Combining the Parallel Consumer with Kafka Streams​​ Using the Parallel Consumer, you can consume from the intermediary topic produced by Kafka  ...
Read more >
Parallel vs Sequential Stream in Java - GeeksforGeeks
Parallel stream leverage multi-core processors, which increases its performance. Using parallel streams, our code gets divide into multiple ...
Read more >
Java Parallel Stream - When to Use it - Xperti
Java parallel stream is a stream API introduced in Java 8 that aims to make use of multiple cores for task execution. This...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found