Use Parallel.ForEach(...) to forward records in child processor inside one topology
See original GitHub issueDescription
At time, forwarding records in child processor inside topology is synchronous.
public virtual void Forward<K1, V1>(K1 key, V1 value)
{
log.Debug($"{logPrefix}Forward<{typeof(K1).Name},{typeof(V1).Name}> message with key {key} and value {value} to each next processor");
foreach (var n in Next)
if (n is IProcessor<K1, V1>)
(n as IProcessor<K1, V1>).Process(key, value);
}
Aims of this ticket is to propose a parallel execution like that :
public virtual void Forward<K1, V1>(K1 key, V1 value)
{
log.Debug($"{logPrefix}Forward<{typeof(K1).Name},{typeof(V1).Name}> message with key {key} and value {value} to each next processor");
Parallel.ForEach(Next, (n) => n.Process(key, value));
}
Test to execute :
- No regression in stateless processor
- No regression in statefull processor
Issue Analytics
- State:
- Created 2 years ago
- Comments:6 (3 by maintainers)
Top Results From Across the Web
Parallel.ForEach - Where is it running on single core ...
I would have to run a test on a single core machine. Since I don't have one, I will use virtual machine and...
Read more >Write a simple Parallel.ForEach loop
This article shows how to use a Parallel.ForEach loop to enable data parallelism over any System.Collections.IEnumerable or System.
Read more >How to use Parallel.For and Parallel.ForEach in C# | ...
The Parallel.ForEach method splits the work to be done into multiple tasks, one for each item in the collection. Parallel.ForEach is like the ......
Read more >Parallel For Each Scope
The Parallel For Each scope enables you to process a collection of messages by splitting the collection into parts that are simultaneously processed...
Read more >Parallel.ForEach not using all my available cpu space
Many requests can be sent out one after another on one thread and the responses can be awaited in parallel, no additional threads...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@LGouellec, We can address the above mentioned issue with the following code,
Parallel.ForEach(Next.OfType<IProcessor<K1, V1>>(), p => p.Process(key, value));
when we have to filter by name we can make use of linq to perform the same
var processors = Next.OfType<IProcessor<K1, V1>>() .Where(p => p.Name.Equals(name));
@LGouellec I oppened a pull request pertaining this issue, let me know what you think! #153