question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Use Parallel.ForEach(...) to forward records in child processor inside one topology

See original GitHub issue

Description

At time, forwarding records in child processor inside topology is synchronous.

        public virtual void Forward<K1, V1>(K1 key, V1 value)
        {
            log.Debug($"{logPrefix}Forward<{typeof(K1).Name},{typeof(V1).Name}> message with key {key} and value {value} to each next processor");
            foreach (var n in Next)
                if (n is IProcessor<K1, V1>)
                    (n as IProcessor<K1, V1>).Process(key, value);
        }

Aims of this ticket is to propose a parallel execution like that :

        public virtual void Forward<K1, V1>(K1 key, V1 value)
        {
            log.Debug($"{logPrefix}Forward<{typeof(K1).Name},{typeof(V1).Name}> message with key {key} and value {value} to each next processor");
            Parallel.ForEach(Next, (n) => n.Process(key, value));
        }

Test to execute :

  • No regression in stateless processor
  • No regression in statefull processor

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
vimal-sivasubramaniancommented, Jun 1, 2021

@LGouellec, We can address the above mentioned issue with the following code,

Parallel.ForEach(Next.OfType<IProcessor<K1, V1>>(), p => p.Process(key, value));

when we have to filter by name we can make use of linq to perform the same

var processors = Next.OfType<IProcessor<K1, V1>>() .Where(p => p.Name.Equals(name));

0reactions
joaovitorpinacommented, Jun 10, 2022

@LGouellec I oppened a pull request pertaining this issue, let me know what you think! #153

Read more comments on GitHub >

github_iconTop Results From Across the Web

Parallel.ForEach - Where is it running on single core ...
I would have to run a test on a single core machine. Since I don't have one, I will use virtual machine and...
Read more >
Write a simple Parallel.ForEach loop
This article shows how to use a Parallel.ForEach loop to enable data parallelism over any System.Collections.IEnumerable or System.
Read more >
How to use Parallel.For and Parallel.ForEach in C# | ...
The Parallel.ForEach method splits the work to be done into multiple tasks, one for each item in the collection. Parallel.ForEach is like the ......
Read more >
Parallel For Each Scope
The Parallel For Each scope enables you to process a collection of messages by splitting the collection into parts that are simultaneously processed...
Read more >
Parallel.ForEach not using all my available cpu space
Many requests can be sent out one after another on one thread and the responses can be awaited in parallel, no additional threads...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found