Multi emit v. single emit

Problem: Currently, we are using two different approaches when it comes to transmitting a list of elements between components.

I. A component emits a message to the queue multiple times to deliver each element of a list as a single element to the next component. multi emit component, mec

II. A component emits one message to the queue holding the full list. Everything will be delivered at once to the next component. single emit component, sec

We want to allow both approaches to keep the framework more flexible. Therefore we need to adjust all components which are doing multiple emits.

To progress on this issue we have to:

1. Detect all mecs within our scope
2. Find a proper way to send currentEmit/allEmits (e.g 3/4)
3. Find a proper way to handle multi emits in orchestrator (evaluate integers)
4. Estimate changes to be done

[winklerj]

I’m not sure where your final decision on this is, but I will weigh in that we make use of both MEC and SEC. Additionally our use case is typically to split and/or batch objects before sending to the rest of the flow steps. So it could be multiple SEC for one object. This is to avoid very large messages going through the flow. The batching gives us the benefit of making efficient API calls on subsequent steps. I don’t want to be asking for one result at a time when the API allows 250.

[SvenHoeffler]

From our perspective, we’d certainly prefer it if SEC remained viable and supported in some fashion, preferably without having to adjust all existing components.

Batching at the receiving end does seem like the most feasible option currently, in part because it would also allow you to have a flow where a single source can distribute data both to single-processing and batch-processing receiving components, possibly through the use of logic gateways also.

The accumulation of batches could principally happen in either the ferryman, or a separate utility component. We’d have to touch the ferryman either way, it’s mostly just a question of how we’d want to distribute the resource load and trust.