Process Manager

What’s the model to do Process Managers/Sagas/Long running processes or in general, transforming event -> command list?

I don’t understand well the proposed framework, sorry 😅

Looks like everything is focused on the Command side (command -> 'state -> event' list) and queries are resolve based on Aggregate state, like “commands with result” (command -> 'state -> result * event' list), am I right?

I understand that propulsion is meant for subscriptions/projections, but then how is the Query part (of CQRS) fall into place within the Equinox Framework?

I’ll give you an example to illustrate my confusion:

• C: Command
• E: Event

A User Service (like a server) sends a command to a Crawler Aggregate to start the crawling, but the HTTP requests need to go through HTTP Aggregate that will schedule each request and process them later (due rate limits).

When the HTTP Aggregate process the requests it issues an event with the response and a Process Manager “should” pick the events and issue commands to the Crawl Aggregate.

• User Service ->[C:Start Crawl Process] -> Crawler Aggregate -> (OK, [E:Crawl Process Started])
• E:Crawl Process Started -> Crawler Process Manager -> [C:Request HTTP] -> HTTP Aggregate -> (OK, [E:Request Scheduled])
• HTTP Aggregate -> [E:Request processed]
• E:Request processed -> HTTP Process Manager -> [C:Response delivery] -> Crawler Aggregate -> (OK, [E: Page analysed; E:Crawl Process finished])

I understand the aggregates (command -> 'state -> event' list) validate the domain and produce events and I’m using the Process Managers(event -> command' list) as asyn communication mechanism between Aggregates.

The parts I don’t understand is how the Crawler Process Manager & HTTP Process Manager should be implemented using Equinox, or if that’s not the right model for Equinox, then what would be a valid model to achieve this example?

[bartelink]

One thing to watch out for is that you do want to arrive at a situation where each stream has a natural end of life in some fashion - talk. The Epoch mechanism in the process manager PRs is a way to synthetically introduce such a thing, but be careful trying to use that as a hammer; thinking deeply about a good way to structure things is not optional - talk.

Finally, there’s your use of complying in reference to my approx reasons why event sourcing is inappropriate for a context - be careful of confirmation bias in justifying a need (but that doesn’t mean you need to go the other way and take official-sounding classification rules literally either).

None of this is a reason not to just build something in order to explore a range of impl techniques. If the intention is to build some hypothetical domain with a view to demonstrating some technical techniques, go for it (but the whole crawler should really not be a single process manager, no matter what!)

[bartelink]

Really appreciate the setting context

This examples (both equinox/samples & jet/dotnet-samples) are however kind of difficult to understand for me, as a newcomer (to equinox), it looks like support material for a presentation given by someone who already understands them.

Yep; its a tough tradeoff - simple, high performance, testable, easy to scan. Diagrams need to happen and will - they just compete with lots of things…

There unfortunately are no tests I can point to (and FsCheck ones are the worst possible start unless that’s something you happen to already grok.)

Based on this, I think I get the spirit of the project, and seems great, I’ll do some tests and write some examples and will try to PR them here to check if I understand correctly the proposal to Equinox and Propulsion.

Looking forward to it; I’ll endeavour to stay responsive.

Regarding jet/dotnet-templates#40 specifically the Process Manager looks like it’s exactly what i’m looking for. The system I’m building has agents to automate tasks, based on time or reacting events. However I’m not sure if I’m reading right or missing something but this Process Manager is actually a stream (event -> event' list) meaning it reads from one event stream and writes to another event stream.

Approximately; however I’d say it’s 'event -> 'action`; you can map the state of the process manager to a “next step with gathered context” as you see fit. How you carry out those actions is you writing code, i.e. while this might appear to just write an event, it’s running a command, whose outcome will dictate what you feed back into the process manager (which in turn influences the next action)

I’m looking for Process Manager with form of event -> command' list because that command goes to an Aggregate that keeps everything sound (as the solely writer of one event stream), I’m yet to familiarize with Streams (Process), I still don’t wrap my head around on how to prevent multiple writers to one event stream.

I’m having trouble unpacking this. Also you don’t “prevent” multiple writers, you define what happens if they happen to conflict (before that you do want to define aggregates as much as possible to minimize such conflicts)

In a Many-to-one scenario with simple Process Managers, many can talk to one Aggregate and that one is the owner of its event stream, but with many Streams then each writes the same output event stream? Or One Stream has to merge input event streams in order to be the only writer to one event stream, but then where to decouple? Sorry for my ignorance with Streams Processes 😅 , I know that those questions have simple answers that I’m yet to discover, but I’m sticking to basic Aggregates/Process Managers until I feel comfortable enough to take the next step (Streams Processes).

On reflection, while its technically possible to model a web crawler as a process manager, it’s not feeling like there’s a fit to warrant applying event sourcing:

• there’s nothing to collaborate on - I’m not seeing/feeling anything that needs to be consistent, agreed and/or have a log of
• there’s no strict ordering constraint; you’re just adding stuff to a queue (not ordered, it could be a bag, could use CRDTs)
• there’s no end to the life of the process, no intrinsic batching
• if we do some double visits at statistically low levels, it’s not the end of the world etc.

I can’t say I’ve an alternate approach to point to off the top of my head (I prefer to do work than on real things than expand my system design interview skills 😸), but I’m sure there’s plenty well documented approaches to managing such an activity a search away - bloom filters, CRDTs, consistent hashing etc. to distribute the work across multiple nodes tc