Support EventStore Projection

For EventStore storage specifically, it looks like this library is designed so that EventStore is only used for storage of events and that subsequent communicating that events occured is offloaded to Kafka. The EventStore product does support messaging via PersistentSubscription and it would be nice for smaller projects to be able to use just EventStore and not have to spin up a Kafka cluster if that kind of scale is not needed.

Specifically I’d love an API where you can subscribe either to a specific aggregate user-1234 or a projection user-* and have the library handle things like checkpointing (storing the last read offset for a client in EventStore itself) similar to how PersistentSubscriptions work in the EventStore API today.

[bartelink]

Yes - using Kafka is definitely not a panacea and having consistent programming models between Equinox.Cosmos and Equinox.EventStore ia part of the idea; such stuff logically fits in an Equinox.EventStore.Projection library, which would match such facilities as presented in Equinox.Cosmos.Projection (which used the MS ChangeFeedProcessor library and stores state in an auxilliary bookmark collection).

As evidenced by this discussion

a) I personally am no expert in such ES facilities b) there are internal users within Jet that are interested in such a facility at a high level c) we’re looking at building out just that

This would manifest as: a) dotnet new eqxprojector -e template which would produce a Projector project that tails an ES instance b) dotnet new eqxprojector -e -k template which would do the same as dotnet new eqxprojector -k (feeding to Kafka in the projector with one adding appropriate handling/filtering as desired) c) dotnet new eqxprojector -e -c template mode which would feed from ES to an Equinox.Cosmos store (thats what the thread is about and what I have shortlisted to build)

Now, to your specific use case:- a) but in general the idea would be to do a high level subscription tailing $all and the batteries included impl would be feeding to Kafka using the same components as used for dotnet new eqxprojector as it stands b) supporting a more constrained subscription would be an extra feature on top of a

So, in short, if you wanted to do a PR to add such a thing a) it could live in Equinox.EventStore.Projection with a sample adjacent to the Cosmos one in https://github.com/jet/dotnet-templates/tree/eqx-prj/equinox-projector-cosmos b) it might move about a little and be merged with the above ES -> Cosmos sync facility sitting alongside

In terms of this organically happening without input from your side, I’d venture: a) in general folks here tend to use Kafka as the consumer abstraction as that’s all rigged and well maintained (i.e. nobody has had a similar ask internally atm) b) whether it makes sense to use a wrapper out of a library if you have ES in place and expertise around working with that API might be debatable

[bartelink]

related: Propulsion.EventStoreDb in V4 implements the reader using gRPC. Leaving this issue open as there’s no in-the box checkpointing that stores the checkpoints in ESDB (supported checkpoint stores are DynamoDB, CosmosDB, Postgres and SQL Server)