In Place Copy-Replace

Hi,

We’ve been using Eveneum as part of our event-sourced application for a while now and so far it has served us well 👍 Thank you @jkonecki !

Over-time we’ve started applying upcasters to convert old event-schemas to new ones (in-memory, at runtime), but we’ve reached the point where we’d like to apply a migration to our store to remove some of these old version and reduce our upcasting code.

To perform this migration we were drawn to an “in place copy-replace” strategy as described in Versioning in an Event Sourced System by Gregory Young. The way we’d like to tackle this is as follows:

• The events are upcasted to their latest version using our existing upcasting code
• The upcasted version of these events are added to the same stream (hence “in place”, since it uses the same stream)
• Additionally, we’d add 2 ‘audit events’, one before and one after the upcasted events respectively (e.g. CopyReplaceStarted and CopyReplaceFinished).
  • The purpose of these is to provide insight in what occurred within the stream for anyone who might check the stream in the future.

Our event consumers are running asynchronously and are heavily reliant on the ‘Cosmos DB Change Feed’. This means that:

• We prefer not to update any “old” event document (as this would trigger the changes to appear in the change feed). So we’d rather not flag them as being ‘soft deleted’.
• In the future we might remove the “old” event documents completely, but to avoid problems with asynchronous processing we don’t want to delete them right away.

To be able to accomplish our flavor of in place copy-replace, we’ve found that we’d need the following features:

• Adding MetaData to the StreamHeader containing the new starting point of the stream
  • This is already possible with Eveneum 👍
• The ability to read the metadata of a stream before reading the actual stream. This would allow us to determine where the stream starts, to then only start reading from that point (skipping the clutter from the past).
  • Reading the header of a specific stream is currently problematic. There seems to only be a way to read multiple headers using LoadStreamHeaders. This problem would probably be fixed if ReadHeader on EventStore was made public (instead of private)
• Finally, we’d need 2 ways of reading a stream from a specific version. One that ignores all Snapshots, so that we could perform a 2nd copy-and-replace starting from a position in the stream. And a second that takes Snapshots into account and so reads as little events as possible (stopping at the first snapshot), to keep our aggregate loading efficient.
  • This is currently not possible because ReadStreamFromVersion ignores the snapshots, and ReadStream doesn’t have an option to read from a certain point. To accomplish this, we’d need a combination of these 2 methods. As an example, this could be an optional RequestOptions on ReadStream (similar to the Cosmos API), where the RequestOptions exposes the following options:
    • bool IgnoreSnapshots (false for rehydration of aggregates, true when reading the entire stream to perform a migration)
    • ulong? Version (to avoid loading events from before the copy-replace migration)

We’re willing to contribute to Eveneum in the form of a PR to add these missing features. But before we start working on this, we wanted to first ask:

• What is your opinion about our approach?
• Are you open to PR’s that would add the features we’re missing?
  • Is there something we should keep in mind?

Thank you in advance!

[jkonecki]

Could you please rename AsOfVersion property on the options class to ToVersion? I think it may be a more clearer alternative, especially when compared to FromVersion.

[jkonecki]

Thank you very much for your kind words, Thomas!

I understand your challenges regarding change feed subscribers.

Your suggestions make sense and I’m happy to accommodate them.

Please give me a moment to think about naming convention and approach regarding ReadStream method - I’ll come back to you by tomorrow.

If you’ve already made some code changes than please start a PR - if not than please wait for feedback from me.

Please rest assured that I’m happy with the approach you’re proposing and will help with getting the new version out as soon as I can.

On Mon, 29 Nov 2021, 18:14 ThomasVandenbon, @.***> wrote:

Hi,

We’ve been using Eveneum as part of our event-sourced application for a while now and so far it has served us well 👍 Thank you @jkonecki https://github.com/jkonecki !

Over-time we’ve started applying upcasters to convert old event-schemas to new ones (in-memory, at runtime), but we’ve reached the point where we’d like to apply a migration to our store to remove some of these old version and reduce our upcasting code.

To perform this migration we were drawn to an “in place copy-replace” strategy as described in Versioning in an Event Sourced System https://leanpub.com/esversioning/read by Gregory Young. The way we’d like to tackle this is as follows:

• The events are upcasted to their latest version using our existing upcasting code
• The upcasted version of these events are added to the same stream (hence “in place”, since it uses the same stream)
• Additionally, we’d add 2 ‘audit events’, one before and one after the upcasted events respectively (e.g. CopyReplaceStarted and CopyReplaceFinished).
  • The purpose of these is to provide insight in what occurred within the stream for anyone who might check the stream in the future.

Our event consumers are running asynchronously and are heavily reliant on the ‘Cosmos DB Change Feed’. This means that:

• We prefer not to update any “old” event document (as this would trigger the changes to appear in the change feed). So we’d rather not flag them as being ‘soft deleted’.
• In the future we might remove the “old” event documents completely, but to avoid problems with asynchronous processing we don’t want to delete them right away.

To be able to accomplish our flavor of in place copy-replace, we’ve found that we’d need the following features:

• Adding MetaData to the StreamHeader containing the new starting point of the stream
  • This is already possible with Eveneum 👍
• The ability to read the metadata of a stream before reading the actual stream. This would allow us to determine where the stream starts, to then only start reading from that point (skipping the clutter from the past).
  • Reading the header of a specific stream is currently problematic. There seems to only be a way to read multiple headers using LoadStreamHeaders. This problem would probably be fixed if ReadHeader on EventStore was made public (instead of private)
• Finally, we’d need 2 ways of reading a stream from a specific version. One that ignores all Snapshots, so that we could perform a 2nd copy-and-replace starting from a position in the stream. And a second that takes Snapshots into account and so reads as little events as possible (stopping at the first snapshot), to keep our aggregate loading efficient.
  • This is currently not possible because ReadStreamFromVersion ignores the snapshots, and ReadStream doesn’t have an option to read from a certain point. To accomplish this, we’d need a combination of these 2 methods. As an example, this could be an optional RequestOptions on ReadStream (similar to the Cosmos API), where the RequestOptions exposes the following options:
    • bool IgnoreSnapshots (false for rehydration of aggregates, true when reading the entire stream to perform a migration)
    • ulong? Version (to avoid loading events from before the copy-replace migration)

We’re willing to contribute to Eveneum in the form of a PR to add these missing features. But before we start working on this, we wanted to first ask:

• What is your opinion about our approach?
• Are you open to PR’s that would add the features we’re missing?
  • Is there something we should keep in mind?

Thank you in advance!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Eveneum/Eveneum/issues/28, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANRNDZE27OR4E6OKT5SES3UOO7J3ANCNFSM5I7Y7CTQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.