question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

In Place Copy-Replace

See original GitHub issue

Hi,

We’ve been using Eveneum as part of our event-sourced application for a while now and so far it has served us well 👍 Thank you @jkonecki !

Over-time we’ve started applying upcasters to convert old event-schemas to new ones (in-memory, at runtime), but we’ve reached the point where we’d like to apply a migration to our store to remove some of these old version and reduce our upcasting code.

To perform this migration we were drawn to an “in place copy-replace” strategy as described in Versioning in an Event Sourced System by Gregory Young. The way we’d like to tackle this is as follows:

  • The events are upcasted to their latest version using our existing upcasting code
  • The upcasted version of these events are added to the same stream (hence “in place”, since it uses the same stream)
  • Additionally, we’d add 2 ‘audit events’, one before and one after the upcasted events respectively (e.g. CopyReplaceStarted and CopyReplaceFinished).
    • The purpose of these is to provide insight in what occurred within the stream for anyone who might check the stream in the future.

Our event consumers are running asynchronously and are heavily reliant on the ‘Cosmos DB Change Feed’. This means that:

  • We prefer not to update any “old” event document (as this would trigger the changes to appear in the change feed). So we’d rather not flag them as being ‘soft deleted’.
  • In the future we might remove the “old” event documents completely, but to avoid problems with asynchronous processing we don’t want to delete them right away.

To be able to accomplish our flavor of in place copy-replace, we’ve found that we’d need the following features:

  • Adding MetaData to the StreamHeader containing the new starting point of the stream
    • This is already possible with Eveneum 👍
  • The ability to read the metadata of a stream before reading the actual stream. This would allow us to determine where the stream starts, to then only start reading from that point (skipping the clutter from the past).
    • Reading the header of a specific stream is currently problematic. There seems to only be a way to read multiple headers using LoadStreamHeaders. This problem would probably be fixed if ReadHeader on EventStore was made public (instead of private)
  • Finally, we’d need 2 ways of reading a stream from a specific version. One that ignores all Snapshots, so that we could perform a 2nd copy-and-replace starting from a position in the stream. And a second that takes Snapshots into account and so reads as little events as possible (stopping at the first snapshot), to keep our aggregate loading efficient.
    • This is currently not possible because ReadStreamFromVersion ignores the snapshots, and ReadStream doesn’t have an option to read from a certain point. To accomplish this, we’d need a combination of these 2 methods. As an example, this could be an optional RequestOptions on ReadStream (similar to the Cosmos API), where the RequestOptions exposes the following options:
      • bool IgnoreSnapshots (false for rehydration of aggregates, true when reading the entire stream to perform a migration)
      • ulong? Version (to avoid loading events from before the copy-replace migration)

We’re willing to contribute to Eveneum in the form of a PR to add these missing features. But before we start working on this, we wanted to first ask:

  • What is your opinion about our approach?
  • Are you open to PR’s that would add the features we’re missing?
    • Is there something we should keep in mind?

Thank you in advance!

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:24 (24 by maintainers)

github_iconTop GitHub Comments

1reaction
jkoneckicommented, Nov 30, 2021

Could you please rename AsOfVersion property on the options class to ToVersion? I think it may be a more clearer alternative, especially when compared to FromVersion.

1reaction
jkoneckicommented, Nov 29, 2021

Thank you very much for your kind words, Thomas!

I understand your challenges regarding change feed subscribers.

Your suggestions make sense and I’m happy to accommodate them.

Please give me a moment to think about naming convention and approach regarding ReadStream method - I’ll come back to you by tomorrow.

If you’ve already made some code changes than please start a PR - if not than please wait for feedback from me.

Please rest assured that I’m happy with the approach you’re proposing and will help with getting the new version out as soon as I can.

On Mon, 29 Nov 2021, 18:14 ThomasVandenbon, @.***> wrote:

Hi,

We’ve been using Eveneum as part of our event-sourced application for a while now and so far it has served us well 👍 Thank you @jkonecki https://github.com/jkonecki !

Over-time we’ve started applying upcasters to convert old event-schemas to new ones (in-memory, at runtime), but we’ve reached the point where we’d like to apply a migration to our store to remove some of these old version and reduce our upcasting code.

To perform this migration we were drawn to an “in place copy-replace” strategy as described in Versioning in an Event Sourced System https://leanpub.com/esversioning/read by Gregory Young. The way we’d like to tackle this is as follows:

  • The events are upcasted to their latest version using our existing upcasting code
  • The upcasted version of these events are added to the same stream (hence “in place”, since it uses the same stream)
  • Additionally, we’d add 2 ‘audit events’, one before and one after the upcasted events respectively (e.g. CopyReplaceStarted and CopyReplaceFinished).
    • The purpose of these is to provide insight in what occurred within the stream for anyone who might check the stream in the future.

Our event consumers are running asynchronously and are heavily reliant on the ‘Cosmos DB Change Feed’. This means that:

  • We prefer not to update any “old” event document (as this would trigger the changes to appear in the change feed). So we’d rather not flag them as being ‘soft deleted’.
  • In the future we might remove the “old” event documents completely, but to avoid problems with asynchronous processing we don’t want to delete them right away.

To be able to accomplish our flavor of in place copy-replace, we’ve found that we’d need the following features:

  • Adding MetaData to the StreamHeader containing the new starting point of the stream
    • This is already possible with Eveneum 👍
  • The ability to read the metadata of a stream before reading the actual stream. This would allow us to determine where the stream starts, to then only start reading from that point (skipping the clutter from the past).
    • Reading the header of a specific stream is currently problematic. There seems to only be a way to read multiple headers using LoadStreamHeaders. This problem would probably be fixed if ReadHeader on EventStore was made public (instead of private)
  • Finally, we’d need 2 ways of reading a stream from a specific version. One that ignores all Snapshots, so that we could perform a 2nd copy-and-replace starting from a position in the stream. And a second that takes Snapshots into account and so reads as little events as possible (stopping at the first snapshot), to keep our aggregate loading efficient.
    • This is currently not possible because ReadStreamFromVersion ignores the snapshots, and ReadStream doesn’t have an option to read from a certain point. To accomplish this, we’d need a combination of these 2 methods. As an example, this could be an optional RequestOptions on ReadStream (similar to the Cosmos API), where the RequestOptions exposes the following options:
      • bool IgnoreSnapshots (false for rehydration of aggregates, true when reading the entire stream to perform a migration)
      • ulong? Version (to avoid loading events from before the copy-replace migration)

We’re willing to contribute to Eveneum in the form of a PR to add these missing features. But before we start working on this, we wanted to first ask:

  • What is your opinion about our approach?
  • Are you open to PR’s that would add the features we’re missing?
    • Is there something we should keep in mind?

Thank you in advance!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Eveneum/Eveneum/issues/28, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANRNDZE27OR4E6OKT5SES3UOO7J3ANCNFSM5I7Y7CTQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Why does "Copy" replace stuff instead of merging it?
In general I think of this as a feature rather than a flaw. It seems very strange to me to expect that two...
Read more >
Windows 7/8/10 - Copy & Replace only when there is a ...
Is there a way - either with a 3rd party software, or with a command-line or batch file, to have just a little...
Read more >
Gradle copy replace text and place into the same folder
This works if I copy the file into a different directory but I want to place the file and place it in the...
Read more >
File copy/replace
Is there any way that we could add functionality to that so that we could keep or replace in the destination file with...
Read more >
Is there a way to copy replace objects in S3? : r/aws
I'm trying to understand S3 for a while now, when we are using GUI S3 you can copy from a dir1 and overwrite...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found