Documents in journal and realtime collection have different _id

My understanding is that _id gets assigned to the document when an insert gets executed by the mongodb client. The fact that the write to realtime collection is happening asynchronously makes me think that it is possible that some ids may end up being out of order between the batches. Besides, _id is used as an offset on the query side and when eventsByTag is used the saved offset (retrieved from the realtime collection) won’t in fact exist in the journal which at least makes it weird as the ordering of ids may still make sense. I think setting _id field explicitly when serializing events makes sense as id generation is happening on the client side anyway so we get the same uniqueness and sequencing guarantees but it ensures the same documents in the two collections have the same ids.

Also wanted to point out how the write to the realtime collection is happening asynchronously with no error handling:

batchFuture.andThen { case _ => doBatchAppend(writes, realtime) }

I think at least some error logging would be useful, but it could also be a good idea if the user of the library could choose if an error to write to realtime collection should be propagated. I think failing all realtime subscriptions can be useful. Then the stream can restart and pick up missed events from the journal and continue listening to realtime events

[scullxbones]

Fixed by #222 … will update with a release version

[scullxbones]

Ah ok yep I get it now, thanks for summarizing. Now I totally understand why the _id is a problem, I was struggling a bit with that. This just further convinces me that #214 / #95 are needed to fix in the ideal way. The _id fix would be good for consistency, and would help in the meantime for sure.

Bullet 2 will probably be a stretch, so i’d think 1 & 3 should take priority. I’m good with this ticket covering both. I’ll pull bullet 2 into a separate ticket.