Redis indexation

In #94, I was talking about some ways to improve redis implementation performance, mostly when there are a large number of events (or snapshots). The problem with actual the implementation is the scan which usually need to scan every events (or snapshots) of an aggregate, and then make the necessary calculations to only take needed events.

I have made a quick test with my production data. I have an aggregate with +46K events with a total amount of +117K I tried to scan all the keys (only scanning, no get or mget) using 2 strategies:

• I made a common scan on all the keys. It took 6181ms.
• I put all the aggregate keys in a sorted set (members score was the event version), and made a zrange on all the keys. It took 415ms

I think that sorted sets could be a good option to index aggregates keys, at least for 2 reasons:

• It seems more efficient to scan all the keys at once, for an aggregate
• It will be more efficient to get keys between 2 revisions, for an aggregate (like you need to do when you reload state from a snapshot and an event)

This means that the database need to be indexed at first, and the index maintained. So whenever a client connects, we need to check if the store has been indexed, and if it’s not the case, scan all the aggregates, and scan for their events and snapshots keys, and finally index the keys (events and snapshots separated obviously). Once done, each time an event is added, its key should be added to the corresponding sorted set.

What do you think about this solution? I’ll try to work on an implementation. But if you have any remarks, objections or suggestions, don’t hesitate!

[rehia]

@TyGuy hi!

Sorry, but I couldn’t take some time to start something on this. You can do it, if you have time for it.

The solution I suggested could be a good one, but as usual, it must first be validated on the field.

I hop you’ll find more time than me 😅 Good luck !

[TyGuy]

@rehia do you still have interest in this, or plans to execute on it?

We are sitting with around 30k events in redis, and noticing it slowing down as well. I think this solution (indexing on aggregateId and using sorted sets) would be awesome!

If you don’t have plans to continue it or work on it, would you be willing to share a WIP branch with your implementation? If/when I find some time, I could pick up where you left off, and carry this forward.