[riot-redis] Live replication not working on production instances. Huge memory usage. Data not replicated.

While testing in our ci and staging environments live replication was working as expected but once we moved the replication to production we observed several issues.

Scenario description

• Source redis version 2.8 with 2.6GB and notify-keyspace-events set to KEA
• Destination redis version 3.2.6 starting with no data, with SSL and Encryption at rest turned on via AWS Elasticache.
• riot-redis v2.2.7

Issue description

Command:

./riot-redis --info -h production-src-redis -p 6379 --db=0 replicate -h production-dst-encrypted-redis -p 6379 -t --db=0 --pass=apasswordhere --no-verify-peer --live

While running that command the output is:

Opening writer
Opening reader
Replicating from redis://production-src-redis█

• The process running riot-redis uses up to 9GB of RAM at times, while the live replication does not work (I write keys into src while the process is running and they are never present in src)
• The src redis memory used is 2.6GB while the dst redis memory used is never over 200MB.
• At times, but not always a log message like the one below is displayed:

Could not get value: Invalid first byte: 53 (5) at buffer index 1 decoding using RESP2

[jruaux]

yes keyspace notifications are queued. I just changed some of the notification queuing mechanism: updates to the same key are now deduplicated so memory usage should be lower if you had repeatedly updated keys. I also added a separate status bar for the live part of the replication process. Can you try the latest release and let me know if things improved?

[jruaux]

Thanks for confirming that this was the issue.