Redis logs null error

We’re having issues with our redis setup, and I reached out to our cloud provider to get more info about it:

We often see this message in our logs:

2019-04-26 16:39:36 [APP/PROC/WEB/0] ERR RedisOplog - Connection to redis ended
2019-04-26 16:39:46 [APP/PROC/WEB/0] ERR RedisOplog - There was an error when re-connecting to redis {"delay":10000,"attempt":1,"error":null,"total_retry_time":0,"times_connected":12}

We only have 128 allowed concurrent connections, so I’m hoping there’s only 1 connection opened per host (we have <10 hosts connected to redis in total).

The answer from their tech support is the following:

“It seems that your application (“RedisOplog”) does not write out a proper error (“error”:null). It would be very helpful to get the detailed error message for the error you are getting. Can you check your application if there is more information available or more information that could be printed in case of an error?”

Any idea what could be causing this? Are lots of connections being opened from the same server? What about the null error?

[ramezrafla]

@edemaine We had this error all the time when we were using the original (i.e this repo). Here is what we discovered:

• The problem in our case was Meteor (actually NodeJS). It’s too busy doing other things OR processing too much data from redis so the connection times out

This is a serious issue as it means you missed important messages and reactivity is broken

We solved it by:

1. Reducing the number of messages (and size) that redis-oplog transmits to the bare minimum (the flag to stop protecting against race conditions makes it worse as it sends the whole doc at each update) – this was done in our fork
2. Using a stronger instance type – on AWS we were using t3.large which is burst-based, we migrated to c5.large to get dedicated CPU
3. Optimizing our app so no method / function takes too long, using this.unblock and Meteor.defer. This is an artifact of the single-thread approach of Node. You have to return quickly from your functions to give Node the chance to read from Redis

Hope that helps

[jamesgibson14]

hmm, I just found this https://devcenter.heroku.com/articles/heroku-redis#timeout my DigitalOcean redis instance might have this timeout set.