ioredis does not detect dead connections

I am using redis as a cache in a production environment. My redis-server is running on a different VM than my client running ioredis. My problem is that ioredis does not properly handle connections that are “lost” (by which I mean connections that are dead, but have not been properly closed). When this happens, the call-backs of all my .get, .set, etc… calls are only reached after about 11 minutes.

For example:

var Redis = require('ioredis');
var redis = new Redis();

redis.set('foo', 'bar');
// connection dies here.
redis.get('foo', function (err, result) {
  // Will only get here after ~11 minutes.
});

As you can imagine, this is unacceptable behavior for my cache as all the code accessing the cache will get stuck and all requests to my server will timeout.

The ‘close’, ‘end’ and ‘error’ events of ioredis are not called, because the tcp connection is not properly closed (no “FIN” tcp packets are sent). This is a common occurrence in a production environment and occurs in the following situations, among others:

• the VM running redis-server is shut down
• the NAT running between the client and the redis-server (e.g. a firewall or docker’s native NATing) eliminates the connection because it has been inactive for some time.

This does not happen if you run the client and redis-server on the same machine and just kill (either SIGINT or SIGKILL) the redis-server, as the OS will perform some clean-up and properly close the tcp connections between the redis-server about to be turned off and all connected clients (the ioredis object receives a ‘close’ event).

Coming back to the first situation where the connection is dead because of e.g. a failure of the VM running redis-server. All ioredis get() requests hang. After about 11 minutes, the OS detects that the connection looks dead, and kills it (seems to be related to the TCP keepalive parameters). At that point, the ‘close’ event will indeed be fired, and the callbacks of the .get requests will be reached.

What I would like to be able to do is to set a parameter that tells ioredis to give up on the redis.get after a certain number of milliseconds and return an error if not successful. E.g.

var redis = new Redis({requestTimeout: 100});
redis.get('foo', function (err, result) {
  // Will get here after at most 100ms, even if the connection died.
});

This would allow me to quickly rely on a fall-back solution (avoid accessing the cache). However, as far as I can tell, this is currently not possible with ioredis.

[luin]

Haven’t come across this issue before. I’m going to look into it the weekend. Currently you can set the timeout of a request by:

redis.get('foo').timeout(100).then(function() {}).catch(function() {});

Possibly related issue #61.

[luin]

Closing this issue since the keepAlive option is added in 1.8.0. Refer to #61 for any further updates about per-operation timeout.