The server crashes with EMFILE NetworkingError

Hello,

I’m running an express.js app using the aws-serverless-express adapter. My app is making some requests to AWS Cognito API using the aws-sdk module. It’s been in production for many months and from time to time it’s crashing with the following stack trace:

{ Error: listen EMFILE /tmp/server-ck6wudgsnfo.sock
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at Server.setupListenHandle [as _listen2] (net.js:1350:19)
at listenInCluster (net.js:1408:12)
at Server.listen (net.js:1503:5)
at startServer (/var/task/node_modules/aws-serverless-express/src/index.js:152:17)
at Object.proxy (/var/task/node_modules/aws-serverless-express/src/index.js:205:14)
at exports.handler (/var/task/src/serverless/index.js:31:33)
code: 'EMFILE',
errno: 'EMFILE',
syscall: 'listen',
address: '/tmp/server-ck6wudgsnfo.sock',
port: -1 
}

It’s caused by the following error in the app. EMFILE is the error code if there’s no more file descriptors available.

{ 
  message: 'getaddrinfo EMFILE cognito-idp.ap-southeast-2.amazonaws.com:443',
  code: 'NetworkingError',
  errno: 'EMFILE',
  syscall: 'getaddrinfo',
  hostname: 'cognito-idp.ap-southeast-2.amazonaws.com',
  host: 'cognito-idp.ap-southeast-2.amazonaws.com',
  port: 443,
  region: 'ap-southeast-2',
  retryable: true,
  time: '2019-05-28T06:04:43.973Z' 
}

Error: getaddrinfo EMFILE cognito-idp.ap-southeast-2.amazonaws.com:443
    at Object._errnoException (util.js:1022:11)
    at errnoException (dns.js:55:15)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:92:26)

[janaz]

@codingyourlife

The reason I had the issue was initializing the internal aws-serverless-express server inside the handler function like this:

const express = require('express');
const awsServerlessExpress = require('aws-serverless-express')

const app = express();
app.get('/', (_, res) => res.json({hello: 'world'}));

const lambdaHandler = (event, context) => {
  // the server is created inside the handler function
  const server = awsServerlessExpress.createServer(app)
  awsServerlessExpress.proxy(server, event, context)
}

The solution to my issue was to move the server initialization to the global scope so that it’s shared across lambda invocations:

// move the server to the global scope
const server = awsServerlessExpress.createServer(app)

const lambdaHandler = (event, context) => {
  awsServerlessExpress.proxy(server, event, context)
}

[janaz]

I checked the module that creates the lambda handler and it has been changed recently. The server creation has been moved from the global scope to the lambda handler. Unfortunately, the change hasn’t been noticed in the code review and we didn’t have any automated tests to detect it. I guess it’s time to write a test for it.

So basically every lambda invocation was creating a new listener and after some time it reached the limit of opened file descriptors (for lambda functions the limit is 1024)