Bluebird Causes HTTP Module to Drop Requests

I have a really, really simple app, but when it experiences a high load of POST requests, it resets the connection for a few of them. I have tried replacing BPromise (my variable holding Bluebird’s promise class) with Promise (i.e. native JavaScript promises) and the problem goes away. This leads me to suspect that Bluebird is involved in causing the error.

Here is the app code:

var http = require('http');
var BPromise = require("bluebird");

function my_route(req, res) {
    var array = [];
    for (var j = 0; j < 1000; ++j)
        array.push(Math.random());
    array.sort();

    BPromise.resolve(true)
    .then(function(){
        res.writeHead(200);
        res.end("true", 'utf-8');
    });
}

http.createServer(function (req, res) {

    my_route(req, res);

}).listen(1390);

console.log('Server running at http://127.0.0.1:1390/');

I have this code running in cluster mode (via PM2) behind a very run-of-the-mill NGINX reverse proxy. To test the configuration, I am hammering it with enough POSTs to fill CPU utilization from a separate server running apache bench:

ab -n 600000 -p body.json -c 32 [my public ip address]

Out of the 600000 requests sent, maybe 1-4 come back with non-2xx responses every couple of trials. The Nginx logs have the following error:

recv() failed (104: Connection reset by peer) while reading response header from upstream

The error that Nginx is providing tells me that the problem lies not with Nginx, but rather with Node or Bluebird. As I stated above, changing BPromise to Promise eliminates the problem.

Can anyone help me figure out why the HTTP module is hanging up the phone when Bluebird is used? Thank you so much for your help!

Here is my precise configuration:

• App Server: Linux 4.4.0-112-generic x86_64; Ubuntu 16.04.3; 8 Cores w/ 15 GB of memory
  • Nginx: 1.13.8
  • Node: 8.9.4
  • Bluebird: 3.5.1 (The problem also occurs in 2.11.0, but does not seem to occur in 1.2.4. I am happy to try other versions if that is helpful.)
• Test Server: Linux 4.4.0-112-generic x86_64; Ubuntu 16.04.3; 4 Cores w/ 4 GB of memory
  • Apache Bench: 2.3

Nginx Files /etc/nginx/nginx.conf:

user                nginx;
worker_processes    auto;

error_log           /var/log/nginx/error.log warn;
pid                 /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include                 /etc/nginx/mime.types;
    default_type            application/octet-stream;

    log_format              main    '$remote_addr - $remote_user [$time_local] "$request" '
                                    '$status $body_bytes_sent "$http_referer" '
                                    '"$http_user_agent" "$http_x_forwarded_for"';

    access_log              /var/log/nginx/access.log  main;

    keepalive_timeout       0;

    underscores_in_headers  on;

    include                 /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/[my public ip address].conf:

upstream api_upstream
{
    server      localhost:1390;
    keepalive   64;
}

server
{
    listen          80 default_server;
    server_name     [my public ip address];

    location /
    {
        proxy_set_header    Host $host;
        proxy_http_version  1.1;
        proxy_set_header    Connection "";
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto $scheme;
        proxy_pass          http://api_upstream/;
    }
}

Thank you again!

[benjamingr]

Hey, very interesting!

For isolating the issue - can you try running:

• with NODE_ENV=production?
• With Promise.setScheduler(fn => process.nextTick(fn))?

Thanks for the report!

[DanielRLevine]

I was able to answer my own question: using the .setScheduler() call you presented will move Bluebird promises from the immediate queue to the next tick queue, which will make them a higher priority than the IO events queue. For anyone else reading this in the future, there was a very helpful series of articles here: https://jsblog.insiderattack.net/promises-next-ticks-and-immediates-nodejs-event-loop-part-3-9226cbe7a6aa

Thank you, @benjamingr again for your help. You aid went a long way in solving a question that had been puzzling me for a few weeks.