How to set max concurrent connections and a queue?

I have a for loop that looks something like this:

var urls = [ 'http://example1.com', 'http://example2.com', 'http://example3.com', 'http://example4.com', ... 'http://example10000.com' ]

for (var i = urls.length - 1; i >= 0; i--) {
    var options = {
        uri: 'http://'+urls[i],
        resolveWithFullResponse: true,
        timeout: 60000,
        jar:true,
        headers: {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
        }
    }
        rp(options).then(function (response) {
                console.log(response);
        });
}

This works fine for the first 500 iterations, but after 500 all iterations fails. I guess too many connections are open. Can you give an example of a queueing the requests with for example 30 concurrent connections? Thanks!

[KpjComp]

This could be solved at the Promise part… eg. Bluebird map has a concurrency option.

eg.

var urls = [ 'http://example1.com', 'http://example2.com', 'http://example3.com', 'http://example4.com', ... 'http://example10000.com' ]

return Promise.map(urls, function (url) {
    var options = {
        uri: 'http://'+url,
        resolveWithFullResponse: true,
        timeout: 60000,
        jar:true,
        headers: {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
        }
    }
    return rp(options).then(function (response) {
        console.log(response);
    });
}, { concurrency: 10});

[analog-nico]

Hi @ahansson89 indeed you have to make sure you don’t overload node.js. At one point there are too many outgoing http connections. AFAIK the most important todo is to use a well configured http.Agent:

var http = require('http') // Do the same with the 'https' module if you do https requests

var httpAgent = new http.Agent()
httpAgent.maxSockets = 15

// Pass the httpAgent to your requests like this:
rp({ uri: 'http://example1.com', pool: httpAgent })

The way how I configured the http.Agent above is the way I use it in my production environment where I do ~100 requests in parallel. Finding a good value for maxSockets helped me to make the requests finish sooner. However, I can’t tell what the perfect configuration is that would be needed in your case. You really have to dig into the details yourself.

As for the parallel execution and queueing of the requests you can approach this as with any other scenario that somehow involves executing tasks in parallel. I did a quick npm search and maybe this library is a good choice for you: https://www.npmjs.com/package/promise-queue

If you queue your requests keep in mind that the rp(...) immediately starts the request. Even if you didn’t call .then(...) yet. So in order to queue requests that are only started once they are dequeued and processed, you have to wrap them into a function. With promise-queue it would look something like this:

queue.add(function () {
    var options = { ... }
    return rp(options)
})
    .then(function (response) {
        console.log(response)
    })

I hope this helps.