Memory leaking with Promise.map concurrency

1. What version of bluebird is the issue happening on? Bluebird 3.5.20 with bluebird-global 3.5.5

2. What platform and version? (For example Node.js 0.12 or Google Chrome 32) Node.js 8.9.

3. Did this issue happen with earlier version of bluebird? As far as I know

I have code that looks like this:

const aBunchOfIds = [ /* a few hundred thousand elements */ ]
Promise.map(aBunchOfIds, async id => {
    const anObj = await getFromDatastore(id);
    const anotherObj = await process(anObj);
    if (isSpecial(anotherObj)) { console.log('so special'); }
}, { concurrency: 1000 });

This dies because node runs out of heap space. However the following modification resolves the issue:

const aBunchOfIds = [ /* a few hundred thousand elements */ ]
for (let i = 0; i < aBunchOfIds.length; i += 10000) {
    const someIds = aBunchOfIds.slice(i, i + 10000);
    Promise.map(someIds, async id => {
        const anObj = await getFromDatastore(id);
        const anotherObj = await process(anObj);
        if (isSpecial(anotherObj)) { console.log('so special'); }
    }, { concurrency: 1000 });
}

So it seems that Promise.map doesn’t free the memory used by its promises until all promises have returned, even when a concurrency is specified.

[0xgeert]

I’m seeing memory-leak issues that are very similar, if not the same.

Came up with the following test-case which shows the issue clearly:

• running below code as-is will result in OOM in a couple of seconds
• running code with then() added (see below) will allow GC to free up mem.

Lodash 3.5.2

/*
 Below gives an OOM in a couple of seconds. 
 Adding the commented-out then() solves it. Why?
 */
const Promise = require("bluebird");
const obj = {a: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"};
return Promise.map(new Array(100000), () => {
  return Promise.resolve()
  .then(() => new Array(10000).map(() => Object.assign({},obj))) //consume some mem
  // .then(() => { //passing in an argument here, makes the OOM-issue reappear
  //   //adding this then(), will allow objects to be GC'ed, thus solving OOM-issue. WHY?
  // });
}, { concurrency: 10 })

It seems that since the result isn’t passed in as argument to the last then() the object can be freed from mem. This seems to suggest that not only the promises themselves but all objects created inside those promises are kept locked in mem until the entire Promise.map is finished.

[krunkosaurus]

Just a heads up that I got “Maximum call stack size exceeded” errors until I finally downgraded this package to 3.5.2

I think it could be related to this issue.