JavaScript heap out of memory on version 5.10.0 and above

Do you want to request a feature or report a bug?

bug

What is the current behavior?

On operations that use a cursor, after some time the following error occurs: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

If the current behavior is a bug, please provide the steps to reproduce.

function queryCursor() {
  return Answer.find({})
    .sort({ _id: -1 })
    .populate({
      path: 'business',
      select: 'name pools tags',
      model: Business,
      populate: [{ path: 'tags', select: 'name', model: Tag }],
    })
    .populate({ path: 'views', select: 'role', model: User })
    .populate({
      path: 'createdBy',
      select: 'name username role tags pools',
      model: User,
      populate: [
        { path: 'tags', select: 'value type', model: Tag },
        { path: 'pools', select: 'name', model: Pool },
      ],
    })
    .populate({ path: 'topic', select: 'subject', model: Topic })
    .populate({ path: 'question', select: 'views type', model: Question })
    .cursor();
}

As seen on the snippet above many document properties have to be populated (this is used to generate a specific report). After some time the error noted above is generated and the process quits.

What is the expected behavior?

The process should complete. The thing is that with any version below 5.10.0, i.e. 5.9.29 (which we have reverted to) the same code works fine. I would guess it has to do with the “upgrade to MongoDB driver 3.6 for full MongoDB 4.4 support”, but have no proof on that.

What are the versions of Node.js, Mongoose and MongoDB you are using? Note that “latest” is not a version.

NodeJS: v14.15.4 Mongoose: 5.10.x (any version starting at 5.10.x) MongoDB: v4.2.9

[vkarpov15]

I did some digging and found that this is an issue with the fix for #8092. https://github.com/Automattic/mongoose/blob/337e3b94ecc56443f17c87b30595c1c22d6c8f6c/lib/model.js#L4466 leads to Mongoose being unable to clean up populated documents in a cursor.

Below is the script I used to repro this:

'use strict';

const mongoose = require('mongoose');

mongoose.set('useFindAndModify', false);

const { Schema } = mongoose;

run().catch(err => console.log(err));

async function run() {
  await mongoose.connect('mongodb://localhost:27017/test', {
    useNewUrlParser: true,
    useUnifiedTopology: true
  });

  const Answer = mongoose.model('Answer', Schema({
    choice: String,
    question: { type: 'ObjectId', ref: 'Question' }
  }));

  const Question = mongoose.model('Question', Schema({
    text: String
  }));

  const numQs = await Question.countDocuments();
  if (numQs < 10000) {
    await Question.deleteMany();
    await Answer.deleteMany();
    for (let i = 0; i < 100000; ++i) {
      const q = await Question.create({ text: 'my question' });
      const answers = [];
      for (let j = 0; j < 10; ++j) {
        answers.push({ choice: 'B', question: q });
      }
      await Answer.create(answers);
      if (i % 100 === 0) console.log(i);
    }
  }

let i = 0;
  await Answer.find().populate('question').cursor().eachAsync((doc) => {
    if (i++ % 100 === 0) {
      console.log('Iterated', i);
    }
  });

  console.log('Done');
}

Sorry for the trouble, the fix will be in v5.12.5 👍

[rsnr]

Thanks so much @vkarpov15 for researching this issue and suggesting these two alternative code changes. I confirm that both alternatives you suggested (options “a” and “b” below) prevent the Out Of Memory state when I use mongoose v5.12.5 on a low-memory Heroku configuration: a) cursor().eachAsync() b) keeping async.whilst but replacing Model.find().exec(function(err, result) {…}) by: Model.find().then(function(result) {…}).catch(err => …) In my specific case the cursor option was about x15 slower than the async.whilst with then(), so I’ll stick with the latter.