question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

QueryStream ending early, after only 1/14 of the query result set

See original GitHub issue

I have a collection with 700K+ records. It contains a legacy geo-location field and I wrote a script, using a QueryStream, to update the field into a GeoJSON compatible format.

The problem is that when using a QueryStream, the stream is ended after only 50635 documents.

The script code is as follows:

var stream = db.propertyModel.find()
.where("address.location").exists(true)
.where("address.location.type").exists(false)
.select("address.location")
.stream();

var count = 0;
stream.on("data", function(property) {
  stream.pause();

  var location = {
    type: "Point",
    coordinates: property.address.location
  };

  property.update({ $set: { "address.location": location }}).exec(function(err, numberAffected, rawResponse) {
    if (err) {
      console.log("\n" + err.message);
    }

    count += 1;
    util.print("\rUpdated property # " + count);

    stream.resume();
  });
}).on("error", function(err) {
  console.log(err);
}).on("close", function() {
  db.mongoose.disconnect();
  console.log("\nStream closed\n");
});

From the Mongo shell, I have confirmed that the same query has a count of 721,938 documents:

db.properties.count({ "address.location": { $exists: true }, "address.location.type": { $exists: false }})

Response: 721938

Why is the QueryStream only streaming 50635 documents? Is it an internal cursor limitation?

Issue Analytics

  • State:closed
  • Created 10 years ago
  • Comments:5

github_iconTop GitHub Comments

1reaction
rickhuizingacommented, Jun 16, 2014

I’ve found the solution for long running cursors: the timeout option needs to be set to false.

Calling setOptions({ timeout: false }) on the query prior to calling stream() will prevent long-running streams from terminating prematurely.

0reactions
vkarpov15commented, Jun 27, 2014

@rickhuizinga thanks!

Read more comments on GitHub >

github_iconTop Results From Across the Web

queryStream hangs if sql has a logical error #1391 - GitHub
The sql hangs/crashes when using queryStream if there's a logical error in the select/result set, but only if the error occurs after the ......
Read more >
postgres query takes longer when streaming in node
This is how I query my data in Node : const {Pool} = require('pg'); const QueryStream = require('pg-query-stream'); const CSVStream ...
Read more >
NodeJS Streams and E2E streaming from Postgres using pg ...
In this demo, we'll use NodeJS and pg- query-stream to implement end -to- end streaming from Postgres to the HTTP client.
Read more >
How does MySQL result set streaming perform vs fetching the ...
First of all, the MySQL result set streaming has the following caveats: ... That being said, you only need to select and process...
Read more >
Query Stream with MariaDB Connector/Node.js (Promise API)
Query Stream with MariaDB Connector/Node.js (Promise API) ... The query() function returns all data in the result set into memory in a single...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found