QueryStream ending early, after only 1/14 of the query result set
See original GitHub issueI have a collection with 700K+ records. It contains a legacy geo-location field and I wrote a script, using a QueryStream, to update the field into a GeoJSON compatible format.
The problem is that when using a QueryStream, the stream is ended after only 50635 documents.
The script code is as follows:
var stream = db.propertyModel.find()
.where("address.location").exists(true)
.where("address.location.type").exists(false)
.select("address.location")
.stream();
var count = 0;
stream.on("data", function(property) {
stream.pause();
var location = {
type: "Point",
coordinates: property.address.location
};
property.update({ $set: { "address.location": location }}).exec(function(err, numberAffected, rawResponse) {
if (err) {
console.log("\n" + err.message);
}
count += 1;
util.print("\rUpdated property # " + count);
stream.resume();
});
}).on("error", function(err) {
console.log(err);
}).on("close", function() {
db.mongoose.disconnect();
console.log("\nStream closed\n");
});
From the Mongo shell, I have confirmed that the same query has a count of 721,938 documents:
db.properties.count({ "address.location": { $exists: true }, "address.location.type": { $exists: false }})
Response: 721938
Why is the QueryStream only streaming 50635 documents? Is it an internal cursor limitation?
Issue Analytics
- State:
- Created 10 years ago
- Comments:5
Top Results From Across the Web
queryStream hangs if sql has a logical error #1391 - GitHub
The sql hangs/crashes when using queryStream if there's a logical error in the select/result set, but only if the error occurs after the ......
Read more >postgres query takes longer when streaming in node
This is how I query my data in Node : const {Pool} = require('pg'); const QueryStream = require('pg-query-stream'); const CSVStream ...
Read more >NodeJS Streams and E2E streaming from Postgres using pg ...
In this demo, we'll use NodeJS and pg- query-stream to implement end -to- end streaming from Postgres to the HTTP client.
Read more >How does MySQL result set streaming perform vs fetching the ...
First of all, the MySQL result set streaming has the following caveats: ... That being said, you only need to select and process...
Read more >Query Stream with MariaDB Connector/Node.js (Promise API)
Query Stream with MariaDB Connector/Node.js (Promise API) ... The query() function returns all data in the result set into memory in a single...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
I’ve found the solution for long running cursors: the timeout option needs to be set to false.
Calling
setOptions({ timeout: false })
on the query prior to callingstream()
will prevent long-running streams from terminating prematurely.@rickhuizinga thanks!