question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Query when using .stream() method does not understand Strings as ObjectIDs or Dates

See original GitHub issue

When running a find() and then stream() as per below

Model.find({
 "_id":{
     "$in":["5aa8f1226744783cd807a817"]
 },
 "date":{
    "$gte":"2019-08-14T23:38:17.123Z"
  }
})
.lean()
.batchSize(100)
.stream();

Mongoose does not cast the “5aa8f1226744783cd807a817” as an ObjectID, neither does it cast “2019-08-14T23:38:17.123Z” as a Date, it runs the query with those values typed as a String and doesn’t return match the records from the database.

However running the same query with the .exec() method works correctly.

Model.find({
 "_id":{
     "$in":["5aa8f1226744783cd807a817"...]
 },
"date":{
    "$gte":"2019-08-14T23:38:17.123Z"
  }
})
.lean()
.exec(function(err, docs) {

});

This has only been an issue since we upgraded to a recent version of Mongoose

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5

github_iconTop GitHub Comments

1reaction
vkarpov15commented, Aug 16, 2019

Which version of mongoose did you upgrade to? stream() is no longer supported, it’s been replaced by cursor() : https://thecodebarbarian.com/cursors-in-mongoose-45

0reactions
vkarpov15commented, Sep 28, 2019

I’m unable to find any indication that cursor() is orders of magnitude slower than stream(). Consider the below script:

'use strict';

const mongoose = require('mongoose');

run().catch(err => console.log(err));

async function run() {
  await mongoose.connect('mongodb://localhost:27017/test');
  await mongoose.connection.dropDatabase();

  const Model = mongoose.model('Test', mongoose.Schema({ name: String }));
  for (let i = 0; i < 10000; ++i) {
    let name = '' + i;
    for (let i = 0; i < 100; ++i) { name += 'a' }
    await Model.create({ name });
  }

  const startTime = Date.now();
  //const cursor = Model.find().lean().batchSize(1000).stream();
  const cursor = Model.find().lean().batchSize(1000).cursor();
  let count = 0;

  await new Promise(resolve => {
    cursor.on('data', () => ++count);
    cursor.on('end', () => resolve());
  });
  console.log('Time elapsed:', Date.now() - startTime);
  console.log('Num docs:', count);
}

Mongoose 5.7 using cursor():

$ node gh-8039.js 
(node:1288) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
(node:1288) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
Time elapsed: 96
Num docs: 10000
^C
$ 

Mongoose 4.13 using stream():

$ node gh-8039.js 
(node:1285) DeprecationWarning: `open()` is deprecated in mongoose >= 4.11.0, use `openUri()` instead, or set the `useMongoClient` option if using `connect()` or `createConnection()`. See http://mongoosejs.com/docs/4.x/docs/connections.html#use-mongo-client
(node:1285) DeprecationWarning: Mongoose: mpromise (mongoose's default promise library) is deprecated, plug in your own promise library instead: http://mongoosejs.com/docs/promises.html
(node:1285) DeprecationWarning: Mongoose: Query.prototype.stream() is deprecated in mongoose >= 4.5.0, use Query.prototype.cursor() instead
Time elapsed: 77
Num docs: 10000
^C
$

How are you iterating through the stream?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Java 8 Streams: How to call once the Collection.stream ...
Is it possible to do a single call to the stream() method and to return the array of objects directly ? Object[] result...
Read more >
A Guide to Java Streams in Java 8: In-Depth Tutorial ... - Stackify
This in-depth tutorial is an introduction to the many functionalities supported by streams, with a focus on simple, practical examples. To ...
Read more >
Processing Data with Java SE 8 Streams, Part 1 - Oracle
We first get a stream from the list of transactions by calling the stream() method. The datasource is the list of transactions and...
Read more >
How to Filter Stream and Collections in Java 8? Example ...
You first need to obtain a stream from Collection by calling stream() method and then you can use the filter() method, which takes...
Read more >
Working with datetime as string - MongoDB
They store this value as an ISO 8601 formatted string ... Can I still run date time queries on it, even though it's...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found