SelectQuery loads all models into memory at once

I have a code like this:

for book in Book.select().where(Book.merchant == m).order_by(Book.last_checked_at):

And there are approximately 100k rows in the table (PostgreSQL). When I don’t use any limit functions, peewee loads all those 100k rows at the very beginning of the loop and that takes 250mb of memory. When I use limit(1000) it only takes 30mb. Is there any way to use cursors to pull models incrementally when they are requested by for loop and not read entire table into memory?

[coleifer]

Try this one:

books = Book.select().where(Book.merchant == m).order_by(Book.last_checked_at).naive()
books_qr = books.execute()
for book in books_qr.iterator():
    # ... etc ...

What I’m curious about is whether this memory usage is related to caching instances on the results wrapper (so that iterating a query multiple times does not cause multiple queries), or is just due to the way psycopg2 handles large result sets.

[extesy]

Much better, thank you!