Several SQL queries for each related item instead of one aggregated query
See original GitHub issueBug report
Describe the bug
When executing very simple GraphQL query with querying data from one relationship field, Keystone executes separate query for each relation, instead of one aggregated query with list of needed ids
To Reproduce
- Launch the “examples/basic” demo, enable database logging via
enableLogging: true
. - Fill 2 authors and 4 posts to database (some linked with author, some - not).
- Execute this GraphQL query:
query {
allPosts {
id, author {id, name}
}
}
- Lookup to the output, you will see several SQL queries:
prisma:query SELECT `main`.`Post`.`id`, `main`.`Post`.`title`, `main`.`Post`.`status`, `main`.`Post`.`content`, `main`.`Post`.`publishDate`, `main`.`Post`.`author` FROM `main`.`Post` WHERE 1=1 LIMIT ? OFFSET ?
prisma:query SELECT 1
prisma:query SELECT `main`.`Author`.`id`, `main`.`Author`.`name`, `main`.`Author`.`email` FROM `main`.`Author` WHERE (`main`.`Author`.`id`) IN (SELECT `t0`.`id` FROM `main`.`Author` AS `t0` INNER JOIN `main`.`Post` AS `j0` ON (`j0`.`author`) = (`t0`.`id`) WHERE (`j0`.`id` = ? AND `t0`.`id` IS NOT NULL)) LIMIT ? OFFSET ?
prisma:query SELECT `main`.`Author`.`id`, `main`.`Author`.`name`, `main`.`Author`.`email` FROM `main`.`Author` WHERE (`main`.`Author`.`id`) IN (SELECT `t0`.`id` FROM `main`.`Author` AS `t0` INNER JOIN `main`.`Post` AS `j0` ON (`j0`.`author`) = (`t0`.`id`) WHERE (`j0`.`id` = ? AND `t0`.`id` IS NOT NULL)) LIMIT ? OFFSET ?
prisma:query SELECT `main`.`Author`.`id`, `main`.`Author`.`name`, `main`.`Author`.`email` FROM `main`.`Author` WHERE (`main`.`Author`.`id`) IN (SELECT `t0`.`id` FROM `main`.`Author` AS `t0` INNER JOIN `main`.`Post` AS `j0` ON (`j0`.`author`) = (`t0`.`id`) WHERE (`j0`.`id` = ? AND `t0`.`id` IS NOT NULL)) LIMIT ? OFFSET ?
prisma:query SELECT `main`.`Author`.`id`, `main`.`Author`.`name`, `main`.`Author`.`email` FROM `main`.`Author` WHERE (`main`.`Author`.`id`) IN (SELECT `t0`.`id` FROM `main`.`Author` AS `t0` INNER JOIN `main`.`Post` AS `j0` ON (`j0`.`author`) = (`t0`.`id`) WHERE (`j0`.`id` = ? AND `t0`.`id` IS NOT NULL)) LIMIT ? OFFSET ?
Problems:
- Why each author queried via separate SQL query? And even when author is empty!
- Why all fields are queried from “Post” table, when we need only
id
andauthor
for query results? - Why author email is queried using inner query and even inner join inside it?
Those problems give large performance degradation on databases with many items!
Expected behavior
- To optimize performance, Keystone must collect id of all relationships, and query them via one SQL query with list of need ids.
- We should query only requested fields, not all available fields from table.
- There must be no inner joins and subqueries for such simple query.
System information
Keystone version: 21.0.2
Issue Analytics
- State:
- Created 2 years ago
- Reactions:3
- Comments:10 (10 by maintainers)
Top Results From Across the Web
Aggregating and Grouping Data
GROUP BY tells SQL what field or fields we want to use to aggregate the data. If we want to group by multiple...
Read more >Writing Subqueries in SQL | Advanced SQL - Mode Analytics
This lesson of the SQL tutorial for data analysis covers using subqueries in SQL with aggregate functions, conditional logic, and joins.
Read more >Learn SQL: Aggregate Functions - SQLShack
Aggregate functions are a very powerful tool in databases. They serve the same purpose as their equivalents in MS Excel.
Read more >Use a union query to combine multiple queries into a single ...
Click the top of the Navigation Pane and then select Object Type to organize all the database objects by their type. Next, expand...
Read more >Why can't you mix Aggregate values and Non ... - Stack Overflow
If you use an aggregate, but doesn't specify any grouping, the query will still be grouped, and the entire result is a single...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Partial fix underway in #6639 🥳
Ok, I’ve talked this though with @timleslie (apparently you and him have been chatting recently too). My current understanding is that, yes, Prisma does proactively apply a data loader pattern to avoid N+1 problems but only does so when it’s really obvious that it can (only
findUnique
calls using the the same parameters). The access control that Keystone applies unintentionally prevents this from occurring – we callfindFirst
instead offindUnique
so we can supply the additional access filters.There may be some low hanging fruit here whereby, maybe we can only call
findFirst
when access filters are actual in effect. This would allow some lists to take advantage of the automatic data loading Prisma provides. It still only solves for querying via many-to-one relationships; coming the direction (one-to-many) will trigger N+1 queries in any case. Otherwise, the longer term solution is to add a data loader layer in our own code and be smarter about how queries are constructed.Regardless, you shouldn’t be relying on Keystone to always build optimal queries. If you have complex queries that are used regularly in your main frontend, you’re probably better moving to custom mutations that do exactly what’s needed. The CRUD schema Keystone provides is intended to support the Admin UI and as a starting point for general app dev.