FindAll and FindAllAndCount with Includes and Limits - [fix proposal for incorrect behavior]

Right now, when you execute a findAll or a findAllAndCount call with includes and add a limit to your options, sequelize will produce something along the lines of

SELECT ... cols ... 
FROM (SELECT ... cols ... FROM `table` AS `table` LIMIT 0, 10) AS `table`
LEFT OUTER JOIN `table2` AS `table2` ON `table`.`col` = `table2`.`col` 
ORDER BY `table2`.`col` DESC;

which is wrong in every single pagination scenario where some sort of filtering and/or sorting needs to be done on the association (i.e. table2). This is not some obscure edge case but a trivial pagination + sorting case. The correct query for any pagination scenario would be

SELECT ... cols ... 
FROM (SELECT ... cols ... FROM `table` AS `table`) AS `table`
LEFT OUTER JOIN `table2` AS `table2` ON `table`.`col` = `table2`.`col` 
ORDER BY `table2`.`col` DESC  LIMIT 0, 10;

I came across this 2 years old thread which seems to have been closed without a solution so I took a quick look at the sequelize source code and while I can’t (and don’t want to) propose an in-depth solution to the query generation logic, I think you can give us a way to work around this problem without major code changes and without forcing us to rely on ugly pure SQL query fixes.

If you take a look at https://github.com/sequelize/sequelize/blob/master/lib/dialects/abstract/query-generator.js#L1329

const limitOrder = this.addLimitAndOffset(options, mainTable.model);
if (limitOrder && !options.groupedLimit) {
	if (subQuery) {
	subQueryItems.push(limitOrder);
	} else {
	mainQueryItems.push(limitOrder);
	}
}

if the findAll or findAllAndCount query has any includes, the subQuery variable will be true and the limit will be added to subQueryItems and thus applied to the subquery.

A very simple workaround would be to just change that if-statement to

if (subQuery && !options.isGlobalLimit) {
	subQueryItems.push(limitOrder);
} else {
	mainQueryItems.push(limitOrder);
}

so that we can simply change our findAll call from

modelA.findAll({ where: {...}, include: [...], order: [ ... ], limit: 10 })

to

modelA.findAll({ where: {...}, include: [...], order: [ ... ], limit: 10, isGlobalLimit: true })

and we would get the correct pagination query with the limit being applied to the main query instead of the subquery.

It’s not a solution to the logical problem but at least it gives us a way to implement correct sorting/filtering behavior using sequelize without any in-depth changes and without introducing any bugs.

Do you think this change could make it into the next release?

[radoslavpetranov]

@krazi3 you don’t remove the limit - you move it to the upper level query. That’s exactly what you want to do when you want to do a join correctly and then limit the results of that join.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 14 days without activity. It will be closed if no further activity occurs within the next 14 days. If this is still an issue, just leave a comment or remove the “stale” label. 🙂