findAndCountAll count disagrees with actual number of rows

I’m running a findAndCountAll query, which is telling me there are 10786 rows, but there are in fact only 5992.

The code I’m running:

Location.findAndCountAll
    include: [
        {
            model: Address
            where: id: gt: 0
        }
        DamageReport
    ]
    limit: 50
    offset: 5990
.done (err, result) ->
    console.log "total: #{result.count}, number after 5990: #{result.rows.length}"

(The where: id: gt: 0 is to skip Locations with no Address – I imagine there’s a cleaner way to do this, but I gave up a few nights ago when I wrote it.)

I’m getting total: 10786, number after 5990: 2

I’m printing the queries which are being run. They are like this:

SELECT COUNT(`Locations`.`id`) as `count`
FROM `Locations`
INNER JOIN `Addresses` AS `Addresses`
ON `Locations`.`id` = `Addresses`.`LocationId`
AND `Addresses`.`id` > 0
LEFT OUTER JOIN `DamageReports` AS `DamageReports`
ON `Locations`.`id` = `DamageReports`.`LocationId`;

and then

SELECT `Locations`.*, [...]
FROM (
    SELECT `Locations`.*
    FROM `Locations`
    WHERE (
        SELECT `LocationId`
        FROM `Addresses`
        WHERE `Locations`.`id` = `Addresses`.`LocationId`
        LIMIT 1
    ) IS NOT NULL
    LIMIT 5990, 50
) AS `Locations`
INNER JOIN `Addresses` AS `Addresses`
ON `Locations`.`id` = `Addresses`.`LocationId`
AND `Addresses`.`id` > 0
LEFT OUTER JOIN `DamageReports` AS `DamageReports`
ON `Locations`.`id` = `DamageReports`.`LocationId`;

The relationships between the models are as follows: Location hasMany Address Location hasMany DamageReport DamageReport belongsTo Location Address belongsTo Location

I played with the queries a little – running the longer one without the limit, for example. It gives me the full 10k rows when I do this. And looking through I notice that some rows are duplicated – I see that the same location IDs come up more than once. I guess this is because of the joins sometimes joining to multiple rows. (It’s too late in the evening for me to be able to mentally parse how exactly the query is supposed to be working.) If I move the LIMIT part to the end of the query rather than the subquery I still get back the full 10k rows.

But honestly I don’t know which of the bigger or smaller number is actually correct (maybe the larger number is due to having duplicates where I don’t actually want them). So I don’t know if that’s the bug – the LIMIT is supposed to be at the end – or if there’s some issue with the count query.

Any ideas what’s going on here?

[tremby]

Thanks regarding required.

This is sequelize 1.7.9.

Just to make sure I’m not accidentally wiping out some of the query I don’t mean to be where I got rid of most of the columns, here are unedited (other than adding line breaks) queries where I’m expecting 150ish rows (due to constraining on the value of Addresses.route).

SELECT COUNT(`Locations`.`id`) as `count`
FROM `Locations`
INNER JOIN `Addresses` AS `Addresses`
ON `Locations`.`id` = `Addresses`.`LocationId`
AND `Addresses`.`route`='West 47th Avenue'
LEFT OUTER JOIN `DamageReports` AS `DamageReports`
ON `Locations`.`id` = `DamageReports`.`LocationId`;

SELECT `Locations`.*,
    `Addresses`.`id` AS `Addresses.id`,
    `Addresses`.`description` AS `Addresses.description`,
    `Addresses`.`subpremise` AS `Addresses.subpremise`,
    `Addresses`.`streetNumber` AS `Addresses.streetNumber`,
    `Addresses`.`route` AS `Addresses.route`,
    `Addresses`.`neighbourhood` AS `Addresses.neighbourhood`,
    `Addresses`.`locality` AS `Addresses.locality`,
    `Addresses`.`administrativeAreaLevel3` AS `Addresses.administrativeAreaLevel3`,
    `Addresses`.`administrativeAreaLevel2` AS `Addresses.administrativeAreaLevel2`,
    `Addresses`.`administrativeAreaLevel1` AS `Addresses.administrativeAreaLevel1`,
    `Addresses`.`country` AS `Addresses.country`,
    `Addresses`.`postalCode` AS `Addresses.postalCode`,
    `Addresses`.`createdAt` AS `Addresses.createdAt`,
    `Addresses`.`updatedAt` AS `Addresses.updatedAt`,
    `Addresses`.`LocationId` AS `Addresses.LocationId`,
    `DamageReports`.`id` AS `DamageReports.id`,
    `DamageReports`.`area` AS `DamageReports.area`,
    `DamageReports`.`damageLevel` AS `DamageReports.damageLevel`,
    `DamageReports`.`description` AS `DamageReports.description`,
    `DamageReports`.`createdAt` AS `DamageReports.createdAt`,
    `DamageReports`.`updatedAt` AS `DamageReports.updatedAt`,
    `DamageReports`.`LocationId` AS `DamageReports.LocationId`,
    `DamageReports`.`PhotographId` AS `DamageReports.PhotographId`
FROM (
    SELECT `Locations`.*
    FROM `Locations`
    WHERE (
        SELECT `LocationId`
        FROM `Addresses`
        WHERE `Locations`.`id` = `Addresses`.`LocationId`
        LIMIT 1
    ) IS NOT NULL
    LIMIT 50
) AS `Locations`
INNER JOIN `Addresses` AS `Addresses`
ON `Locations`.`id` = `Addresses`.`LocationId`
AND `Addresses`.`route`='West 47th Avenue'
LEFT OUTER JOIN `DamageReports` AS `DamageReports`
ON `Locations`.`id` = `DamageReports`.`LocationId`;

This one gives me the ~150 rows for the count, but the data gives me 0 rows.

If it makes any difference, I’ve checked and confirmed that the findAll call instead of findAndCountAll generates exactly the same query.

Whether the count query is inefficient or broken in some way or not, it’s giving me a result at least resembling what I expect. (It seems to me it’s right, but I don’t know my data well enough to say for sure.) If I remove the limit, I get the results I expect, and no results with the limit.

I need to sleep now, and I likely won’t be back on this project for a couple of days. I can try to put a test together after that if you haven’t gotten to it already.

[tremby]

That’s great to hear. 😃 Thanks very much. Hoping to be on a project where I can use Node and Sequelize again soon.