Subqueries and missing fields, again

Note: This post is to describe the problem, I posted a reply with a suggested solution approach below.

What you are doing?

• do a find
• include 1:n association
• limit the outer query
• filter on the outer query (top-level where) with a “nested” key, e.g. $child.childAttribute$: 1337
  • no, I cannot filter on the include, because I need to AND/OR-combine the childAttribute condition with some other condition on the outer/parent model

var Parent = sequelize.define('Parent', {name: Sequelize.STRING}, {timestamps: false});
var Child = sequelize.define('Child', {name: Sequelize.STRING}, {timestamps: false});
Parent.hasMany(Child, {as: 'children', foreignKey: 'parentId'});
Child.belongsTo(Parent, {as: 'child', foreignKey: 'parentId'});

async function run() {
    await sequelize.sync();
    const p1 = await Parent.create({name: 'p1'});
    const p2 = await Parent.create({name: 'p2'});
    const c1 = await Child.create({name: 'c1', parentId: p1.id});
    const c2 = await Child.create({name: 'c2', parentId: p1.id});
    const c3 = await Child.create({name: 'c3', parentId: p2.id});
    const c4 = await Child.create({name: 'c4', parentId: p2.id});

    const p = await Parent.findAll({
        where: {
            $or: {
                name: 'p1',
                '$children.name$': 'c3',
            },
        },
        include: [{model: Child, as: 'children'}],
        limit: 1,
        order: [['name', 'ASC']],
    });

    console.log(JSON.stringify(p.map(p => p.toJSON()), null, 4));
}

run().catch((err) => console.error(err));

What do you expect to happen?

I would like to receive one parent (p1, thanks to name ordering) for which the filter holds (the name matches ‘p1’), including ALL their children (c1 and c2) (since I’m not filtering on them, just filtering on the parent relation).

What is actually happening?

It breaks with ER_BAD_FIELD_ERROR: Unknown column because the LIMIT is applied in the subquery, but the JOIN isn’t.

I cannot move the where condition for the children into the include, since then I would only receive those children, not ALL children of the matching parents. Also I could not combine that condition using $or with the name condition on the parent.

Dialect: mysql Database version: irrelevant Sequelize version: 4.0.0-0

[opatut]

Bump.

[opatut]

I fiddled around a little with the SQL generated. I think one possible solution would be to

• LEFT OUTER JOIN in the subquery, just like in the outer query
• but!?! you say, what’s the point of the subquery then, if that join yields an increased count of rows that we cannot LIMIT on
• luckily we can break that count down again to after WHERE, but before LIMIT by using GROUP BY on primary key attributes of the main table
• then, outside the subquery, as per normal we join again to find ALL instances of associated items, not just those that we filtered by
• of course, a where inside the include would generate the WHERE condition outside the subquery, nothing changed there either

Before

SELECT
    Parent.*,
    children.*,
FROM (
    SELECT Parent.*
    FROM Parents AS Parent
    WHERE (Parent.name = "p1" OR children.name = "c3") -- here is our problem
    LIMIT 1
) as Parent
LEFT OUTER JOIN Children AS children on Children.parentId = Parent.id;

After

SELECT
    Parent.*,
    children.*,
FROM (
    SELECT Parent.*
    FROM Parents AS Parent
    LEFT OUTER JOIN Children AS children on Children.parentId = Parent.id     -- look at this!
    WHERE (Parent.name = "p1" OR children.name = "c3") -- no more problem
    GROUP BY Parent.id     -- and this!
    LIMIT 1
) as Parent
LEFT OUTER JOIN Children AS children on Children.parentId = Parent.id;

Things to decide

• should this “feature” be always-on or opt-in
• does this really work?
• does this impact performance (more joins!)?
• does this break any of those complicated features like attributes etc.?
• does the subquery sometimes serve another purpose (other than limiting and ordering) which would break if there were more joins?

How to implement

Apart from that, the implementation seems not too bad, simply add a “default” GROUP BY on each subquery, and all mainJoinQueries to the subJoinQueries in the abstract query generator. I tried it, it worked for my use case.

diff --git a/lib/dialects/abstract/query-generator.js b/lib/dialects/abstract/query-generator.js
index 2a34e6f..7a083cd 100644
--- a/lib/dialects/abstract/query-generator.js
+++ b/lib/dialects/abstract/query-generator.js
@@ -1030,7 +1030,7 @@ const QueryGenerator = {
     }

     // Escape attributes
-    mainAttributes = mainAttributes && mainAttributes.map(attr => {
+    const escapeAttribute = attr => {
       let addTable = true;

       if (attr._isSequelizeMethod) {
@@ -1058,7 +1058,9 @@ const QueryGenerator = {
         attr = mainTableAs + '.' + attr;
       }
       return attr;
-    });
+    };
+
+    mainAttributes = mainAttributes && mainAttributes.map(escapeAttribute);

     // If no attributes specified, use *
     mainAttributes = mainAttributes || (options.include ? [mainTableAs + '.*'] : ['*']);
@@ -1377,6 +1379,7 @@ const QueryGenerator = {
     if (subQuery) {
       subQueryItems.push(this.selectFromTableFragment(options, model, subQueryAttributes, table, mainTableAs));
       subQueryItems.push(subJoinQueries.join(''));
+      subQueryItems.push(mainJoinQueries.join(''));

     // Else do it the reguar way
     } else {
@@ -1435,6 +1438,8 @@ const QueryGenerator = {
       } else {
         mainQueryItems.push(' GROUP BY ' + options.group);
       }
+    } else if (subQuery) {
+      subQueryItems.push(' GROUP BY ' + model.primaryKeyAttributes.map(escapeAttribute).join(', '));
     }

     // Add HAVING to sub or main query