Prisma silently misses nested elements on MariaDB when it triggers a huge IN clause

Bug description

I stumbled upon an issue when trying to perform a findMany query that included nested relations.

What I tried to achieve is to retrieve the list of number of “Donnee” per "Commune which are related via “Inventaire” and “Lieudit”.

See my query below for reference.

The issue I am facing is that at no matching “Donnee” is returned at all. No log related to the engine is displayed at all, so it fails silently.

I have spent a lot of time investigating and found the following:

• I noticed that the latest generated query was containing a WHERE clause with thousands of values in the IN parameter. This is expected given the base I am using.
• While trying to find a similar issue, like https://github.com/prisma/prisma/issues/9132 I have found that setting QUERY_BATCH_SIZE=999 was working while QUERY_BATCH_SIZE=1000 was not.

So after a bit of digging with the engine code ( https://github.com/prisma/prisma-engines/pull/2293 ) and MariaDB doc, I found out that my issue was related to the following MariaDB parameter: https://mariadb.com/docs/reference/mdb/system-variables/in_predicate_conversion_threshold/ which is 1000 by default.

This parameter seems specific to MariaDB >=10.3. My understanding is that once the number of parameters in the IN clause reaches this threshold, it internally generates subqueries that so not seem to be handled properly by the Prisma engine.

In order to validate this, I changed this threshold to 10000 : set in_predicate_conversion_threshold=10000; and managed to have the request successfully working until QUERY_BATCH_SIZE=9999.

My understanding is that the default value for QUERY_BATCH_SIZE is currently set here : https://github.com/prisma/prisma-engines/blob/c9f86866d2fb27b2066e5447ee7f6f65c46c5707/query-engine/connectors/sql-query-connector/src/sql_info.rs#L48

So the issue could be resolved by lowering this value, however it would probably not make sense for MySQL users. Another way would be to investigate why these subqueries are not handled properly by the engine.

How to reproduce

1. Setup any MariaDB instance with default parameters
2. Create 4 tables A, B , C and D where A is related to B , B to C and C to D
3. Perform any findManyQuery that includes nested elements and that will trigger a IN clause with more than 1000 parameters.

Expected behavior

The nested related are properly returned without workaround on QUERY_BATCH_SIZE

Prisma information

Relevant excerpt of the schema:

model Donnee {
  id                   Int                   @id @default(autoincrement()) @db.UnsignedMediumInt
  inventaireId         Int                   @map("inventaire_id") @db.UnsignedMediumInt
  inventaire           Inventaire            @relation(fields: [inventaireId], references: [id], onDelete: Cascade, onUpdate: Restrict, map: "fk_donnee_inventaire_id")

  @@index([inventaireId], map: "fk_donnee_inventaire_id")
}

model Inventaire {
  id                 Int                  @id @default(autoincrement()) @db.UnsignedMediumInt
  lieuDitId          Int                  @map("lieudit_id") @db.UnsignedMediumInt
  lieuDit            Lieudit              @relation(fields: [lieuDitId], references: [id], onDelete: Cascade, onUpdate: Restrict, map: "fk_inventaire_lieudit_id")
  donnee             Donnee[]

  @@index([lieuDitId], map: "fk_inventaire_lieudit_id")
}

model Lieudit {
  id                Int               @id @default(autoincrement()) @db.UnsignedMediumInt
  communeId         Int               @map("commune_id") @db.UnsignedSmallInt
  commune           Commune           @relation(fields: [communeId], references: [id], onDelete: Cascade, onUpdate: Restrict, map: "fk_lieudit_commune_id")
  inventaire        Inventaire[]

  @@unique([communeId, nom], map: "unique_commune_nom")
}

model Commune {
  id            Int         @id @default(autoincrement()) @db.UnsignedSmallInt
  lieudit       Lieudit[]
}

Example of query that silently fails by returning empty nested relations:

prisma.commune.findMany({
    select: {
      id: true,
      lieudit: {
        select: {
          inventaire: {
            select: {
              _count: {
                select: {
                  donnee: true
                }
              }
            }
          }
        }
      }
    }
  });

Environment & setup

• OS: MacOS
• Database: MariaDB 10.7.3
• Node.js version: 16.14.0

Prisma Version

prisma                  : 3.10.0
@prisma/client          : 3.10.0
Current platform        : darwin-arm64
Query Engine (Node-API) : libquery-engine 73e60b76d394f8d37d8ebd1f8918c79029f0db86 (at node_modules/@prisma/engines/libquery_engine-darwin-arm64.dylib.node)
Migration Engine        : migration-engine-cli 73e60b76d394f8d37d8ebd1f8918c79029f0db86 (at node_modules/@prisma/engines/migration-engine-darwin-arm64)
Introspection Engine    : introspection-core 73e60b76d394f8d37d8ebd1f8918c79029f0db86 (at node_modules/@prisma/engines/introspection-engine-darwin-arm64)
Format Binary           : prisma-fmt 73e60b76d394f8d37d8ebd1f8918c79029f0db86 (at node_modules/@prisma/engines/prisma-fmt-darwin-arm64)
Default Engines Hash    : 73e60b76d394f8d37d8ebd1f8918c79029f0db86
Studio                  : 0.458.0

[d-Rickyy-b]

Hi, I randomly stumbled across this GitHub issue, because I was having similar issues with WHERE IN () statements where the IN clause contains >= 1000 values. I found the MariaDB bug tracker and it seems like this is not related to any ORM (I am using Golang & Gorm), but only to MariaDB.

See https://jira.mariadb.org/browse/MDEV-27937

It seems to only affect prepared statements. The bug seems to be fixed in these versions: 10.3.35, 10.4.25, 10.5.16, 10.6.8, 10.7.4, 10.8.3.

Hope this helps.

[jkomyno]

Closing this issue as it’s independent of Prisma: it’s instead related to a bug in some MariaDB versions. This reproduction runs test cases inspired by issue #14539 (which comprises #12338 and #14019 as well) on a “bugged” MariaDB instance (mariadb:10.7.3), on a “stable” MariaDB instance (mariadb:10.8.4), and on a recent MySQL instance (mysql:8.0.30).

Thanks to @kkomelin and @HendrikJan for their independent reproductions, and to @d-Rickyy-b for spotting https://jira.mariadb.org/browse/MDEV-27937.