Ballooning of Prepared_stmt_count on MYSQL

Bug description

Hey guys, I hope you all are doing well!

I’m running prisma @ 2.14 on MySQL, and after updating to and trying multiple versions (2.18, 2.20.1, 2.21.1), I’m running into issues where my Prepared_stmt_count global MySQL variable is increased by a huge factor compared to 2.14, which leads to me hitting the MAX_PREPARED_STATEMENTS limit. It seems like there is a difference in behavior of these prepared statements from 2.14 and onwards.

Just to reiterate, this is not a problem on 2.14, and was I wondering if there was anything I could do to get the old behavior back on a newer version.

Thank you!

How to reproduce

I’m running this on a production kube deployment, so I’m unsure what specific queries may be triggering this, but I do know that 2.14 and previous never had this behavior

1. Be on similar env ?
2. Use normally (create/update/delete) (many)

Expected behavior

Expect the Prepared_stmt_count to stay low, this is the current behavior on 2.14.

Actual behavior

Prepared statement count balloons, and stays that way for a long enough time to where my set MAX_PREPARED_STMT_COUNT gets triggered and hangs the db. Note that the actual value in prod gets to 64000+.

Prisma information

Environment & setup

• OS: Mac / Ubuntu
• Database: MySQL 8.0.21
• Node.js version: 12.16.1
• Prisma version: Tested on 2.17, 2.18, 2.20.x, 2.21.x

prisma               : 2.17.0
@prisma/client       : 2.17.0
Current platform     : darwin
Query Engine         : query-engine 3c463ebd78b1d21d8fdacdd27899e280cf686223 (at node_modules/@prisma/engines/query-engine-darwin)
Migration Engine     : migration-engine-cli 3c463ebd78b1d21d8fdacdd27899e280cf686223 (at node_modules/@prisma/engines/migration-engine-darwin)
Introspection Engine : introspection-core 3c463ebd78b1d21d8fdacdd27899e280cf686223 (at node_modules/@prisma/engines/introspection-engine-darwin)
Format Binary        : prisma-fmt 3c463ebd78b1d21d8fdacdd27899e280cf686223 (at node_modules/@prisma/engines/prisma-fmt-darwin)
Studio               : 0.353.0

[olif]

The problem is that prisma is using prepared statements when making the request to the mysql db and never drops them. Mysql has a default limit of about 16000 prepared statements which puts an upper limit on how many different prepared statements are allowed to be used globally for the db engine.

For instance, when querying the /post/:id endpoint the resulting parameterized query is

prisma:query SELECT `db`.`Post`.`id`, `db`.`Post`.`createdAt`, `db`.`Post`.`updatedAt`, `db`.`Post`.`title`, `db`.`Post`.`content`, `db`.`Post`.`published`, `db`.`Post`.`viewCount`, `db`.`Post`.`authorId` FROM `db`.`Post` WHERE `db`.`Post`.`id` = ? LIMIT ? OFFSET ?

and it will be sent by the prisma engine to the database as a prepared statement. If the exact same query is run again but with other parameters the prepared statement will be reused by mysql. However, if the query is slightly modified, for instance by adding a new statement in the WHERE clause, it will be cached by mysql as a new prepared statement.

The prepared statements belongs to the session/connection so if we have multiple instances of the same app and also a connection pool, each session/connection will have their prepared statements cached and these statements will no be dropped until the connection is closed.

This means that the overall number of different queries that an application can have is limited to 16000/(nr of app instances * size of connection pool).

To make things worse, prepared statements will be created for each size of parameters of an IN statement. For instance, this handler:

app.get(`/posts`, async(req, res) => {
  const queryParams = req.query
  const ids = queryParams.ids as string[]
  const post = await prisma.post.findMany({
    where: {
      id: {
        in: ids.map(x => Number(x))
      }
    }
  })

  res.json(post)
})

will issue a new prepared statement for every number of provided ids.

$> curl 'localhost:3000/posts?ids[]=1'

SELECT `db`.`Post`.`id`, `db`.`Post`.`createdAt`, `db`.`Post`.`updatedAt`, `db`.`Post`.`title`, `db`.`Post`.`content`, `db`.`Post`.`published`, `db`.`Post`.`viewCount`, `db`.`Post`.`authorId` FROM `db`.`Post` WHERE `db`.`Post`.`id` IN (?)

Prepared_stmt_count=1

$> curl 'localhost:3000/posts?ids[]=1&ids[]=2'

SELECT `db`.`Post`.`id`, `db`.`Post`.`createdAt`, `db`.`Post`.`updatedAt`, `db`.`Post`.`title`, `db`.`Post`.`content`, `db`.`Post`.`published`, `db`.`Post`.`viewCount`, `db`.`Post`.`authorId` FROM `db`.`Post` WHERE `db`.`Post`.`id` IN (?,?)

Prepared_stmt_count=2

which further reduces the number of queries possible in the application since they are treated as different prepared statements.

As I said earlier, the reason this problem arises in versions of prisma > 2.14 is that the connection pool did not work in earlier versions which resulted in a new connection per request (which then dropped the associated prepared statements).

I don’t know if I would classify this as a bug but since we don’t have control over which queries are sent as prepared statements or not it has a severe negative impact on our scaling possibilities. Also, since IN clauses also are sent as prepared_statements it is hard to determine an upper bound that is guaranteed to cover the Prepared_stmt_count.

[pimeys]

So, this is a bug in the mysql library we use. Their LRU leaks statements, and I need to probably fix it and do a PR for them.

Meanwhile, for the next release, I’m introducing this parameter statement_cache_size. By default it’s 1000 and this is a reason you’re probably the first one that gets this problem. If you reach this limit, it starts dropping queries, replacing them with new ones. This drop doesn’t actually clean them from the database, meaning when the cache gets full, every subsequent query will leak one from the cache. Eventually you hit the hard limit and crash.

I’d suggest, if I don’t get a proper fix through to the library today (which I highly suspect), that you try to set this parameter to something like 5000 or 10000 until we fix the leak.