Timed out fetching a new connection from the pool

Bug description

After running my application for around 1-2 days, I start getting the following error:

prisma:client Error: Timed out fetching a new connection from the pool. Please consider reducing the number of requests or increasing the `connection_limit` parameter (https://www.prisma.io/docs/concepts/components/prisma-client/connection-management#connection-pool). Current limit: 10.

The same happened before with a connection limit of 3. Raising it to 10 seemed to have no effect.

After the error occurs once, no query will succeed anymore, the error will be thrown on any query and the application requires a restart (prisma will never recover from the error).

I checked whether the database has an issue, but it is working fine. SHOW PROCESSLIST shows that no connection is actually established.

How to reproduce

I have not found a way to reliably reproduce this yet. I always need to wait 1-2 days until the error randomly appears. I am running my application in debug mode with DEBUG=* for the last days, so if you need any additional information, I can look it up in the logs.

Expected behavior

No response

Prisma information

generator client {
    provider        = "prisma-client-js"
    binaryTargets   = ["native"]
}

datasource db {
    provider = "mysql"
    url      = env("DATABASE_URL")
}

I can’t post my complete prisma schema here. It doesn’t matter which queries are run, all of them fail with the same error.

Environment & setup

• OS: Linux (official node:14 docker image)
• Database: MySQL 8.0
• Node.js version: 14

Prisma Version

prisma               : 2.21.2
@prisma/client       : 2.21.2
Current platform     : darwin
Query Engine         : query-engine e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/prisma/node_modules/@prisma/engines/query-engine-darwin)
Migration Engine     : migration-engine-cli e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/prisma/node_modules/@prisma/engines/migration-engine-darwin)
Introspection Engine : introspection-core e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/prisma/node_modules/@prisma/engines/introspection-engine-darwin)
Format Binary        : prisma-fmt e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/prisma/node_modules/@prisma/engines/prisma-fmt-darwin)
Default Engines Hash : e421996c87d5f3c8f7eeadd502d4ad402c89464d
Studio               : 0.371.0

[TLadd]

I’ve been experiencing this as well. We get a temporary spike in requests to our app servers that increases memory and cpu near 100% for a short time. Prisma then seems to get stuck in a state of every query failing to fetch a connection and never recovers, even as cpu and memory usage drop. Restarting the servers returns things to normal. Wondering if https://github.com/prisma/prisma/issues/9584 is related since both seem to involve the client connections getting stuck in particular failure states.

[janpio]

We actually have a reproduction now (essentially load spike or resource constraint with specific parameters) and are working on understanding the problem (non surprisingly something with the connection pool) and hopefully fixing it (no ETA - but if you are able to somehow reproduce this, we are looking for testers of our fixes of course).

Some related issues: https://github.com/prisma/prisma/issues/11868 https://github.com/prisma/prisma-private/issues/160 (private with reproduction)