Prisma connection doesn't recover from idle in transaction state

Bug description

Hi there,

As the title mention it, I encountered a bug when using both interactives and non-interactives transactions with prisma.

Sometimes, the prisma client get stuck into a ‘idle in transaction’ mode, this mainly happens under loads (eg: truncate all tables in a database) when postgres is allowed little ressources (eg: in CI, 1 CPU for postgres container).

From my preliminary investigation:

1. This bug is random and ressources related (with 4 CPU allowed to postgres it doesn’t appears), with less query load it doesn’t appears. And when it does, it doesn’t appears at the same point.
2. It seems to be postgres “lock” related. For us, this show up during our “integrations testing” in which we truncate all the tables, then seed them with controlled data. Sometimes it’s one of the “flush” (TRUNCATE) query who get stuck in the ‘idle in transaction’ mode, sometimes it’s one of the “seed” (INSERT INTO). The common denominator seems to be the fact that some locks aren’t unlocked by postgres.

The main issue is that, once it get into this state, the prisma client never get out of it. In our case, requiring to restart the webserver to instantiate a new connection to the database.

It’s not very clear for me how to make a MRE from my issue since it’s heavily related our code and only appears under loads.

How to reproduce

Here is a MRE reproducing this issue: https://github.com/avallete/prisma-9584-MRE

Expected behavior

Prisma could throw a “transaction timeout” error after X seconds passed in a transaction which doesn’t end.

Prisma information

nothing specific

Environment & setup

• OS: OSX/Debian
• Database: postgres:12
• Node.js version: v14.17.1

Prisma Version

prisma                : 2.30.3
@prisma/client        : 2.30.3
Current platform      : darwin
Query Engine (Binary) : query-engine b8c35d44de987a9691890b3ddf3e2e7effb9bf20 (at ../../node_modules/@prisma/engines/query-engine-darwin)
Migration Engine      : migration-engine-cli b8c35d44de987a9691890b3ddf3e2e7effb9bf20 (at ../../node_modules/@prisma/engines/migration-engine-darwin)
Introspection Engine  : introspection-core b8c35d44de987a9691890b3ddf3e2e7effb9bf20 (at ../../node_modules/@prisma/engines/introspection-engine-darwin)
Format Binary         : prisma-fmt b8c35d44de987a9691890b3ddf3e2e7effb9bf20 (at ../../node_modules/@prisma/engines/prisma-fmt-darwin)
Default Engines Hash  : b8c35d44de987a9691890b3ddf3e2e7effb9bf20
Studio                : 0.423.0
Preview Features      : interactiveTransactions, nApi, referentialActions

Also tried with the latest 3.X version, the issue is the same.

[ssukienn]

Hi. I will add my bits to it.

I am facing a similar problem in MySql 8. This works locally (or at least I didn’t encounter the problem yet) on 4-core 32GB device but hangs (promise do not return) in i.e. preemptible GCE server with 1 core and low amount of RAM. Seems like the database is not locked afterwards just the Prisma promise does not return, nor throw/rejects.

In my scenario that prevents ack’ing broker messages at some point which creates the service unresponsive (15 is its prefetch/concurrent limit):

Code:

   //called by message broker consumer for each message
    async saveMessage(input: MessageWithoutOrderInput, order?: number) {
        const uuid = randomUUID();
        const promise = this._prismaSaveMessage([input], order, uuid);

        const interval = this.checkForPending(promise, uuid);
        intervals.set(uuid, interval);

        const debug = (await promise)[0];
        this.logger.debug(debug);
        return debug;
    }
    
    //for logging pending promises
    private checkForPending(promise: Promise<any>, uuid: string) {
        return setInterval(() => {
            const isPending = util.inspect(promise) === 'Promise { <pending> }';
            this.logger.debug(`isPending: ${isPending}`, uuid);
            if (!isPending) {
                intervals.get(uuid)();
            }
        }, 2000);
    }

    private _prismaSaveMessage(inputArray: MessageWithoutOrderInput[], explicitOrder?: number) {
        return this.db.$transaction(async (prisma) => {
            let lastMessageOrder =
                explicitOrder ||
                ((
                    await prisma.assistantMessage.aggregate(...)
                )._max.order || 0) + 1;

            return Promise.all(
                inputArray.map((input) => {
                    return prisma.assistantMessage.upsert({
                        where: { uuid: input.uuid || '' },
                        create: {
                            ...input,
                            order: lastMessageOrder++,
                        },
                        update: input,
                    });
                }),
            );
        });
    }

output: 15 lines of pending promise output each 2 seconds indefinitely

Changing from interactive to array-based transaction method does not change the outcome.

Creating my own timeout over this to prevent it, seems like overkill and it should be rather taken care of by the framework. It would not fix the problem though just prevent zombie service.

Prisma:

prisma : 3.3.0 @prisma/client : 3.3.0 Current platform : debian-openssl-1.1.x Query Engine (Node-API) : libquery-engine 33838b0f78f1fe9052cf9a00e9761c9dc097a63c (at node_modules/prisma/node_modules/@prisma/engines/libquery_engine-debian-openssl-1.1.x.so.node) Migration Engine : migration-engine-cli 33838b0f78f1fe9052cf9a00e9761c9dc097a63c (at node_modules/prisma/node_modules/@prisma/engines/migration-engine-debian-openssl-1.1.x) Introspection Engine : introspection-core 33838b0f78f1fe9052cf9a00e9761c9dc097a63c (at node_modules/prisma/node_modules/@prisma/engines/introspection-engine-debian-openssl-1.1.x) Format Binary : prisma-fmt 33838b0f78f1fe9052cf9a00e9761c9dc097a63c (at node_modules/prisma/node_modules/@prisma/engines/prisma-fmt-debian-openssl-1.1.x) Default Engines Hash : 33838b0f78f1fe9052cf9a00e9761c9dc097a63c Studio : 0.437.0 Preview Features : interactiveTransactions

[awinograd]

@janpio The issue in my MRE is resolved as of 3.9.x! Thanks