issues when upserting several thousand records

Bug description

I have around 3500 records to insert into a - mysql - DB. I want to check they aren’t there already, and only insert the ‘new’ ones. For this I am using prisma’s upsert method:

    await Promise.all(
        currencies.map(async ({ name, symbol, id: nomicsId }) => {
            try {
                // await prisma.$connect()
                await prisma.currency.upsert({
                    where: { nomicsId },
                    update: { name, symbol },
                    create: { nomicsId, name, symbol },
                })
                // await prisma.$disconnect()
            } catch (err) {
                console.error('err adding', err)
            }
        }),
    )
    await prisma.$disconnect()

I thought by blocking via async/await would ensure I wasn’t straining the database unduly. However I get a stream of these errors:

How to reproduce

In the above snippet, my model is quite simple:

Expected behavior

I want all records to be inserted into the database that aren’t already there. I expect prisma executes one query at a time, such that I don’t have connection issues as only one would be in use.

Prisma information

As above.

Environment & setup

• OS: Linux (ubuntu) but running prisma within docker container (mhart/alpine-node)
• Database: MySQL (via dockerhub mysql:5.7)
• Node.js version: 14.4
• Prisma version: 2.15

[laubonghaudoi]

@samthomson I am having the exact same issue. Really disappointed that Prisma hasn’t fixed this.

[dthyresson]

Try running the operation with p-map and limit concurrency. https://www.npmjs.com/package/p-map.

FYI - When processing 100s/1000s of records I have been using p-map as well but also p-retry and delay:

https://github.com/sindresorhus/p-retry

and

https://github.com/sindresorhus/delay

so that each Prisma create (or typically upsert) is wrapped in a retry with a slight delay on an error – like too many connections or timeouts.

Note that this pattern behaved better after

https://github.com/prisma/prisma/issues/4906

was fixed in 2.15.