Efficiently create many records

We need a way to efficiently insert many rows of data.

For small amounts of data, a simple batch API might be sufficient, as the database can handle all the operations in a short time. This could either be exposed through operations similar to our nested create API as described below, or through a new Batch API as described in https://github.com/prisma/prisma-client-js/issues/667.

For large amounts of data, we need to support breaking the large number of create operations into smaller batches. In order to optimise throughput, you need to balance two things: provide large enough work packages such that network latency doesn’t dominate the actual work, and keep the work packages small enough to not overwhelm the database. This could be exposed through a batch API with configurable batch size, or through a streaming API with buffering support.

Original issue

I think is quite common the need of having to create or upsert many records on a database at the same time, at least is a feature I see in many other ORMs.

Right now is posible to create many records using a nested write like this

const newUser: User = await photon.users.create({
  data: {
    email: '[e-mail]',
    posts: {
      create: [
        { title: 'Join the Prisma Slack on https://slack.prisma.io' },
        { title: 'Follow Prisma on Twitter' },
      ],
    },
  },
})

But sometimes you just want to create many records, in this case the only thing I managed to find to accomplish is by doing this

const newPosts = [
  { id: 1, title: 'Join the Prisma Slack on https://slack.prisma.io' },
  { id: 2, title: 'Follow Prisma on Twitter' },
];

// create
const createManyPosts = newPosts.map((post) =>
  photon.posts.create({
    data: post,
  }),
);

Promise.all(createManyPosts);

// upsert
const upsertManyPosts = newPosts.map((post) =>
  photon.posts.upsert({
    where: { id: post.id },
    create: post,
    update: post,
  }),
);

Promise.all(upsertManyPosts);

This with 2 records works fine, but as soon as you need to create 1000 doesn’t seems right to fire 1000 promises.

Maybe I’m missing something and there is a good reason for a feature like this to not be available, or maybe is already planned to be added in future releases?

I think prisma2 is awesome! Thank you all very much for this product 😃 it’s totally an step forward.

[robmurtagh]

Wanted to add my vote to this ticket. This is currently quite a deal-breaker for us. Our app isn’t huge, but at one point it requires creating around 1000 rows in a table. It would be great if this was possible via Photon.

@timsuchanek any idea on timelines for this? I guess it would be sometime after Prisma2 GA?

[matthewmueller]

Hey folks, we’ve started designing this feature.

Please have a read through the proposal and let us know what you think! Also feel free to schedule time with me to chat about this proposal.

Proposal

Database Support

• Postgres
• MySQL
• SQL Server

Operations

Bulk Create

• You can efficiently insert many records in one query
• Type-safe inputs
• On success, we return a count of the records created
• If there’s an error, we throw that error

Example

const result = await client.user.createMany({
  data: [
		{ email: "[e-mail]" },
		{ email: "[e-mail]" }
	]
})

// result contains the number of rows affected
console.log(result.count) // 2

Idempotent Creates

If you try creating many records at once, the whole operation can fail if there are any constraint violations. The skipDuplicates key helps avoid this.

const result = await client.user.createMany({
  skipDuplicates: true,
  data: [
		{ id: uuid(), email: "[e-mail]" },
		{ id: uuid(), email: "[e-mail]" }
	]
})

When skipDuplicates: true, if a record already exists it, the record will be ignored. This feature is only supported by databases that support INSERT ON CONFLICT DO NOTHING.

Currently, Microsoft’s SQL Server does not fully support this feature, so we’ve disabled skipDuplicates.

Nested Create Many

• Nested create many operation within create and update operations.
• Accepts a type-safe array of records

create with createMany Example

const result = await client.user.create({
  data: { 
    // id autoincrement
    email: "[e-mail]",
    posts: {
      createMany: {
		skipDuplicates: true,
		data: [
	      { title: "1st post", body: "" },
	      { title: "2nd post" }
        ]
      },
    },
    addresses: {
	  // We'll continue to support nested creates
      create: [
	    { zip: 12345 }
	  ]
    }
  }
})

One important distinction is that we will still continue to support a nested create: [...]. This is functionally similar to createMany: [...], but has a few subtle differences:

• create: [...] creates records one query at a time
• create: [...] supports nesting additional relations
• create: [...] cannot skip over duplicate records
• create: [...] supports has-many and many-to-many relations
• createMany: [...] inserts all the records in one query
• createMany: [...] does not support nesting additional relations
  • You can use foreign keys though (e.g. addressId: 10)
• createMany: [...] can skip over duplicate records
• createMany: [...] supports has-many but not many-to-many relations

Limitations

• No SQLite support
  • SQLite doesn’t support DEFAULT values on INSERT, so we decided to disable support for createMany for this release.
  • SQLite doesn’t have roundtrip network latency, so we’re not sure if createMany is all that much worse than create.
  • We’ll open an issue to discuss this further
• SQL Server can’t override a field that has @default(autoincrement())
  • You most likely won’t run into this because you don’t usually want to manually override a field that auto-increments.

Out of Scope

• Bulk upserts
  • The mechanisms that govern upsert are not applicable to a many operation.
  • We’d have to rewrite it to be a completely separate operation (doable, but out of scope).
• Return newly created records from a createMany.
  • Only Postgres supports RETURNING natively.
  • Workaround: If you need to read the records from a createMany, call findMany after.
• select and include options not supported
  • Since we don’t return newly created records, select and include operations do not apply.
• Nested create, connect, connectOrCreate
  • data won’t allow nested relations to be created or connected.
• Batch inserts across unrelated models. You can only batch one type of record at a time. (e.g. no support for prisma.createMany([post, update]))
  • Example: you can’t batch insert a user and an unrelated post.
  • This doesn’t apply to related models where you can use nested writes.
• Copying data into the table even faster by omitting integrity checks. (e.g. COPY IN)
  • Some databases offer “copy in” functionality. This is a separate feature.
  • Import and Export

FAQ

How do I get the records back after a createMany?

To get the records back, you can use a findMany after your createMany. This will require you to findMany by a unique field in the record or precompute your IDs.

So for the following Prisma Schema:

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

generator client {
  provider = "prisma-client-js"
}

model User {
  id    String @id @default(uuid())
  email String
}

You can pre-compute the IDs and search by them after:

import { PrismaClient, User } from '@prisma/client'
import { v4 as uuid } from 'uuid';

const client = new PrismaClient()

async function main() {
  const records: User[] = [
    { id: uuid(), email: "[e-mail]" },
    { id: uuid(), email: "[e-mail]" }
  ]

  const result = await client.user.createMany({
    data: records
  })

  await client.user.findMany({
    where: {
      id: {
        in: records.map(record => record.id)
      }
    }
  })
}