Imperative Migrations with a TypeScript DSL

Problem

Not all migrations can be declarative. For example, we need to add a custom index for performance reasons. We also need this as an escape hatch for anything prisma schema doesn’t support. Lastly, we also need this for data migrations.

This is very important for any serious apps using prisma migrate.

Suggested solution

A new command: prisma migrate create

This will create the necessary migration files. Similar as prisma migrate save but the files are just stubbed out.

Then the user will open the stubbed migration file and add their custom migration code.

Perhaps the imperative migration file can look something like this:

import {migrate} from '@prisma/migrate`

migrate({
  async up(prisma) {
    // SQL migration
    await prisma.executeRaw(`some SQL stuff`)

    // Data migration
    await prisma.someModel.create(/*..*/)
  },

  async down(prisma) {
    await prisma.someModel.delete(/*..*/)

    await prisma.executeRaw(`undo some SQL stuff`)
  }
})

[LMaxence]

TL;DR

From my own user perspective, and for a very basic first version of the feature we are discussing:

• I don’t really care about typings for this
• I don’t really care about using the prisma client for this
• I just would like to use JS/TS/whatever nice scripting language code to write modifications of my data, regardless of this data being typed or not
• I’m completely fine with writing raw SQL queries to do the part “fetching the existing data”, and “saving the migrated data”

Detailed point of view

I completely understand the problem with using the client to manipulate data across migrations, as typings may change over time and invalidate existing migrations.

That’s why I suggested a first basic approach of the issue, where this feature would have nothing to do with the Prisma client. The API we are talking about can be really simple and only go down to generated SQL statements. For instance:

ALTER TABLE "users" ADD COLUMN "something_new" VARCHAR NOT NULL;

could go down to

async up() {
   db.table("users").addColumn("something_new", {type: PSQLTypes.VARCHAR, notNull: true})
}

I agree this is not a big improvement, but that can become one once I can do stuff like:

async up() {
   await db.table("myEntity").addColumn("somethingPreiouslyStoredInAJsonbCol")
   const rows = await db.table("myEntity").select("id", "myHorribleJSONbColumn").rows()
   // I'm 100% fine with the two lines above being `$executeRaw` statements
   
   for (row of rows) {
       // This is what takes hours to do when manipulating complex data with SQL statements
       const { removedAttribute, ...remainingJSONb } = row.myHorribleJSONbColumn
       // I'm 100% fine with this line below being an `$executeRaw` statement
       await db.table("myEntity").update({
           set: {somethingPreiouslyStoredInAJsonbCol: removedAttribute, myHorribleJSONbColumn: remainingJSONb}, 
           where: {id: row.id}
       })
   }

}

The problem is not about writing structure migrations in SQL, I mean it’s totally fine to write ALTER TABLE statements, or UPDATE statements for scalar columns. However, things always get messy once I try to modify columns with a complex data structure (as I mentionned, JSONb that’s you I’m looking at 🤬).

[marek-hanzal]

Hi,

So one last thing: I made a solution compatible with Prisma Migrations:

• you can generate Prisma Migration (even with applying it)
• you can create your own TS migration (applying it by Umzug)
• migration history is saved in Prisma, so it will not break things and it’s maximally compatible with Prisma, basically like it is it’s feature
• there are a few limitations, but not critical; also it’s not battle tested, so there could be some problems

If anybody is interested, I will create a sample repo or something like that.

See ya!