[Proposal] New migrate primitives to resolve failed migrations

tl;dr

Migrations can sometimes fail and this usually leads to situations tricky to solve. Some solutions offer automatic rollbacks but we believe they give a wrong sense of security as rollbacks might or might not succeed.

As we still wanted to improve our support around failed migrations, we decided to propose some new tooling around that and this document explains what we have in mind.

Please read through the proposed solution below and help us understand if this would be helpful to you.

For feedback, simply leave a comment. Alternatively, book 30 minutes with us.

Example of a failed migration

Imagine the following scenario: You created a migration with prisma migrate dev which was saved in a *.sql file containing several SQL statements. These statements are:

ALTER TABLE `Users` DROP `birthDate`;
ALTER TABLE `Users` ADD CONSTRAINT  `unique_emails` UNIQUE (`email`);
ALTER TABLE `Users` ADD COLUMN `signup_date` Date;

Applying the migration succeeded in your dev environment and now it is time to release to production. Unfortunately, the operation fails, since there is non-unique data in your production database (e.g. two users with the same email address).

Migrate returned the error directly from the database which lets you know about the violation of the unique constraint.

ERROR 1062 (23000): Duplicate entry '[e-mail]' for key 'unique_emails'

You now need to recover manually from the partially executed migration (“partially” as the removal of birthDate from table Users was successfully executed).

Automatic rollbacks

Other tools sometimes auto-generate rollbacks. The problem with rollbacks is that they would not be of help here as your migration partially applied already. You now might argue that transactions could have helped but unfortunately transactions are not supported across different databases for DDL statements (SQL statements altering the schema). Additionally, they also come with performance penalties especially on larger datasets.

Proposal: Migrate Primitives

We could offer migrate primitives to resolve failed migrations. Fundamentally you have two options how to proceed, forward and backwards:

• fix forward: apply the failed part of the migration or
• fix backwards: abort the migration and go back to the state before trying to run the failed migration

The two new commands we are using to fix the scenario from above are:

prisma db execute

# Applies a SQL script to the database without interacting 
# with the prisma migrations table.
prisma db execute --target myDB --sql script.sql

prisma migrate diff

# Diffs two diffable targets (prisma schema, database schema or
# migrations history). Output transforms one schema into another
# and can be human-readable or an executable SQL script.
prisma migrate diff --from $PRODUCTION_DATABASE_URL --to prisma/schema.prisma --output fix-migration.sql

Move Forward

The existence of non-unique data in your database is unintentional and you want to fix that. Once fixed, you would be able to go ahead with the migration as planned:

• You need to figure out which rows contain duplicate values
• You need to delete / alter them to be able to apply the unique constraint
• You then can continue applying the rest of the failed migration
  • You migrate diff from: prodDB to: prismaSchema output: migrationScript
  • You db execute the resulting script against your prod database
  • You db migrate resolve —applied to mark the previously failed migration as succeeded in prod

Your local migration history now yields the same result as the state your prod db is in.

Move Backwards

You realize that non-unique data is valid and you cannot move forward with the migration with the unique constraint included:

• You need to create a migration that takes your prod db to the state of your datamodel before the last migration
• You get the previous datamodel from git, or you get it from the local migrations history
  • You migrate diff from: prod to: prev datamodel output: migration script
  • You db execute that migration against prod.
  • You db migrate resolve —rolled_back the failed migration marking it as rolled back.
  • You now delete the folder of the failed migration to prevent it from being run locally again.

Your local migration history now yields the same result as the state your prod db is in. You can now modify the datamodel again to create a migration that suits your new understanding of the feature you’re working on (with non-unique mail addresses).

Conclusion

Resolving failed migrations is hard as every situation is different. Often, there is no automatic fix available through rollbacks. We believe offering the described commands gives users the right level of control to address their specific needs.

We can’t wait to hear what you think.

For feedback, book 30 minutes with us or leave a comment.

Thanks

Edit 1: Updated diff example

[do4gr]

We’ve spend some time fleshing out the design for the proposed commands. Below is what we plan to implement for the initial version. Please let us know if you have any more feedback.

prisma db execute

Helptext

Preview Feature (Warning)

Run a database native database command on the specified database. You need to specify the datasource and the command input.

The output of the command is connector-specific, and is not meant for returning data, but only to report success or failure.

On SQL databases, this command takes as input a SQL script. The whole script will be sent as a single command to the database.

This command is currently not supported on MongoDB.

Examples:

# Standard input as source and explicit database url.
$ echo 'SELECT `hello world`;' | \
  prisma db execute \
  --preview-feature \
  --stdin \
  --url="mysql://home/testdb"

# SQL file as input and database url from schema.
script.sql
 'SELECT `hello world`;'

$ prisma db execute \
  --preview-feature \
  --file-path ./script.sql \
  --schema schema.prisma

Arguments

• The datasource:
  • --url: the URL of the datasource to run the command on.
  • --schema: path to the Prisma schema file to take the datasource URL from.
  • Exactly one of --url or --schema must be provided. None is an error, and more than one is also an error.
• The command input:
  • --file-path : the path towards the file containing the commands
  • --stdin : take input from stdin
  • Exactly one of --file-path or --stdin must be provided. None is an error, and more than one is also an error.

Output

• Free form, depending on the connector.
• Exit code:
  • 0: ok
    • 26 statements executed in 1.4s? (MySQL only)
  • 1: error (red)

Error Handling

• engine will error on mongoDB for example

prisma migrate diff

Helptext

Compares the database schema from two arbitrary sources, and displays the differences either as a human-readable summary (by default) or executable script (For SQL databases: SQL, not supported on MongoDB).

[Link to expanded use cases in docs]

prisma migrate diff is a read-only command that does not write anything to any database (except the shadow database in special cases, see below).

A diff command takes a source (--from) and a destination (--to) and compares the source with the destination to generate the diff. The diff can be interpreted as generating a migration that brings the source schema (from) to the shape of the destination schema (to). The following targets are supported:

• --from-url / --to-url : a connection string pointing to a live database.
• --from-empty /--to-empty : an empty schema (takes no parameter)
• --from-datamodel/ --to-datamodel: the datamodel in a schema file (takes a path).
• --from-schema/--to-schema: the database specified via the datasource in a schema file (takes a path).
• --from-migrations/ --to-migrations: a migrations directory path used with Prisma migrate.

prisma migrate diff will ask you to provide a --shadow-database-url argument when necessary. For example when you diff migration directories with --from-migrations or --to-migrations, Prisma needs a shadow database to determine a schema from the migration files. You can read more about shadow databases at [link]

Examples:

Starting point options:
--from-empty
--from-schema prisma/schema.prisma
--from-datamodel prisma/schema.prisma
--from-url postgres://....
--from-migrations ./migrations...

End point options:
--to-empty
--to-schema prisma/schema.prisma
--to-datamodel prisma/schema.prisma
--to-url postgres://....
--to-migrations ./migrations...

Shadow db:
optional in most cases
--shadow-database-url "$SHADOW_DB"

Output format:
defaults to human readable summary on stdout
--script will render a sql script to stdout

Examples

# From database to database as summary
# e.g. compare two live databases
$ prisma migrate diff \
  --preview-feature \
  --from-url "$DATABASE_URL" \
  --to-url "postgresql://login:password@localhost:5432/db2"

# From empty to current datamodel as a SQL script
# e.g. to squash migrations
$ prisma migrate diff \
  --preview-feature \
  --from-empty \
  --to-datamodel schema.prisma \
  --script

# From current database specified in the schema to empty as a SQL script
# e.g. to teardown a database
$ prisma migrate diff \
  --preview-feature \
  --from-schema schema.prisma \
  --to-empty \
  --script

# From a live db to a Prisma datamodel
# e.g. roll forward after a migration failed in the middle
$ prisma migrate diff \
  --preview-feature \
  --shadow-database-url "$SHADOW_DB" \
  --from-url "$PROD_DB" \
  --to-datamodel=next_datamodel.prisma \
  --script

# From a live database to a datamodel
# e.g. roll backward after a migration failed in the middle
$ prisma migrate diff \
  --preview-feature \
  --shadow-database-url "$SHADOW_DB" \
  --from-url "$PROD_DB" \
  --to-datamodel=previous_datamodel.prisma \
  --script

# From migrations directory to another database 
# e.g. generate a migration for a hotfix already applied on production
$ prisma migrate diff \
  --preview-feature \
  --shadow-database-url "$SHADOW_DB" \
  --from-migrations ./migrations \
  --to-url "$PROD_DB" \
  --script

Arguments

• Diff targets:
  • --from / --to
    • schema — url from schema, takes an optional file path argument, but defaults to the project’s schema.prisma
    • datamodel — datamodel from schema, takes an optional file path argument, but defaults to the project’s schema.prisma
    • empty — an empty database
    • url — the schema of the database behind the url
    • migrations — can also target up to a specific migration (later)
• --shadow-database-url:
  • required when you use --from-migrations or --to-migrations
  • or are on a cloud provider for example
• --script
  • defaults to best-effort human-readable summary of the diff
  • if --script is passed - a script executable on the datasource to bring from --from to --to

Output

• The summary or script goes to stdout. This is the only output in the success case.
• Exit code:
  • 0: ok
  • 1: error (red)

[floelhoeffel]

hey @maoosi - we envision diffable to support:

• Prisma schema files
• Migrations history (e.g. files in your migrations folder)
  • maybe allowing to specify up to which migration e.g. HEAD-1 or something similar
• any database we support

Diffing between databases would work as long as the connector is the same. E.g. you could diff a MariaDB and a MySQL database as they use the same prisma connector under the hood.