Separating the partial-rollback from roll forward in migration scripts

Problem statement / motivation

As a new user introducing graphile-migrate to a team, I’ve noticed that code review of migrations is complicated by the need to thoroughly check the DROP statements that are required for idempotency. This is a point of friction for introducing the tooling, as people responsible for operational databases are nervous about having migrations containing a lot of DROP statements. It seems we currently have the following tradeoff

• If someone writes an errant DROP statement for a resource in the existing schema, data loss will occur in production.
• However, various DROP statements are a necessary part of idempotency and having graphile-migrate watch work during development.

Feature description

I find that it’s often possible to separate migrations into two halves which seem like they could be treated differently in production and development:

1. The DROP (or other) statements which remove any resources previously created during development. This is like a partial rollback which is happy to destroy data, and only needs to get the DB in a state where the roll forward phase can reliably be reapplied.
2. The roll forward migration statements needed for production.

If graphile-migrate guaranteed to only apply (2) in production (and for the shadow db rollforward check phase) it would seem unnecessary for (1) to be thoroughly vetted in code review, and reduce overall risk of applying a migration.

I’m not sure what this feature would look like. Could it be a comment which separates migration files into two sections? That seems a little fragile (eg, what if the comment is spelled wrong?), but it would be minimally breaking.

Supporting development

I could potentially help build this.

• am interested in building this feature myself
• am interested in collaborating on building this feature
• am willing to help testing this feature before it’s released
• am willing to write a test-driven test suite for this feature (before it exists)
• am a Graphile sponsor ❤️
• have an active support or consultancy contract with Graphile

[c42f]

we could even expose it in an envvar and move rather than delete

True! I think this is the right idea.

[benjie]

Of course, your beforeCommit could also figure out what the next commit number will be (we could even expose it in an envvar) and move rather than delete e.g. migrations/rollbacks/000036.sql.

We could even have an afterUncommit that does the reverse. (I don’t think we have uncommit hooks currently, but they wouldn’t be much work to add I suspect.)