Migrations: reset & improvements

Migration Refactoring

Migrations need revisiting for the alpha, they’ve currently effectively been removed, only a fresh DB population will work. We are going to refactor some parts of our migration system. The goal is to extract the execution of migration out of Ghost. The Ghost-CLI tool will help us achieving our goal.

Using third party tools

https://github.com/db-migrate/node-db-migrate

Cons:

• https://github.com/db-migrate/node-db-migrate/issues/410

http://knexjs.org/#Migrations

Pros:

• transactions are configurable, which is nice (config.transacting=true)
• can be used via shell out of the box

Cons:

• how to make a backup before running the migrations?
• how to execute from JS level?

We already have a full working migration system, i personally don’t see any reason to switch to a third party tool. Migration functionality like seed, create, up, down are in general not hard to implement. The whole wrapping logic around is important.

I am open for opinions. We can also make an experimental testing with knex migrate!

Creating the database

Right now Ghost never controlled creating the database (in case of mysql).

I would love to get support for that.

So before populating the database (see next section), we could execute a command schema.commands.createDatabase (notation might change).

Mysql will execute a raw query to the database, sqlite3 will just skip.

We can control with which encoding we create the database. If database exists, it just skips.

Populating the database

When Ghost-CLI installs a Ghost Version and starts Ghost, it can use an internal Ghost-CLI command before starting the application to check if the database is initialised. This command can be used as single shell command as well.

ghost db —init

Will create all needed tables.

If database is already populated, it just throws/shows an error.

The code for running db —init will still live in Ghost.

So it’s for now actually a shortcut for node core/server/data/migration/bin/runner (or similar).

Running populations in a transaction

We need that. We had trouble in the past, that a database was in a broken state, because a container was destroyed while running the populations. (already PR’d). This is also important for LTS . Does not work in mysql, but we have a workaround PR for it.

Seed the database

In Ghost we have a set of initial data, which we call fixtures. (Welcome post, owner, permissions). This can/should be treated as a migration. It is the first migration for Ghost.

Running migrations

Options for inter process communication:

Fork process and communicate via node communication channel.

https://github.com/weixiyen/messenger.js

https://github.com/RIAEvangelist/node-ipc

When Ghost-CLI starts Ghost, it can receive an event.

Ghost can trigger an event like process.emit('started') (or similar) , which Ghost-CLI listens on. So when Ghost is started, Ghost-CLI can execute an internal command to execute migrations. This command can be used as single shell command as well.

Ghost-CLI executes the migrations and if migrations needs to happen, it will send an event to the Ghost child process to boot into maintenance mode.

If finished, it will send another event to boot into normal mode. The whole inter process communication is for now optional, we can also just hack it so it works.

ghost db —migrate

(or even ghost db —migrate —version XXX )

Database versions

Right now we have used the naming pattern 001 - 00X .

Database versions and Ghost source code are very tide together.

Each Ghost Version depends on a specific database schema. So there is only a handful database versions per Ghost Version.

I would keep numbers, but extend/change them a bit:

1.0-1 - is the first migration for Ghost Version 1.0.X

1.0-2 - is the second migration for Ghost Version 1.0.X

1.1-1 - is the first migration for Ghost Version 1.1.X

We don’t do migrations for patch updates, so we don’t need the whole Ghost version.

Combine fixtures and normal migrations into one folder.

Yes i think that is a good idea.

Make Ghost updates smarter!

Right now, if an error occurs while updating from 1.0.0 to 1.10, your database rolls back to 1.0.0-2 and you cannot start Ghost anymore in worst case.

This can be avoided by making the update process different:

When updating a Ghost Version via Ghost-CLI (ghost update ), we are building a new folder with the new version next to your current version.

We copy the database (in case of sqlite), the config files and the content folder. Then Ghost-CLI will bring the current (!) process into maintenance mode to avoid getting new data. Then Ghost-CLI will execute the migrations on the new folder, if available. If it was successful, the new current version is the new Ghost Version. If it was not successful, database commands rollback and current version stays. So the blog will still run in old version without any trouble and same source code.

Auto updates for Ghost: In case of auto updates, this strategy won’t work.

Migrate down?

You have Ghost running with version 1.5.0. And you would like to use your current database with Ghost 1.4.0. So if you install Ghost 1.4.0 and you copy the database of 1.5.0, it should downgrade the database. So each migration file get’s an up and down function. That is an optional idea.

Architecture changes?

Basically not so much will change. The biggest change is executing migrations and population from outside and making fixtures being a migration We will tidy up the files a bit (fixtures, bootup etc). Effort is still 2-3 days to have everything in place.

Tasks:

• Running database population in a transaction
• Reset migrations (remove all old migrations)
• Change the versioning
• add migration runner
• seeding
• populate tables
• knex-migrator migrate
• reset/backup
• knex-migrator as npm
• integrate with Ghost-CLI
• integrate knex-migrator into Ghost completely
• look for kate-migrations in the code
• remove old migrations code 🕵🏻
• look at https://github.com/TryGhost/knex-migrator/issues/13
• test import/export database
• hooks audit
• rename .knex-migrator file
• revert https://github.com/TryGhost/Ghost/pull/7591
• look at https://github.com/TryGhost/Ghost/issues/7615
• fix https://github.com/TryGhost/knex-migrator/issues/26
• simulate using the old migration tasks with knex-migrator
• think about a nested folder “upgrades” and then have nested folders for each version

@acburdine I would like to hear your opinion 😃

[ErisDS]

On 3rd party solutions:

The major downside to knex migrations is that it requires you to cycle through the migrations step by step in order to create a fresh DB. This is designed for long-running single-install software, rather than mass distributed & installed software like Ghost. If we switched to knex migrations in its current form, fresh installs would get slower and slower as time went on.

Additionally, as much as I like the idea of knex seed files, knex only supports the concept of there being one seed. Which is kind of annoying, because I could see us using it for both the original DB seed, and for seeding various tests.

Those are my two major reasons for not switching to it. I wrote some more stuff in this issue ages ago but I’m leaning towards keeping our tailor-made solution, after all we have it now and it doesn’t need much.

---

On how the Ghost-CLI will interact with migrations,

The steps for the cli would be something like:

• install the new version of ghost
• before switching the versions over, use a predetermined method to see if a migration is required
• if no, switch. Done!
• if yes, then:
• restart the current Ghost process with the maintenance mode flag
• run the migration
• then switch, starting the new process in normal maintenance mode
• if something goes wrong with the migration, start the old ghost version in normal mode, and show an error to indicate the upgrade failed

The addition of a Process.emit() on startup is a nice-to-have so that the CLI can tell if everything is ok or not.

This leaves 2 slightly tricky things:

• how to determine if a migration is needed
• how to run a migration

---

Other things:

• migrate down is a 👎 from me, I think it is an edge case and just IMAGINE the manual testing you’d need to do. Also, no one has ever asked for it 😉
• new versioning is 👍
• Combine fixtures and normal migrations into one folder 👍

[ErisDS]

@kirrg001 have created this as an issue so that we can start to document what is gonna be done - feel free to edit it with tasks and details 👍