The new roadmap to v1.0.0

Foreword

This year is special to me, since I am finally able to provide more of my time to open source. First of all due to the commitment https://github.com/db-migrate/node-db-migrate/issues/605 of sponsoring time and money from my own company (which currently means effectively plan in whole days/weeks of time that I do not spend on paid projects and invest those into open source instead). This is important, since nothing is free in this world and this is my new way, to give something back to the open source community, since a lot changed for me and I do not have as much free time as I wish I would have.

Further I added opencollective and sponsoring slots, from which funds also solely progress and work on db-migrate will be paid. So this way also people that are not able to contribute to the project in terms of providing PRs and their workpower, can contribute to the success and further development by providing resources otherwise.

That being said, i am very grateful for all the contributors of the past and of the future of any kind. All support matters!

The roadmap to v1.0.0

That being said, the new roadmap which we try to achieve over the next few month consists out of the following:

• Adding a concurrency manager
• Adding a state manager
• Adding internal schema representation
• Make db-migrate learn about the schema
• Introduce the new v2 migration schema
• Rework db-migrates structure and components
• Deprecate node <8
• Add chainable steps for new migration schemas
• Add support for multiple driver instances
• Add new methods to the drivers for the state table creation (this time without any future need to adjust again)
• Make sql migrations first class, but separate them into plugins (no more javascript loaders, wont be in the standard features anymore, and will be needed to installed once as a plugin)
• Adding new templates and move them to plugins
• Clearly define the new concept for seeders (in work, but will integrate more seamlessly than the current approach)
• Adding default columns to table creation, such as an indexed seeder key column to reference and control seeded content and as well support easier support for reproducible system in integration testing
• Add separate static seeds for very simple needs, without any version control (still making use of the default columns)
• Enhance the documentation in general
• Add the new features to the documentation
• Remove transitioner from the project

Work In Progress

The good news here are, a lot of this is done or in progress (master branch, not released yet). For example the structure has been reworked completely already. Yet I am thinking to also replace the CLI, but in favor of a shorter release time I might drop that into the next step. What is in its current state already ready is the new migration schema with a lot of new features. More to that later. The next step will be the templates and the default columns for table creation. After that I either go on the seeders or the concurrency manager, which highly depends on how the dependencies of this task resolve to each other (it turns out, most if not all tasks are somehow related to each other and require each other).

Also this is a very intensive rework and a lot of work put into thoughts and new processes, I am looking to keep the breaking changes to a minimum. In fact, I will never drop the version 1 migration schema. So expect that most should continue working, except that we’re completely dropping support for old node versions, to move the db-migrate code base to a more modern state. This should be fine for the majority, for everyone else that does not integrate the db-migrate api, this change shouldn’t affect to badly either. But it is finally time to move on and drop the support at this stage (also noting that node v6 is almost out of the maintenance cycle anyways.

The new migration schema

So the biggest thing in this release will be that I will finally clean up with the missing seeders and get the strings straight on this point and of course the migration schema. Which is leveling up the game a lot.

So to give you a preview of what is coming (and already working, if you’re feeling fancy checkout the master branch and pg driver which already carry support). Here is what it is about:

The migration schema v2

The migration schema v2 is a very strict new standard, in which ultimately only reproducible components are allowed to be executed. It adds support for atomicity without transactions, it is to a certain degree self healing and it massively decreases the complexity of creating migrations. It allows an as seamless process of migrating up, as rolling back. It clearly defines guidelines for zero downtime compatible migrations and some more compelling features which will incorporate the DDL as well as the DML.

To give you a more alive example, lets take a series of actually working v2 migrations.

So assume this two migrations:

Migration 1

'use strict';

exports.migrate = async (db, opt) => {
  const type = opt.dbm.dataType;
  await db.createTable('test', {
    id: {
      type: 'UUID',
      primaryKey: true
    },
    test: {
      type: type.INTEGER
    },
    test2: {
      type: type.INTEGER,
      notNull: true
    }
  });
  return db.createTable('test2', {
    id: {
      type: 'UUID'
    },
    test_id: {
      type: 'UUID',
      foreignKey: {
        name: 'namename',
        table: 'test',
        mapping: 'id'
      }
    },
    test: {
      type: type.INTEGER
    }
  });
};

exports._meta = {
  version: 2
};

Migration 2

'use strict';

exports.migrate = async db => {
  await db.dropTable('test2');
  await db.removeColumn('test', 'test2');
};

exports._meta = {
  version: 2
};

Explanation

So what you can see here is, migration 1 creates two tables and migration 2 deletes one of the table completely and and removes a column from the other one. The first thing you will notice is that the interface looks different. First of all there is no up and down anymore. Only a migrate and a required _meta export are provided. The latter will later also allow to introduce some more features like specific driver instances per migration later on. The second change in the interface is, that callbacks are dead. Instead the migrate function now gets passed the driver and an options object. The options object contains the objects you used to retrieve through the setup function and might also hold some other toolings in the future.

So all in all the migrations look and feel very simple now and you might ask, where is my definition of rolling back? The answer is, there is none and you don’t need any. The reason for this is the same reason why the new standard is so strict and the same reason why the v1 standard will keep alive forever. Certainly everything can be covered in the v2 schema, but in the meantime it will always be possible to execute operations that are not yet executable on the v2 schema. But that aside. Why you don’t need the down operation anymore?

db-migrate got a few new components exclusively available to the v2 schema (with some exceptions such as manual schema feeding which will be available for inter compatibility to the v1 as well). These components are for example a component that learns about the manipulation of the DDL you’re applying currently and another that stores the state of the new manipulated schema and information about the alteration that has been made. When you now execute an action in the new schema you’re not accessing the driver directly anymore. Instead db-migrate will inject a chain of actions before it sends your operation finally off to the driver beneath. This allows db-migrate not only to intercept but also to correct actions. For example the new migration v2 schema also comes with failureStrategies. Currently the only available and also the defacto standard is rollback the current migration on failure automatically. Should the migration being interrupted by for example a failure of the database in the process of migrating, then db-migrate will be able to recognize this on the next run and rollback the executed actions before continuing by default. The rollback is being recreated from the stored information. That also means, when you decide to delete a table for example, db-migrate knows exactly how your table looked before it was deleted and can completely automatically revert to the before state. An exception to this, is of course the data in the table. This is why a dropTable option is not safe by the zero downtime guidelines and a new deprecateX function for tables and columns will be added to the driver. Those will support a full zero downtime compatible cycle. Which will consist out of the following step:

• 1. mark the entity to be deleted on the first round. Now you will make your first release in which you will implement support for the new schema coming up.
• 1. rename the entity with an unique name Now you will release the new version which uses the new schema instead and your old version already respects this in which way ever necessary.
• 1. after a cycle of by default 4 releases (including the 2 before already) db-migrate will remove the table

These deprecation functions will also get later on support for signaling from the application directly, to shorten the process and they require information about a new release. For this there will be _meta export signal to define a certain migration as a start for a new release. This also might be improved in the future, since there are plans to also implement new migration patterns, to organize migrations in features and versions directly (for example a long requested feature like semver versioning might become reality, but this is yet to be defined).

Some final words about the standard itself. There will be a definition of the standard, which might be extended with features, as long as it is not altered. So already defined components might only change in a new version of the schema, which shall be avoided when possible. But new components might be added to make it more feature rich.

Some more notes

As always I want to express, I welcome any help. Be it documentation, implementation or helping out others with their issues. I want to thank all the people that helped in the process and all the contributors that added value to the project and I am looking forward to make db-migrate a migration framework that is not staying in your way, but helping you to do the right things and get the things done. Especially important to me is to search for a way to transfer knowledge, especially on the zero downtime section. I don’t know yet, how I best accomplish this, but I consider this being even more important than everything else since I might provide a powerful tool, but what does it help you if you don’t know how to handle it. Whoever has ideas or wants to help out on that section. I greatly appreciate your input!

Post v1.0.0

So to answer the question, what comes post v1.0.0. I can’t answer yet, but there will always be room for improvement and i want to encourage you the community, to take active part in creating meaningful value for the project.

Notable references

• #508
• #324
• #401
• #518
• #538
• #539
• #537
• #514
• #464
• #338
• #233
• #215
• #230

Issues to be checked further

• #664

<bountysource-plugin>

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource. </bountysource-plugin>

[wzrdtales]

some significant progress lately.

• templates are there now instead of any templating language, we will just let the plugins decide how to render the template. a hook for cli options will be added as well. Internal templates will just use es6 templating and we move each template to its dedicated file.
• v2 schema is maturing and a lot of bugs are fixed, it is being used in production right now in some of my biggest projects and clients. still in dog feeding phase, but the worst bugs have been identified and fixed by now
• state manager is fully working by now, just missing the concurrency manager

next up is

• concurrency manager
• seeder concept (and maybe implementation already)

[dimitriy-tomilovskiy]

@wzrdtales Any news on concurrency manager? It could be mighty useful for some of the things we are looking to do.