Allow current.sql to be split into several files

Hello,

After working for more than 1 month with graphile migrate, I would like to be able to split a big current.sql into several files smaller files. In practice, it’s actually more useful than named commits as I initially requested in #28.

During the initial development phase of a project, we don’t want to commit/track migrations. Our schema changes a lot, so we don’t want to start building a messy migration history. We’d rather share a big current.sql between the development team, and commit a clean initial migration when our schema is stable. Also during that time, we don’t release anything on a production/public environment. We’d rather enjoy the flexibility of being able to reset the database (+run the seeds) every time current.sql changes.

It works great so far, except for the fact that current.sql grew to more than 2k lines, which is not great for readability. It would be useful to be able to split current into smaller files. For instance custom_1_utils.sql, current_2_create_user.sql, current_3_mutations.sql, etc.

This feature is much needed for us so I plan to work on it. If you feel it’s too specific or beyond the scope of migrate I will just make a fork. But if you’re interested in a PR, here’s how I see the specs:

• Automatically detect between a single or multiple current files, do not allow both;
• Not breaking, optional addition;
• Filename for split current is current_[int]_[description].sql;
• [int] defines the order in which the sql files are run (doesn’t have to be sequential, but must be >= 1 and unique)
• [description] is just a text without any implication;
• On commit, it merges all current_ files into a single sql commit. The commit file is formated in such way that it’s possible to re-create all current_ files on uncommit.

I didn’t think too much about the format of the commit. It definitely needs to be specific enough so it doesn’t clash with a user’s own comments. For instance:

--! Previous:
--! Hash:
--! Merged:
--!   1: utils
--!   2: users
 
--! Part: 1 - utils
...

--! Part: 2 - users
...

Let me know what you think. Thanks!

[benjie]

current/ folder, split on --! split, just numbers, names optional.

[hcharley]

fwiw, I wrote a small script for this:

import { readdirSync, readFileSync, writeFileSync } from 'fs';
import { resolve, join } from 'path';

import { textSync } from 'figlet';

const run = () => {
  const GENERATED_FILE_NAME = 'current.generated.sql';

  const PATH_TO_MIGRATIONS_DIR = resolve(
    join(__dirname, '../../..', 'migrations')
  );

  const GENERATED_FILE_PATH = resolve(
    join(PATH_TO_MIGRATIONS_DIR, GENERATED_FILE_NAME)
  );

  const PATH_TO_CURRENT_DIR = resolve(join(PATH_TO_MIGRATIONS_DIR, 'current'));

  const COMMENT_DIVIDER_LENGTH = 60;

  const COMMENT_DIVIDER = `${Array.from({ length: COMMENT_DIVIDER_LENGTH })
    .map(() => '-')
    .join('')}\n`;

  const replaceNewLines = (str: string) => {
    return str.replace(/\n/gm, `\n${pfix}`).replace(/^\s/, `${pfix}`);
  };

  const pfix = '----    ';

  const toCommentHeader = (content: string) => {
    return `${COMMENT_DIVIDER}${COMMENT_DIVIDER}${replaceNewLines(
      content
    )}\n${COMMENT_DIVIDER}${COMMENT_DIVIDER}\n\n`;
  };

  const readdirSyncWithPath = (path: string) => {
    return readdirSync(path).map((n) => join(path, n));
  };

  const pathToFileName = (path: string) => {
    return path.replace(PATH_TO_CURRENT_DIR, '').replace(/^\//, '');
  };

  const getFileNumberFromPath = (path: string) => {
    const fname = pathToFileName(path);
    const idx = fname.split('-');
    if (typeof idx[0] !== 'string') {
      throw new Error(
        'Filename does not start with a string/number followed by a hyphen'
      );
    }
    try {
      return parseInt(idx[0], 10);
    } catch (e) {
      throw new Error(
        'Filename does not start with a string/number followed by a hyphen'
      );
    }
  };

  const mergeFilesWithPath = (path: string) => {
    return readdirSyncWithPath(path)
      .sort((a, b) => {
        return getFileNumberFromPath(a) - getFileNumberFromPath(b);
      })
      .reduce((acc, fpath, idx) => {
        const body = readFileSync(fpath).toString('utf-8');
        const fname = fpath.replace(PATH_TO_CURRENT_DIR, '').replace(/^\//, '');
        const header = toCommentHeader(`---- File #${idx + 1}: ${fname}`);
        return `${acc}${header}${body}\n\n`;
      }, '');
  };

  const GENERATED_HEADER = toCommentHeader(
    `${textSync(`Current.sql`, {
      font: 'Soft',
    })}`
  );

  //;
  const MERGED_CURRENT_BODY = mergeFilesWithPath(PATH_TO_CURRENT_DIR);
  const GENERATED_BODY = `${GENERATED_HEADER}\n\n${MERGED_CURRENT_BODY}`;

  // eslint-disable-next-line no-console
  console.log(
    'Writing to',
    GENERATED_FILE_PATH,
    'With:',
    `\n\n${GENERATED_BODY}`
  );

  writeFileSync(GENERATED_FILE_PATH, GENERATED_BODY);
};

run();