Add support for alembic migrations

Scrolling through the issue as I was setting this up, I found the following that appear to be relevant. #43 #7 #20 and #39 .

Basically it comes down to the way that the triggers are added into the DB is via the after_create signal in SQLAlchemy. This unfortunately only triggers the first time a table is created and it only triggers when you are using the table metadata.

Working with Alembic, Flask-Migrate or other wrappers around alembic this means that the postgres triggers are never properly added and as such nothing ever gets added to the activity table.

#46 is my first pass at attempting to resolve this problem. Its not perfect but it is a start.

Basically at this point all the SQL calls back been removed to external functions from the base VersionManager. In addition, new functions have been added to the migrations file.

manually adding init_before_create_transaction, init_activity_table_triggers, rollback_create_transaction and register_table allow most things to function.

Basically the first migration will look something like this:

def upgrade():
    # ### commands auto generated by Alembic - please adjust! ###
    op.execute(text("CREATE EXTENSION btree_gist;"))
    init_before_create_transaction(op)

    op.create_table('ipam_vrfs',
    sa.Column('DateCreated', sa.DateTime(), nullable=True),
    sa.Column('LastModified', sa.DateTime(), nullable=True),
    sa.Column('id', sa.Integer(), nullable=False),
    sa.Column('name', sa.String(), nullable=True),
    sa.PrimaryKeyConstraint('id')
    )
    op.create_table('transaction',
    sa.Column('id', sa.BigInteger(), nullable=False),
    sa.Column('native_transaction_id', sa.BigInteger(), nullable=True),
    sa.Column('issued_at', sa.DateTime(), nullable=True),
    sa.Column('client_addr', postgresql.INET(), nullable=True),
    sa.Column('actor_id', sa.Text(), nullable=True),
    postgresql.ExcludeConstraint((sa.column('native_transaction_id'), '='), (text("tsrange(issued_at - INTERVAL '1 hour', issued_at)"), '&&'), using='gist', name='transaction_unique_native_tx_id'),
    sa.PrimaryKeyConstraint('id')
    )
    op.create_table('activity',
    sa.Column('id', sa.BigInteger(), nullable=False),
    sa.Column('schema_name', sa.Text(), nullable=True),
    sa.Column('table_name', sa.Text(), nullable=True),
    sa.Column('relid', sa.Integer(), nullable=True),
    sa.Column('issued_at', sa.DateTime(), nullable=True),
    sa.Column('native_transaction_id', sa.BigInteger(), nullable=True),
    sa.Column('verb', sa.Text(), nullable=True),
    sa.Column('old_data', postgresql.JSONB(astext_type=sa.Text()), server_default='{}', nullable=True),
    sa.Column('changed_data', postgresql.JSONB(astext_type=sa.Text()), server_default='{}', nullable=True),
    sa.Column('transaction_id', sa.BigInteger(), nullable=True),
    sa.ForeignKeyConstraint(['transaction_id'], ['transaction.id'], ),
    sa.PrimaryKeyConstraint('id')
    )
    op.create_index(op.f('ix_activity_native_transaction_id'), 'activity', ['native_transaction_id'], unique=False)

    init_activity_table_triggers(op.get_bind())
    register_table(op, "ipam_vrfs", ["DateCreated", "LastModified"])

    # ### end Alembic commands ###


def downgrade():
    # ### commands auto generated by Alembic - please adjust! ###
    rollback_create_transaction(op)
    op.drop_index(op.f('ix_activity_native_transaction_id'), table_name='activity')
    op.drop_table('activity')
    op.drop_table('transaction')
    op.drop_table('ipam_vrfs')
    # ### end Alembic commands ###

All future migrations will need lines like register_table(op, "ipam_vrfs", ["DateCreated", "LastModified"]) added for all tables that have had either their `excluded_columns modified or a new table for revision tracking is added.

Excluded columns are explicitly added and not read from the models to allow full history tracking of what was ignored.

[wakemaster39]

PR has been updated to include support for alembic autogenerated migrations. To support this, __versioned__ needed to be moved into the __table_args__['info'] dict under the versioned key. This appeared to be an acceptable move by comments in #7.

To enable alembic support in your env.py file you need to add a couple lines.

import postgresql_audit.alembic

postgresql_audit.alembic.writer.process_revision_directives needs to be passed in to the context.configure call under the process_revision_directives keyword arg.

If you are using flask-migrate, process_revision_directives function can be slightly modified to

    def process_revision_directives(context, revision, directives):
        if getattr(config.cmd_opts, "autogenerate", False):
            script = directives[0]
            if script.upgrade_ops.is_empty():
                directives[:] = []
                logger.info("No changes in schema detected.")
                return
        postgresql_audit.alembic.writer.process_revision_directives(context, revision, directives)

[vvvrrooomm]

op.execute(text(“CREATE EXTENSION btree_gist;”))

this needs superuser privileges. [1] suggests to use the following instead: op.execute(text("CREATE EXTENSION IF NOT EXISTS btree_gist;"))

[1]https://stackoverflow.com/a/51395480/10617111