How can I best monitor the execution of each migration in 3.x when we previously used the Announcer functionality in 1.4.0

Describe the question With 1.4.0 we used a custom IAnnouncer (code after body for further explanation) and we could capture all the data coming out of the migrations as they happened.

As best I can follow, this is all deprecated behavior so I should shift my paradigm. (If this is where I’m wrong, I can probably get away with registering my own IAnnouncer implementation and that’s that)

What we were tracking before is each migration, that it applied successfully, what was applied, etc, and then we logged that out as a final big bulk thing on our TeamCity logs, but on local machines we just dumped the relevant bits.

Now with IMigrationRunner.MigrateUp, I’m not sure I follow how to capture that information except by using a custom logger that looks for specific triggers or the like.

Documentation Pages You’ve Read So Far Honestly I’ve cloned the repo down for both 1.4.0 and 3.3.1 and I see where you’ve marked a ton of things deprecated and I think I have a handle on it but this is a big project to keep in the head, yeah? Especially comparing two vastly different codebases at this point. I’ve read a ton of posts and other people’s questions, and never kept track, but clearly I may have missed an obvious answer somewhere.

Expected benefit I would like to capture details in memory of each migration as a tiny encapsulation of what happened as well as the migration sequence number and SQL that was attempted to be applied, and if it errored or was successful, if it was up or down, etc. (This is so we can sort the errors and log concise recaps at the end, etc. Yay business needs.)

I hope the code to follow is helpful in understanding what we were doing before.

Information (please complete the following information):

• OS: Windows 10/2016/2019
• Platform: .Net Core 3.1.x
• FluentMigrator version: 3.3.1
• FluentMigrator runner: In process, similar to [https://github.com/fluentmigrator/fluentmigrator/blob/master/samples/FluentMigrator.Example.Migrator/Program.DependencyInjection.cs] but with a bit more plumbing around the edges 😄
• Database Management System: SQLServer, potentially others
• Database Management System Version: Sql Server Enterprise, Developer

Additional context

First: I’m using a custom ILoggerFactory and ILogger due to the rest of my setup and all the things it has to do (fetch packages, extract things, do validation, etc etc) with one of two loggers injected in Startup.cs, so I wasn’t really planning on using .AddLogging(lb => lb.AddFluentMigratorConsole()) because as per [https://github.com/fluentmigrator/fluentmigrator/issues/930] so long as I’m already supplying the ILogger etc, you don’t need me to register yours (besides, I want to intercept the logs, not write them to console only, etc etc, yay business needs). I can absolutely play around with the ILogger implementation I have, but I would rather keep those implementations cleaner, since they are just receiving messages and then doing the appropriate thing based on their implementation, as a good logger should.

Here are some selected files of how we were doing this in 1.4.0 FluentMigrator with an in-process runner

1.4.0 MigrationAnnouncer

using System;
using System.Linq;
using FluentMigrator.Runner;

namespace Our.Internal.Models
{
    /// <summary>
    /// The purpose of a MigrationAnnouncer is to print things appropriately during runtime.
    /// </summary>
    public class MigrationAnnouncer : IAnnouncer
    {
        private readonly MigrationResult _result;
        private Migration _migration = null;

        public MigrationAnnouncer(MigrationResult result)
        {
            this._result = result;
        }

        public void Heading(string message)
        {
            // We use an accumulator pattern so we can always modify the same migration in flight across many methods
            if (_migration != null)
            {
                _result.Migrations.Add(_migration);
            }

            // This indicates a new migration
            string[] split = message.Split(new[] { ' ', ':' }, StringSplitOptions.RemoveEmptyEntries); // Should have 3

            if (split.Length != 3)
            {
                // this indicates that we are not looking at <internal> migration, because <internal> migrations always have a triple-part
                // This is the migration sequence number (date-time string), the migration description, and the status of the migration (up/down)

                _migration = new Migration
                {
                    Description = message,
                    MigrationType = MigrationType.Unknown
                };
            }
            else
            {
                _migration = new Migration
                {
                    Description = split.Skip(1).First(),
                    MigrationType = string.Equals(split.Last(), "reverting", StringComparison.InvariantCultureIgnoreCase) ? MigrationType.Down : MigrationType.Up
                };
            }
        }

        public void Say(string message)
        {
            _migration.Say.Add(message);
        }

        public void Emphasize(string message)
        {
            _migration.Emphasize.Add(message);
        }

        public void Sql(string sql)
        {
            _migration.Sql.Add(sql);
        }

        public void ElapsedTime(TimeSpan timeSpan)
        {
            _result.ElapsedTime = timeSpan;
        }

        public void Error(string message)
        {
            _result.Errors.Add(message);
        }

        public void Error(Exception exception)
        {
            _result.Errors.Add(exception.ToString());
        }

        public void Complete()
        {
            if (_migration?.Sql != null && _migration.Sql.Any(x => !string.IsNullOrWhiteSpace(x)))
            {
                _result.Migrations.Add(_migration);
            }
        }

        public void Write(string message, bool escaped)
        {
            _migration.Write.Add($"Escaped: {escaped} Message: {message}");
        }
    }
}

1.4.0 Migration (ours, not yours)

using System.Collections.Generic;

namespace Our.Internal.Models
{
    /// <summary>
    /// A tracking model used to help keep track of what was done
    /// </summary>
    public class Migration
    {
        public Migration()
        {
            Sql = new List<string>();
        }

        /// <summary>
        /// The migration description
        /// </summary>
        public string Description { get; set; }

        /// <summary>
        /// The actual sql commands run by the target migration
        /// </summary>
        public List<string> Sql { get; set; }

        /// <summary>
        /// The type of migration run, such as "up"
        /// </summary>
        public MigrationType MigrationType { get; set; }

        public List<string> Say { get; set; } = new List<string>();

        public List<string> Emphasize { get; set; } = new List<string>();

        public List<string> Write { get; set; } = new List<string>();
    }
}

1.4.0 MigrationResult (accumulates migration run results by AppliesTo - multitenancy targets)

using System;
using System.Collections.Generic;
using System.Linq;

namespace Our.Internal.Models
{
    /// <summary>
    /// Keep track of which migrations were run for which AppliesTo
    /// </summary>
    public class MigrationResult
    {
        public MigrationResult(AppliesTo appliesTo)
        {
            AppliesTo = appliesTo;
        }

        /// <summary>
        /// Show a summary
        /// </summary>
        public string Summary => !Migrations.Any() || Migrations.All(x => x.Sql.All(string.IsNullOrWhiteSpace))
                    ? "No migrations!"
                    : Errors.Any() ? "There were errors running migrations." : "Success";

        /// <summary>
        /// Track the detail of the migration
        /// </summary>
        public string Detail { get; set; }

        /// <summary>
        /// Track which AppliesTo this was run for
        /// </summary>
        public AppliesTo AppliesTo { get; private set; }

        /// <summary>
        /// Track how long the migration took to run (so we can do analytics)
        /// </summary>
        public TimeSpan ElapsedTime { get; set; }

        /// <summary>
        /// Track which errors we got back. 
        /// </summary>
        public List<string> Errors { get; set; } = new List<string>();

        /// <summary>
        /// The migrations run for this AppliesTo.
        /// </summary>
        public List<Migration> Migrations { get; set; } = new List<Migration>();
    }
}

All of which eventually was consumed like so:

<snip>
                    var announcer = new MigrationAnnouncer(response);
                    var runner = new RunnerContext(announcer)
                    {
                        ApplicationContext = string.Empty,
                        Connection = parameter.Tenant.ConnectionString.ConnectionString
                    };
                    var processor = new SqlServerProcessor(
                        parameter.Tenant.ConnectionString,
                        new SqlServer2014Generator() { },
                        announcer,
                        new ProcessorOptions() { PreviewOnly = parameter.WhatIf },
                        new SqlServerDbFactory() { }
                    );

                    var migrationRunner = new FluentMigrator.Runner.MigrationRunner(assembly, runner, processor);

                    try
                    {
                        migrationRunner.MigrateUp();
                    }
<snip>

I think that’s enough context to show how we were using it in 1.4.0, and I would just like to basically do the same thing in 3.x and later, without relying on deprecated structures, because clearly that’s a bad design.

If you say "nah, implement the above and use the AnnouncerFluentMigratorLogger then I can do that, it just seems like that’s the wrong way to go here.

What did I not remember to provide here to help explain this more clearly at 3am on a Saturday? 😬

[fubar-coder]

First, the IAnnouncer is (IIRC) still fully functional even though the output might not be 100% compatible, but there’s still a class that isn’t deprecated yet and which might be something you can build on: FluentMigratorLogger.

So, you can still register your own ILoggerProvider, which returns your custom (derived) implementation of FluentMigratorLogger. This class is indeed the foundation of the IAnnouncer compatibility 😉 . You can find an example for the logging provider in the sources, which in turn uses AnnouncerFluentMigratorLogger.

If I were you, I’d just copy the code, remove the Obsolete attribute, register the logger provider and go with it.

[jcolebrand]

Oh, yeah, no, I wouldn’t be doing something like that for logging checking. I see where you’re going with that now re: activities, which isn’t quite what I’m looking for. I just want to know basically when migrations finish and against what metadata did they finish. Those activity graphs on the ASN.1 would be intense, and more than I suspect most of what would be useful to me for my goals.

It’s more about, like, uhhh … consider this migration class (where ASN.1 wouldn’t even come into play as no actions have occurred on the target database, but a record marker would certainly show up in the metadata table to prevent it being rerun, which is the principle use-case of FM for us – most of our runs are )

using System.Threading;
using FluentMigrator;

namespace Framework.Multiple.Migration1
{
    [Migration(1, "First")]
    public class Class1 : Migration
    {
        public override void Down()
        {
            Thread.Sleep(RandomTimeSpan.Get());
        }

        public override void Up()
        {
            Thread.Sleep(RandomTimeSpan.Get());
        }
    }
}

And here’s the implementation for that static class

using System;

namespace Framework.Multiple.Migration1
{
    public static class RandomTimeSpan
    {
        public static Random Random = new Random();

        public static TimeSpan Get()
        {
            return new TimeSpan(0, 0, Random.Next(1,10));
        }
    }
}

I want to, in parallel, run that across 10 target databases, and collect the information about which of the 10 databases had the migration applied at the runtime (I could, in fact, interrogate the databases for the applied date, but I already would have gotten that from the existing run, and I would lose any ancillary data that may have been generated, such as runtime per migration applied, connection string of the target environment, etc). Did it get applied to all 10 databases? To only one of them? The default console logger gives me almost none of the metadata, and definitely doesn’t tell me the target that I was writing to/migrating.

Now, this is a highly contrived example, so of course it’s not really useful for a lot of information. But let’s say that I had 10 real migrations to each apply to 10 target databases, but of those, 3 had already received all the migrations in the set, 2 had only received 7 of the previous migrations, 3 had only received 5 of the previous migrations, and 2 had received none of the migrations.

Again, we are doing this in parallel as a metarunner but in a single controlling process (hence the O(1) parts of the above call chain). So we would take as long as the 10 migrations to run, but 3 of the databases would effectively end up with “List<migrationresult>.Count = 0” as it were, and the others at 3 applied and 5 applied, etc.

If I only wrote those to console, or to Serilog -> file, then I don’t have a way to batch them by target database, only by runtime. So I could reprocess those logs somehow, and provided the record told me the metadata (which connection string got the migration?) I could reconstruct my results graph.

But by using the approach above, I’m keeping all of that in memory, so I can just work with an in-memory list of results.

A zoom call would probably go a long way to you understanding in depth more of what I’m trying to accomplish. I would love to just throw it all up in a github repo somewhere but I know the company would not appreciate that. The easiest gmail to reach me at is the same as this handle, but it also gets a lot of other regular mail, so I miss unexpected emails. (in this case it would be expected tho, yeah? but future readers may not realize that if they tried to reach out to me)

Our codebase has thousands of migration classes across hundreds of projects, and when I say I’m applying them in parallel across a number of target databases, I do mean in some cases 250+ databases in a single push, with as many as 30-40 active migrations per target invocation, with hundreds of dlls being migrated from in the course of a deploy window on the worst of times. Being able to aggregate the data in-situ is ideal for me, much more so than trying to hope I can parse console.writeline results in realtime back into events in the call stream. And clearly doing (250 * 30) in serial takes much longer than (250 * 30 / parallelization) just so I can get the results from log per run in a clean rebuildable fashion.

Again, ASN.1 wouldn’t apply, insofar as I can tell from your example above, as this is what we generally end up with on our platform:

using CompanyName.Migrations;
using FluentMigrator;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace CompanyName.SomeProjectName.Migrations.Migrations.DEV_121345
{
    [ServiceMigration("DEV-121345", 2021, Month.July, 26)]
    public class MigrateSomeClassDescriptiveName : Migration
    {
        public override void Down()
        {
            Execute.EmbeddedScript("Dev-121345.down.sql"); // Needs to be embedded in the dll
        }

        public override void Up()
        {
            Execute.EmbeddedScript("Dev-121345.up.sql"); // Needs to be embedded in the dll
        }
    }
}

So in this case I would be getting the descriptive information from the metadata (they like to have their own overriding attributes that implement over your attribute class to make it easier for the teams to parse what becomes a YYYYMMDD style database identifier on the base migration attribute, with description, etc) and I would pick up whatever else gets logged today (cos I did originally share the IAnnouncer codepath from 1.4.0 that we use in production way up above) but clearly you would have no way to infer the actual operations on the database to generate any of the ASN.1 type data that you referenced.

If you want to ping me on email and we can set something up, I’m happy to do so. I know we all have busy schedules, but I’m pretty flexible on when I can chat. We can definitely nail something down, I’m sure.

And like I say, if you guys are open to seeing something like what I’m proposing I am more than happy to do the work to setup the PR, just not sure how useful it would be for anyone who isn’t me. I’m sure we aren’t the only people doing something this strange, but are there 10 of us? 100? 🤷

Thanks again fellas!