Upgrade from v7 to v8.6+ throws exception on media URL migration

Overview

In trying to upgrade a database that uses RTEs and Grid heavily, the upgrade fails if I got directly to v8.6.0. It works if I first go to an earlier version, and then try to go to v8.6.0. The error is in the ConvertTinyMceAndGridMediaUrlsToLocalLink migration, on line 113.

Details

Umbraco version

I am seeing this issue on Umbraco version: v8.6.0, from v7.15.3

Bug summary

This is caused by the ConvertTinyMceAndGridMediaUrlsToLocalLink migration using a service reference to retrieve data instead of using direct database access. Thus when new columns were introduced in the AddPropertyTypeValidationMessageColumns migration, the service is trying to read from these new columns that don’t yet exist in the database.

Steps to reproduce

Create a v7.15.3 database, and add Grid and RTE data that contain media links. Then try to upgrade the database to v8.6.0.

Expected result

The site is upgraded to v8.6.0 successfully.

Actual result

The upgrade wizard fails with the message “The database failed to upgrade. ERROR: The database configuration failed with the following message: Invalid column name ‘mandatoryMessage’. Invalid column name ‘validationRegExpMessage’. Please check log file for additional information (can be found in ‘/App_Data/Logs/’)”. Reviewing the trace log shows the following stack trace for that error:

System.Data.SqlClient.SqlException (0x80131904): Invalid column name 'mandatoryMessage'.
Invalid column name 'validationRegExpMessage'.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at StackExchange.Profiling.Data.ProfiledDbCommand.ExecuteDbDataReader(CommandBehavior behavior) in C:\projects\dotnet\src\MiniProfiler.Shared\Data\ProfiledDbCommand.cs:line 223
   at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in C:\projects\umbraco-cms\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172
   at NPoco.Database.ExecuteReaderHelper(DbCommand cmd)
   at NPoco.Database.ExecuteDataReader(DbCommand cmd)
   at NPoco.Database.<QueryImp>d__164`1.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Umbraco.Core.Persistence.Repositories.Implement.ContentTypeCommonRepository.MapGroupsAndProperties(IDictionary`2 contentTypes) in C:\projects\umbraco-cms\src\Umbraco.Core\Persistence\Repositories\Implement\ContentTypeCommonRepository.cs:line 192
   at Umbraco.Core.Persistence.Repositories.Implement.ContentTypeCommonRepository.GetAllTypesInternal() in C:\projects\umbraco-cms\src\Umbraco.Core\Persistence\Repositories\Implement\ContentTypeCommonRepository.cs:line 114
   at Umbraco.Core.Cache.AppCacheExtensions.<>c__DisplayClass0_0`1.<GetCacheItem>b__0() in C:\projects\umbraco-cms\src\Umbraco.Core\Cache\AppCacheExtensions.cs:line 22
   at Umbraco.Core.Cache.FastDictionaryAppCacheBase.<>c__DisplayClass21_0.<GetSafeLazy>b__0() in C:\projects\umbraco-cms\src\Umbraco.Core\Cache\FastDictionaryAppCacheBase.cs:line 285
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Umbraco.Core.Cache.WebCachingAppCache.GetInternal(String key, Func`1 factory, Nullable`1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, String[] dependentFiles) in C:\projects\umbraco-cms\src\Umbraco.Core\Cache\WebCachingAppCache.cs:line 174
   at Umbraco.Core.Cache.WebCachingAppCache.Get(String key, Func`1 factory, Nullable`1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, String[] dependentFiles) in C:\projects\umbraco-cms\src\Umbraco.Core\Cache\WebCachingAppCache.cs:line 40
   at Umbraco.Core.Cache.DeepCloneAppCache.Get(String key, Func`1 factory, Nullable`1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, String[] dependentFiles)
   at Umbraco.Core.Persistence.Repositories.Implement.ContentTypeCommonRepository.GetAllTypes() in C:\projects\umbraco-cms\src\Umbraco.Core\Persistence\Repositories\Implement\ContentTypeCommonRepository.cs:line 48
   at Umbraco.Core.Persistence.Repositories.Implement.MediaTypeRepository.PerformGetAll(Int32[] ids) in C:\projects\umbraco-cms\src\Umbraco.Core\Persistence\Repositories\Implement\MediaTypeRepository.cs:line 54
   at Umbraco.Core.Cache.FullDataSetRepositoryCachePolicy`2.GetAllCached(Func`2 performGetAll) in C:\projects\umbraco-cms\src\Umbraco.Core\Cache\FullDataSetRepositoryCachePolicy.cs:line 166
   at Umbraco.Core.Cache.FullDataSetRepositoryCachePolicy`2.Get(TId id, Func`2 performGet, Func`2 performGetAll)
   at Umbraco.Core.Persistence.Repositories.Implement.MediaRepository.MapDtoToContent(ContentDto dto) in C:\projects\umbraco-cms\src\Umbraco.Core\Persistence\Repositories\Implement\MediaRepository.cs:line 538
   at Umbraco.Core.Services.Implement.MediaService.GetMediaByPath(String mediaPath) in C:\projects\umbraco-cms\src\Umbraco.Core\Services\Implement\MediaService.cs:line 631
   at Umbraco.Core.Migrations.Upgrade.V_8_1_0.ConvertTinyMceAndGridMediaUrlsToLocalLink.<>c__DisplayClass3_0.<UpdateMediaUrls>b__0(Match match) in C:\projects\umbraco-cms\src\Umbraco.Core\Migrations\Upgrade\V_8_1_0\ConvertTinyMceAndGridMediaUrlsToLocalLink.cs:line 113
   at System.Text.RegularExpressions.RegexReplacement.Replace(MatchEvaluator evaluator, Regex regex, String input, Int32 count, Int32 startat)
   at System.Text.RegularExpressions.Regex.Replace(String input, MatchEvaluator evaluator)
   at Umbraco.Core.Migrations.Upgrade.V_8_1_0.ConvertTinyMceAndGridMediaUrlsToLocalLink.UpdateMediaUrls(Regex mediaLinkPattern, String value, Boolean& changed) in C:\projects\umbraco-cms\src\Umbraco.Core\Migrations\Upgrade\V_8_1_0\ConvertTinyMceAndGridMediaUrlsToLocalLink.cs:line 103
   at Umbraco.Core.Migrations.Upgrade.V_8_1_0.ConvertTinyMceAndGridMediaUrlsToLocalLink.Migrate() in C:\projects\umbraco-cms\src\Umbraco.Core\Migrations\Upgrade\V_8_1_0\ConvertTinyMceAndGridMediaUrlsToLocalLink.cs:line 59
   at Umbraco.Core.Migrations.MigrationBase.Umbraco.Core.Migrations.IMigration.Migrate() in C:\projects\umbraco-cms\src\Umbraco.Core\Migrations\MigrationBase.cs:line 73
   at Umbraco.Core.Migrations.MigrationPlan.Execute(IScope scope, String fromState, IMigrationBuilder migrationBuilder, ILogger logger) in C:\projects\umbraco-cms\src\Umbraco.Core\Migrations\MigrationPlan.cs:line 316
   at Umbraco.Core.Migrations.Upgrade.Upgrader.Execute(IScopeProvider scopeProvider, IMigrationBuilder migrationBuilder, IKeyValueService keyValueService, ILogger logger) in C:\projects\umbraco-cms\src\Umbraco.Core\Migrations\Upgrade\Upgrader.cs:line 67
   at Umbraco.Core.Migrations.Install.DatabaseBuilder.UpgradeSchemaAndData(MigrationPlan plan) in C:\projects\umbraco-cms\src\Umbraco.Core\Migrations\Install\DatabaseBuilder.cs:line 501

[benjaminc]

Its pretty disrespectful to throw shade on the publicly available package I created to fix the problem I identified for you when your internal testing failed, and then submitted a tested fix for which you ignored, and so finally created a community package to fix the issue instead of just fixing it for myself, all because my one-off package to fix your hole doesn’t have as much documentation as you’d like to see. That’s pretty low.

And as has been said above, this isn’t affecting just a few people. This bug actually affect EVERY upgrade that tries to go directly from v7 to v8.6 or above. So either you were an early adopter, or you hit this bug. There isn’t any way around it, and it doesn’t only affect you if you have a specific set of data. It affects every upgrade. If you haven’t had more reports of it, that means one of two things: either your clients aren’t upgrading to v8 or they are so used to poor service that they aren’t even bothering to mention it to you anymore. Both of those should really be pretty chilling options to anyone who cares about their product.

[benjaminc]

The problem comes because a migration (designed to interact directly with the database while the database is in a state of flux) is calling into the service layer (designed to interact with a stable database to apply current business logic). If you look at what it is trying to get from the service layer, it is just trying to get a single piece of information about media items. This could very easily be achieved with a simple DTO query, my guess is that the migration developer was simply more familiar with how to get it via the service layer, and so went with what they were familiar with. Without any standards on what should or shouldn’t be done in migrations, this approach completely makes sense.

I think the long term fix for this is that migrations included as part of the core should simply not be allowed to use the service layer. Any data access should be done with queries and version-specific DTOs, similar to what was done for the bulk of the 8.0 and 8.1 migrations. It was done right for the bulk of the 8.X upgrade migrations, I think this one just slipped through because there wasn’t a standard that a reviewer could check against, because the problem wasn’t well understood. Now that the impact of this is seen, just add a note for reviewers to check for and not allow service layer access in migrations, and rewrite this one existing migration to not use the service layer, but to do a direct DB query. Problem solved.