The MARS TDS header contained errors using ASP.NET Core and EF Core connecting to Azure SQL Server

Copied from here:

I have an ASP.NET Core app [Microsoft.AspNetCore.App 2.1.4] with EF Core 2.1.4 [DbContext pooling enabled] and data stored on an Azure SQL database.

Occasionally [once in 1-2 days] I get unrecoverable error stating System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.

Once I restart the app, everything resumes working until it happens again.

Probably import to note is that I don’t have MultipleActiveResultSets enabled in my connection string at all, which makes the issue even more strange.

Has anyone else noticed something similar? Are there any ways I can trace the problem?

Stack trace:

System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.Execute(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues)
   at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteReader(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(DbContext _, Boolean buffer)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[TState,TResult](Func`3 operation, Func`3 verifySucceeded, TState state)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Inviton.Web.Portal.Controllers.EventListController.GetEvents(BL bl, Int32 venueId, Int32 promoterId) 

[mscrivo]

These errors have completely disappeared for us after upgrading to EF Core 3.1/Microsoft.Data.SqlClient 1.1.0.

[brunolau]

@mshenoy83 Sorry it took a bit longer, here goes my dirty hack. In order to use it, please check the “Custom execution strategy” section of this documentation https://docs.microsoft.com/en-us/ef/core/miscellaneous/connection-resiliency

class CustomExecutionStrategy:

using Brulasoft.Inviton.Data.Internal.SQL;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Storage;
using System;
using System.Data.SqlClient;

namespace Brulasoft.Inviton.Data.Internal
{
    /// <summary>
    /// Custom SQL execution
    /// </summary>
    internal class CustomExecutionStrategy : SqlServerRetryingExecutionStrategy
    {
        public CustomExecutionStrategy(ExecutionStrategyDependencies dependencies) : base(dependencies)
        {
        }

        protected override bool ShouldRetryOn(Exception exception)
        {
            if (base.ShouldRetryOn(exception))
            {
                return true;
            }

            if (exception is SqlException sqlException)
            {
                if (SqlRecoveryAttemptor.ShouldTryRecover(exception, ExceptionsEncountered.Count) && Dependencies != null && Dependencies.CurrentDbContext != null)
                {
                    var context = Dependencies.CurrentDbContext.Context;
                    if (context != null)
                    {
                        var sqlConn = context.Database.GetDbConnection() as SqlConnection;
                        if (sqlConn != null)
                        {
                            SqlRecoveryAttemptor.TryRecover(sqlConn, ExceptionsEncountered.Count);
                            return true;
                        }
                    }
                }
            }

            return false;
        }
    }
}

class SqlRecoveryAttemptor

using System;
using System.Collections.Generic;
using System.Data.Common;
using System.Data.SqlClient;
using System.Text;

namespace Brulasoft.Inviton.Data.Internal.SQL
{
    public class SqlRecoveryAttemptor
    {
        public const int RetryCount = 4;


        public static bool ShouldTryRecover(Exception ex, int retryAttemptsCount)
        {
            return retryAttemptsCount < RetryCount && IsMARSError(ex);
        }

        public static bool IsMARSError(Exception ex)
        {
            return ex.Message.Contains("MARS TDS header");
        }

        public static void TryRecover(SqlConnection sqlConn, int retryAttemptsCount)
        {
            if (retryAttemptsCount > 2)
            {
                System.Threading.Thread.Sleep(1500);
            }

            if (sqlConn.State != System.Data.ConnectionState.Closed)
            {
                TryClose(sqlConn);
                TryClearPool(sqlConn);
                TryOpen(sqlConn);
            }
            else
            {
                TryOpen(sqlConn);
                TryClose(sqlConn);
                TryClearPool(sqlConn);
                TryOpen(sqlConn);
                TryClose(sqlConn);
            }
        }

        private static void TryClearPool(SqlConnection conn)
        {
            try
            {
                SqlConnection.ClearPool(conn);
            }
            catch (Exception)
            {
            }
        }

        private static void TryClose(DbConnection conn)
        {
            try
            {
                conn.Close();
            }
            catch (Exception)
            {
            }
        }

        private static void TryOpen(DbConnection conn)
        {
            try
            {
                conn.Open();
            }
            catch (Exception)
            {
            }
        }
    }
}