"The connection is closed" errors in System.Data.SqlClient on Linux without MARS.

We’ve recently run into an issue where executing SQL commands results in "Invalid operation. The connection is closed." exceptions being thrown.

• This seems to happen exclusively on Async operations.
• This seems to have started happening since we upgrade to .NET Core 2.1. We’ve seen occurrences in our logs before the upgrade, but that was a big burst of them (probably real, but transient, connectivity errors), but they’ve started showing up consistently literally the hour after we migrated to 2.1.
• It looks like this happens only on Linux. We can’t reproduce it on our local Windows machines. To rule out this is purely Azure related we reproduced it on Linux by running our application in a Linux Docker container on our local machines.
• Turning on MultipleActiveResultSets (MARS) in the connection string seems to eliminate this error entirely.
• The fact that it doesn’t happen on Windows and that MARS solves it, rules out transient connectivity errors.
• Our usage of SqlConnection is pretty standard: they come from a connection pool, and we don’t use them concurrently.
• I’ve attempted to reproduce the issue in an isolated project, but I’ve been unable to, so I’m not sure where the difference between that project and our full application lies.

The exception that gets thrown is this one:

https://github.com/dotnet/corefx/blob/a74bf4200926c47c11ba0383e2bddeaa59227266/src/System.Data.SqlClient/src/System/Data/Common/AdapterUtil.SqlClient.cs#L395

The exception gets thrown (or rethrown) from these three points in the code:

1. https://github.com/dotnet/corefx/blob/e0ba7aa8026280ee3571179cc06431baf1dfaaac/src/System.Data.SqlClient/src/System/Data/SqlClient/TdsParserStateObject.cs#L3098

Example stacktrace:

at Task System.Data.SqlClient.TdsParserStateObject.WritePacket(byte flushMode, bool canAccumulate)
   at Task System.Data.SqlClient.TdsParserStateObject.WriteByteArray(byte[] b, int len, int offsetBuffer, bool canAccumulate, TaskCompletionSource<object> completion)
   at void System.Data.SqlClient.TdsValueSetter.SetString(string value, int offset, int length)
   at void Microsoft.SqlServer.Server.ValueUtilsSmi.SetString_LengthChecked(SmiEventSink_Default sink, ITypedSettersV3 setters, int ordinal, SmiMetaData metaData, string value, int offset)
   at void Microsoft.SqlServer.Server.ValueUtilsSmi.SetCompatibleValue(SmiEventSink_Default sink, ITypedSettersV3 setters, int ordinal, SmiMetaData metaData, object value, ExtendedClrTypeCode typeCode, int offset)
   at void Microsoft.SqlServer.Server.ValueUtilsSmi.SetDataTable_Unchecked(SmiEventSink_Default sink, SmiTypedGetterSetter setters, int ordinal, SmiMetaData metaData, DataTable value)
   at void System.Data.SqlClient.TdsParser.WriteSmiParameter(SqlParameter param, int paramIndex, bool sendDefault, TdsParserStateObject stateObj)
   at Task System.Data.SqlClient.TdsParser.TdsExecuteRPC(_SqlRPC[] rpcArray, int timeout, bool inSchema, SqlNotificationRequest notificationRequest, TdsParserStateObject stateObj, bool isCommandProc, bool sync, TaskCompletionSource<object> completion, int startRpc, int startParam)
   at SqlDataReader System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, bool returnStream, bool async, int timeout, out Task task, bool asyncWrite, SqlDataReader ds)
   at Task System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource<object> completion, bool sendToPipe, int timeout, bool asyncWrite, string methodName)
   at IAsyncResult System.Data.SqlClient.SqlCommand.BeginExecuteNonQuery(AsyncCallback callback, object stateObject)
   at Task<TResult> System.Threading.Tasks.TaskFactory<TResult>.FromAsyncImpl(Func<AsyncCallback, object, IAsyncResult> beginMethod, Func<IAsyncResult, TResult> endFunction, Action<IAsyncResult> endAction, object state, TaskCreationOptions creationOptions)
   at Task<int> System.Data.SqlClient.SqlCommand.ExecuteNonQueryAsync(CancellationToken cancellationToken)
   at async Task<int> Dapper.SqlMapper.ExecuteImplAsync(IDbConnection cnn, CommandDefinition command, object param) in C:/projects/dapper/Dapper/SqlMapper.Async.cs:line 678

1. https://github.com/dotnet/corefx/blob/e0ba7aa8026280ee3571179cc06431baf1dfaaac/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlCommand.cs#L1638

Example stacktrace:

at Task<DbDataReader> System.Data.SqlClient.SqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)+(Task<SqlDataReader> result) => { }
   at void System.Threading.Tasks.ContinuationResultTaskFromResultTask<TAntecedentResult, TResult>.InnerInvoke()
   at async Task<IEnumerable<T>> Dapper.SqlMapper.QueryAsync<T>(IDbConnection cnn, Type effectiveType, CommandDefinition command) in C:/projects/dapper/Dapper/SqlMapper.Async.cs:line 419

1. https://github.com/dotnet/corefx/blob/a74bf4200926c47c11ba0383e2bddeaa59227266/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlInternalConnectionTds.cs#L612

Example stacktrace:

 at void System.Data.SqlClient.SqlInternalConnectionTds.ValidateConnectionForExecute(SqlCommand command)
   at void System.Data.SqlClient.SqlInternalTransaction.Rollback()
   at void System.Data.SqlClient.SqlTransaction.Dispose(bool disposing)
   at void System.Data.Common.DbTransaction.Dispose()

So it either happens when writing data to SQL Server or reading a result set and it seems to happen on longer running operations, but probably only because a longer query makes it more likely for the error to show up.

Our connection string looks like this:

Server=tcp:<DB>.database.windows.net,1433;Initial Catalog=<DB>;Persist Security Info=False;User ID=<USER>;Password=<PW>;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;

[mathobbs]

Thank you @JulianRooze for the great write up. We also see this intermittently on services running in linux docker containers (dotnetcore 2.2.3 on alpine 3.9). Enabling MARS on the connection as a workaround also worked.

The SQLConnection’s StateChange event is called unexpectedly indicating a transition from Open to Closed, and the same “connection is closed” exception is thrown to the code executing the sql call. We have logic verifying that a single SQLConnection is not being used concurrently.

With a retry added in the code, before enabling the workaround we would sometimes see multiple retries fail with the same, on other occasions one retry worked.

Currently we have only seen this on calls that have table-valued parameters (it may be more likely for larger parameter payloads). Sometimes (but not always) on a fail there is a sql-server logged error message which is usually: “The data for table-valued parameter “@TheUserDefinedTypeName” doesn’t conform to the table type of the parameter” (error code 8061) but is sometimes this: “The incoming tabular data stream (TDS) protocol stream is incorrect. The stream ended unexpectedly.” (error code 4002)

[cheenamalhotra]

Hi @alexmg

You may access nightly build feed here: https://dev.azure.com/sqlclientdrivers-ci/sqlclient/_packaging?_a=feed&feed=Microsoft.Data.SqlClient.dev

We’ll be releasing this fix soon! Thanks!