Azure SQL: System.ObjectDisposedException: Cannot access a disposed object on OpenAsync

Describe the bug

On Azure SQL, after short server unavailability / failover, subsequent call to OpenAsync fails with System.ObjectDisposedException: Cannot access a disposed object.

Stack trace:

Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 35 - An internal exception was caught)
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.
   at System.Net.Sockets.Socket.SetSocketOption(SocketOptionLevel optionLevel, SocketOptionName optionName, Int32 optionValue)
   at Microsoft.Data.SqlClient.SNI.SNITCPHandle..ctor(String serverName, Int32 port, Int64 timerExpire, Object callbackObject, Boolean parallel)
   at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)
   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()

Exception is re-thrown with throw e at CheckPoolBlockingPeriod for Azure SQL connections, so only two first lines seems to be related. MultiSubnetFailover attribute is not set, so it probably follows non-parallel connect path.

To reproduce

Synthetic repo (based on SNITCPHandle.Connect code +Thread.Sleep to force exact timing): https://github.com/PashaPash/ConnectRace/blob/master/ConnectRace/Program.cs

Unable to clearly reproduce it without manual Thread.Sleeps - seems to happen on Azure only, and during (or immediately after) Azure SQL downtime.

Expected behavior

OpenAsync either fails with real network-related error or connects successfully.

Further technical details

Microsoft.Data.SqlClient version: 1.1.1, with #359 fix applied .NET target: Core 3.0.3 SQL Server version: Azure SQL, listed as 12.0.2000.8 in SSMS Operating system: Docker Image mcr.microsoft.com/dotnet/core/aspnet:3.0, running on AKS

---

Same stack trace reported by @TheDruidsKeeper: https://github.com/dotnet/SqlClient/issues/359#issuecomment-575787541

[cheenamalhotra]

Hi @RCTycooner

We received update on another thread regarding issues in AKS environments ( https://github.com/dotnet/SqlClient/issues/647#issuecomment-806831567), could you check with Azure Support if this is related to your case too? As I see socket exceptions in your stacktrace, that may be pointing to same issue.

[PashaPash]

@cheenamalhotra I was able to reproduce it. Here is a sample code: https://github.com/PashaPash/ConnectRace/blob/master/ConnectRace/Program.cs

It is basically SNITCPHandle.Connect with few Thread.Sleep added to get an exact execution order.

Conditions seems to be Connect Timeout < real host response timeout + bad luck on simultaneous Dispose / Connect calls.