CPU-bound tasks seem to starve network-bound tasks

Investigative information

Please provide the following:

• Timestamp: 12/12/20 2:56:49 PM ET
• Function App version: 3
• Function App name: NpgsqlHeavyCPU
• Function name(s) (as appropriate): NpgsqlOrchestration
• Invocation ID: 9948a45e-dcf9-410f-a167-ea81f2dc4682
• Region: East US 2

Repro steps

I created a sample project to demonstrate the problem: https://github.com/Methuselah96/NpgsqlHeavyCpu

The sample only shows Npgsql, but I’ve also seen issues with Cosmos and Blob Storage as well.

Expected behavior

The Azure Function to complete without failure.

Actual behavior

Various socket-related errors: Npgsql:

Exception message: Npgsql.NpgsqlException (0x80004005): Exception while writing to stream
Stack trace:
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)

at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)

at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnection.c__DisplayClass32_0.g__OpenLong|0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at Dapper.SqlMapper.QueryRowAsync[T](IDbConnection cnn, Row row, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 482

Npgsql:

Exception message: System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
Stack trace:
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)

at System.Net.Security.SslStream.ThrowIfExceptional()

at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)

at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)

at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)

at System.Net.Security.SslStream.c.b__64_2(IAsyncResult iar)

at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)

--- End of stack trace from previous location where exception was thrown ---

at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnection.c__DisplayClass32_0.g__OpenLong|0>d.MoveNext()

Cosmos:

System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.

---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

--- End of inner exception stack trace ---

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)

at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at Microsoft.Azure.Cosmos.DocumentClient.HttpRequestMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

at Microsoft.Azure.Cosmos.GatewayAccountReader.GetDatabaseAccountAsync(Uri serviceEndpoint)

at Microsoft.Azure.Cosmos.Routing.GlobalEndpointManager.GetDatabaseAccountFromAnyLocationsAsync(Uri defaultEndpoint, IList`1 locations, Func`2 getDatabaseAccountFn)

at Microsoft.Azure.Cosmos.GatewayAccountReader.InitializeReaderAsync()

at Microsoft.Azure.Cosmos.CosmosAccountServiceConfiguration.InitializeAsync()

at Microsoft.Azure.Cosmos.DocumentClient.InitializeGatewayConfigurationReaderAsync()

at Microsoft.Azure.Cosmos.DocumentClient.GetInitializationTaskAsync(IStoreClientFactory storeClientFactory)

at Microsoft.Azure.Cosmos.DocumentClient.EnsureValidClientAsync()

at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.EnsureValidClientAsync(RequestMessage request)

at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)

at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(Uri resourceUri, ResourceType resourceType, OperationType operationType, RequestOptions requestOptions, ContainerCore cosmosContainerCore, Nullable`1 partitionKey, Stream streamPayload, Action`1 requestEnricher, CosmosDiagnosticsContext diagnosticsContext, CancellationToken cancellationToken)

at Microsoft.Azure.Cosmos.CosmosClient.c__DisplayClass27_0.b__0>d.MoveNext()

Blob storage:

Microsoft.WindowsAzure.Storage.StorageException: An attempt was made to access a socket in a way forbidden by its access permissions.

---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.

---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

--- End of inner exception stack trace ---

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)

at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)

Known workarounds

For Npgsql, if I opened all the connections before running the CPU-bound code, then I wouldn’t have any errors. I haven’t found any workarounds for Cosmos or Blob storage yet.

[v-bbalaiagar]

Hi @Methuselah96 , The bot has closed the issue due to in-activity. We shall keep the issue open.

[v-bbalaiagar]

Hi @Methuselah96 , We have added this issue to projects to investigate further.