Timeouts at 0 seconds

I’m seeing intermittent timeouts after unexpected times like 0, 1, and 59 seconds. The 59 seems awfully suspicious to me. Is there another concurrency issue here like there was with the where clause? I turned on Keepalive only as a guess in my troubleshooting but it didn’t help. The query below is a bit complex and seems to trigger this issue more often. It is a function. Occasionally simpler queries generated by EF Core show this error as well.

Connection String: Server=127.0.0.1;Port=5432;User ID=user;Password=password;Include Error Detail=true;Command Timeout=300;Internal Command Timeout=300;Keepalive=15;Database=database;

[ERR] 01:08 | Failed executing DbCommand ("307,002"ms)
[Parameters=["@__p_0='754100018209620019' (Nullable = true),
@__since_1='2021-02-05T00:00:00.0000000Z' (Nullable = true) (DbType =
DateTime)"], CommandType='Text', CommandTimeout='300']"\r\n""SELECT
a.attacker_clan_tag, a.attacker_discord_id, a.attacker_hidden,
a.attacker_map_position, a.attacker_tag, a.attacker_townhall,
a.clan_no_stat, a.clan_tag, a.defender_clan_tag,
a.defender_discord_id, a.defender_hidden, a.defender_map_position,
a.defender_tag, a.defender_townhall, a.destruction_percentage,
a.downloaded_at, a.end_time, a.fresh, a.id, a.league_id,
a.manually_entered, a.opponent_no_stat, a.opponent_tag, a.\"order\",
a.preparation_start_time, a.season, a.stars, a.stars_gained,
a.start_time, a.state, a.war_id, a.war_tag, a.war_type\r\nFROM
attack_details AS a\r\nWHERE (((((((((a.attacker_discord_id = @__p_0)
OR (a.defender_discord_id = @__p_0)) AND CASE\r\n    WHEN
(a.attacker_discord_id = @__p_0) AND (a.attacker_clan_tag =
a.clan_tag) THEN NOT (a.clan_no_stat)\r\n    ELSE TRUE\r\nEND) AND
CASE\r\n    WHEN (a.defender_discord_id = @__p_0) AND
(a.defender_clan_tag = a.clan_tag) THEN NOT (a.clan_no_stat)\r\n
ELSE TRUE\r\nEND) AND CASE\r\n    WHEN (a.attacker_discord_id =
@__p_0) AND (a.attacker_clan_tag = a.opponent_tag) THEN NOT
(a.opponent_no_stat)\r\n    ELSE TRUE\r\nEND) AND CASE\r\n    WHEN
(a.defender_discord_id = @__p_0) AND (a.defender_clan_tag =
a.opponent_tag) THEN NOT (a.opponent_no_stat)\r\n    ELSE TRUE\r\nEND)
AND CASE\r\n    WHEN a.attacker_discord_id = @__p_0 THEN NOT
(a.attacker_hidden)\r\n    ELSE TRUE\r\nEND) AND CASE\r\n    WHEN
a.defender_discord_id = @__p_0 THEN NOT (a.defender_hidden)\r\n
ELSE TRUE\r\nEND) AND (a.league_id IS NOT NULL)) AND (a.downloaded_at
> @__since_1)"
[ERR] 01:08 | An exception occurred while iterating over the results
of a query for context type
'"MinionBot.db.MbDbContext"'."\r\n""System.InvalidOperationException:
An exception has been raised that is likely due to a transient
failure.\r\n ---> Npgsql.NpgsqlException (0x80004005): Exception while
reading from stream\r\n ---> System.TimeoutException: Timeout during
reading attempt\r\n   at
Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector
connector, Boolean async, DataRowLoadingMode dataRowLoadingMode,
Boolean readingNotifications, Boolean isReadingPrependedMessage)\r\n
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean
isConsuming, CancellationToken cancellationToken)\r\n   at
Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean
async, CancellationToken cancellationToken)\r\n   at
Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean
async, CancellationToken cancellationToken)\r\n   at
Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior
behavior, CancellationToken cancellationToken)\r\n   at
Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject
parameterObject, CancellationToken cancellationToken)\r\n   at
Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject
parameterObject, CancellationToken cancellationToken)\r\n   at
Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext
_, Boolean result, CancellationToken cancellationToken)\r\n   at
Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState
state, Func`4 operation, Func`4 verifySucceeded, CancellationToken
cancellationToken)\r\n   --- End of inner exception stack trace
---\r\n   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState
state, Func`4 operation, Func`4 verifySucceeded, CancellationToken
cancellationToken)\r\n   at
Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()"
System.InvalidOperationException: An exception has been raised that is
likely due to a transient failure.
 ---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
 ---> System.TimeoutException: Timeout during reading attempt
   at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector
connector, Boolean async, DataRowLoadingMode dataRowLoadingMode,
Boolean readingNotifications, Boolean isReadingPrependedMessage)
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean
isConsuming, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior,
Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior,
Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior
behavior, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject
parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject
parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext
_, Boolean result, CancellationToken cancellationToken)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState
state, Func`4 operation, Func`4 verifySucceeded, CancellationToken
cancellationToken)
   --- End of inner exception stack trace ---
   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState
state, Func`4 operation, Func`4 verifySucceeded, CancellationToken
cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()

Further technical details

Npgsql version: 5.0.2 PostgreSQL version:11 Operating system: Windows 10

[roji]

@devhl-labs at the end of the day, the root cause here - which you probably need to figure out - is that you have certain queries which seem to take a very long time. For the time that a query is in progress (54 seconds is a lot!), the connection is tied up, so if you have lots of concurrent requests happening, you can reach the maximum pool size very quickly. The typical process here would be to isolate the query (or queries) which are taking a long time (by examining timeouts and/or completion times in the logs), and then trying to understand why they’re taking so long (e.g. by analyzing their query plan). It could be that adding the right index could make the culprit execute in milliseconds instead of minutes.

Otherwise, if there’s a legitimate reason why your queries are taking so long (actual huge data, no possibility to speed up the queries with index or more efficient SQL), then you could increase max_connections in your postgresql.conf to 400, allowing each program to have 200.

[devhl-labs]

So I saw issues on another program that uses the same database. It had these errors:

An exception has been raised that is likely due to a transient failure. | The connection pool has been exhausted, either raise MaxPoolSize (currently 100) or Timeout (currently 15 seconds)

So I increased the Max Pool Size in the connection string to 200 and then the error changed to too many clients. My postgresql.conf max connections is 200, so I suppose this is right give that I have two programs touching the database.

I reduced the concurrency a bit by reducing the number of events that fire at once, and this did reduce the number of timeout errors, but I still do get them. I’m not really sure what to do.