Npgsql.NpgsqlReadBuffer is causing console application to hang?

Hi, I have a .NET Core 3.1 console application that is running some background cron jobs. There are different jobs doing different things and they will start every x minute, do its thing and then stop. Most of the time they run fine, but lately these jobs have been starting to hang. They just never finish, the process is stuck.

I don’t really know how to debug this or how to figure out what is causing it to hang. What I’ve done is:

• Create a DMP file from the task manager
• Load this using dotnet dump analyze myfile.DMP
• Run dumpasync -stacks

This gives a list of all the stacks. I’ve created a few DMP files from different workers (doing different things), but they all have in common that there is one task on Npgsql.NpgsqlReadBuffer.

This is one example:

000001ed01aa18c8 00007ffca0c9e3d8      128          1 Npgsql.NpgsqlReadBuffer+<>c__DisplayClass34_0+<<Ensure>g__EnsureLong|0>d
Async "stack":
.000001ed01aa1988 (1) Npgsql.NpgsqlConnector+<>c__DisplayClass160_0+<<DoReadMessage>g__ReadMessageLong|0>d
..000001ed01aa1a40 (5) Npgsql.NpgsqlDataReader+<NextResult>d__44
...000001ed01aa1ae0 (0) Npgsql.NpgsqlCommand+<ExecuteReaderAsync>d__102
....000001ed01aa1b90 (0) Npgsql.NpgsqlCommand+<ExecuteDbDataReaderAsync>d__97
.....000001ed01aa1c10 (1) Dapper.SqlMapper+<QueryAsync>d__33`1[[System.__Canon, System.Private.CoreLib]]
......000001ed021ac5d8 (3) Acme.Common.Data.Dapper.Repositories.AccountItems.GetDapperAccountItemsHandlerSql+<GetAccountItemsAsync>d__3
.......000001ed021ac638 (0) Acme.Common.Data.Dapper.Repositories.ItemRepository`1+<GetAccountItemsHigherThanIdAsync>d__9[[Acme.Core.Db.Dapper.DapperReaderConnection, Acme.Core.Db.Dapper]]
........000001ed021ac698 (1) Acme.Common.Services.EmailReport.ReportDataService+<MakeInstantAlertDto>d__20
.........000001ed00badd90 (3) Acme.Common.Services.EmailReport.ReportDataService+<GetReportDtoAsync>d__19
..........000001ed0105f968 (2) Acme.Common.Services.EmailReport.InstantAlertReportService+<SendInstantAlertReportAsync>d__6
...........000001ed0105f9c8 (0) Acme.Common.Services.EmailReport.EmailReportWorkerService+<SendInstantAlertReportsAsync>d__10
............000001ed01b902d0 System.Threading.Tasks.TaskFactory+CompleteOnInvokePromise

I don’t know if this means that npgsql is the cause of the hang, but it seems to be what is common between all of them.

The connection is created like this:

public async Task<IEnumerable<MyDto>> GetData()
{
    using (var dbConnection = await _dapperConnection.OpenAsync())
    {
        var sql = "SELECT * FROM ....";
        var result = await dbConnection.QueryAsync<MyDto>(sql);
        return result;
    }
}

private async Task<NpgsqlConnection> OpenAsync(CancellationToken cancellationToken = default)
{
    var connection = new NpgsqlConnection(_connectionString);
    await connection.OpenAsync(cancellationToken);

    return connection;
}

The connection string looks like this: User ID=<userid>;Password=<password>;Host=<host>;Port=5432;Database=<databasename>;Pooling=true;Maximum Pool Size=200;Keepalive=30;

How can I debug this further? What would help?

I cross posted this on Stackoverflow in hope to get an answer there.

Further technical details

Npgsql version: 4.1.3 PostgreSQL version: 9.6 Operating system: Windows

[roji]

Cross-posted on stackoverflow: https://stackoverflow.com/questions/60550923/net-core-3-1-console-application-hangs

What you’re seeing is basically Npgsql waiting for a response from PostgreSQL on an SQL statement which it sent (#2860 is about connecting, not about executing a command). This could happen from any number of reasons: a network failure, a deadlock (when different transactions lock rows in different orders), a very slow query… There are various ways for you to analyze further what’s going on:

• Check out pg_stat_activity while the application is hanging.
• Turn on Npgsql logging
• Check the PostgreSQL logs around the time of the hang for suspicious messages. You may want to turn on statement logging (but that may cause lots of logging, be careful).
• Use wireshark (or similar) to get a TCP packet dump and then locate where the problematic packets are (probably a last resort).

[roji]

Am going to close this for now as no further details have been provided, but can reopen if new information is given.