PostgreSQL connections spike

Describe the bug In production we have a large invoices flow. Every ~10-30 minutes the number of connections to the database increases dramatically (from ~10 to maximum of connection pool size).

I suppose the problem is that there is no limit on the number of parallel threads. With a large number of pending invoices, the application will open a large number of database connections. At the moment when too many connections to the database are open, other operations with invoices take too long or fall with a HTTP timeout. It is critical for our payments flow.

In my opinion, using System.Linq.ParallelEnumerable or Microsoft.TPL.DataFlow will limit the number of parallel threads and database connections.

To Reproduce Generate a large number of rows in database table “PendingInvoices”.

Expected behavior BtcPay server does not take unlimited number of connections.

Screenshots

Your BTCPay Environment (please complete the following information): Deployment method: Docker Browser: all browsers Bitcoin Network: mainnet NBXplorer version: v2.1.36 BtcPay server version: v1.0.5.7 Database engine: PostgreSQL v11.8 Database hosting: AWS RDS Pending invoices count: 3162 Connection pool size: 255

SQL Query that runs in each connection SELECT i."Id", i."Archived", i."Blob", i."Created", i."CurrentRefundId", i."CustomerEmail", i."ExceptionStatus", i."ItemCode", i."OrderId", i."Status", i."StoreDataId", p."Id", p."Accounted", p."Blob", p."InvoiceDataId" FROM "Invoices" AS i LEFT JOIN "Payments" AS p ON i."Id" = p."InvoiceDataId" WHERE i."Id" = $1 ORDER BY i."Id", p."Id"

[Kukks]

@NicolasDorier There were updates of invoice status. I’ve created 500 invoices simultaneously for testing.

SQL query: INSERT INTO "InvoiceEvents" ("InvoiceDataId", "UniqueId", "Message", "Severity", "Timestamp") VALUES ($1, $2, $3, $4, $5)

On production environment spikes are gone but I continue to monitor the number of connections.

These are less important, we can create a batching tool to schedule a mass insert every x interval

[Rheopyrin]

@pavlenex I’ll ask our customer support team to provide actual feedback.