Postgres NOTIFY message queue overflow

We have problem with Postgres LISTEN/NOTIFY. I’ve added some own triggers with pg_notify and now, when hasura is listening for messages in hasura_schema_update channel, Postgres NOTIFY message queue is overflowing.

> SELECT pg_notification_queue_usage();
> 0.79306648254394531

Postgres logs at this moment:

2020-11-26 20:33:26.686 UTC [24866] [user=fastcup,db=fastcup,app=[unknown]] WARNING:  NOTIFY queue is 79% full
2020-11-26 20:33:26.686 UTC [24866] [user=fastcup,db=fastcup,app=[unknown]] DETAIL:  The server process with PID 30986 is among those with the oldest transactions.
2020-11-26 20:33:26.686 UTC [24866] [user=fastcup,db=fastcup,app=[unknown]] HINT:  The NOTIFY queue cannot be emptied until that process ends its current transaction.

I checked PID 30986, and it was connection from hasura with LISTEN hasura_schema_update;. I tried to terminate all connections with this workaround query:

SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE query = 'LISTEN  hasura_schema_update;';

And after that I see that SELECT pg_notification_queue_usage() returns 0, but for 1-2 minutes, after that it is growing again;

From Postgres documentation:

There is a queue that holds notifications that have been sent but not yet processed by all listening sessions. If this queue becomes full, transactions calling NOTIFY will fail at commit. The queue is quite large (8GB in a standard installation) and should be sufficiently sized for almost every use case. However, no cleanup can take place if a session executes LISTEN and then enters a transaction for a very long time. Once the queue is half full you will see warnings in the log file pointing you to the session that is preventing cleanup. In this case you should make sure that this session ends its current transaction so that cleanup can proceed.

The function pg_notification_queue_usage returns the fraction of the queue that is currently occupied by pending notifications. See Section 9.25 for more information.

[osseonews]

We are having this same exact problem with the NOTIFY message queue overflow. How did you fix this? We simply can’t use Hasura anymore. Every time we turn it back on Hasura overwhlems the message queue with connections from LISTEN hasura_schema_update;

[maxpain]

@rakeshkky @0x777 Could you please check this?