Event triggers use too many postgres connections

Tested with hasura 1.3.3 and 1.3.4-beta.2

We were surprised to see Hasura using upwards of 50 postgres sockets/connections with around 5000 transactions per second.

1. create a simple table and attach a dummy event trigger
2. insert a batch of 5000 events (using generate_series for example)
3. monitor postgresql sockets with select count(*) from pg_stat_activity;
4. observe 50+ additional sockets being used by hasura
5. observe some warnings in hasura logs: “Events processor may not be keeping up with events generated in postgres, or we’re working on a backlog of events. Consider increasing HASURA_GRAPHQL_EVENTS_HTTP_POOL_SIZE”

Is this expected behavior?

Are event triggers not a viable solution in a “high” throughput situation?

I can confirm the sockets do get released once idle (within 8-10 minutes)

[tirumaraiselvan]

Event Trigger mechanism is fully configurable to achieve the throughput you need. But nothing comes free, so you may need to give resources.

As the logs indicate, there are possibly many events pending and hence increasing the number of max concurrent http workers via: HASURA_GRAPHQL_EVENTS_HTTP_POOL_SIZE will help here. Further, each http worker will mark the event as “delivered” so it acquires a PG connection, and hence it will also take resources from the Postgres connection pool whose size you can configure via HASURA_GRAPHQL_PG_CONNECTIONS. Since you are running a test bench, the delivery is perhaps happening instantly and hence there is very high contention for the pool.

[cpursley]

@jflambert ~ it depends ~ what backend stack are you using? I’m using Elixir which is particularly well suited for that type of workload (I stole the code from here: https://github.com/supabase/realtime)

Here’s an example in Go: https://github.com/ihippik/wal-listener

You could also probably do in Node.