Slow performance of Hasura behind nginx proxy for https, as keep-alive is not ticking

Some discussions here in https://github.com/hasura/graphql-engine/discussions/6154#discussioncomment-131629

Had to use nginx for https support. But, doing so it drastically reduced the speed by around 50x.

Based on the note from nginx mailing list, http vs https performance degradation to 100x is normal. But, keep-alive should improve performance.

Further noted that keep-alive for other REST services like https://github.com/netlify/gotrue are improving performance; but not for Hasura as there is no keep-alive connections.

Copied relevant details from above discussions:

Benchmarks and factors

	Time per request (ms)	Factor
Hasura - ab - Direct	36.579	1
Hasura - ab - nginx - simple	1929.93	52.76060034
Hasura - ab - nginx - optimized	1939.657	53.02651795
Hasura - ab -k - Direct	26.944	0.7365975013
Hasura - ab -k - nginx - simple	1919.033	52.46269718
Hasura - ab -k - nginx - optimized	1938.435	52.9931108

	Time per request (ms)	Factor
Go True - ab - Direct	1.407	1
Go True - ab - nginx - simple	441.281	313.6325515
Go True - ab - nginx - optimized	411.653	292.5749822
Go True - ab -k - Direct	1.513	1.075337598
Go True - ab -k - nginx - simple	117.077	83.21037669
Go True - ab -k - nginx - optimized	141.344	100.4577114

Benchmark - detailed ab results

nginx - hasura performance - comparision on ab - All data.csv.txt

Refer to ‘Keep-Alive requests:’ row

---

nginx conf files used (simple and optimized)

conf.d_used.zip

Endpoint and payload

It is a simple users list query:

{"operationName":null,"variables":{"offset":0,"sortBy":{"id":"desc"},"where":{}},"query":"query ($where: users_bool_exp, $offset: Int!, $sortBy: [users_order_by!]) {\n  members: users(where: $where, order_by: $sortBy, limit: 20, offset: $offset) {\n    id\n    created_at\n    username\n    email\n    name\n    is_active\n    last_online_at\n    last_login_time\n    last_login_ip\n  }\n  users_aggregate(where: $where) {\n    aggregate {\n      count\n      __typename\n    }\n    __typename\n  }\n}\n"}

Note: for ab. Formatted below for easy viewing:

{"operationName":null,"variables":{"offset":0,"sortBy":{"id":"desc"},"where":{}},"query":"query ($where: users_bool_exp, $offset: Int!, $sortBy: [users_order_by!]) {
  members: users(where: $where, order_by: $sortBy, limit: 20, offset: $offset) {
    id
    created_at
    username
    email
    name
    is_active
    last_online_at
    last_login_time
    last_login_ip
  }
  users_aggregate(where: $where) {
    aggregate {
      count
      __typename
    }
    __typename
  }
}
"}

ab commands

ab -n 200 -c 10 -v 4 -H 'accept: */*' -H 'Authorization: Bearer [xxx]' -H \
'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) \
Chrome/86.0.4240.75 Safari/537.36' -H 'Origin: http://localhost' -H \
'Accept-Language: en-US,en;q=0.9' -H 'Accept-Language: en-US,en;q=0.9' \
-p users_gql_query.txt 'https://localhost/api/query'

Also, added -k switch for the keep-alive

[tirumaraiselvan]

Hi @rrjanbiah Apologies for the delay in responding.

TLDR: Use a benchmarking tool which uses HTTP 1.1 as this is the most common protocol used by http clients.

The main issue arises because ab is using HTTP 1.0 protocol to make a connection with the server: https://serverfault.com/questions/317103/make-apachebenchab-send-http1-1-request/1034856 (can also be seen in tcpdump). It does appear warp (the web server graphql-engine uses) does not respond with a Connection: Keep-Alive header in HTTP 1.0 (even with -k) so it’s probably a bug in warp. This makes the client close the connection as per the HTTP spec: https://stackoverflow.com/a/17438094/1911889

The good news is almost ALL modern clients will use HTTP 1.1 to make a request ( https://stackoverflow.com/questions/2073392/is-http-1-0-still-in-use) which by default keeps the connection alive at the server side.

I used wrk to run a simple test and graphql-engine keeps the connection alive as expected:

wrk -d10 -t10 -c100 -s post_wrk.lua http://localhost:8080/v1/graphql

where post_wrk.lua is the following:

wrk.method = "POST"
wrk.body = '{"query":"query {\n  users {\n    id\n    email\n    username\n    created_at\n  }\n}"}'
wrk.headers["Content-Type"] = "application/json"

[tirumaraiselvan]

@rrjanbiah Sorry, haven’t had the time to look at this but I will do so soon.