nginx + sanic problems

Describe the bug

Hi, I’m using nginx as reverse-proxy to handle TLS and route to multiple applications on same server. My sanic app is the main route that handles most of the traffic. The traffic is pretty high (~5000 requests per second, on 16 workers) and I get some random errors in the nginx every few minutes. The errors are from these 3 types:

2022/08/18 12:05:20 [error] 42#42: *34037 writev() failed (32: Broken pipe) while sending request to upstream, client: 10.100.13.56, server: , request: "POST /engine HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "10.100.13.154"
2022/08/18 12:05:40 [error] 42#42: *35312 upstream prematurely closed connection while reading response header from upstream, client: 10.100.13.82, server: , request: "POST /engine HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "10.100.13.154"
2022/08/18 12:10:52 [error] 40#40: *55659 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.100.13.66, server: , request: "POST /engine HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "10.100.13.154"

On the sanic side there’s nothing in the logs. access_log=True and also debug=True didn’t show any error or anything suspicious, I also tried verbosity=5 but still nothing relevant.

Going deeper, I started tshark to try to see what actually happens, and I saw that the Sanic side actually sent tcp RST in the middle of a POST request being sent to it.

I tried setting SANIC_NOISY_EXCEPTIONS, increase SANIC_KEEP_ALIVE_TIMEOUT and REQUEST_BUFFER_SIZE - but still nothing helps.

I saw https://github.com/sanic-org/sanic/issues/991 which is quite old and has similar errors, but I’m using latest version and still have them.

Any idea what can cause it? Or how to fix it?

Environment (please complete the following information):

• OS: Ubuntu 18.04
• Sanic Version: 22.6.2 (Running inside docker using FROM python:3.8-slim-buster)
• Docker version: 20.10.16, running containers with host network

[Tronic]

There is an open issue (with a hacky workaround too) https://github.com/MagicStack/uvloop/issues/241

[ahopkins]

Nice find.