Sporatic errors with nginx connection pipelining
See original GitHub issueThere seems to be some weird issue with pipelining and or connection reuse and nginx. I have nginx sitting in front of my uvicorn/starlette app, and on patch/post requests I occasionally get 502’s from nginx (about 30-40% of the time). The nginx logs say:
upstream prematurely closed connection while reading response header from upstream
all over the place
and my app logs say:
WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.
with not much more information.
I forced connection reuse to not work by adding the default header Connection: close
, which forces nginx to kill the connection. Performance drops significantly, but at least I don’t get 502’s.
uvicorn.run(app, host='0.0.0.0', port=8445, headers=[('Server', 'custom'), ('Connection', 'close')], proxy_headers=True)
potentially relevant: the 502 doesn’t seem to ever happen on a GET
Issue Analytics
- State:
- Created 4 years ago
- Reactions:13
- Comments:67 (23 by maintainers)
Top Results From Across the Web
Avoiding the Top 10 NGINX Configuration Mistakes
Errors include insufficient file descriptors per worker, disabling proxy buffering, and not using upstream groups and keepalive connections.
Read more >Occasional pipelined requests failing against localhost nginx ...
Pretty stock Debian nginx config. Recently I noticed repeated sbuild failures during package installation, due to connection resets.
Read more >Intermittent 502s with nginx+gunicorn+django - Stack Overflow
Over the past few weeks we've been getting more and more 502 errors. Currently our stack is nginx + gunicron + django on...
Read more >NGINX / Cloudflare random 520 HTTP errors when HTTP2 is ...
After lots of troubleshooting, I've discovered if I remove the two http2 parameters after listen , then the issue goes away, and Cloudflare ......
Read more >Common Nginx Connection Errors - DigitalOcean
Common Nginx Connection Errors · Inspecting Your Nginx Error Log · Addressing “This site can't be reached” Error · Verifying your Nginx Web...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I did some tcpdump’ing and found that uvicorn does not send the tcp-FIN packet to nginx and so nginx assumes the connection is still useable. nginx then sends the next http request to the open socket but uvicorn replies with a tcp-FIN packet and logs the invalid request error message. So uvicorn did not have an open connection??? Even though it did not send the FIN…
Workarounds:
gunicorn -k uvicorn.workers.UvicornWorker -c /gunicorn_conf.py main:app --keep-alive=0
gunicorn -k uvicorn.workers.UvicornH11Worker -c /gunicorn_conf.py main:app
But I prefer to have reuse of connection available and working. Still investigating how to get that working.
If people impacted could look at their logs (if running 0.13.2 🍏 ) and tell us if they see something about a potentially malformed request that would help
note to self, reproduction steps, hopefully what we see here
docker run --rm --name nginx_uvicorn -p 80:80 -v /tmp/uvicorn.sock:/tmp/uvicorn.sock -v $PWD/nginx.conf:/etc/nginx/nginx.conf:ro nginx
with quite a minimal nginx.confrun uvicorn :
uvicorn app:app --uds /tmp/uvicorn.sock
send incorrect request with
curl -i 'http://localhost/?x=y z'
client result:
nginx result:
uvicorn result: