question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Sporatic errors with nginx connection pipelining

See original GitHub issue

There seems to be some weird issue with pipelining and or connection reuse and nginx. I have nginx sitting in front of my uvicorn/starlette app, and on patch/post requests I occasionally get 502’s from nginx (about 30-40% of the time). The nginx logs say:

upstream prematurely closed connection while reading response header from upstream

all over the place

and my app logs say:

WARNING: Invalid HTTP request received.
WARNING: Invalid HTTP request received.

with not much more information.

I forced connection reuse to not work by adding the default header Connection: close, which forces nginx to kill the connection. Performance drops significantly, but at least I don’t get 502’s.

uvicorn.run(app, host='0.0.0.0', port=8445, headers=[('Server', 'custom'), ('Connection', 'close')], proxy_headers=True)

potentially relevant: the 502 doesn’t seem to ever happen on a GET

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:13
  • Comments:67 (23 by maintainers)

github_iconTop GitHub Comments

7reactions
MilovdZeecommented, Sep 9, 2019

I did some tcpdump’ing and found that uvicorn does not send the tcp-FIN packet to nginx and so nginx assumes the connection is still useable. nginx then sends the next http request to the open socket but uvicorn replies with a tcp-FIN packet and logs the invalid request error message. So uvicorn did not have an open connection??? Even though it did not send the FIN…

Workarounds:

  • Setting keep-alive to 0 fixes this while uvicorn then sends a tcp-FIN after each request. gunicorn -k uvicorn.workers.UvicornWorker -c /gunicorn_conf.py main:app --keep-alive=0
  • Also switching to the H11 worker fixes the issue because this also closes the connection after each request. gunicorn -k uvicorn.workers.UvicornH11Worker -c /gunicorn_conf.py main:app

But I prefer to have reuse of connection available and working. Still investigating how to get that working.

3reactions
euri10commented, Dec 28, 2020

If people impacted could look at their logs (if running 0.13.2 🍏 ) and tell us if they see something about a potentially malformed request that would help


note to self, reproduction steps, hopefully what we see here

  1. run nginx with: docker run --rm --name nginx_uvicorn -p 80:80 -v /tmp/uvicorn.sock:/tmp/uvicorn.sock -v $PWD/nginx.conf:/etc/nginx/nginx.conf:ro nginx with quite a minimal nginx.conf
events {
}

http {
  server {
    listen 80;
    location / {
      proxy_pass http://uvicorn;
    }
  }
  upstream uvicorn {
    server unix:/tmp/uvicorn.sock fail_timeout=0;
  }
}

  1. run uvicorn : uvicorn app:app --uds /tmp/uvicorn.sock

  2. send incorrect request with curl -i 'http://localhost/?x=y z'


client result:

- ❯ curl -i 'http://localhost/?x=y z'
HTTP/1.1 502 Bad Gateway
Server: nginx/1.19.1
Date: Mon, 28 Dec 2020 19:27:46 GMT
Content-Type: text/html
Content-Length: 157
Connection: keep-alive

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.19.1</center>
</body>
</html>

nginx result:

2020/12/28 19:27:46 [error] 29#29: *4 upstream prematurely closed connection while reading response header from upstream, client: 172.17.0.1, server: , request: "GET /?x=y z HTTP/1.1", upstream: "http://unix:/tmp/uvicorn.sock:/?x=y z", host: "localhost"
172.17.0.1 - - [28/Dec/2020:19:27:46 +0000] "GET /?x=y z HTTP/1.1" 502 157 "-" "curl/7.64.0"

uvicorn result:

WARNING:  Invalid HTTP request received.
Traceback (most recent call last):
  File "/home/lotso/PycharmProjects/uvicorn/uvicorn/protocols/http/httptools_impl.py", line 165, in data_received
    self.parser.feed_data(data)
  File "httptools/parser/parser.pyx", line 193, in httptools.parser.parser.HttpParser.feed_data
httptools.parser.errors.HttpParserError: invalid constant string
Read more comments on GitHub >

github_iconTop Results From Across the Web

Avoiding the Top 10 NGINX Configuration Mistakes
Errors include insufficient file descriptors per worker, disabling proxy buffering, and not using upstream groups and keepalive connections.
Read more >
Occasional pipelined requests failing against localhost nginx ...
Pretty stock Debian nginx config. Recently I noticed repeated sbuild failures during package installation, due to connection resets.
Read more >
Intermittent 502s with nginx+gunicorn+django - Stack Overflow
Over the past few weeks we've been getting more and more 502 errors. Currently our stack is nginx + gunicron + django on...
Read more >
NGINX / Cloudflare random 520 HTTP errors when HTTP2 is ...
After lots of troubleshooting, I've discovered if I remove the two http2 parameters after listen , then the issue goes away, and Cloudflare ......
Read more >
Common Nginx Connection Errors - DigitalOcean
Common Nginx Connection Errors · Inspecting Your Nginx Error Log · Addressing “This site can't be reached” Error · Verifying your Nginx Web...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found