question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

gunicorn keeps booting workers when exceptions raised at startup events

See original GitHub issue

Hi, I have the same issues when deploy FastAPI using gunicorn + uvicorn,it keeps booting workers when exceptions raised at startup events.

main.py

from fastapi import FastAPI

app = FastAPI()


@app.on_event("startup")
async def startup():
    raise Exception("error")


@app.get("/")
async def hello():
    return {"msg": "hello world"}

gunicorn.conf.py

import json
import os

worker_class = "uvicorn.workers.UvicornWorker"

host = os.getenv("HOST", "0.0.0.0")
port = os.getenv("PORT", "8000")
bind_env = os.getenv("BIND", None)
use_loglevel = os.getenv("LOG_LEVEL", "info")
if bind_env:
    use_bind = bind_env
else:
    use_bind = f"{host}:{port}"


accesslog_var = os.getenv("ACCESS_LOG", "-")
use_accesslog = accesslog_var or None
errorlog_var = os.getenv("ERROR_LOG", "-")
use_errorlog = errorlog_var or None
graceful_timeout_str = os.getenv("GRACEFUL_TIMEOUT", "120")
timeout_str = os.getenv("TIMEOUT", "120")
keepalive_str = os.getenv("KEEP_ALIVE", "5")

# Gunicorn config variables
loglevel = use_loglevel
bind = use_bind
errorlog = use_errorlog
accesslog = use_accesslog
graceful_timeout = int(graceful_timeout_str)
timeout = int(timeout_str)
keepalive = int(keepalive_str)

# For debugging and testing
log_data = {
    "loglevel": loglevel,
    "bind": bind,
    "graceful_timeout": graceful_timeout,
    "timeout": timeout,
    "keepalive": keepalive,
    "errorlog": errorlog,
    "accesslog": accesslog,
}
print(json.dumps(log_data))

gunicorn log

{"loglevel": "info", "bind": "0.0.0.0:8000", "graceful_timeout": 120, "timeout": 120, "keepalive": 5, "errorlog": "-", "accesslog": "-", "host": "0.0.0.0", "port": "8000"}
[2021-06-01 21:13:05 +0800] [40946] [INFO] Starting gunicorn 20.0.4
[2021-06-01 21:13:05 +0800] [40946] [INFO] Listening at: http://0.0.0.0:8000 (40946)
[2021-06-01 21:13:05 +0800] [40946] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-06-01 21:13:05 +0800] [40949] [INFO] Booting worker with pid: 40949
[2021-06-01 21:13:05 +0800] [40949] [INFO] Started server process [40949]
[2021-06-01 21:13:05 +0800] [40949] [INFO] Waiting for application startup.
[2021-06-01 21:13:05 +0800] [40949] [ERROR] Traceback (most recent call last):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 526, in lifespan
    async for item in self.lifespan_context(app):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 467, in default_lifespan
    await self.startup()
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 502, in startup
    await handler()
  File "./app/main.py", line 16, in startup
    raise Exception("error")
Exception: error

[2021-06-01 21:13:05 +0800] [40949] [ERROR] Application startup failed. Exiting.
[2021-06-01 21:13:05 +0800] [40949] [INFO] Worker exiting (pid: 40949)
[2021-06-01 21:13:05 +0800] [40950] [INFO] Booting worker with pid: 40950
[2021-06-01 21:13:06 +0800] [40950] [INFO] Started server process [40950]
[2021-06-01 21:13:06 +0800] [40950] [INFO] Waiting for application startup.
[2021-06-01 21:13:06 +0800] [40950] [ERROR] Traceback (most recent call last):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 526, in lifespan
    async for item in self.lifespan_context(app):
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 467, in default_lifespan
    await self.startup()
  File "./envs/test/lib/python3.7/site-packages/starlette/routing.py", line 502, in startup
    await handler()
  File "./app/main.py", line 16, in startup
    raise Exception("error")
Exception: error

[2021-06-01 21:13:06 +0800] [40950] [ERROR] Application startup failed. Exiting.

I hope the gunicorn would exit when exceptions raised, but it keeps booting workers. Can someone helps me, thanks?

_Originally posted by @fanchunke1991 in https://github.com/encode/uvicorn/issues/1031#issuecomment-852121760_

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:12 (9 by maintainers)

github_iconTop GitHub Comments

1reaction
Kludexcommented, Sep 6, 2021

Oh. I think you’ve confused the current issue. This issue is for gunicorn exit behavior when startup fails. We don’t have a matching behavior for uvicorn yet.

We have this PR from some long time ago: https://github.com/encode/uvicorn/pull/835 Which I don’t think we should merge, as I’d prefer to match the exit code from gunicorn worker.

0reactions
vackosarcommented, Sep 6, 2021

@Kludex thanks, yes. Any non zero exit code there would be great (even exit code 3 like you suggest here if I understand well!). Otherwise, how to find out that the app failed? Returning 0 is destroying information for me. Even if it required setting a switch or running through gunicorn, just to have some way to find out that it failed on startup.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Gunicorn worker timeout error - Stack Overflow
In Docker I keep trained LightGBM model + Flask serving requests. As HTTP server I used gunicorn 19.9.0 . When I run my...
Read more >
Gunicorn Documentation - Read the Docs
Gunicorn 'Green Unicorn' is a Python WSGI HTTP Server for UNIX. It's a pre-fork worker model ported from Ruby's. Unicorn project.
Read more >
Settings — Gunicorn 20.1.0 documentation
Restart workers when code changes. This setting is intended for development. It will cause workers to be restarted whenever application code changes. The ......
Read more >
Azure Web App Exception in Worker Process "LookupError ...
The problem is that the server doesn't keep an instance of the project code running. Instead, there is some sort of interceptor that...
Read more >
How To Set Up an ASGI Django App with Postgres, Nginx ...
To use uvicorn workers with the gunicorn server, enter your project directory and use the following gunicorn command to load the project's ASGI ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found