Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Gunicorn+UvicornWorker,access log not working after logrotate access log file.

See original GitHub issue

Checklist

  • [ YES ] The bug is reproducible against the latest release and/or master.
  • [ YES ] There are no similar issues or pull requests to fix it yet.

Describe the bug

Use FastAPI+Uvicorn+Gunicorn to deploy the production environment, When use logrotate to compressed log, access.log file will be reopen, but UvicornWorker not write access log to access.log.

To reproduce

  1. start server with gunicorn+UvicornWorker
(venv) [root@localhost task_registry_center]# gunicorn -w 2 -b 0.0.0.0:8000 -k uvicorn.workers.UvicornWorker --access-logfile log/access.log main:app
[2020-12-16 17:09:41 +0800] [3330] [INFO] Starting gunicorn 20.0.4
[2020-12-16 17:09:41 +0800] [3330] [INFO] Listening at: http://0.0.0.0:8000 (3330)
[2020-12-16 17:09:41 +0800] [3330] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-12-16 17:09:41 +0800] [3332] [INFO] Booting worker with pid: 3332
[2020-12-16 17:09:41 +0800] [3333] [INFO] Booting worker with pid: 3333
  1. now access log file is empty
[root@localhost log]# pwd
/data/task_registry_center/log
[root@localhost log]# ll
total 20
-rw-r--r--. 1 root root    0 Dec 16 16:56 access.log
-rw-r--r--. 1 root root    0 Dec 16 16:41 client.log
-rw-r--r--. 1 root root    0 Dec 16 16:56 debug.log
-rw-r--r--. 1 root root    0 Dec 16 16:41 elk_client.log
-rw-r--r--. 1 root root 5402 Dec 16 17:08 error.log
-rw-r--r--. 1 root root    0 Dec 16 16:41 id_server.log
-rw-r--r--. 1 root root 4029 Dec 16 17:09 main.log
-rw-r--r--. 1 root root 4480 Dec 16 17:10 tasks.log
  1. call one api 10 times, now log path is :
[root@localhost log]# ll
total 24
-rw-r--r--. 1 root root  720 Dec 16 17:11 access.log

access.log context is

192.168.11.1:60477 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60478 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60479 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60480 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60481 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60482 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60483 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60484 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60485 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60486 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
  1. manual call logrotate
[root@localhost ~]# /usr/sbin/logrotate -f /etc/logrotate.d/task_registry_center

logrotate config is :

[root@localhost ~]# cat /etc/logrotate.d/task_registry_center                                                                                                                                                   
/data/task_registry_center/log/*.log {                                                                                                                                  
        size 1M                                                                                                                                                         
        rotate 20                                                                                                                                                       
        compress                                                                                                                                                        
        nodelaycompress                                                                                                                                                 
        dateformat -%Y-%m-%d-%s                                                                                                                                         
        dateext                                                                                                                                                         
        missingok                                                                                                                                                       
        notifempty                                                                                                                                                      
        postrotate                                                                                                                                                      
            killall -s USR1 gunicorn                                                                                                                  13,9          All
        endscript
}
  1. gunicorn receive USR1 signal to reopen access log file and log this
[2020-12-16 17:12:16 +0800] [3330] [INFO] Handling signal: usr1
[2020-12-16 17:12:16 +0800] [3330] [INFO] Handling signal: usr1
[2020-12-16 17:12:16 +0800] [3330] [INFO] Handling signal: usr1
[2020-12-16 17:12:16 +0800] [3330] [INFO] Handling signal: usr1
  1. now log path is:
[root@localhost log]# ll
total 20
-rw-r--r--. 1 root root   0 Dec 16 17:12 access.log
-rw-r--r--. 1 root root 125 Dec 16 17:11 access.log-2020-12-16-1608109936.gz
  1. then call one api 10 times, log path is:
[root@localhost log]# ll
total 20
-rw-r--r--. 1 root root   0 Dec 16 17:12 access.log
-rw-r--r--. 1 root root 125 Dec 16 17:11 access.log-2020-12-16-1608109936.gz

access.log-2020-12-16-1608109936.gz context still is

192.168.11.1:60477 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60478 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60479 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60480 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60481 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60482 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60483 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60484 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60485 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200
192.168.11.1:60486 - "GET /api/client?page_size=10&page=1 HTTP/1.1" 200

this 10 times does not log to any access log.

Expected behavior

the lastest 10 times api call will log access log to reopened access.log file.

Actual behavior

the lastest 10 times api call does not log to any access log file.

Debugging material

I have manual send USR1 signal instead of call logrotate, the result is same , new log file is created,but no access record log to it.

Environment

  • OS is centos7
  • Python 3.8.3
  • gunicorn 20.0.4
  • uvicorn 0.13.1
  • fastapi 0.62.0

Additional context

I have find someone has same issue as me on so,see https://stackoverflow.com/questions/61794303/why-doesnt-gunicorn-write-log-to-reopened-file-while-rotating-a-log-file

I have try to rewirte UvicornWorker like:

class RequestUvicornWorker(UvicornWorker):

    def __init__(self, *args, **kwargs):
        super(RequestUvicornWorker, self).__init__(*args, **kwargs)
        logger = logging.getLogger("uvicorn.access")
        handler = WatchedFileHandler(filename='log/access.log')
        logger.handlers = self.log.access_log.handlers
        logger.setLevel(self.log.access_log.level)

change log handler to WatchedFileHandler, still not working. And i have a flask app deploy with gunicorn, this will correctly log access log to new access log file after logrotate.

Hope for reply,THANK YOU!

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:14 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
yinkhcommented, Jan 19, 2021

I wrote a very simple Starlette app to test the log rotate behavior.

umain.py
import logging

from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route

logging.basicConfig(level=logging.DEBUG, filename='./log.txt')

logger = logging.getLogger(__name__)


async def some_startup_task():
    logger.info("startup")


async def some_shutdown_task():
    logger.info("shutdown")


async def homepage(request):
    logger.debug("homepage()")
    return JSONResponse({'hello': 'world'})


app = Starlette(debug=False,
                routes=[Route('/', homepage)],
                on_startup=[some_startup_task],
                on_shutdown=[some_shutdown_task])

Start this Starlette app with Gunicorn and UvicornWorker.

$ gunicorn -w 2 -k uvicorn.workers.UvicornWorker umain:app

[2021-01-12 17:26:17 +0900] [287] [INFO] Starting gunicorn 20.0.4
[2021-01-12 17:26:17 +0900] [287] [INFO] Listening at: http://127.0.0.1:8000 (287)
[2021-01-12 17:26:17 +0900] [287] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-01-12 17:26:17 +0900] [289] [INFO] Booting worker with pid: 289
[2021-01-12 17:26:17 +0900] [290] [INFO] Booting worker with pid: 290
[2021-01-12 17:26:17 +0900] [289] [INFO] Started server process [289]
[2021-01-12 17:26:17 +0900] [290] [INFO] Started server process [290]
[2021-01-12 17:26:17 +0900] [289] [INFO] Waiting for application startup.
[2021-01-12 17:26:17 +0900] [290] [INFO] Waiting for application startup.
[2021-01-12 17:26:17 +0900] [290] [INFO] Application startup complete.
[2021-01-12 17:26:17 +0900] [289] [INFO] Application startup complete.

Invoking kill -USR1 287 in order to rotate the log file, then the console log is as follows:

[2021-01-12 17:26:33 +0900] [287] [INFO] Handling signal: usr1
[2021-01-12 17:26:33 +0900] [300] [INFO] Booting worker with pid: 300
[2021-01-12 17:26:33 +0900] [311] [INFO] Booting worker with pid: 311
[2021-01-12 17:26:33 +0900] [300] [INFO] Started server process [300]
[2021-01-12 17:26:33 +0900] [300] [INFO] Waiting for application startup.
[2021-01-12 17:26:33 +0900] [300] [INFO] Application startup complete.
[2021-01-12 17:26:33 +0900] [311] [INFO] Started server process [311]
[2021-01-12 17:26:33 +0900] [311] [INFO] Waiting for application startup.
[2021-01-12 17:26:33 +0900] [311] [INFO] Application startup complete.

Worker processes were restarted.

The log.txt indicates that the on_startup task were called but the on_shutdown task were not called.

$ cat log.txt
INFO:umain:startup
INFO:umain:startup
INFO:umain:startup
INFO:umain:startup

How can we reopen a log file without restarting worker processes for log rotation? And, on restarting worker processes, why the on_shutdown task are not called?

Environment

  • macOS 10.15.7
  • Python 3.8.6
  • uvicorn 0.13.3
  • gunicorn 20.0.4
  • starlette 0.13.6

I has same question. uvicorn 0.13.3 will restart gunicorn worker for USR1 signal,which is different from gunicorn. Change init_signals to original will work:

    def init_signals(self):
        # reset signaling
        for s in self.SIGNALS:
            signal.signal(s, signal.SIG_DFL)
        # init new signaling
        signal.signal(signal.SIGQUIT, self.handle_quit)
        signal.signal(signal.SIGTERM, self.handle_exit)
        signal.signal(signal.SIGINT, self.handle_quit)
        signal.signal(signal.SIGWINCH, self.handle_winch)
        signal.signal(signal.SIGUSR1, self.handle_usr1)
        signal.signal(signal.SIGABRT, self.handle_abort)

        # Don't let SIGTERM and SIGUSR1 disturb active requests
        # by interrupting system calls
        signal.siginterrupt(signal.SIGTERM, False)
        signal.siginterrupt(signal.SIGUSR1, False)

        if hasattr(signal, 'set_wakeup_fd'):
            signal.set_wakeup_fd(self.PIPE[1])
1reaction
euri10commented, Jan 12, 2021

How can we reopen a log file without restarting worker processes for log rotation?

no idea, someone more familiar with gunicorn internals might have a clue

And, on restarting worker processes, why the on_shutdown task are not called?

it’s because, I suppose, we have a SIGTERM issue (see #852) , I just tested your snippet on #853 and I have the shutdown logrotate in files

Read more comments on GitHub >

github_iconTop Results From Across the Web

Gunicorn holding onto old logs after logrotate - Stack Overflow
The logs get rotated correctly, compressed, and a new access.log gets created. However, gunicorn does not release a 'pointer' to the old log...
Read more >
Logrotation on output log file causes empty log files #106
After logrotate runs overnight, it copies sends it off to the file fine, but forever then doesn't continue writing to the main log...
Read more >
How to Manage Log Files Using Logrotate - Datadog
Log rotation, which is managed by tools like logrotate, solves these problems by performing routine maintenance on log files.
Read more >
Logrotate not working - Server Fault
I think that weekly means that logrotate wants to see at least a week old entry for your access.log file in order to...
Read more >
Setting up logrotate in Linux | Enable Sysadmin - Red Hat
However, this poses a problem as large file sizes make working with log files a cumbersome process for both the system and the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

uvirorn.workers.UvicornWorker fails with certain ASGI middleware

Next page

uvicorn.run with reload or workers in ProcessPoolExecutor fails with Bad file descriptor error