Issues with rotate

I’m aware I could just be misusing loguru, but have noticed some non-intuitive behavior around ‘rotation’.

I’m running: Python 3.7.6 (default, Feb 26 2020, 20:54:15) [GCC 7.3.1 20180712 (Red Hat 7.3.1-6)] on linux and loguru 0.5.1

I have a web-app built in bottle that uses the gunicorn webserver and has 6 worker processes.

In my app I have the following code: logger.add("pods.log", rotation="sunday at 03:00")

I think I’m running into problems when I have multiple users on the website simultaneously and gunicorn runs multithreaded copies of the website. This is actually kind of unusual; the site is kind of sandbox and the user volume is really low… we have maybe 5 visitors per day. I have a bunch of single-threaded cron jobs that also rotate their logs like this one does, and it works fine. However this is the only multithreaded program that I have and It seems to create lots more log files than I ask for. Here’s what the last few weeks look like:

-rw-r--r-- 1 root     root     135829 Oct 31 17:55 pods.2020-10-26_04-57-05_964506.log
-rw-r--r-- 1 root     root       6893 Nov  2 20:11 pods.2020-11-01_18-20-42_710982.log
-rw-r--r-- 1 root     root       1384 Nov  2 03:23 pods.2020-11-02_03-20-20_265978.log
-rw-r--r-- 1 root     root       2900 Nov  2 07:36 pods.2020-11-02_03-42-34_356316.log
-rw-r--r-- 1 root     root       3303 Nov  2 20:16 pods.2020-11-02_07-38-17_428417.log
-rw-r--r-- 1 root     root      92259 Nov  7 20:07 pods.2020-11-02_08-14-48_165825.log
-rw-r--r-- 1 root     root       5989 Nov 10 21:55 pods.2020-11-09_07-38-13_902695.log
-rw-r--r-- 1 root     root       4503 Nov 10 00:55 pods.2020-11-09_08-36-40_989960.log
-rw-r--r-- 1 root     root       6215 Nov 11 03:45 pods.2020-11-09_17-55-30_725461.log
-rw-r--r-- 1 root     root       5524 Nov 11 03:42 pods.2020-11-09_18-03-11_396160.log
-rw-r--r-- 1 root     root       4538 Nov 11 06:05 pods.2020-11-10_01-11-39_544732.log
-rw-r--r-- 1 root     root        562 Nov 10 01:56 pods.2020-11-10_01-56-49_697171.log
-rw-r--r-- 1 root     root      25173 Nov 13 17:08 pods.log

We can see on Nov 11, it created a lot of logs… not sure why; it’s not even a Sunday.

Are my expectations wrong or am I doing something wrong?

Regardless – I love this tool! Thanks for making it!

[kylinat2688]

@cestes Did that fix your problem? Can I close this issue?

I also meet a problem like this, and I have used the parameter equeue=True, the difference is that I add log in gunicorn hood when_ready. when the log rotate, gunicorn crash,. my version is 0.5.3, here is my code:

from loguru import logger
proc_name = "picture_finger_api"
workers = 8
worker_class = 'gevent'
bind = "0.0.0.0:7410"
def when_ready(server):
    log_config = {'enqueue': True, 'rotation': '00:00', 'retention': '2 weeks', 'compression': 'tar.gz'}
    logger.add('log/app.log', level='DEBUG', **log_config)

crash core file has nothing,like this: #0 0x00007f2388dcf14b in ?? () #1 0x00007f2388df3b94 in ?? () #2 0x0000000000000009 in ?? () #3 0x0000000000000000 in ?? ()

[Delgan]

Well, you are indirectly calling logger.add() in the if __name__ == '__main__' block as it’s part of your main() function. So, moving logger.add() around would not change anything.

I think the problem does come from multiprocessing, but not exactly for the reason I first mentioned.

from loguru import logger
import multiprocessing
import time

def worker():
    while True:
        time.sleep(0.01)
        logger.info("Logging...")



if __name__ == "__main__":
    logger.add("foo.log", rotation="200 KB")

    multiprocessing.set_start_method("fork")
    processes = [multiprocessing.Process(target=worker) for _ in range(4)]

    for p in processes:
        p.start()

    for p in processes:
        p.join()

The new processes are probably created using the "fork" method. In the end, the result is the same: each child process ends up with an independent handler but logging to the same file. This means every handler will try to rotate the file separately, causing the duplicated files you’re seeing.

What you need is a way to synchronize the workers so that they effectively use one common handler. That way, the rotation will be triggered only once, and then all processes will keep logging to the same shared file.

You should try to use enqueue=True while configuring the logger:

logger.add("pods.log", enqueue=True, rotation="sunday at 03:00")

I also advise you to read this paragraph in the documentation that may help you understand how this parameter works: Compatibility with multiprocessing using enqueue argument.