Messages are getting overlapped when logging in the sys.stderr using multiprocessing

I am trying to log to sys.stderr using multiprocessing but the log messages are getting overlapped. Is this the intended behavior?

2019-07-01 15:19:49.197 | SUCCESS  | __mp_main__:worker:14 - My function executed successfully
2019-07-01 15:19:49.2052019-07-01 15:19:49.205 |  | SUCCESS SUCCESS  |  | __mp_main____mp_main__::workerworker::1414 -  - My function executed successfullyMy function executed successfully

2019-07-01 15:19:50.198 | SUCCESS  | __mp_main__2019-07-01 15:19:50.200: | workerSUCCESS : | 14__mp_main__ - :My function executed successfullyworker
:14 - My function executed successfully
2019-07-01 15:19:50.205 | SUCCESS  | __mp_main__:worker:14 - My function executed successfully
2019-07-01 15:19:50.209 | SUCCESS  | __mp_main__:worker:14 - My function executed successfully
2019-07-01 15:19:51.207 | SUCCESS  | __mp_main__:worker:14 - My function executed successfully
2019-07-01 15:19:52.198 | SUCCESS  | __mp_main__:worker:14 - My function executed successfully
2019-07-01 15:19:52.209 | SUCCESS  | __mp_main__:worker:14 - My function executed successfully

Here is the code that generated the above output.

import sys
import time
import random
from concurrent.futures import ProcessPoolExecutor

from loguru import logger

logger.remove()
logger.add(sys.stderr, enqueue=True)


def worker(seconds):
    time.sleep(seconds)
    logger.success("My function executed successfully")


def main():

    with ProcessPoolExecutor() as executor:
        seconds = [random.randint(1, 3) for i in range(10)]
        executor.map(worker, seconds)


if __name__ == "__main__":
    main()

Cheers, Chris

[Delgan]

I can finally close this issue! v0.4.0 has just been released and should solve this problem. 👍

As a reminder:

I added some code snippets and explanation in the documentation, I highly recommend to read it if you need to deal with multiprocessing on Windows:

• Compatibility with multiprocessing using enqueue argument

[Delgan]

Ok, I finally managed to fix that. 😛

Dealing with multiprocessing on Windows is really painful, especially considering that Loguru heavily relies on one unique global logger.

I had to rethink the management of handlers (see #157), it’s not perfect but the final solution is satisfactory enough in my opinion.

Basically, you will need to make your child processes “inherit” from the parent logger. It should not be passed as an argument once the child is started, it should be passed during construction of the child. To do so, most multiprocessing functions offer initializer and initargs arguments. This also requires added handlers to be picklable or, alternatively, to be added with enqueue=True.

So, the initial snippet of this issue may look like this once updated:

import sys
import time
import random
from concurrent.futures import ProcessPoolExecutor

from loguru import logger

def set_logger(logger_):
    global logger
    logger = logger_

def worker(seconds):
    time.sleep(seconds)
    logger.success("My function executed successfully")


def main():

    with ProcessPoolExecutor(initializer=set_logger, initargs=(logger, )) as executor:
        seconds = [random.randint(1, 3) for i in range(10)]
        executor.map(worker, seconds)


if __name__ == "__main__":
    logger.remove()
    logger.add(sys.stderr, enqueue=True)

    main()

I added some code snippets and explanation in the documentation, I highly recommend to read it if you need to deal with multiprocessing on Windows:

• Compatibility with multiprocessing using enqueue argument

I will leave this issue open until next loguru release. In the meantine, feel free to suggest improvements to the documentation or ask any question.