Feature: multiple independent bound Loggers wich have separate set of handlers

Hi. In my app I found a need to create a Logger with its own set of handlers, disjoint with the root logger. In my usecase, I am trying to spawn many tasks (async functions), so that each task logs into its own file. Minimal code snippet:

import asyncio
from loguru import logger

async def runtasks():
    async def worker(taskid, log):
        log.info(f'ping {taskid}')
        
    tasks = []
    for taskid in (1,2,3,1):
        log = logger.bind(taskid=taskid)   # Bound logger still shares handlers with root logger :(
        log.add(f'{taskid}.log')
        tasks.append(asyncio.create_task(worker(taskid, log)))
    await asyncio.gather(*tasks)

asyncio.run(runtasks())

In this code snippet, i spawn 4 tasks, with identifiers 1, 2, 3, and 1 again. I want each task to write into its log file, named {taskid}.log. Ufortunately it is hard to accomplish right with any of the logging libraries i tried - loguru, structlog, stdlib. The above snippet fails to achieve what i want, and the contents of 2.log contains the log messages from the wrong tasks:

> cat 2.log 
2019-03-28 17:20:14.000 | INFO     | __main__:worker:178 - ping 1
2019-03-28 17:20:14.001 | INFO     | __main__:worker:178 - ping 2
2019-03-28 17:20:14.001 | INFO     | __main__:worker:178 - ping 3
2019-03-28 17:20:14.002 | INFO     | __main__:worker:178 - ping 1

It would work if the call to log = logger.bind(taskid=taskid) would return a bound logger instance, not connected to the root logger, so that subsequent call to log.add() only affected this bound logger instance. That would be a nice feature to have.

[Delgan]

Next Loguru release will add a way to use independent loggers, finally. 🎉

I refactored the logger so that handlers are bound to the logger instances and not class (see #157). I also removed some internal functions which caused the logger to not be picklable. Now that logger can be pickled, it also means it can be deep copied using copy.deepcopy().

So, if you want to use independent loggers with independent handlers, you just have to call logger_2 = copy.deepcopy(logger). Updating your initial code snippet, it may look like this for example:

import asyncio
import copy
from loguru import logger

async def runtasks():
    async def worker(taskid, log):
        log.info(f'ping {taskid}')

    loggers = {}

    logger.remove()

    for taskid in (1, 2, 3):
        task_logger = copy.deepcopy(logger)
        task_logger.add(f'{taskid}.log')
        loggers[taskid] = task_logger

    tasks = []
    for taskid in (1,2,3,1):
        log = loggers[taskid]
        tasks.append(asyncio.create_task(worker(taskid, log)))
    await asyncio.gather(*tasks)

asyncio.run(runtasks())

This is intentionally not part of the Loguru API (it requires to import the copy module) because I think this goes against the design of Loguru (one global logger to manage everything). However, I’m very happy there exists a workaround for some use cases like the one of this issue.

Note that most of the handlers are not picklable by default, so it’s better to remove() all handlers before copying the logger.

I also added this to the documentation:

• Creating independent loggers with separate set of handlers

I will close this issue once next Loguru release is published. 😉

[Delgan]

@lsabi I just published the v0.4.0!

The deepcopy() workaround should work fine. Please, tell me if you have any problem. 👍