Watchtower + Multiprocessing

The logger is not successfully writing to CloudWatch when using multiprocessing. I tested to see whether this was my configuration by dropping a watchtower handler and using a file handler. This logged perfectly, however, when switching back to the watchtower handler only messages before and after outputs = pool.map(worker, inputs) worked.

Any idea how to fix this? Setting use_queues to True didn’t help.

Sample code:

import watchtower
import logging
from multiprocessing import Lock, Process, Queue, current_process, Manager, Pool

def worker(var):
    logger.debug("Incoming variable: %s" % var)
    logger.debug("Outgoing variable: %s" % (var+1))
    return var+1

def main():
    inputs = []
    for i in xrange(1000):
        inputs.append(i)

    logger.debug("Starting run now!")
    pool = Pool(processes=3)
    outputs = pool.map(worker, inputs)
    pool.close()
    pool.join()
    logger.debug("Just finished run")

if __name__ == "__main__":
    logger = logging.getLogger("multi")
    logger.setLevel(logging.DEBUG)

    fh = logging.FileHandler("test.log")
    fh.setLevel(logging.DEBUG)
    logger.addHandler(fh)

    wt_project_handler = watchtower.CloudWatchLogHandler(stream_name="test",
                                                         use_queues=True)
    wt_project_handler.setLevel(logging.DEBUG)
    logger.addHandler(wt_project_handler)

    main()

[Audace]

Solved this issue by using this repo: https://github.com/jruere/multiprocessing-logging, which was spun out of this post: http://stackoverflow.com/questions/641420/how-should-i-log-while-using-multiprocessing-in-python.

All it resulted in was importing multiprocessing_logging and then adding multiprocessing_logging.install_mp_handler(logger)

[kislyuk]

The suggested way to use logging in multiprocessing pools is to share nothing. Use one logger per worker process (or thread) and initialize the logger after forking. A shared logger will not work correctly with multiprocessing due to the stateful nature of the logger and race conditions that will arise between different copies of the logger in the different processes.