Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

New backend fails if logger set up with additional handlers

See original GitHub issue

Consider this code:

import logging
import time
import sys

# from sklearn.externals.joblib import Parallel, delayed
from joblib import Parallel, delayed

logger = logging.getLogger()
formatter = logging.Formatter(
    '%(asctime)s %(levelname)s: %(message)s '
    '[in %(pathname)s:%(lineno)d]')
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(formatter)
logger.addHandler(handler)


def func(i):
    logger.warning(f"Function calling {i}")
    time.sleep(0.5)


with Parallel(n_jobs=4) as parallel:
    parallel(delayed(func)(i) for i in range(10))

When running with the sklearn version of joblib it produces the following output:

2018-02-20 16:30:50,960 WARNING: Function calling 0 [in log_test.py:18]
2018-02-20 16:30:50,963 WARNING: Function calling 1 [in log_test.py:18]
2018-02-20 16:30:50,963 WARNING: Function calling 2 [in log_test.py:18]
2018-02-20 16:30:50,965 WARNING: Function calling 3 [in log_test.py:18]
2018-02-20 16:30:51,461 WARNING: Function calling 4 [in log_test.py:18]
2018-02-20 16:30:51,464 WARNING: Function calling 5 [in log_test.py:18]
2018-02-20 16:30:51,464 WARNING: Function calling 6 [in log_test.py:18]
2018-02-20 16:30:51,466 WARNING: Function calling 7 [in log_test.py:18]
2018-02-20 16:30:51,962 WARNING: Function calling 8 [in log_test.py:18]
2018-02-20 16:30:51,965 WARNING: Function calling 9 [in log_test.py:18]

If running it on the current git head, it fails:

joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/loky/backend/queues.py", line 150, in _feed
    obj, reducers=reducers)
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/loky/backend/reduction.py", line 145, in dumps
    p.dump(obj)
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/parallel.py", line 140, in __getstate__
    for func, args, kwargs in self.items]
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/parallel.py", line 140, in <listcomp>
    for func, args, kwargs in self.items]
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 881, in dumps
    cp.dump(obj)
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 268, in dump
    return Pickler.dump(self, obj)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 405, in save_function
    self.save_function_tuple(obj)
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 553, in save_function_tuple
    save(state)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 781, in save_list
    self._batch_appends(obj)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 808, in _batch_appends
    save(tmp[0])
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 496, in save
    rv = reduce(self.proto)
TypeError: can't pickle _thread.RLock objects
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "log_test.py", line 24, in <module>
    parallel(delayed(func)(i) for i in range(10))
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/parallel.py", line 813, in __call__
    self.retrieve()
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/parallel.py", line 714, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 468, in wrap_future_result
    return future.result(timeout=timeout)
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/loky/_base.py", line 424, in result
    return self.__get_result()
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/loky/_base.py", line 382, in __get_result
    raise self._exception
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/loky/backend/queues.py", line 150, in _feed
    obj, reducers=reducers)
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/loky/backend/reduction.py", line 145, in dumps
    p.dump(obj)
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/parallel.py", line 140, in __getstate__
    for func, args, kwargs in self.items]
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/parallel.py", line 140, in <listcomp>
    for func, args, kwargs in self.items]
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 881, in dumps
    cp.dump(obj)
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 268, in dump
    return Pickler.dump(self, obj)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 405, in save_function
    self.save_function_tuple(obj)
  File "/home/javier/anaconda3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 553, in save_function_tuple
    save(state)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 781, in save_list
    self._batch_appends(obj)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 808, in _batch_appends
    save(tmp[0])
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/home/javier/anaconda3/lib/python3.6/pickle.py", line 496, in save
    rv = reduce(self.proto)
TypeError: can't pickle _thread.RLock objects

The error originates from the assignment of a handler to the logger (code works if the handler is removed, obviously without the fancy formatting). This was the cause of the second error mentioned in https://github.com/joblib/joblib/issues/629#issuecomment-364158151

Issue Analytics

  • State:open
  • Created 6 years ago
  • Reactions:1
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

5reactions
lestevecommented, Feb 21, 2018

As a work-around, your best bet is to define the logger in a helper function:

import logging
import time
import sys

# from sklearn.externals.joblib import Parallel, delayed
from joblib import Parallel, delayed

def get_logger():
    logger = logging.getLogger()
    formatter = logging.Formatter(
        '%(asctime)s %(levelname)s: %(message)s '
        '[in %(pathname)s:%(lineno)d]')
    handler = logging.StreamHandler(sys.stdout)
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    return logger


def func(i):
    logger = get_logger()
    logger.warning(f"Function calling {i}")
    time.sleep(0.5)


with Parallel(n_jobs=4) as parallel:
    parallel(delayed(func)(i) for i in range(10))

More details: another work-around is to give a name to the logger. cloudpickle seems to have a bug when pickling the root logger that it does not have when pickling other loggers. In this case though, the formatter is lost with the loky backend, so the get_logger function is probably the best thing to do.

1reaction
tommctcommented, Sep 18, 2020

Following from https://github.com/joblib/joblib/issues/634#issuecomment-367321025, I was still unable to get a StreamHandler working when using loky in a Jupyter notebook, but when naming the logger as suggested, as well as setting the level to the logger, the following works for writing a log file.

import logging
import time

from joblib import Parallel, delayed
    
def get_logger(name=None, userlevel=logging.DEBUG, fh_kwargs=dict(filename='parallel-logging.log')):
    logger = logging.getLogger(name)
    formatter = logging.Formatter(
        '%(asctime)s %(levelname)s: %(message)s '
        '[in %(pathname)s:%(lineno)d]')
    handler = logging.FileHandler(**fh_kwargs)
    handler.setLevel(userlevel)
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    logger.setLevel(userlevel)
    return logger   

def func(i):
    logger = get_logger(f'{i}_log')
    logger.warning(f"Function calling {i}")
    time.sleep(0.5)

with Parallel(n_jobs=4) as parallel:
    parallel(delayed(func)(i) for i in range(10))
Read more comments on GitHub >

github_iconTop Results From Across the Web

Logger v1.11.2 - HexDocs
The event manager and all added event handlers are automatically supervised by Logger . Note that if a backend fails to start by...
Read more >
Python Logging: In-Depth Tutorial - Toptal
The Python logging module comes with the standard library and provides basic logging features. By setting it up correctly, a log message can...
Read more >
Python Logging Guide - Best Practices and Hands-on Examples
An alert handler that formats ERROR and CRITICAL log requests via the detailed formatter and emails the resulting log entry to the given...
Read more >
ValueError: Unable to configure handler 'file': [Errno 2] No ...
What happens is there is some problem with your code happening. Handler catches this error to save it to your log file so...
Read more >
Best Practices for Client-Side Logging and Error Handling in ...
The only downside of this method is that you must set up your ... your own logging service, they require a lot of...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Bug when passing a function as parameter in a delayed function

Next page

Joblib pickling performance for pandas DataFrame