Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Joblib silently ignores pickle exceptions and does not use cache

See original GitHub issue

I’m trying to use memory.cache on a function that returns a bunch of Pandas named tuples made with list(dataframe.itertuples()). Apparently, such tuples cannot be pickled. I’m fine with that, however I’m not fine with the fact that Joblib doesn’t even log the issue. Because of the empty except at https://github.com/joblib/joblib/blob/master/joblib/_store_backends.py#L193 the pickle exception is just ignored, and the hint about race condition doesn’t really help to debug the issue.

Issue Analytics

State:
Created 4 years ago
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

tomMoralcommented, Oct 13, 2019

Indeed, the except to catch the directory creation race condition should probably be moved in the create_location method. I also run into some race conditions when calling in parallel a cached function that has just been modified. The workers are all trying to delete the directory at once, creating errors. I will try to fix this this week.

0reactions

p-himikcommented, Oct 14, 2019

Right, sorry - I was just reproducing it from my head. In the real code, I have location specified.

Top Results From Across the Web

Do not swallow PicklingError by Nielius · Pull Request #1359

Currently, when caching fails because the function output can't be pickled, there is no warning, but the output will not be cached.

joblib Documentation - Read the Docs

It works by explicitly saving the output to a file and it is designed to work with non-hashable and potentially large input and...

Development — joblib 1.3.0.dev0 documentation

Rely on the built-in exception nesting system of Python 3 to preserve traceback information when an exception is raised on a remote worker...

Changelog — Dask.distributed 2022.12.1 documentation

Set sizing mode on Tabs to avoid layout collapse (GH#7365) Mateusz Paprocki ... Pickle worker state machine exceptions (GH#6702) crusaderky.

Release History — scikit-learn 0.20.4 documentation

MissingIndicator which was not supported while sklearn.impute.SimpleImputer was supporting ... Fix datasets.fetch_openml to correctly use the local cache.