Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Using Memory and Parallel with cached function defined inside Jupyter notebook results in not using the cache

See original GitHub issue

I’m running a Jupyter notebook on a machine with multiple cores. I’m trying to use both caching of intermediate results and parallelism for this. I get a lot of:

python3.7/site-packages/joblib/parallel.py:256: JobLibCollisionWarning: Cannot detect name collisions for function 'cached_two' for func, args, kwargs in self.items]

I believe these are not appearing when I just run single threade or not in a notebook. Sample code below to run in jupyter notebook. joblib 0.14.1 notebook 6.0.3

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:8 (5 by maintainers)

Top GitHub Comments

3reactions

dseverocommented, Jun 17, 2020

I was having the same problem in #1069. Moving the functions to a python file as @lesteve suggested worked. Just wanted to report it.

2reactions

pierreglasercommented, Jul 1, 2020

Thank you very much for the report @dsevero and @jontis. I’m going to work on a fix for this.

Top Results From Across the Web

Checkpoint using joblib.Memory and joblib.Parallel

This example illustrates how to cache intermediate computing results using joblib.Memory within joblib.Parallel . Embed caching within parallel processing¶. It ...

IPython: turn off %timeit message about "This could mean that ...

Since upgrading to IPython version 3 (I have 3.1.0), every time I use the %timeit command, it prints "The slowest run took [number]...

Execute and cache your pages - Jupyter Book

Jupyter Book can automatically run and cache any notebook pages. Notebooks can either be run each time the documentation is built, or cached...

Using joblib to speed up your Python pipelines

Now, let's use joblib's Memory function with a location defined to store a ... I have shown how Parallel jobs can be run...

joblib Documentation - Read the Docs

caching methods: memory is designed for pure functions and it is not recommended to use it for methods. If one wants to use...