Silent recalculation of result when trying to cache a lambda with dill

I have @memory.cache’d a function that returns a lambda expression. This is not possible to pickle, which makes joblib not cache the result and instead rerun it everytime. dill however is able to pickle such functions, and so I set it as a loky pickler. Yet, the function is still not cached and rerun everytime.

This took me quite a while to find out in two ways. First, to see that if pickling fails, joblib silences this error and does not cache. To mitigate, I use dill as a pickler but that still fails silently.

I would expect

• joblib to crash if it cannot pickle a result, instead of silently going on,
• joblib to crash if I explicitly set a pickler, but that is apparently not used.

I am using joblib-1.1.0, dill-0.3.4 Python 3.9.9 on Mac OS Monterey, M1. Here is what I run:

from joblib.memory import Memory
from joblib.externals.loky import set_loky_pickler
set_loky_pickler('dill')

memory = Memory('./cache')

@memory.cache
def do_stuff():
    return lambda: 2


x = do_stuff()

[GaelVaroquaux]

I hence take away from the conversation that there currently is no way to make joblib work with lambdas.

Not with regards to the outputs of functions in mem.cache.

Doing so would require quite some work in numpy_pickle, I suppose.

I would agree. We would probably rather use cloud_pickle than dill, because we really in cloud_pickle elsewhere.

[bayerj]

Ok, excuse my misunderstanding and thank you for your patience.

I hence take away from the conversation that there currently is no way to make joblib work with lambdas. Doing so would require quite some work in numpy_pickle, I suppose.