Memory release after Parallel

I did a test and found objects returned by parallel processing cannot be del and thus memory cannot be released. simplified code:

### Measure memory usage
import psutil, gc
ps = psutil.Process()
def memory_usage_psutil(ps):
    gc.collect()
    return ps.get_memory_info().rss / (2 ** 20)

### a function making a list from 0 to 10**6
def make_list(i): return [x for x in xrange(10 ** 6)]

print 'start', memory_usage_psutil()
### create 5 such lists w/o parallel and delete
single = [make_list(i) for i in range(5)]
print 'have single list', memory_usage_psutil()
del single
print 'delete single list', memory_usage_psutil()

### create 5 such lists with parallel and delete
parallel = Parallel(n_jobs=5)(delayed(make_list)(i) for i in range(5))
print 'have parallel list', memory_usage_psutil()
del parallel
print 'delete parallel list', memory_usage_psutil()

### =========Output==========
start 19.1796875
have single list 174.328125
delete single list 19.3203125
have parallel list 194.50390625
delete parallel list 154.9609375

[tomMoral]

This issue should be resolved in the latest version of joblib which feature memory leaks safeguards with the loky backend:

• If psutil is installed on the system, a worker process is shutdown and a new worker is re-spawn if its memory usage grows by more than 100Mb between two tasks (the checks are performed at most once every second).
• else, we call gc.collect periodically between 2 tasks.

Please try to use joblib==0.12.4 and report if you still see some memory leaked.

[ogrisel]

I went back to old versions of joblib 0.8.4. And I cannot reproduce the bug either. It might have been fixed on the Python side (I am using Python 3.7) and is therefore unrelated to joblib as Gael initially suggested…