Pickle error in Parallel

Hey,

I get an error with 0.12 that didn’t occur in 0.11

snippet of code: num_cores = multiprocessing.cpu_count() chunks = _get_chunks(ids, 100) logger.info('cores: {0}, chunks: {1}'.format(num_cores, len(chunks))) Parallel(n_jobs=num_cores)(delayed(_clean_text)(c) for c in chunks)

The resulting error is: Cannot pickle files that are not opened for reading: a

[ogrisel]

@EloyRoura the point of using the loky backend in joblib is to replace the multiprocessing.Pool backend. With the loky backend you no longer need to protect your code with the if __name__ == '__main__' condition.

Here is an example:

>>> from joblib import Parallel, delayed
>>> def some_funtion(p):
...     return p*p
... 
>>> parameters = range(10)
>>> num_cores=4#multiprocessing.cpu_count()
>>> Parallel(n_jobs=num_cores, verbose=1)(delayed(some_funtion)(p) for p in parameters)
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done  10 out of  10 | elapsed:    0.4s finished
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

[lesteve]

For completeness, here is a stand-alone snippet for the pymc3-related issue:

import numpy as np

import pymc3 as pm

rng = np.random.RandomState(0)
count_data = rng.randint(1, 70, 100).astype(dtype=np.float64)
n_count_data = len(count_data)

with pm.Model() as model:
    alpha = 1.0/count_data.mean()  # Recall count_data is the
                                   # variable that holds our txt counts
    lambda_1 = pm.Exponential("lambda_1", alpha)
    lambda_2 = pm.Exponential("lambda_2", alpha)
    
    tau = pm.DiscreteUniform("tau", lower=0, upper=n_count_data - 1)

with model:
    step = pm.Metropolis()
    trace = pm.sample(5, tune=10,step=step)