Parallel job takes more time than non-parallel job?

It seems that my batches are executed one by one rather than been parallel? I’m using iPython. It’s 32 core machine.

import pandas, numpy, hashlib
from joblib import Parallel, delayed

d = pandas.DataFrame({'a': numpy.random.randn(1000 * 1000)})

def hash(x):
    return hashlib.sha256(x.encode('utf-8')).hexdigest()

The output:

In [35]: Parallel(n_jobs=-1, batch_size=100 * 1000, verbose=20) (delayed(hash) (str(row[1:])) for row in (d.itertuples()))
[Parallel(n_jobs=-1)]: Done  32 tasks      | elapsed:    0.0s
[Parallel(n_jobs=-1)]: Done 100032 tasks      | elapsed:    6.8s
[Parallel(n_jobs=-1)]: Done 200032 tasks      | elapsed:   13.5s
[Parallel(n_jobs=-1)]: Done 300032 tasks      | elapsed:   20.1s
[Parallel(n_jobs=-1)]: Done 400032 tasks      | elapsed:   26.9s
[Parallel(n_jobs=-1)]: Done 500032 tasks      | elapsed:   33.9s
[Parallel(n_jobs=-1)]: Done 600032 tasks      | elapsed:   40.8s
[Parallel(n_jobs=-1)]: Done 700032 tasks      | elapsed:   47.5s
[Parallel(n_jobs=-1)]: Done 800032 tasks      | elapsed:   54.2s

[chengguangnan]

It worked. Thanks. The execution time is 2 seconds rather than 12 seconds.

    chunks = []
    for i in range(1, len(d), 1000):
       chunks.append(d[i:min(i + 1000, len(d))])

    res = Parallel(n_jobs=-1) (delayed(hash) (d) for d in chunks)

    def hash(d):
        return [hashlib.sha256(str(x).encode('utf-8')).hexdigest()[:30] for x in d.itertuples(index=False, name=None)]

[Debabrataadak]

from math import sqrt from joblib import Parallel, delayed from numpy import square import numpy as np import time

n=10000000 def sim(x):

time.sleep(100)

return x**2

chunk=[] arr=np.arange(n) for i in range(0,len(arr),100): chunk.append(arr[i:(i+100)])

if name == “main”:

result = Parallel(n_jobs=1, backend="threading", verbose=5) \
    (delayed(square)(x) for x in chunk)

#print result

I did the same. But the time taken by a single core is less than using 8 cores.