pickling error when doing using dataframe with multiprocessing

When using python 3.4.3 or 3.5.0 and pandas 0.16.2, this simple example works

import pandas as pd
import multiprocessing as mp

def squareIt(row):
    return row[0]**2, row[1]**2

df = pd.DataFrame({'a': range(100), 'b': range(100)})

with mp.Pool(2) as pool:
    sq = pool.imap(squareIt, df.itertuples(False), chunksize=10)
    for x in sq:
        print(x)

when upgrading to 0.17.1 there is this error.

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/opt/Python-3.4.3/lib/python3.4/multiprocessing/pool.py", line 295, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/opt/Python-3.4.3/lib/python3.4/multiprocessing/pool.py", line 689, in next
    raise value
  File "/opt/Python-3.4.3/lib/python3.4/multiprocessing/pool.py", line 383, in _handle_tasks
    put(task)
  File "/opt/Python-3.4.3/lib/python3.4/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/opt/Python-3.4.3/lib/python3.4/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'pandas.core.frame.Pandas'>: attribute lookup Pandas on pandas.core.frame failed

Is there a better way to do some simple multi-threaded processing of a dataframe. This had worked very well for me in the past. I would rather not rewrite all of my analysis, but if there is a better way I could be convinced.

Thanks.

[shoyer]

You can use df.itertuples(name=None) to make these regular rather than named tuples.

This does seem to be an unfortunate downside of switching itertuples to return named tuples.

[max-sixty]

I believe that’s trying to pickle a namedtuple. You can either use a different pickler (dill works great), or use a different method than itertuples - you could just send each Series?

Note that pickling is fairly expensive, and so to make this worthwhile you’re going to need to be doing a lot of compute on each piece of data. That example is going to be much faster in a single process.