Plasma Store Fails when defaulting to pickle

I tried to put an array of CustomObjects - and got this warning:

WARNING: Falling back to serializing objects of type <class 'scipy.interpolate._cubic.CubicSpline'> by using pickle. This may be inefficient.

Followed by this error:

---------------------------------------------------------------------------
ArrowCapacityError                        Traceback (most recent call last)
<ipython-input-35-4fa2fc614b22> in <module>()
      4 for names, obj in obj_dict.items():
      5     print(names)
----> 6     _pinned_objects[names] = pin_in_object_store(obj)

<ipython-input-34-a2184fde3bd2> in pin_in_object_store(obj)
      1 def pin_in_object_store(obj):
----> 2     obj_id = ray.put(_to_pinnable(obj))
      3     return "{}{}".format(
      4         PINNED_OBJECT_PREFIX, base64.b64encode(obj_id.id()).decode("utf-8"))

~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/worker.py in put(value, worker)
   2384         object_id = worker.local_scheduler_client.compute_put_id(
   2385             worker.current_task_id, worker.put_index)
-> 2386         worker.put_object(object_id, value)
   2387         worker.put_index += 1
   2388         return object_id

~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/worker.py in put_object(self, object_id, value)
    409         # Serialize and put the object in the object store.
    410         try:
--> 411             self.store_and_register(object_id, value)
    412         except pyarrow.PlasmaObjectExists:
    413             # The object already exists in the object store, so there is no

~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/worker.py in store_and_register(self, object_id, value, depth)
    344                     memcopy_threads=self.memcopy_threads,
    345                     serialization_context=self.get_serialization_context(
--> 346                         self.task_driver_id))
    347                 break
    348             except pyarrow.SerializationCallbackError as e:

~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/utils.py in _wrapper(*args, **kwargs)
    402                 def _wrapper(*args, **kwargs):
    403                     with self.lock:
--> 404                         return orig_attr(*args, **kwargs)
    405 
    406                 self._wrapper_cache[attr] = _wrapper

~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/_plasma.pyx in pyarrow._plasma.PlasmaClient.put()

~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/serialization.pxi in pyarrow.lib.serialize()

~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowCapacityError: BinaryArray cannot contain more than 2147483646 bytes, have 2147486312

Is there an easy fix for this?

[dmadeka]

That works! I get an even more interesting error:

~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/scipy/interpolate/interpolate.py in _evaluate(self, x, nu, extrapolate, out)
    986     def _evaluate(self, x, nu, extrapolate, out):
    987         _ppoly.evaluate(self.c.reshape(self.c.shape[0], self.c.shape[1], -1),
--> 988                         self.x, x, nu, bool(extrapolate), out)
    989 
    990     def derivative(self, nu=1):

_ppoly.pyx in scipy.interpolate._ppoly.evaluate()

~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/scipy/interpolate/_ppoly.cpython-36m-x86_64-linux-gnu.so in View.MemoryView.memoryview_cwrapper()

~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/scipy/interpolate/_ppoly.cpython-36m-x86_64-linux-gnu.so in View.MemoryView.memoryview.__cinit__()

ValueError: buffer source array is read-only

It seems I have to copy the array to avoid this which blows the memory - Anyway, its a SciPy issue! https://github.com/scipy/scipy/issues/9331

Thanks!

[edoakes]

Stale - please open new issue if still relevant