Error importing ray in colab due to pyarrow

System information

• colab
• pip install ray[rllib]
• python 3.7

Describe the problem

Very simple, just pip install ray, then import ray:

!pip install tensorflow  # or tensorflow-gpu
!pip install ray[rllib]  # also recommended: ray[debug]
import ray

on Google colab and it fails with:

ImportError: Ray must be imported before pyarrow because Ray requires a specific version of pyarrow (which is packaged along with Ray).

Recreate

Take a look at this gist running on colab

[pcmoritz]

Thanks for reporting this! This error happens because colab is preloading certain libraries (more on this below). To fix it, you can do the following:

!pip install tensorflow  # or tensorflow-gpu
!pip install ray[rllib]  # also recommended: ray[debug]
!pip uninstall -y pyarrow

and then click on Runtime/Restart runtime. After that, the Ray cell will work.

Now as to why colab’s preloading is problematic: Ray depends on a library called pyarrow for serialization and some other things. Because TensorFlow’s binary package is not compliant with the manylinux1 standard [1], we need to ship a custom version of pyarrow that is compatible with TensorFlow. Until TensorFlow is fixed, we don’t really have a choice here, so sorry for that.

[1] https://groups.google.com/a/tensorflow.org/d/msg/developers/TMqRaT-H2bI/J1Wiu8_XCQAJ

[richardliaw]

Just a note - the way we get around this on our tutorial notebooks is by using this block of code before running anything:

print("Setting up colab environment")
!pip uninstall -y -q pyarrow
!pip install -q ray[debug]

# A hack to force the runtime to restart, needed to include the above dependencies.
print("Done installing! Restarting via forced crash (this is not an issue).")
import os
os._exit(0)