[KED-1056] pyarrow imported by package created by `kedro new` is problematic

Description

Package created by kedro new command seems to import pyarrow and cause a problem.

Context

I tried to use a Python package called “ray” inside the package created by kedro new command, but import ray returns the following error.

    import ray
  File "/usr/local/lib/python3.6/dist-packages/ray/__init__.py", line 9, in <module>
    raise ImportError("Ray must be imported before pyarrow because Ray "
ImportError: Ray must be imported before pyarrow because Ray requires a specific version of pyarrow (which is packaged along with Ray).

This issue was reported at ray’s GitHub repo (https://github.com/ray-project/ray/issues/5497), but hasn’t been resolved yet.

This error itself is ray’s issue, but I found that it does not reproduce outside of a package created by kedro new.

Steps to Reproduce

Here are 3 experiments:

[Case 1] Code in a module in the package creaed by kedro new (package_created_by_kedro_new.nodes.exmaple.py)

import ray 

ImportError: Ray must be imported before pyarrow ... as shown above.

[Case 2] Code in a module outside of the package created by kedro new

import kedro
print(">>> imported kedro: ", kedro.__version__)
import pyarrow
print(">>> pyarrow NOT imported by ray: ", pyarrow.__version__)
import ray
print(">>> imported ray: ", ray.__version__)

Output:

>>> imported kedro:  0.15.1
>>> pyarrow NOT imported by ray:  0.12.0
Traceback (most recent call last):
  File "/home/u/Minyus/_Python_scratch/scratch.py", line 5, in <module>
    import ray
  File "/usr/local/lib/python3.6/dist-packages/ray/__init__.py", line 9, in <module>
    raise ImportError("Ray must be imported before pyarrow because Ray "
ImportError: Ray must be imported before pyarrow because Ray requires a specific version of pyarrow (which is packaged along with Ray).

Process finished with exit code 1

[Case 3] Code in a module outside of the package created by kedro new

import kedro
print(">>> imported kedro: ", kedro.__version__)
# import pyarrow
# print(">>> pyarrow NOT imported by ray: ", pyarrow.__version__)
import ray
print(">>> imported ray: ", ray.__version__)
import pyarrow
print(">>> pyarrow imported by ray: ", pyarrow.__version__)
print(">>> No error is returned!")

Output:

>>> imported kedro:  0.15.1
>>> imported ray:  0.7.3
>>> pyarrow imported by ray:  0.14.0.RAY
>>> No error is returned!

Process finished with exit code 0

In summary, importing kedro is harmless, but the package created by kedro new command seems to import pyarrow and causes a problem. Are there any workaround?

Your Environment

• Kedro version used (pip show kedro or kedro -V): 0.15.1 (installed by pip install kedro)
• Python version used (python -V): Python 3.6.8
• Operating system and version: Ubuntu 18.04.3 LTS

[Minyus]

Thanks @Minyus, could it possibly be a duplicate of this issue?

Hi @Pet3ris, I believe my issue is technically different although both are related to pyarrow. While https://github.com/quantumblacklabs/kedro/issues/94 is installation failure using pipenv, I do not use pipenv and I can install and import Kedro without errors. I can use kedro new command as well.

[ghost]

From what I can understand, it’s not kedro new that is the problem, but running any code that includes import ray with kedro run, since kedro/cli.py imports stuff that imports stuff, that eventually imports kedro.io which imports datasets which imports s3fs which imports fsspec which imports pyarrow, sopyarrow will be present in sys.modules long before kedro run runs import ray.

We’re restructuring our contrib and dependencies significantly, which would have potentially solved your problem, but we’re also moving all our datasets to use fsspec (which imports pyarrow) – so if you want to use any fsspec dataset, you’ll be forced to have pyarrow.

I think this is really something on ray’s side to fix.