Cannot install pyarrow on macOS "Big Sur" with pyenv and Python 3.9

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 11.1 “Big Sur”
• Poetry version: 1.1.4
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/maxcountryman/4640caa6ac48acb0eb4f19cb5dfcb40e

Issue

I’m unable to install pyarrow via poetry add pyarrow on macOS 11.1 “Big Sur” into a virtualenv which uses Python 3.9.0 via pyenv.

Here’s the error I see:

    ...
    installing to build/bdist.macosx-11.1-x86_64/wheel
    running install
    running install_lib
    creating build/bdist.macosx-11.1-x86_64
    creating build/bdist.macosx-11.1-x86_64/wheel
    creating build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/public-api.pxi -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/io.pxi -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/serialization.pxi -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/orc.py -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/_compute.pyx -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/_orc.pxd -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/builder.pxi -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/array.pxi -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/_generated_version.py -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/compat.py -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/_cuda.pyx -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/benchmark.py -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/parquet.py -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/_csv.pyx -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/config.pxi -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/_dataset.pyx -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/_flight.pyx -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/ipc.py -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/util.py -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/flight.py -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/tensor.pxi -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/__init__.pxd -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    copying build/lib.macosx-11.1-x86_64-3.9/pyarrow/cffi.py -> build/bdist.macosx-11.1-x86_64/wheel/pyarrow
    creating build/bdist.macosx-11.1-x86_64/wheel/pyarrow/include
    error: can't copy 'build/lib.macosx-11.1-x86_64-3.9/pyarrow/include/arrow': doesn't exist or not a regular file
    ----------------------------------------
    ERROR: Failed building wheel for pyarrow
  Failed to build pyarrow
  ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly

  at ~/.poetry/lib/poetry/utils/env.py:1074 in _run
      1070│                 output = subprocess.check_output(
      1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1072│                 )
      1073│         except CalledProcessError as e:
    → 1074│             raise EnvCommandError(e, input=input_)
      1075│
      1076│         return decode(output)
      1077│
      1078│     def execute(self, bin, *args, **kwargs):

It seems there is a wheel for macOS 11, but it’s not being used by Poetry. For instance, I am able to install pyarrow via pip via pip install pyarrow, so I’m wondering why Poetry isn’t able to manage.

[ybroze]

I’m actually encountering the same problem using pip, on Big Sur 11.2.3 and a fresh pyenv-managed Python 3.9.4 installation, on M1 Silicon:

...
  running install
  running install_lib
  creating build/bdist.macosx-11.2-arm64
  creating build/bdist.macosx-11.2-arm64/wheel
  creating build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/public-api.pxi -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/io.pxi -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/serialization.pxi -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/orc.py -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/_compute.pyx -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/_orc.pxd -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/builder.pxi -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/array.pxi -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/_generated_version.py -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/compat.py -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/_cuda.pyx -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/benchmark.py -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/parquet.py -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/_csv.pyx -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/config.pxi -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/_dataset.pyx -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/_flight.pyx -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/ipc.py -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/util.py -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/flight.py -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/tensor.pxi -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/__init__.pxd -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  copying build/lib.macosx-11.2-arm64-3.9/pyarrow/cffi.py -> build/bdist.macosx-11.2-arm64/wheel/pyarrow
  creating build/bdist.macosx-11.2-arm64/wheel/pyarrow/include
  error: can't copy 'build/lib.macosx-11.2-arm64-3.9/pyarrow/include/arrow': doesn't exist or not a regular file
  ----------------------------------------
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly

This makes me think the issue is not with Poetry, but in the pyarrow package itself.

[dbanty]

@ybroze I’m fairly certain the issue that was in Poetry has been fixed in the latest versions. There are a couple related issues though:

1. Everything but the latest versions of pip don’t know how to install things for Big Sur. Any time I create a virtual env at this point I have to do a pip install --upgrade pip to be confident.
2. Some packages that have native code may not have wheels for Apple Silicon, which might mean you have to build from source. Building from source definitely will require the latest pip if that package uses PEP 517 (which pyarrow appears to).