Finding the right python with cmake on mac

Describe the bug

I have a project that use cmake to build and python with numpy in some of the tests. Since a couple of days ago the macOS build action started failing due to not being able to load python modules installed. It seems that cmake has started finding the system python, where as pip uses the hostedtoolcache python. Non macos builds are not effected. This results in the error

ModuleNotFoundError: No module named 'numpy'

The yaml file is https://github.com/Merlin-Collaboration/Merlin/blob/2b4fd134e9066230b094b1f2991ca3c8b40439d3/.github/workflows/build-cmake.yaml

These are 2 nightly runs with against the same git version 2b4fd13

Before https://github.com/Merlin-Collaboration/Merlin/runs/898329811?check_suite_focus=true

-- Found Python3: /Users/runner/hostedtoolcache/Python/3.8.3/x64/bin/python3.8 (found version "3.8.3") found components: Interpreter

after https://github.com/Merlin-Collaboration/Merlin/runs/901845108?check_suite_focus=true

-- Found Python3: /usr/local/Frameworks/Python.framework/Versions/3.8/bin/python3.8 (found version "3.8.4") found components: Interpreter

It looks like the change from Xcode_11.5.app to Xcode_11.6.app may have triggered it,

I appreciate that this might not be consider a bug in setup-python. But a report here might help other users that hit the issue.

Workaround

I have worked around by telling cmake which python to use by adding the argument

-DPython3_ROOT_DIR=${pythonLocation}

Which version of the action are you using?

• v1
• v2
• Some other tag (such as v2.0.1 or master)

Environment

• self-hosted
• Linux
• Windows
• Mac

If applicable, please specify if you’re using a container

Python Versions 3.8

To Reproduce Steps to reproduce the behavior:

1. Install python with actions/setup-python
2. Install numpy with pip install numpy
3. In cmake use find_package(Python3) to find the python executable
4. Wrong python executable found

Run/Repo Url https://github.com/Merlin-Collaboration/Merlin

[sgatto]

I don’t know if this can help anyone but as today I had to use a different solution than the one proposed by @nikita-bykov or @jiridanek

I simply hinted the python location in the Cmake command line using -DPython3_ROOT_DIR. setup-python defines the right environment variable pythonLocation:

cmake -S . -B build -DPython3_ROOT_DIR="${{ env.pythonLocation }}"  ...

This allows not to change the cmakelist.txtand adapt the yaml file to the specific behaviours of the GH runners.

On windows runners we have a default python v3.7.9, plus the version that can be installed using setup-python plus the others python versions that seem to be cached in the runners. In particular a 3.10 (C:\Users\runneradmin\AppData\Roaming\Python\Python310\Scripts) that I did not aks and that CMake always finds no matter the version I selected in setup-python.

[jiridanek]

I am currently experiencing similar issues. Ubuntu works as intended, but CMake keeps picking up Python 3.9 on Windows and macOS, despite the fact I use actions/setup-python to set Python 3.6.

I believe I understand the problem now. CMake 3.15 (with policy CMP0094 set to NEW) locates Python in the following order

1. It looks for an active Virtual environment
2. on Windows it examines default Python settings in the system registry, on macOS it looks for the default Framework
3. finally it examines environment, looking for the first Python on PATH

CMake 3.12 and newer, up to and not including 3.15, looks up Python similarly, except it homes in on the newest available Python in all three locations, or maybe it tries to pick up the virtual environment, and then goes for the newest Python in the remaining two? (This is my understanding from reading the docs, I am not using FindPython on these versions of CMake.)

Since actions/setup-python does not touch Windows registries nor macOS frameworks, I am going to do the following to give preference to the environment settings (which this action does set).

https://cmake.org/cmake/help/v3.15/module/FindPython.html#module:FindPython

    if (POLICY CMP0094)  # https://cmake.org/cmake/help/latest/policy/CMP0094.html
        cmake_policy(SET CMP0094 NEW)  # FindPython should return the first matching Python
    endif ()

    # needed on GitHub Actions CI: actions/setup-python does not touch registry/frameworks on Windows/macOS
    # this mirrors PythonInterp behavior which did not consult registry/frameworks first
    if (NOT DEFINED Python_FIND_REGISTRY)
        set(Python_FIND_REGISTRY "LAST")
    endif ()
    if (NOT DEFINED Python_FIND_FRAMEWORK)
        set(Python_FIND_FRAMEWORK "LAST")
    endif ()

    find_package(Python
        REQUIRED COMPONENTS Interpreter
        OPTIONAL_COMPONENTS Development)

This should hopefully land for me in https://github.com/apache/qpid-proton/pull/294 and Bob is the uncle.