terryfy macOS 2.7.17 and virtualenv 20.0: unable to find 'coverage' command (zope.interface)

(We found a workaround so this isn’t super critical, but several other projects also would have to change to use the workaround.)

zope.interface uses https://github.com/MacPython/terryfy to install the python.org distribution of Python on Travis macOS for testing for 2.7, and 3.5+. Here’s the relevant .travis.yml configuration for the builds discussed. terrypy

A few days ago, there was a successful build of all versions using virtualenv 16.7.9. (Build log).

This morning, a round of builds installed virtualenv 20.0. While all the Python 3 builds worked, Python 2.7.17 failed. It seems that the $PATH wasn’t setup correctly, and so the coverage command provided by the coverage wheel wasn’t available. This build failed repeatedly, so it doesn’t seem to be a simple fluke. (Build log). As far as I can see, that’s the only relevant difference.

terryfy uses the get_macpython_environment function from https://github.com/matthew-brett/multibuild to create and activate the virtual environment after installing Python:

function get_macpython_environment {
    # Set up MacPython environment
    # Parameters:
    #     $version : [implementation-]major[.minor[.patch]]
    #         The Python implementation to install, e.g. "3.6" or "pypy-5.4"
    #     $venv_dir : {directory_name|not defined}
    #         If defined - make virtualenv in this directory, set python / pip
    #         commands accordingly
    #     $py_osx_ver: {major.minor | not defined}
    #         if defined, the macOS version that Python is built for, e.g.
    #         "10.6" or "10.9", if not defined, use the version from MB_PYTHON_OSX_VER
    #
    # Installs Python
    # Sets $PYTHON_EXE to path to Python executable
    # Sets $PIP_CMD to full command for pip (including sudo if necessary)
    # If $venv_dir defined, Sets $VIRTUALENV_CMD to virtualenv executable
    # Puts directory of $PYTHON_EXE on $PATH
    local version=$1
    local venv_dir=$2
    local py_osx_ver=${3:-$MB_PYTHON_OSX_VER}

    if [ "$USE_CCACHE" == "1" ]; then
        activate_ccache
    fi

    remove_travis_ve_pip
    install_macpython $version $py_osx_ver
    install_pip

    if [ -n "$venv_dir" ]; then
        install_virtualenv
        make_workon_venv $venv_dir
        source $venv_dir/bin/activate
    else
        export PATH="`dirname $PYTHON_EXE`:$PATH"
    fi
    export PYTHON_EXE PIP_CMD

We’re calling terryfy and passing a venv_dir, and I verified that in fact the virtualenv Python is the Python that’s active.

before_install:
  - |
    if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
      git clone https://github.com/MacPython/terryfy
      source terryfy/travis_tools.sh
      get_python_environment $TERRYFY_PYTHON venv
    fi

With virtualenv 20.0, we get lots of errors from pip install coverage about scripts not being on the PATH:

Installing collected packages: chardet, idna, pycparser, cffi, ipaddress, enum34, cryptography, pyOpenSSL, urllib3, requests, coverage, docopt, coveralls
202  WARNING: The script chardetect is installed in '/Library/Frameworks/Python.framework/Versions/2.7/bin' which is not on PATH.
203  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
204  WARNING: The scripts coverage, coverage-2.7 and coverage2 are installed in '/Library/Frameworks/Python.framework/Versions/2.7/bin' which is not on PATH.
205  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
206  WARNING: The script coveralls is installed in '/Library/Frameworks/Python.framework/Versions/2.7/bin' which is not on PATH.
207  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

Is this an expected change of some sort? Does terryfy/multibuild need to adapt? Are we doing something wrong? (Possibly https://github.com/pypa/virtualenv/pull/1550/files might be related?) Thanks for any insight!

Workaround

In a subsequent build, I modified the commands in .travis.yml to use python -m pip install... and python -m coverage run …. The python -m pip install command still produced warnings about scripts not being on the PATH, but the python -m coverage run … command executed as expected. That will let us limp along.

[gaborbernat]

so looked into this; seems MacOs customizes framework builds so much that they just don’t work without rewriting the mach-o (the magic from here https://github.com/pypa/virtualenv/blob/legacy/virtualenv.py#L2492-L2631); will take a while to get this done, but I’m on it 🤷‍♂

[matthew-brett]

Thanks - and thanks for the quick release to PyPI.