New pre-release of numpy breaks an old repository with pinned versions

Since a few days ago our CI started failing for one of the 'frozen in time" repos we use to check if we can still build old images.

The problem is that a new version of numpy was pre-released a few days ago. Since then one of our “check we don’t break old repos in the wild” tests is now installing the latest version of numpy, which requires Python 3 but the repo wants Python 2.

The weird thing is that the repository specifies a version of numpy that it would like to install, a Python version and a pinned matplotlib version as well.

This is the end of the trace back we see:

      File "/srv/conda/envs/kernel/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-VBFt7K/numpy-1.17.0rc1/setup.py", line 31, in <module>

    RuntimeError: Python version >= 3.5 required.
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-sgij5M/matplotlib/

This is while it is trying to install matplotlib-1.5.1. It is odd that this is trying to install numpy 1.17 from source instead of 1.10.4 which is what is specified in requirements.txt

Figuring out why this happens is high priority as it will block all PRs.

[minrk]

So we’ve been building numpy from source for matplotlib all along but now the version of numpy that gets pulled in for that doesn’t work with Python 2 any more which is why we see the failure now?

Yes, that’s exactly right. This is why truly reproducible builds are hard!

[minrk]

It’s related to setup_requires since there’s no wheel for matplotlib 1.5.1 which has numpy in setup_requires, which is handled by setuptools (not pip). Since there’s no wheel for matplotlib, it’s building from source, pulling in numpy during build. At the time it does this, it doesn’t know that numpy is topologically before matplotlib (it can’t know when building from source, since the install_requires metadata hasn’t been determined yet). Since setup_requires and easy_install are deprecated functionality, they’ve never been fixed to support things like python_requires metadata that would forbid the new prerelease which has dropped py2.

The basic answer is: pinning matplotlib 1.5.1 doesn’t work for Python 2.7 unless numpy is installed first, and for packages installed from source at least, requirements.txt must itself be in topological order.

Related: https://github.com/pypa/setuptools/issues/917 which proposes PEP 518 as the replacement for setup_requires, but no real plan to fix the deprecated setup_requires to use pip (since it could break other things).