BUG: PyPI requires on 1.7.1 should allow Python 3.10+

Describe your issue.

With Python 3.10rc2 installed, pip install scipy installs scipy 1.6.1 (the most recent version on PyPI that does not restrict Python ❤️.10) and not 1.7.1, resulting in a rather broken install. I get a lot of

Undefined symbol: _PyGen_Send

when running ordinary code. It looks like on PyPI scipy 1.7.1 requires Python < 3.10.

Installing master pip install git+https://github.com/scipy/scipy fixes the problems.

Reproducing Code Example

see above

Error message

see above

SciPy/NumPy/Python version information

3.6.1

[rgommers]

We will release 1.7.2 with Python 3.10 support very soon.

The other part of this is: it would be great to at some point prevent older SciPy versions from being picked up. This is a little tricky to achieve, but can be done by adding a new 1.6.x release which aborts a from-source build immediately with a clear error message. That’s something I may get around to at some point, but it needs thorough testing to ensure we’re not breaking anything.

[henryiii]

The problem @NickleDave is complaining about is not the fact SciPy doesn’t work on 3.10, but rather that SciPy is setting python_requires = "...,<3.10", which is a misunderstanding of how Requires-Python metadata slot works. This was added to allow packages to drop a Python version (2.7) without breaking people on that version[^1] - pip would simply scroll back in time and select the final release that included that Python version. It was never intended to cap Python; it was designed to allow dropping Python. If you want to cap Python, you can add a check in setup.py that throws an error if Python is too high. You should not use this metadata slot to cap Python. If you do use it that way, you must always use it or yank any old releases that don’t cap, otherwise you’d get the problem above, that pip installing SciPy on Python 3.10 will “work” but get the oldest version that didn’t have this cap. This is exactly the intended behavior, and it’s exactly wrong.

The good news is this bad solve is only on Python 3.10, where even a good solve would likely not work (assuming SciPy really doesn’t support Python 3.10).

This has another unpleasant effect. If you use a locking package manager (Poetry, PDM), then the package manager will see that the lock file has a Python cap in it. It will then force the user to add the matching cap, since the lock file will be invalid on Python 3.10. If you are developing a library and don’t have a strong cap on SciPy, this is wrong, but it doesn’t matter, these systems are prioritizing the lock file over the library dependencies.

The bad news is this affects all downstream libraries using a locking package manager on any Python version.

IMO, a library’s Requires-Python should not depend on the lock file; if a system wants to force you to be truthful for a lock file’s sake, that should be a separate way to do this that does not get exported. Or it could just print a warning (“this lock file will only be valid on Python x to y”) But nether @frostming’s PDM nor Poetry support this, as far as I know. Until this is fixed, I think the only advice I can give is to avoid locking package managers for developing libraries.

[^1]: This would probably have been a lot better as a “min_python” slot, but at the time, it was important to allow older Python 3’s to be dropped while keeping 2.7 as well, so it was allowed to be an open expression. But it was never supposed to upper-cap!