Building from 0.20.1 source distribution on Py3.7 fails because it includes C files generated with old Cython

Description

Building from 0.20.1 source distribution on Py3.7 fails because it includes C files generated with old Cython.

Steps/Code to Reproduce

In a clean Py3.7 venv, install numpy, scipy, and cython (e.g. 1.15.4, 1.1.0, and 0.29.1, i.e. latest as of now), then

$ pip install --no-binary=:all: scikit-learn
<elided>
    sklearn/cluster/_hierarchical.cpp:24789:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
         tmp_tb = tstate->exc_traceback;
                          ^~~~~~~~~~~~~
                          curexc_traceback
    sklearn/cluster/_hierarchical.cpp:24790:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
         tstate->exc_type = *type;
                 ^~~~~~~~
                 curexc_type
    sklearn/cluster/_hierarchical.cpp:24791:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
         tstate->exc_value = *value;
                 ^~~~~~~~~
                 curexc_value
    sklearn/cluster/_hierarchical.cpp:24792:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
         tstate->exc_traceback = *tb;
                 ^~~~~~~~~~~~~
                 curexc_traceback
    error: Command "g++ -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fstack-protector-strong -fno-plt -flto=4 -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -march=x86-64 -mtune=generic -O3 -pipe -fstack-protector-strong -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fstack-protector-strong -fno-plt -fPIC -I/tmp/testenv/lib/python3.7/site-packages/numpy/core/include -I/tmp/testenv/lib/python3.7/site-packages/numpy/core/include -I/tmp/testenv/include -I/usr/include/python3.7m -c sklearn/cluster/_hierarchical.cpp -o build/temp.linux-x86_64-3.7/sklearn/cluster/_hierarchical.o -MMD -MF build/temp.linux-x86_64-3.7/sklearn/cluster/_hierarchical.o.d" failed with exit status 1
    
    ----------------------------------------
Command "/tmp/testenv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-auq3meb_/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-lfcer_r1/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/testenv/include/site/python3.7/scikit-learn" failed with error code 1 in /tmp/pip-install-auq3meb_/scikit-learn/

Essentially, this is due to the sdist on PyPI (scikit-learn-0.20.1.tar.gz) containing C-files generated by Cython 0.23.5, which are incompatible with Py3.7.

I would suggest not including the C files in the sdist: if an user is knowledgeable enough to compile scikit-learn themselves (and thus went through the bother of setting up a compilation system, etc.) then hopefully they should be able to install Cython themselves too (or you can declare Cython as a setup_requires, up to you). Or at least please regenerate these files using the latest version of Cython (but I would consider that a more brittle option).

Expected Results

scikit-learn is installed.

Actual Results

scikit-learn fails to install.

Versions

scikit-learn is not installed, so heh 😃 (But Py3.7/Linux and cython 0.29.1 (in the venv).)

[jnothman]

I’m not sure what you mean by “tested” though

I mean that if we say “We require the latest cython version” then release Scikit-learn in January but Cython’s release in March breaks our compilation, we would not force ourselves to do a compatibility release.

[jnothman]

I’m happy with always requiring the latest Cython that is tested.