BLD: Compatibility issues with master branch cython

Continuation of issue #33224

---

With the latest version of cython, running python setup.py build_ext --inplace -j 0 gives the following errors:

---

pandas/_libs/join.pyx:

building 'pandas._libs.join' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/klib -I/home/user/Venvs/cythonM-pandas/lib/python3.8/site-packages/numpy/core/include -I/home/user/Venvs/cythonM-pandas/include -I/usr/include/python3.8 -c pandas/_libs/join.c -o build/temp.linux-x86_64-3.8/pandas/_libs/join.o
In file included from /usr/include/python3.8/tupleobject.h:41,
                 from /usr/include/python3.8/Python.h:105,
                 from pandas/_libs/join.c:32:
pandas/_libs/join.c: In function ‘__pyx_pf_6pandas_5_libs_4join_12inner_join_indexer.isra.0’:
/usr/include/python3.8/cpython/tupleobject.h:27:60: warning: array subscript -1 is below array bounds of ‘PyObject *[1]’ {aka ‘struct _object *[1]’} [-Warray-bounds]
   27 | #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
      |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
pandas/_libs/join.c:204503:27: note: in expansion of macro ‘PyTuple_GET_ITEM’
204503 |             PyObject *r = PyTuple_GET_ITEM(o, n);
       |                           ^~~~~~~~~~~~~~~~
pandas/_libs/join.c: In function ‘__pyx_pf_6pandas_5_libs_4join_10left_join_indexer.isra.0’:
/usr/include/python3.8/cpython/tupleobject.h:27:60: warning: array subscript -1 is below array bounds of ‘PyObject *[1]’ {aka ‘struct _object *[1]’} [-Warray-bounds]
   27 | #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
      |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
pandas/_libs/join.c:204503:27: note: in expansion of macro ‘PyTuple_GET_ITEM’
204503 |             PyObject *r = PyTuple_GET_ITEM(o, n);
       |                           ^~~~~~~~~~~~~~~~
pandas/_libs/join.c: In function ‘__pyx_pf_6pandas_5_libs_4join_8left_join_indexer_unique.isra.0’:
/usr/include/python3.8/cpython/tupleobject.h:27:60: warning: array subscript -1 is below array bounds of ‘PyObject *[1]’ {aka ‘struct _object *[1]’} [-Warray-bounds]
   27 | #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
      |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
pandas/_libs/join.c:204503:27: note: in expansion of macro ‘PyTuple_GET_ITEM’
204503 |             PyObject *r = PyTuple_GET_ITEM(o, n);
       |                           ^~~~~~~~~~~~~~~~
pandas/_libs/join.c: In function ‘__pyx_pf_6pandas_5_libs_4join_14outer_join_indexer.isra.0’:
/usr/include/python3.8/cpython/tupleobject.h:27:60: warning: array subscript -1 is below array bounds of ‘PyObject *[1]’ {aka ‘struct _object *[1]’} [-Warray-bounds]
   27 | #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
      |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
pandas/_libs/join.c:204503:27: note: in expansion of macro ‘PyTuple_GET_ITEM’
204503 |             PyObject *r = PyTuple_GET_ITEM(o, n);
       |                           ^~~~~~~~~~~~~~~~

---

pandas/_libs/reshape.pyx:

building 'pandas._libs.reshape' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 
-pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtun
e=generic -O3 -pipe -fno-plt -fPIC -DNPY_NO_DEPRECATED_API=0 -I/home/user/Venvs/cythonM-pandas/lib/python3.8/
site-packages/numpy/core/include -I/home/user/Venvs/cythonM-pandas/include -I/usr/include/python3.8 -c pandas
/_libs/reshape.c -o build/temp.linux-x86_64-3.8/pandas/_libs/reshape.o
In file included from /usr/include/python3.8/tupleobject.h:41,
                 from /usr/include/python3.8/Python.h:105,
                 from pandas/_libs/reshape.c:30:
pandas/_libs/reshape.c: In function ‘__pyx_pf_6pandas_5_libs_7reshape_unstack.isra.0’:
/usr/include/python3.8/cpython/tupleobject.h:27:60: warning: array subscript -1 is below array bounds of ‘PyObject *[1]’ {aka ‘struct _object *[1]’} [-Warray-bounds]
   27 | #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
      |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
pandas/_libs/reshape.c:29728:27: note: in expansion of macro ‘PyTuple_GET_ITEM’
29728 |             PyObject *r = PyTuple_GET_ITEM(o, n);
      |                           ^~~~~~~~~~~~~~~~

---

join_zip.zip

reshape_zip.zip

---

Output of pd.show_versions():

INSTALLED VERSIONS

commit : bd91f4577c5b54a72a4e092a7d7db43fec1b46bb python : 3.8.2.final.0 python-bits : 64 OS : Linux OS-release : 5.5.13.b-1-hardened machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.1.0.dev0+1159.gbd91f4577 numpy : 1.18.2 pytz : 2019.3 dateutil : 2.8.1 pip : 20.0.2 setuptools : 46.1.3 Cython : 3.0a0 (https://github.com/cython/cython/tree/30c891e0dc36a553455df9064ecb1f13414a12d2) pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : None IPython : 7.13.0 pandas_datareader: None bs4 : None bottleneck : None fastparquet : None gcsfs : None lxml.etree : None matplotlib : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pytables : None pytest : None pyxlsb : None s3fs : None scipy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None xlsxwriter : None numba : None

[da-woods]

@WillAyd It’s definitely reproducible, and the warnings are pointing towards something that’ll actually crash (so it isn’t just a case of silencing warnings). I’ve submitted a fix for Cython so hopefully should be resolved soon. It’s obvious exactly where it’s coming from so doesn’t need further diagnostics here I think

The Cython fused dispatch code was changed to make it faster, but that’s clearly exposed a few other issues which need sorting.

[ShaharNaveh]

This should be fixed in Cython now.

I can confirmed that this is fixed!

Thank you @da-woods!