[BUG] Using types.CodeType in Python 3.11 fails with ValueError: types.CodeType size changed, may indicate binary incompatibility. Expected 168 from C header, got 160 from PyObject

Versions:

• Cython 0.29 branch: commit c0dfb9c2f592b7e0ece90ec83fbadcf6e6da3d47 (May 17, 2022)
• Python 3.11.0b3+: commit a848a9894d (Jun 6 2022)

Reproducer, myext.pyx:

# cython: language_level=3
cdef extern from "Python.h":
    ctypedef class types.CodeType [object PyCodeObject]:
        pass

Build the extension (I make the assumption that cythonize uses Python 3.11):

cythonize -i myext.pyx

Importing the C extension fails:

$ python3.11 -c 'import myext'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "myext.pyx", line 1, in init myext
    # cython: language_level=3
                               
ValueError: types.CodeType size changed, may indicate binary incompatibility. Expected 168 from C header, got 160 from PyObject

To get Python 3.11 with Cython, you can use for example:

wget https://www.python.org/ftp/python/3.11.0/Python-3.11.0b3.tar.xz
tar -xf Python-3.11.0b3.tar.xz
cd Python-3.11.0b3
./configure --prefix /opt/py311
make
make install
cd ..
/opt/py311/bin/python3.11 -m pip install Cython

Then use /opt/py311/bin/cythonize. You can use a different prefix.

---

The problem is that tp_basicsize of &PyCode_Type is 160 bytes, whereas and sizeof(PyCodeObject) is 168 bytes.

---

Issue discovered when on my PR to update gevent to Python 3.11: https://github.com/gevent/gevent/pull/1872#issuecomment-1146149244

@hroncok proposed a Cython pull request in Fedora: https://src.fedoraproject.org/rpms/Cython/pull-request/35

I am working on reworking @hroncok’s PR to propose a PR to the master branch of Cython.

cc @encukou who helped to debug this tedious issue.

[scoder]

This has been resolved by weakening the conditions in https://github.com/cython/cython/pull/4894

[encukou]

I wonder if sizeof() also adds padding after the last struct field?

Yes, it does 😦 It seems that the upper bound should be rounded up to the nearest alignof(the struct) boundary. (But alignof is C11. For older C… well, people tend to get away with using sizeof(long long) as maximum alignment, or even hardcoding it to 64 – even on platforms where long long has 128-byte alignment, since long long is rare in complex structs.)

Adding an option always means admitting defeat regarding a proper solution. In the end, this is a safety check meant to produce error messages instead of crashes.

It’s good to know there is possibility to admit defeat 😃