Inspection of some cython produced functions causes segfault

Description

Trying to access __kwdefaults__ on some Cython produced functions, either by getattr or hasattr, segfaults the Python process. This occurs at least on macOS when scikit-learn is installed from PyPI using pip install scikit-learn. (I’m reporting a similar issue on pandas, where I can install via pip install pandas --no-binary :all: and no longer get the segfault, but I’m having trouble compiling it on my Mac.) There may be other functions which do this, but this is the first we encounter.

This impacts our Python language server as it uses the inspect library to examine libraries without Python source. The code sample below is a minimal repro, but in reality it’s being called by inspect.getfullargspec() (which eventually does this). When it segfaults, our process crashes (and on some OSs like macOS produces a visible popup as the OS is tracking these sorts of crashes). See: Microsoft/python-language-server#740

cython/cython#1470 looks to be related, and would be fixed in Cython 0.29.6, so maybe a version bump is all that would be required. (I’m currently working on building it locally to see if it goes away, which does work for pandas, as previously mentioned.)

Steps/Code to Reproduce

from slearn.cluster._k_means_elkan import k_means_elkan
getattr(k_means_elkan, "__kwdefaults__", None)

Run with -X faulthandler to get more info.

Expected Results

Anything, just no crash.

Actual Results

Python segfaults at getattr. Here’s what macOS’s crash reporter says:

Process: Python [80492] Path: /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python Identifier: Python Version: 3.7.3 (3.7.3) Code Type: X86-64 (Native) Parent Process: zsh [72384] Responsible: Python [80492] User ID: 501

Date/Time: 2019-04-17 16:16:08.689 -0700 OS Version: Mac OS X 10.14.4 (18E226) Report Version: 12 Bridge OS Version: 3.0 (14Y674) Anonymous UUID: 5A957B3E-4E8F-3DE2-C606-5B11FE48E6DD

Sleep/Wake UUID: 02FDA72B-8D53-471B-80AE-6514E0B386FB

Time Awake Since Boot: 23000 seconds Time Since Wake: 2100 seconds

System Integrity Protection: disabled

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000008 Exception Note: EXC_CORPSE_NOTIFY

Termination Signal: Segmentation fault: 11 Termination Reason: Namespace SIGNAL, Code 0xb Terminating Process: exc handler [80492]

VM Regions Near 0x8: –> __TEXT 000000010e316000-000000010e318000 [ 8K] r-x/rwx SM=COW /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 _k_means_elkan.cpython-37m-darwin.so 0x00000001260712bf pyx_pf_7sklearn_7cluster_14_k_means_elkan_12__defaults + 47 1 sparsefuncs_fast.cpython-37m-darwin.so 0x0000000125693410 __Pyx_CyFunction_get_kwdefaults + 48 2 org.python.python 0x000000010e33b587 getset_get + 58 3 org.python.python 0x000000010e362c2f _PyObject_GenericGetAttrWithDict + 180 4 org.python.python 0x000000010e362b20 _PyObject_LookupAttr + 166 5 org.python.python 0x000000010e3bfd66 builtin_getattr + 141 6 org.python.python 0x000000010e3366f2 _PyMethodDef_RawFastCallKeywords + 495 7 org.python.python 0x000000010e335c8e _PyCFunction_FastCallKeywords + 44 8 org.python.python 0x000000010e3cadb2 call_function + 636 9 org.python.python 0x000000010e3c3c35 _PyEval_EvalFrameDefault + 6594 10 org.python.python 0x000000010e3cb6d3 _PyEval_EvalCodeWithName + 1867 11 org.python.python 0x000000010e3c21d0 PyEval_EvalCode + 51 12 org.python.python 0x000000010e3f079b run_mod + 54 13 org.python.python 0x000000010e3ef7c5 PyRun_FileExFlags + 163 14 org.python.python 0x000000010e3eee6b PyRun_SimpleFileExFlags + 263 15 org.python.python 0x000000010e4079b0 pymain_main + 5367 16 org.python.python 0x000000010e408088 _Py_UnixMain + 56 17 libdyld.dylib 0x00007fff5dead3d5 start + 1

Thread 1: 0 libsystem_kernel.dylib 0x00007fff5dfe586a __psynch_cvwait + 10 1 libsystem_pthread.dylib 0x00007fff5e09e56e _pthread_cond_wait + 722 2 libopenblasp-r0.3.5.dev.dylib 0x000000011a621a3b blas_thread_server + 619 3 libsystem_pthread.dylib 0x00007fff5e09b2eb _pthread_body + 126 4 libsystem_pthread.dylib 0x00007fff5e09e249 _pthread_start + 66 5 libsystem_pthread.dylib 0x00007fff5e09a40d thread_start + 13

Thread 2: 0 libsystem_kernel.dylib 0x00007fff5dfe586a __psynch_cvwait + 10 1 libsystem_pthread.dylib 0x00007fff5e09e56e _pthread_cond_wait + 722 2 libopenblasp-r0.3.5.dev.dylib 0x000000011a621a3b blas_thread_server + 619 3 libsystem_pthread.dylib 0x00007fff5e09b2eb _pthread_body + 126 4 libsystem_pthread.dylib 0x00007fff5e09e249 _pthread_start + 66 5 libsystem_pthread.dylib 0x00007fff5e09a40d thread_start + 13

Thread 3: 0 libsystem_kernel.dylib 0x00007fff5dfe586a __psynch_cvwait + 10 1 libsystem_pthread.dylib 0x00007fff5e09e56e _pthread_cond_wait + 722 2 libopenblasp-r0.3.5.dev.dylib 0x000000011a621a3b blas_thread_server + 619 3 libsystem_pthread.dylib 0x00007fff5e09b2eb _pthread_body + 126 4 libsystem_pthread.dylib 0x00007fff5e09e249 _pthread_start + 66 5 libsystem_pthread.dylib 0x00007fff5e09a40d thread_start + 13

Thread 0 crashed with X86 Thread State (64-bit): rax: 0x0000000000000000 rbx: 0x0000000125f54118 rcx: 0x000000012569cda0 rdx: 0x00007ffee18e8a18 rdi: 0x0000000125f54118 rsi: 0x0000000000000000 rbp: 0x00007ffee18e89e0 rsp: 0x00007ffee18e89b0 r8: 0x50e9a9e0fed7166b r9: 0x00007ffee18e8a90 r10: 0x00007f9c0d2f6e68 r11: 0x00007ffee18e8ae8 r12: 0x0000000125f54118 r13: 0x000000012569ce18 r14: 0x0000000000000000 r15: 0x0000000125f54118 rip: 0x00000001260712bf rfl: 0x0000000000010246 cr2: 0x0000000000000008

Logical CPU: 2 Error Code: 0x00000004 Trap Number: 14

Versions

System:
    python: 3.7.3 (default, Mar 27 2019, 09:23:15)  [Clang 10.0.1 (clang-1001.0.46.3)]
executable: /Users/jake/skseg2/venv/bin/python
   machine: Darwin-18.5.0-x86_64-i386-64bit

BLAS:
    macros: NO_ATLAS_INFO=3, HAVE_CBLAS=None
  lib_dirs: 
cblas_libs: cblas

Python deps:
       pip: 19.0.3
setuptools: 40.8.0
   sklearn: 0.20.3
     numpy: 1.16.2
     scipy: 1.2.1
    Cython: None
    pandas: 0.24.2

[jakebailey]

Oh, I see, even though that repo’s in the MacPython org, it builds all of the wheels.

[jakebailey]

@rth I believe this affects all platforms, not just macOS. I tested my cases on Linux, and we’ve had people report this on Windows as well. Will the version bump be done everywhere it’s needed?