CircleCI fails on pypy3 build

Several pypy3 builds on circleci have been failing recently, example: https://app.circleci.com/pipelines/github/scikit-learn/scikit-learn/8663/workflows/a0349bb9-2c2f-4334-85f2-c48f09a1a30d/jobs/122623

This is kind of a blocker for the 0.24 release since conda-forge now builds for pypy.

Ping @rth

platform linux -- Python 3.6.9[pypy-7.2.0-final], pytest-6.1.1, py-1.9.0, pluggy-0.13.1
rootdir: /root/project, configfile: setup.cfg
collecting 3 items                                                             Fatal Python error: Segmentation fault

Stack (most recent call first, approximate line numbers):
  File "/root/project/pypy-env/site-packages/numpy/core/_asarray.py", line 16 in asarray
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File "/root/project/sklearn/manifold/_t_sne.py", line 11 in <module>
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File "/root/project/sklearn/manifold/__init__.py", line 1 in <module>
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File "/root/project/sklearn/cluster/_spectral.py", line 2 in <module>
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File "/root/project/sklearn/cluster/__init__.py", line 1 in <module>
  File ???
  File ???
  File ???
  File ???
  File ???
  File ???
  File "/usr/local/lib-python/3/importlib/__init__.py", line 108 in import_module
  File "/root/project/pypy-env/site-packages/_pytest/pathlib.py", line 444 in import_path
  File "/root/project/pypy-env/site-packages/_pytest/doctest.py", line 491 in collect
  File "/root/project/pypy-env/site-packages/_pytest/runner.py", line 340 in <lambda>
  File "/root/project/pypy-env/site-packages/_pytest/runner.py", line 299 in from_call
  File "/root/project/pypy-env/site-packages/_pytest/runner.py", line 339 in pytest_make_collect_report
  File "/root/project/pypy-env/site-packages/pluggy/callers.py", line 157 in _multicall
  File "/root/project/pypy-env/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/root/project/pypy-env/site-packages/pluggy/manager.py", line 90 in _hookexec
  File "/root/project/pypy-env/site-packages/pluggy/hooks.py", line 272 in __call__
  File "/root/project/pypy-env/site-packages/_pytest/runner.py", line 454 in collect_one_node
  File "/root/project/pypy-env/site-packages/_pytest/main.py", line 791 in genitems
  File "/root/project/pypy-env/site-packages/_pytest/main.py", line 571 in perform_collect
  File "/root/project/pypy-env/site-packages/_pytest/main.py", line 322 in pytest_collection
  File "/root/project/pypy-env/site-packages/pluggy/callers.py", line 157 in _multicall
  File "/root/project/pypy-env/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/root/project/pypy-env/site-packages/pluggy/manager.py", line 90 in _hookexec
  File "/root/project/pypy-env/site-packages/pluggy/hooks.py", line 272 in __call__
  File "/root/project/pypy-env/site-packages/_pytest/main.py", line 309 in _main
  File "/root/project/pypy-env/site-packages/_pytest/main.py", line 244 in wrap_session
  File "/root/project/pypy-env/site-packages/_pytest/main.py", line 305 in pytest_cmdline_main
  File "/root/project/pypy-env/site-packages/pluggy/callers.py", line 157 in _multicall
  File "/root/project/pypy-env/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/root/project/pypy-env/site-packages/pluggy/manager.py", line 90 in _hookexec
  File "/root/project/pypy-env/site-packages/pluggy/hooks.py", line 272 in __call__
  File "/root/project/pypy-env/site-packages/_pytest/config/__init__.py", line 130 in main
  File "/root/project/pypy-env/site-packages/_pytest/config/__init__.py", line 180 in console_main
  File "/root/project/pypy-env/site-packages/pytest/__main__.py", line 1 in <module>
  File "/usr/local/lib-python/3/runpy.py", line 62 in _run_code
  File "/usr/local/lib-python/3/runpy.py", line 167 in _run_module_as_main
  File "<builtin>/app_main.py", line 88 in run_toplevel
  File "<builtin>/app_main.py", line 560 in run_command_line
  File "<builtin>/app_main.py", line 862 in entry_point
./build_tools/circle/build_test_pypy.sh: line 46:  7987 Segmentation fault      (core dumped) python -m pytest sklearn/

[ogrisel]

It’s possible to reproduce easily with an env dedicated to pypy from conda-forge as follows:

conda create -n pypy -c conda-forge pypy numpy scipy joblib threadpoolctl cython pytest
conda activate pypy
python setup.py develop  # python is an alias for pypy in this env
python -c "import sklearn.cluster"

yields:

Fatal error in cpyext, CPython compatibility layer, calling PyMemoryView_FromObject
Either report a bug or consider not using this particular extension
<OperationError object at 0x564dd9c3d038>
RPython traceback:
  File "pypy_module_cpyext.c", line 5650, in wrapper_second_level__star_1
  File "pypy_module_cpyext_1.c", line 35424, in make_ref
  File "pypy_module_cpyext_2.c", line 10168, in create_ref
  File "pypy_module_cpyext_3.c", line 23729, in memory_attach
  File "pypy_module_cpyext_4.c", line 9653, in fill_Py_buffer
  File "pypy_module_cpyext_1.c", line 41590, in unwrapper__StdObjSpaceConst_SomeInstance_star_1_2
  File "pypy_interpreter.c", line 22498, in BuiltinCodePassThroughArguments0_funcrun
  File "pypy_module__frozen_importlib.c", line 70, in import_with_frames_removed
  File "rpython_rlib_rvmprof.c", line 105, in __vmprof_eval_vmprof
  File "pypy_interpreter_1.c", line 29197, in execute_frame
  File "rpython_jit_metainterp.c", line 8574, in ll_portal_runner__Unsigned_Bool_pypy_interpreter
  File "pypy_module_pypyjit.c", line 181, in portal_7
  File "pypy_interpreter_2.c", line 33958, in handle_bytecode__AccessDirect_None
  File "pypy_interpreter_3.c", line 14663, in dispatch_bytecode__AccessDirect_None
  File "pypy_interpreter_3.c", line 38521, in CALL_FUNCTION__AccessDirect_None
  File "rpython_rlib_rvmprof.c", line 105, in __vmprof_eval_vmprof
  File "pypy_interpreter_1.c", line 29197, in execute_frame
  File "rpython_jit_metainterp.c", line 8574, in ll_portal_runner__Unsigned_Bool_pypy_interpreter
  File "pypy_module_pypyjit.c", line 181, in portal_7
  File "pypy_interpreter_2.c", line 33958, in handle_bytecode__AccessDirect_None
  File "pypy_interpreter_3.c", line 14663, in dispatch_bytecode__AccessDirect_None
  File "pypy_interpreter_3.c", line 38521, in CALL_FUNCTION__AccessDirect_None
  File "rpython_rlib_rvmprof.c", line 105, in __vmprof_eval_vmprof
  File "pypy_interpreter_1.c", line 29197, in execute_frame
  File "rpython_jit_metainterp.c", line 8600, in ll_portal_runner__Unsigned_Bool_pypy_interpreter
  File "rpython_jit_metainterp.c", line 21618, in handle_jitexception_7
Segmentation fault (core dumped)

However if you replace the pypy artifacts generated by this PR:

https://github.com/conda-forge/pypy3.6-feedstock/pull/30#issuecomment-721819190

then it works:

unzip ~/Downloads/conda_artifacts_linux_64__656edd35-690f-5c53-9ba3-09c10d0bea97.zip
conda install conda_artifacts_linux_64__656edd35-690f-5c53-9ba3-09c10d0bea97/linux-64/pypy3.6-7.3.2-h45e8706_2.tar.bz2
python -c "import sklearn.cluster"

[cmarmo]

I have had a look to the segfaults. For the sake of completness here is the list of the import giving a segfault with pypy:

• import sklearn.cluster
• import sklearn.ensemble
• import sklearn.inspection
• import sklearn.manifold
• import sklearn.mixture
• import sklearn.tree