pytest segfaults but python -m pytest does not

tl;dr - read the title of this issue.

Related information

• Related issue: #2564
• Is triggered by: diana-hep/pyhf#385
• Related documentation: https://docs.pytest.org/en/latest/pythonpath.html#invoking-pytest-versus-python-m-pytest

This occurs as part of a travis-CI job. This is very hard to provide a MWE, so I will instead provide enough background information.

Environment

OS

travis@travis-job-2326826f-1987-469e-80ed-956889b8a731:~/build/diana-hep/pyhf$ uname -a

Linux travis-job-2326826f-1987-469e-80ed-956889b8a731 4.4.0-101-generic #124~14.04.1-Ubuntu SMP Fri Nov 10 19:05:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

pip list

Package                            Version     Location                         
---------------------------------- ----------- ---------------------------------
absl-py                            0.6.1       
adal                               1.2.0       
alabaster                          0.7.12      
ansiwrap                           0.8.3       
asn1crypto                         0.24.0      
aspy.yaml                          1.1.1       
astor                              0.7.1       
atomicwrites                       1.2.1       
attrs                              18.2.0      
awkward                            0.7.1       
azure-common                       1.1.16      
azure-datalake-store               0.0.40      
azure-nspkg                        3.0.2       
azure-storage-blob                 1.4.0       
azure-storage-common               1.4.0       
azure-storage-nspkg                3.1.0       
Babel                              2.6.0       
backports-abc                      0.5         
backports.functools-lru-cache      1.5         
backports.shutil-get-terminal-size 1.0.0       
backports.shutil-which             3.5.2       
backports.weakref                  1.0.post1   
bleach                             3.1.0       
boto3                              1.9.77      
botocore                           1.12.77     
bumpversion                        0.5.3       
cachetools                         3.0.0       
certifi                            2018.11.29  
cffi                               1.11.5      
cfgv                               1.4.0       
chardet                            3.0.4       
Click                              7.0         
colorama                           0.4.1       
configparser                       3.5.0       
contextlib2                        0.5.5       
coverage                           4.5.2       
cryptography                       2.4.2       
cycler                             0.10.0      
decorator                          4.3.0       
defusedxml                         0.5.0       
docutils                           0.14        
entrypoints                        0.3         
enum34                             1.1.6       
funcsigs                           1.0.2       
functools32                        3.2.3.post2 
future                             0.17.1      
futures                            3.2.0       
gast                               0.2.1.post0 
gitdb2                             2.0.5       
GitPython                          2.1.11      
graphviz                           0.10.1      
grpcio                             1.17.1      
h5py                               2.9.0       
identify                           1.1.8       
idna                               2.6         
imagesize                          1.1.0       
iminuit                            1.3.3       
importlib-metadata                 0.8         
importlib-resources                1.0.2       
ipaddress                          1.0.22      
ipykernel                          4.10.0      
ipython                            5.8.0       
ipython-genutils                   0.2.0       
ipywidgets                         7.4.2       
Jinja2                             2.10        
jmespath                           0.9.3       
jsonpatch                          1.23        
jsonpointer                        2.0         
jsonschema                         3.0.0a4     
jupyter                            1.0.0       
jupyter-client                     5.2.4       
jupyter-console                    5.2.0       
jupyter-core                       4.4.0       
Keras-Applications                 1.0.6       
Keras-Preprocessing                1.0.5       
kiwisolver                         1.0.1       
latexcodec                         1.0.5       
m2r                                0.2.1       
Markdown                           3.0.1       
MarkupSafe                         1.1.0       
matplotlib                         2.2.3       
mistune                            0.8.4       
mock                               2.0.0       
more-itertools                     5.0.0       
mxnet                              1.3.1       
nbconvert                          5.4.0       
nbdime                             1.0.4       
nbformat                           4.4.0       
nbsphinx                           0.4.1       
nodeenv                            1.3.3       
nose                               1.3.7       
notebook                           5.7.4       
numpy                              1.14.6      
oset                               0.1.3       
packaging                          18.0        
pandas                             0.23.4      
pandocfilters                      1.4.2       
papermill                          0.16.2      
pathlib2                           2.3.3       
pbr                                5.1.1       
pexpect                            4.6.0       
pickleshare                        0.7.5       
pip                                18.1        
pkginfo                            1.5.0.1     
pluggy                             0.8.1       
pockets                            0.7.2       
pre-commit                         1.14.1      
prometheus-client                  0.5.0       
prompt-toolkit                     1.0.15      
protobuf                           3.6.1       
ptyprocess                         0.6.0       
py                                 1.7.0       
py-cpuinfo                         4.0.0       
pybtex                             0.22.0      
pybtex-docutils                    0.2.1       
pycparser                          2.19        
pyflakes                           2.0.0       
pygal                              2.4.0       
pygaljs                            1.0.1       
Pygments                           2.3.1       
pyhf                               0.0.15      /home/travis/build/diana-hep/pyhf
PyJWT                              1.7.1       
pyparsing                          2.3.0       
pyrsistent                         0.14.9      
pytest                             3.10.1      
pytest-benchmark                   3.2.0       
pytest-console-scripts             0.1.7       
pytest-cov                         2.6.1       
pytest-mock                        1.10.0      
pytest-runner                      4.2         
python-coveralls                   2.9.1       
python-dateutil                    2.7.5       
pytz                               2018.9      
PyYAML                             3.13        
pyzmq                              17.1.2      
qtconsole                          4.4.3       
readme-renderer                    24.0        
requests                           2.18.4      
requests-toolbelt                  0.8.0       
s3transfer                         0.1.13      
scandir                            1.9.0       
scipy                              1.2.0       
Send2Trash                         1.5.0       
setuptools                         40.6.3      
simplegeneric                      0.8.1       
singledispatch                     3.4.0.3     
six                                1.12.0      
smmap2                             2.0.5       
snowballstemmer                    1.2.1       
Sphinx                             1.8.3       
sphinx-issues                      1.2.0       
sphinx-rtd-theme                   0.4.2       
sphinxcontrib-bibtex               0.4.2       
sphinxcontrib-napoleon             0.7         
sphinxcontrib-websupport           1.1.0       
statistics                         1.0.3.5     
subprocess32                       3.5.3       
tensorboard                        1.12.2      
tensorflow                         1.12.0      
tensorflow-probability             0.5.0       
termcolor                          1.1.0       
terminado                          0.8.1       
testpath                           0.4.2       
textwrap3                          0.9.1       
toml                               0.10.0      
torch                              1.0.0       
tornado                            5.1.1       
tqdm                               4.29.0      
traitlets                          4.3.2       
twine                              1.12.1      
typing                             3.6.6       
uproot                             3.3.2       
uproot-methods                     0.3.3       
urllib3                            1.22        
virtualenv                         16.2.0      
wcwidth                            0.1.7       
webencodings                       0.5.1       
Werkzeug                           0.14.1      
wheel                              0.32.3      
widgetsnbextension                 3.4.2       
zipp                               0.3.3       

python

travis@travis-job-2326826f-1987-469e-80ed-956889b8a731:~/build/diana-hep/pyhf$ which python
/opt/pyenv/shims/python
travis@travis-job-2326826f-1987-469e-80ed-956889b8a731:~/build/diana-hep/pyhf$ python --version
Python 2.7.14

python path

>>> import sys
>>> sys.path
['', '/home/travis/virtualenv/python2.7.14/lib/python27.zip', '/home/travis/virtualenv/python2.7.14/lib/python2.7', '/home/travis/virtualenv/python2.7.14/lib/python2.7/plat-linux2', '/home/travis/virtualenv/python2.7.14/lib/python2.7/lib-tk', '/home/travis/virtualenv/python2.7.14/lib/python2.7/lib-old', '/home/travis/virtualenv/python2.7.14/lib/python2.7/lib-dynload', '/opt/python/2.7.14/lib/python2.7', '/opt/python/2.7.14/lib/python2.7/plat-linux2', '/opt/python/2.7.14/lib/python2.7/lib-tk', '/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages', '/home/travis/build/diana-hep/pyhf']

Background

pyhf is a statistics tool that uses multiple backends for performing tensor calculations. Some of these backends include tensorflow, mxnet, pytorch, etc… We initially observed a segfault:

Segmentation fault: 11
Stack trace returned 10 entries:
[bt] (0) /home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x1e945a) [0x7f4fec78545a]
[bt] (1) /home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2f891f6) [0x7f4fef5251f6]
[bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0) [0x7f5061593cb0]
[bt] (3) /lib/x86_64-linux-gnu/libpthread.so.0(sem_wait+0) [0x7f5061934640]
[bt] (4) /opt/python/2.7.14/lib/libpython2.7.so.1.0(PyThread_acquire_lock+0x68) [0x7f5061c92578]
[bt] (5) /lib/x86_64-linux-gnu/libcrypto.so.1.0.0(+0x610c0) [0x7f505e3140c0]
[bt] (6) /lib/x86_64-linux-gnu/libcrypto.so.1.0.0(+0x61749) [0x7f505e314749]
[bt] (7) /lib/x86_64-linux-gnu/libssl.so.1.0.0(SSL_CTX_free+0x6e) [0x7f505bd1f8de]
[bt] (8) /home/travis/virtualenv/python2.7.14/lib/python2.7/lib-dynload/_ssl.so(+0xa06b) [0x7f505bf5106b]
[bt] (9) /opt/python/2.7.14/lib/libpython2.7.so.1.0(+0xba17a) [0x7f5061bfe17a]

which we initially believed to be due to changes in pathlib vs pathlib2 in ionelmc/pytest-benchmark#8469a5.

However, upon doing some debug session with travis, I’ve stumbled upon the fact that pytest invocation causes a segfault while python -m pytest does not, even including updating the PYTHONPATH to take into account the current directory as suggested by pytest documentation. In summary, the following commands produce the segfault:

• pytest tests/test_notebooks.py
• PYTHONPATH=. pytest tests/test_notebooks.py
• python $(which pytest) tests/test_notebooks.py

The following commands do not produce a segfault:

• python -m pytest tests/test_notebooks.py
• PYTHONPATH=. python -m pytest tests/test_notebooks.py
• /usr/bin/env python -m pytest tests/test_notebooks.py
• /home/travis/virtualenv/python2.7.14/bin/python -m pytest tests/test_notebooks.py

Here’s a quick checklist in what to include:

• Include a detailed description of the bug or suggestion
• pip list of the virtual environment you are using
• pytest and operating system versions
• Minimal example if possible

[asottile]

I can reproduce this outside of travis-ci 🎉

dockerfile

FROM ubuntu:trusty
RUN : \
    && apt-get update \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        g++ \
        gcc \
        gdb \
        git-core \
        libbz2-dev \
        libreadline-dev \
        libsqlite3-dev \
        libssl-dev \
        make \
        python-dev \
        python-virtualenv \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN : \
    && git clone https://github.com/pyenv/pyenv ~/.pyenv \
    && bash -euxc ' \
        : \
        && export PYENV_ROOT=/root/.pyenv \
        && export PATH=/root/.pyenv/bin:$PATH \
        && eval "$(pyenv init -)" \
        && pyenv install 2.7.14 \
        && pyenv global 2.7.14 \
        && test "$(python2.7 --version 2>&1)" = "Python 2.7.14" \
    '

WORKDIR /pyhf

ENV PATH=/venv/bin:$PATH
RUN : \
    && git clone https://github.com/diana-hep/pyhf . \
    && git checkout c3dd4592e54392ea0af25132c029df855423211b \
    && virtualenv /venv -p /root/.pyenv/shims/python2.7 \
    && pip install --upgrade setuptools pip \
    && pip install --no-cache-dir -e .[complete]

build

docker build -t test .

runtime

$ docker run --rm --privileged -ti test bash
root@dacd63468da6:/pyhf# gdb $(which python)
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /venv/bin/python...done.
(gdb) run /venv/bin/pytest tests/test_notebooks.py
Starting program: /venv/bin/python /venv/bin/pytest tests/test_notebooks.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffee627700 (LWP 46)]
[New Thread 0x7fffede26700 (LWP 47)]
[New Thread 0x7fffeb625700 (LWP 48)]
[New Thread 0x7fffe8e24700 (LWP 49)]
[New Thread 0x7fffe6623700 (LWP 50)]
[New Thread 0x7fffe3e22700 (LWP 51)]
[New Thread 0x7fffe1621700 (LWP 52)]
[New Thread 0x7fff7c4db700 (LWP 53)]
[New Thread 0x7fff7bcda700 (LWP 54)]
[New Thread 0x7fff7b4d9700 (LWP 55)]
[New Thread 0x7fff7acd8700 (LWP 56)]
[New Thread 0x7fff7a4d7700 (LWP 57)]
[New Thread 0x7fff79cd6700 (LWP 58)]
[New Thread 0x7fff794d5700 (LWP 59)]
[New Thread 0x7fff78cd4700 (LWP 60)]
[New Thread 0x7fff5bfff700 (LWP 61)]
[New Thread 0x7fff5b7fe700 (LWP 62)]
[New Thread 0x7fff5affd700 (LWP 63)]
[New Thread 0x7fff5a7fc700 (LWP 64)]
[New Thread 0x7fff59ffb700 (LWP 65)]
[New Thread 0x7fff597fa700 (LWP 66)]
[New Thread 0x7fff58ff9700 (LWP 67)]
[New Thread 0x7fff3bfff700 (LWP 68)]
[New Thread 0x7fff3b7fe700 (LWP 69)]
[New Thread 0x7fff3affd700 (LWP 70)]
[New Thread 0x7fff1d42a700 (LWP 71)]
[New Thread 0x7fff1cc29700 (LWP 72)]
[New Thread 0x7fff1c428700 (LWP 73)]
[Thread 0x7fff1c428700 (LWP 73) exited]
[Thread 0x7fff1cc29700 (LWP 72) exited]
[New Thread 0x7fff1cc29700 (LWP 74)]
[New Thread 0x7fff1c428700 (LWP 75)]
[New Thread 0x7fff1bc27700 (LWP 76)]
[New Thread 0x7fff1b426700 (LWP 77)]
[New Thread 0x7fff1ac25700 (LWP 78)]
[New Thread 0x7fff1a424700 (LWP 79)]
[New Thread 0x7fff19c23700 (LWP 80)]
[Thread 0x7fff1a424700 (LWP 79) exited]
[Thread 0x7fff19c23700 (LWP 80) exited]
[Thread 0x7fff1ac25700 (LWP 78) exited]
[Thread 0x7fff1b426700 (LWP 77) exited]
[Thread 0x7fffe1621700 (LWP 52) exited]
[Thread 0x7fffe3e22700 (LWP 51) exited]
[Thread 0x7fffede26700 (LWP 47) exited]
[Thread 0x7fffe6623700 (LWP 50) exited]
[Thread 0x7fffe8e24700 (LWP 49) exited]
[Thread 0x7fffeb625700 (LWP 48) exited]
[Thread 0x7fffee627700 (LWP 46) exited]
[New Thread 0x7fffe1621700 (LWP 82)]
[New Thread 0x7fffe3e22700 (LWP 87)]
[New Thread 0x7fffe6623700 (LWP 88)]
[New Thread 0x7fffe8e24700 (LWP 89)]
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.10.1, py-1.7.0, pluggy-0.8.1
benchmark: 3.2.1 (defaults: timer=time.time disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /pyhf, inifile: pytest.ini
plugins: mock-1.10.0, cov-2.6.1, console-scripts-0.1.7, benchmark-3.2.1
collecting ... 
Program received signal SIGSEGV, Segmentation fault.
update_refs (containers=0x7cf080 <generations+96>) at Modules/gcmodule.c:332
332	Modules/gcmodule.c: No such file or directory.
(gdb) bt
#0  update_refs (containers=0x7cf080 <generations+96>)
    at Modules/gcmodule.c:332
#1  collect (generation=2) at Modules/gcmodule.c:924
#2  0x0000000000503461 in _PyObject_GC_Malloc (basicsize=<optimized out>)
    at Modules/gcmodule.c:1511
#3  _PyObject_GC_New (tp=tp@entry=0x7e1a00 <PyFunction_Type>)
    at Modules/gcmodule.c:1521
#4  0x0000000000533a84 in PyFunction_New (code=code@entry=0x7ffff2376ab0, 
    globals=0x7ffff236ee88) at Objects/funcobject.c:12
#5  0x00000000004b3c8f in PyEval_EvalFrameEx (f=f@entry=0x27f4be0, 
    throwflag=throwflag@entry=0) at Python/ceval.c:3051
#6  0x00000000004b8c66 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=3, pp_stack=0x7fffffff37d0, func=0x7ffff1c0aa28)
    at Python/ceval.c:4442
#7  call_function (oparg=<optimized out>, pp_stack=0x7fffffff37d0)
    at Python/ceval.c:4377
#8  PyEval_EvalFrameEx (f=f@entry=0x7ffff17657b0, throwflag=throwflag@entry=0)
    at Python/ceval.c:2994
#9  0x00000000004baf58 in PyEval_EvalCodeEx (co=<optimized out>, 
    globals=<optimized out>, locals=locals@entry=0x0, 
    args=args@entry=0x7fffedc7e248, argcount=3, kws=kws@entry=0x0, 
    kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, 
    closure=0x0) at Python/ceval.c:3589
#10 0x0000000000533190 in function_call (func=0x7ffff1c0ab18, 
    arg=0x7fffedc7e230, kw=0x0) at Objects/funcobject.c:523
#11 0x00000000004231fa in PyObject_Call (func=func@entry=0x7ffff1c0ab18, 
    arg=arg@entry=0x7fffedc7e230, kw=kw@entry=0x0) at Objects/abstract.c:2547
#12 0x00000000004276e5 in instancemethod_call (func=0x7ffff1c0ab18, 
    arg=0x7fffedc7e230, kw=0x0) at Objects/classobject.c:2602
#13 0x0000000000423de0 in PyObject_Call (kw=0x0, arg=0x7fffedc84e18, 
    func=0x7ffff16b7aa0) at Objects/abstract.c:2547
#14 PyObject_CallFunctionObjArgs (callable=0x7ffff16b7aa0)
    at Objects/abstract.c:2774
#15 0x00007ffff1c8011d in CTracer_handle_call (frame=0x664b0e0, 
    self=0x7ffff173e9f0) at coverage/ctracer/tracer.c:398
#16 CTracer_trace (self=0x7ffff173e9f0, frame=0x664b0e0, what=<optimized out>, 
    arg_unused=<optimized out>) at coverage/ctracer/tracer.c:871
#17 0x00000000004b0aad in call_trace (arg=0x7ba070 <_Py_NoneStruct>, what=0, 
    frame=0x664b0e0, obj=0x7ffff173e9f0, func=0x7ffff1c7f810 <CTracer_trace>)
    at Python/ceval.c:4028
#18 call_trace_protected (func=0x7ffff1c7f810 <CTracer_trace>, 
    obj=0x7ffff173e9f0, frame=frame@entry=0x664b0e0, what=what@entry=0, 
    arg=0x7ba070 <_Py_NoneStruct>) at Python/ceval.c:4004
#19 0x00000000004b1ae5 in PyEval_EvalFrameEx (f=f@entry=0x664b0e0, 
    throwflag=throwflag@entry=0) at Python/ceval.c:993
#20 0x00000000004baf58 in PyEval_EvalCodeEx (co=co@entry=0x7fffedc821b0, 
    globals=globals@entry=0x7fffedcd6d70, locals=locals@entry=0x7fffedcd6d70, 
    args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, 
    kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, 
    closure=closure@entry=0x0) at Python/ceval.c:3589
#21 0x00000000004bb082 in PyEval_EvalCode (co=co@entry=0x7fffedc821b0, 
    globals=globals@entry=0x7fffedcd6d70, locals=locals@entry=0x7fffedcd6d70)
    at Python/ceval.c:669
#22 0x00000000004d4a88 in PyImport_ExecCodeModuleEx (
    name=0x6649e70 "wcwidth.table_wide", co=0x7fffedc821b0, 
    pathname=<optimized out>) at Python/import.c:731
#23 0x00000000004d4de6 in load_source_module (
    name=0x6649e70 "wcwidth.table_wide", 
    pathname=0x664de00 "/venv/lib/python2.7/site-packages/wcwidth/table_wide.pyc", fp=<optimized out>) at Python/import.c:1121
#24 0x00000000004d5c77 in import_submodule (mod=mod@entry=0x7fffedc7f210, 
    subname=subname@entry=0x6649e78 "table_wide", 
    fullname=fullname@entry=0x6649e70 "wcwidth.table_wide")
    at Python/import.c:2725
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) run /venv/bin/pytest tests/test_notebooks.py
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: /venv/bin/python /venv/bin/pytest tests/test_notebooks.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffee627700 (LWP 101)]
[New Thread 0x7fffede26700 (LWP 102)]
[New Thread 0x7fffeb625700 (LWP 103)]
[New Thread 0x7fffe8e24700 (LWP 104)]
[New Thread 0x7fffe6623700 (LWP 105)]
[New Thread 0x7fffe3e22700 (LWP 106)]
[New Thread 0x7fffe1621700 (LWP 107)]
[New Thread 0x7fff7c4db700 (LWP 108)]
[New Thread 0x7fff7bcda700 (LWP 109)]
[New Thread 0x7fff7b4d9700 (LWP 110)]
[New Thread 0x7fff7acd8700 (LWP 111)]
[New Thread 0x7fff7a4d7700 (LWP 112)]
[New Thread 0x7fff79cd6700 (LWP 113)]
[New Thread 0x7fff794d5700 (LWP 114)]
[New Thread 0x7fff78cd4700 (LWP 115)]
[New Thread 0x7fff5bfff700 (LWP 116)]
[New Thread 0x7fff5b7fe700 (LWP 117)]
[New Thread 0x7fff5affd700 (LWP 118)]
[New Thread 0x7fff5a7fc700 (LWP 119)]
[New Thread 0x7fff59ffb700 (LWP 120)]
[New Thread 0x7fff597fa700 (LWP 121)]
[New Thread 0x7fff58ff9700 (LWP 122)]
[New Thread 0x7fff33fff700 (LWP 123)]
[New Thread 0x7fff3bfff700 (LWP 124)]
[New Thread 0x7fff3b7fe700 (LWP 125)]
[New Thread 0x7fff38ffd700 (LWP 126)]
[New Thread 0x7fff317fe700 (LWP 127)]
[New Thread 0x7fff30ffd700 (LWP 128)]
[Thread 0x7fff317fe700 (LWP 127) exited]
[Thread 0x7fff30ffd700 (LWP 128) exited]
[New Thread 0x7fff30ffd700 (LWP 129)]
[New Thread 0x7fff317fe700 (LWP 130)]
[New Thread 0x7fff1b42a700 (LWP 131)]
[New Thread 0x7fff1ac29700 (LWP 132)]
[New Thread 0x7fff1a428700 (LWP 133)]
[New Thread 0x7fff19c27700 (LWP 134)]
[New Thread 0x7fff19426700 (LWP 135)]
[Thread 0x7fff19426700 (LWP 135) exited]
[Thread 0x7fff19c27700 (LWP 134) exited]
[Thread 0x7fff1ac29700 (LWP 132) exited]
[Thread 0x7fff1a428700 (LWP 133) exited]
[Thread 0x7fffe3e22700 (LWP 106) exited]
[Thread 0x7fffe1621700 (LWP 107) exited]
[Thread 0x7fffede26700 (LWP 102) exited]
[Thread 0x7fffe6623700 (LWP 105) exited]
[Thread 0x7fffee627700 (LWP 101) exited]
[Thread 0x7fffe8e24700 (LWP 104) exited]
[Thread 0x7fffeb625700 (LWP 103) exited]
[New Thread 0x7fffe1621700 (LWP 137)]
[New Thread 0x7fffe3e22700 (LWP 142)]
[New Thread 0x7fffe6623700 (LWP 143)]
[New Thread 0x7fffe8e24700 (LWP 144)]
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.10.1, py-1.7.0, pluggy-0.8.1
benchmark: 3.2.1 (defaults: timer=time.time disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /pyhf, inifile: pytest.ini
plugins: mock-1.10.0, cov-2.6.1, console-scripts-0.1.7, benchmark-3.2.1
collecting ... [Thread 0x7fffe1621700 (LWP 137) exited]
[Thread 0x7fffe8e24700 (LWP 144) exited]
[Thread 0x7fffe6623700 (LWP 143) exited]
[Thread 0x7fffe3e22700 (LWP 142) exited]
[New Thread 0x7fffe8e24700 (LWP 146)]
[New Thread 0x7fffe6623700 (LWP 151)]
[New Thread 0x7fffe3e22700 (LWP 152)]
[New Thread 0x7fffe1621700 (LWP 153)]
collected 1 item                                                               

tests/test_notebooks.py [New Thread 0x7fff189be700 (LWP 154)]
[Thread 0x7fffe8e24700 (LWP 146) exited]
[Thread 0x7fffe6623700 (LWP 151) exited]
[Thread 0x7fffe1621700 (LWP 153) exited]
[Thread 0x7fffe3e22700 (LWP 152) exited]
[New Thread 0x7fffe1621700 (LWP 156)]
[New Thread 0x7fffe3e22700 (LWP 161)]
[New Thread 0x7fffe6623700 (LWP 162)]
[New Thread 0x7fffe8e24700 (LWP 163)]
[Thread 0x7fffe6623700 (LWP 162) exited]
[Thread 0x7fffe8e24700 (LWP 163) exited]
[Thread 0x7fffe3e22700 (LWP 161) exited]
[Thread 0x7fffe1621700 (LWP 156) exited]
[New Thread 0x7fffe8e24700 (LWP 165)]
[New Thread 0x7fffe6623700 (LWP 170)]
[New Thread 0x7fffe3e22700 (LWP 171)]
[New Thread 0x7fffe1621700 (LWP 172)]
[New Thread 0x7fff03fff700 (LWP 173)]
[New Thread 0x7fff037fe700 (LWP 174)]
[New Thread 0x7fff02ffd700 (LWP 175)]
[Thread 0x7fff02ffd700 (LWP 175) exited]
[Thread 0x7fff189be700 (LWP 154) exited]
[New Thread 0x7fff189be700 (LWP 194)]
[Thread 0x7fffe1621700 (LWP 172) exited]
[Thread 0x7fffe3e22700 (LWP 171) exited]
[Thread 0x7fffe8e24700 (LWP 165) exited]
[Thread 0x7fffe6623700 (LWP 170) exited]
[New Thread 0x7fffe1621700 (LWP 196)]
[New Thread 0x7fffe3e22700 (LWP 201)]
[New Thread 0x7fffe6623700 (LWP 202)]
[New Thread 0x7fffe8e24700 (LWP 203)]
[New Thread 0x7fff02ffd700 (LWP 204)]
[Thread 0x7fff02ffd700 (LWP 204) exited]
[Thread 0x7fff189be700 (LWP 194) exited]
[New Thread 0x7fff189be700 (LWP 235)]
[Thread 0x7fffe3e22700 (LWP 201) exited]
[Thread 0x7fffe8e24700 (LWP 203) exited]
[Thread 0x7fffe6623700 (LWP 202) exited]
[Thread 0x7fffe1621700 (LWP 196) exited]
[New Thread 0x7fffe8e24700 (LWP 237)]
[New Thread 0x7fffe6623700 (LWP 242)]
[New Thread 0x7fffe3e22700 (LWP 243)]
[New Thread 0x7fffe1621700 (LWP 244)]
[New Thread 0x7fff02ffd700 (LWP 245)]
[Thread 0x7fff02ffd700 (LWP 245) exited]
[Thread 0x7fff189be700 (LWP 235) exited]
[New Thread 0x7fff189be700 (LWP 264)]
[Thread 0x7fffe6623700 (LWP 242) exited]
[Thread 0x7fffe1621700 (LWP 244) exited]
[Thread 0x7fffe3e22700 (LWP 243) exited]
[Thread 0x7fffe8e24700 (LWP 237) exited]
[New Thread 0x7fffe1621700 (LWP 266)]
[New Thread 0x7fffe3e22700 (LWP 271)]
[New Thread 0x7fffe6623700 (LWP 272)]
[New Thread 0x7fffe8e24700 (LWP 273)]
[New Thread 0x7fff02ffd700 (LWP 274)]
[Thread 0x7fff02ffd700 (LWP 274) exited]
[Thread 0x7fff189be700 (LWP 264) exited]
[New Thread 0x7fff189be700 (LWP 293)]
[Thread 0x7fffe8e24700 (LWP 273) exited]
[Thread 0x7fffe6623700 (LWP 272) exited]
[Thread 0x7fffe1621700 (LWP 266) exited]
[Thread 0x7fffe3e22700 (LWP 271) exited]
[New Thread 0x7fffe8e24700 (LWP 295)]
[New Thread 0x7fffe6623700 (LWP 300)]
[New Thread 0x7fffe3e22700 (LWP 301)]
[New Thread 0x7fffe1621700 (LWP 302)]
[New Thread 0x7fff02ffd700 (LWP 303)]
[Thread 0x7fff02ffd700 (LWP 303) exited]
[Thread 0x7fff189be700 (LWP 293) exited]
.                                                [100%]

---------- coverage: platform linux2, python 2.7.14-final-0 ----------
Name                                Stmts   Miss  Cover   Missing
-----------------------------------------------------------------
pyhf/__init__.py                       31      5    84%   54-58, 60, 79
pyhf/commandline.py                    78     78     0%   1-137
pyhf/constraints.py                    86      2    98%   127-128
pyhf/events.py                         31      5    84%   11-12, 15, 91, 99
pyhf/exceptions/__init__.py            25     11    56%   20-34
pyhf/interpolators/__init__.py         27     19    30%   2-15, 27-37
pyhf/interpolators/code0.py            50     13    74%   51-54, 83-89, 92, 95-96
pyhf/interpolators/code1.py            55     41    25%   27-43, 46-57, 60-72, 75-93, 98-104, 107, 110-111
pyhf/interpolators/code2.py            60     46    23%   31-45, 48-56, 59-64, 67-108, 113-121, 124, 127-128
pyhf/interpolators/code4.py            85     70    18%   35-126, 129-146, 149-164, 167-226, 269-343, 346-347, 350-351
pyhf/modifiers/__init__.py             60     12    80%   20, 40, 48, 58, 135, 138, 158-171
pyhf/modifiers/histosys.py             36      0   100%
pyhf/modifiers/lumi.py                 34      5    85%   14, 55-62
pyhf/modifiers/normfactor.py           34      1    97%   55
pyhf/modifiers/normsys.py              36      6    83%   15, 54, 69-76
pyhf/modifiers/shapefactor.py          39      0   100%
pyhf/modifiers/shapesys.py             69      0   100%
pyhf/modifiers/staterror.py            65     32    51%   14, 55-72, 85-90, 95-123, 129-142
pyhf/optimize/__init__.py              34     16    53%   14-22, 27-35, 47-48
pyhf/optimize/opt_minuit.py            38     25    34%   18-46, 50-53, 59-64
pyhf/optimize/opt_pytorch.py           44     44     0%   1-58
pyhf/optimize/opt_scipy.py             23      6    74%   18-20, 38-40
pyhf/optimize/opt_tflow.py             46     46     0%   1-76
pyhf/paramsets.py                      54      4    93%   23, 75, 91, 97
pyhf/pdf.py                           219     22    90%   26-32, 58-64, 79, 87, 124, 273-281, 296, 444-450, 453-454
pyhf/readxml.py                       116    116     0%   1-257
pyhf/simplemodels.py                    4      0   100%
pyhf/tensor/__init__.py                34      6    82%   21-22, 34-35, 47-48
pyhf/tensor/mxnet_backend.py          127     90    29%   39-40, 52-57, 70-82, 91, 94-95, 107-113, 126-130, 143-147, 150-151, 163, 175, 189-191, 203-204, 217-219, 231-232, 244-245, 262, 289-292, 308, 334-350, 356, 359, 374-375, 378-380, 406-412, 418-420, 449-456, 479-485
pyhf/tensor/numpy_backend.py           83     16    81%   40-43, 46-48, 80, 95, 98, 101, 143, 197-199, 235
pyhf/tensor/pytorch_backend.py        121     85    30%   34-35, 38-43, 46-48, 60-62, 65, 68-69, 72, 75, 78-79, 86-87, 90-91, 94, 97, 100-102, 105-106, 109-111, 114-115, 118-119, 122, 125-128, 142, 145, 168-190, 204-205, 208-210, 236-238, 241-245, 270-274, 295-299
pyhf/tensor/tensorflow_backend.py     143    106    26%   40-45, 48-71, 74-86, 89, 92, 95, 107-118, 121-122, 129-130, 137-138, 141, 144, 147-149, 152-153, 156, 159, 162-164, 167-168, 171-172, 175, 178-181, 195, 219-244, 261, 290-292, 321-323, 351-355, 383-387, 413-417
pyhf/utils.py                          69      3    96%   40-41, 229
pyhf/version.py                         1      0   100%
pyhf/writexml.py                       30     30     0%   1-45
-----------------------------------------------------------------
TOTAL                                2087    961    54%
Coverage HTML written to dir htmlcov


=============================== warnings summary ===============================
/venv/lib/python2.7/site-packages/nbconvert/exporters/exporter_locator.py:28
  /venv/lib/python2.7/site-packages/nbconvert/exporters/exporter_locator.py:28: DeprecationWarning: `nbconvert.exporters.exporter_locator` is deprecated in favor of `nbconvert.exporters.base` since nbconvert 5.0.
    DeprecationWarning)

/venv/lib/python2.7/site-packages/tornado/web.py:1747
  /venv/lib/python2.7/site-packages/tornado/web.py:1747: DeprecationWarning: @asynchronous is deprecated, use coroutines instead
    DeprecationWarning)

tests/test_notebooks.py::test_notebooks
  /venv/lib/python2.7/site-packages/jupyter_client/session.py:348: DeprecationWarning: Session._session_changed is deprecated in traitlets 4.1: use @observe and @unobserve instead.
    def _session_changed(self, name, old, new):

-- Docs: https://docs.pytest.org/en/latest/warnings.html
==================== 1 passed, 3 warnings in 37.19 seconds =====================

Program received signal SIGSEGV, Segmentation fault.
update_refs (containers=0x7cf080 <generations+96>) at Modules/gcmodule.c:332
332	in Modules/gcmodule.c
(gdb) bt
#0  update_refs (containers=0x7cf080 <generations+96>)
    at Modules/gcmodule.c:332
#1  collect (generation=generation@entry=2) at Modules/gcmodule.c:924
#2  0x00000000005031c8 in PyGC_Collect () at Modules/gcmodule.c:1440
#3  0x00000000004ebe0b in Py_Finalize () at Python/pythonrun.c:448
#4  0x00000000004eb9c8 in Py_Exit (sts=sts@entry=0) at Python/pythonrun.c:1783
#5  0x00000000004149f3 in handle_system_exit () at Python/pythonrun.c:1151
#6  0x00000000004ebd25 in handle_system_exit () at Python/pythonrun.c:1192
#7  PyErr_PrintEx (set_sys_last_vars=<optimized out>)
    at Python/pythonrun.c:1161
#8  0x00000000004ec783 in PyErr_Print () at Python/pythonrun.c:1064
#9  PyRun_SimpleFileExFlags (fp=<optimized out>, filename=<optimized out>, 
    closeit=1, flags=0x7fffffffe5f0) at Python/pythonrun.c:952
#10 0x0000000000415f11 in Py_Main (argc=<optimized out>, argv=<optimized out>)
    at Modules/main.c:645
#11 0x00007ffff7107f45 in __libc_start_main ()
   from /lib/x86_64-linux-gnu/libc.so.6
#12 0x00000000004150fe in _start ()

Note that this shows two different crashes – the first crashes on a “fresh” run during collection, the second crashes during final-gc. Running a third time shows the same as the final-gc crash.

Interestingly enough, python -m pytest is also crashing for me:

root@dacd63468da6:/pyhf# python -m pytest tests/test_notebooks.py 
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.10.1, py-1.7.0, pluggy-0.8.1
benchmark: 3.2.1 (defaults: timer=time.time disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /pyhf, inifile: pytest.ini
plugins: mock-1.10.0, cov-2.6.1, console-scripts-0.1.7, benchmark-3.2.1
collecting ... *** Error in `python': double free or corruption (!prev): 0x0000000001adc2a0 ***
Aborted (core dumped)

[asottile]

The vonly difference I could imagine there is the setuptools entry point machinery somehow having a negative side effect. This one sounds like it’s going to be pretty difficult to reproduce!