Crash on `array_dealloc` when running test suite

This is happening when running the full test suite (numpy.test('full', verbose=3)) for NumPy v1.11.0 in win64 pydebug version.

It hangs right after printing Test two arrays with different shapes are found not equal. ..., which means it is probably on test_utils.py. When the tests there are run isolated, it does not crash (crashes only when running the full test suite).

I’ve managed to attach a debugger, and this is the detailed call stack inside multiarray_d.pyd:

    multiarray_d.pyd!PyDataMem_FREE(void * ptr=0x00000000120034f0) Line 214 C
    multiarray_d.pyd!_npy_free_cache(void * p=0x00000000120034f0, unsigned __int64 nelem=0, unsigned int msz=1024, cache_bucket * cache=0x000007fee1bf7280, void (void *) * dealloc=0x000007fee1941540) Line 60 C
    multiarray_d.pyd!npy_free_cache(void * p=0x00000000120034f0, unsigned __int64 sz=0) Line 95 C
>   multiarray_d.pyd!array_dealloc(tagPyArrayObject * self=0x00000000126fa0c0) Line 428 C
    python27_d.dll!_Py_Dealloc(_object * op=0x00000000126fa0c0) Line 2263   C
    python27_d.dll!frame_dealloc(_frame * f=0x0000000004dc63d8) Line 460    C
    python27_d.dll!_Py_Dealloc(_object * op=0x0000000004dc63d8) Line 2263   C
    python27_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x0000000002714b98, _object * globals=0x0000000000000002, _object * locals=0x0000000000000002, _object * * args=0x0000000000000000, int argcount=2, _object * * kws=0x0000000003a38118, int kwcount=0, _object * * defs=0x0000000000000000, int defcount=0, _object * closure=0x0000000000000000) Line 3594  C

It seems to be a “HEAP CORRUPTION DETECTED” error.

Here is what is happening in detail:

npy_free_cache(fa->data, PyArray_NBYTES(self)); being called for array with refcount == 0, and nbytes == 0. Inside _npy_free_cache, nelem == 0, cache[nelem].available == 7 == NCACHE, so it arrives at dealloc(p), which is trying to free a probably already freed pointer.

Here is fa:

0x00000000126fa0c0 {_ob_next=0x0000000000000000 <NULL> _ob_prev=0x0000000000000000 <NULL> ob_refcnt=...}
    _ob_next: 0x0000000000000000 <NULL>
    _ob_prev: 0x0000000000000000 <NULL>
    ob_refcnt: 0
    ob_type: 0x000007fee1b70f60 {multiarray_d.pyd!_typeobject PyArray_Type} {_ob_next=0x000000000254a698 {_ob_next=...} ...}
    data: 0x00000000120034f0 ""
    nd: 1
    dimensions: 0x0000000013ccd250 {0}
    strides: 0x0000000013ccd258 {4}
    base: 0x0000000000000000 <NULL>
    descr: 0x000007fee1b77720 {multiarray_d.pyd!_PyArray_Descr LONG_Descr} {_ob_next=0x0000000000000000 <NULL> ...}
    flags: 1287
    weakreflist: 0x0000000000000000 <NULL>

fa->descr:

fa->descr
0x000007fee1b77720 {multiarray_d.pyd!_PyArray_Descr LONG_Descr} {_ob_next=0x0000000000000000 <NULL> ...}
    _ob_next: 0x0000000000000000 <NULL>
    _ob_prev: 0x0000000000000000 <NULL>
    ob_refcnt: 1043
    ob_type: 0x000007fee1b91fa0 {multiarray_d.pyd!_typeobject PyArrayDescr_Type} {_ob_next=0x00000000025670d8 {_ob_next=...} ...}
    typeobj: 0x000007fee1bf3b40 {multiarray_d.pyd!_typeobject PyLongArrType_Type} {_ob_next=0x0000000002556fa8 {_ob_next=...} ...}
    kind: 105 'i'
    type: 108 'l'
    byteorder: 61 '='
    flags: 0 '\0'
    type_num: 7
    elsize: 4
    alignment: 4
    subarray: 0x0000000000000000 <NULL>
    fields: 0x000000001e3cd2d8 {python27_d.dll!_object _Py_NoneStruct} {_ob_next=0x0000000001e31cf8 {_ob_next=0x0000000001e32040 {...} ...} ...}
    names: 0x0000000000000000 <NULL>
    f: 0x000007fee1b775a0 {multiarray_d.pyd!PyArray_ArrFuncs _PyLong_ArrFuncs} {cast=0x000007fee1b775a0 {0x000007fee195a620 {multiarray_d.pyd!LONG_to_BOOL(void *, void *, __int64, void *, void *)}, ...} ...}
    metadata: 0x0000000000000000 <NULL>
    c_metadata: 0x0000000000000000 <NULL>
    hash: -1299418611

    [0]: 0x00000004
    [1]: 0x00000000
    [2]: 0x00342731
    [3]: 0xfdfdfdfd
    [4]: 0x00000000    <-- fa->data points here
    [5]: 0x00000000
    [6]: 0x0133e83a
    [7]: 0x88000047
    [8]: 0x04ca6990
    [9]: 0x00000000
    [10]: 0x132c2db0
    [11]: 0x00000000
    [12]: 0x00000000
    [13]: 0x00000000
    [14]: 0x00000000
    [15]: 0x00000001
    [16]: 0x00000004
    [17]: 0x00000000
    [18]: 0x00339900
    [19]: 0xfdfdfdfd
    [20]: 0x00000001
    [21]: 0xfdfdfdfd
    [22]: 0x0133e826

[mattip]

Closing this. If the issue persists, please provide a way to reproduce with a more modern build system / python.

[tadeu]

We don’t use pydebug versions anymore, it’s painful because all packages must be built with pydebug, and there’s no standard python distribution with this feature (we used to build them “in-house”, but dropped this in favor of using conda and conda-forge before moving to Python 3).