free(): invalid next size (normal)

Environment

$ python --version
Python 3.6.8

$ pip list
Package         Version
--------------- -------
cycler          0.10.0 
decorator       4.4.1  
imageio         2.6.1  
kiwisolver      1.1.0  
matplotlib      3.1.2  
networkx        2.4    
numpy           1.18.1 
Pillow          7.0.0  
pip             18.1   
pyparsing       2.4.6  
python-dateutil 2.8.1  
PyWavelets      1.1.1  
scikit-image    0.16.2 
scipy           1.4.1  
setuptools      40.6.2 
six             1.14.0 

$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.6.1810 (Core) 
Release:	7.6.1810
Codename:	Core

Steps to reproduce

$ gdb python3
(gdb) run -c "import numpy as np; from scipy.ndimage.morphology import binary_dilation; from skimage.morphology import disk; inds = np.array(np.random.rand(45, 25) > 0.2); sm = 32; struct = disk(sm, dtype=bool); zvals2 = ~binary_dilation(inds,struct, iterations=2)"
Starting program: /disk/home/ecarter/py3/bin/python3 -c "import numpy as np; from scipy.ndimage.morphology import binary_dilation; from skimage.morphology import disk; inds = np.array(np.random.rand(45, 25) > 0.2); sm = 32; struct = disk(sm, dtype=bool); zvals2 = ~binary_dilation(inds,struct, iterations=2)"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffedd1a700 (LWP 11983)]
*** Error in `/disk/home/ecarter/py3/bin/python3': free(): invalid next size (normal): 0x0000000000a6fc70 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81489)[0x7ffff6c41489]
/disk/home/ecarter/py3/lib64/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so(+0x2a82c)[0x7fffefdd082c]
/disk/home/ecarter/py3/lib64/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so(+0x2df97)[0x7fffefdd3f97]
/lib64/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x7604)[0x7ffff7a02544]
/lib64/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x22a)[0x7ffff7a070da]
/lib64/libpython3.6m.so.1.0(PyEval_EvalCode+0x3b)[0x7ffff7a07d7b]
/lib64/libpython3.6m.so.1.0(+0x1dcf0e)[0x7ffff7a8ef0e]
/lib64/libpython3.6m.so.1.0(PyRun_StringFlags+0x72)[0x7ffff7a8f592]
/lib64/libpython3.6m.so.1.0(PyRun_SimpleStringFlags+0x3c)[0x7ffff7939b27]
/lib64/libpython3.6m.so.1.0(Py_Main+0x4ca)[0x7ffff7a951da]
/disk/home/ecarter/py3/bin/python3(main+0x119)[0x400a99]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6be23d5]
/disk/home/ecarter/py3/bin/python3[0x400c20]
======= Memory map: ========
00400000-00401000 r-xp 00000000 fd:00 692370                             /usr/bin/python3.6
00601000-00602000 r--p 00001000 fd:00 692370                             /usr/bin/python3.6
00602000-00603000 rw-p 00002000 fd:00 692370                             /usr/bin/python3.6
00603000-02de4000 rw-p 00000000 00:00 0                                  [heap]
7fffcc000000-7fffcc021000 rw-p 00000000 00:00 0 
7fffcc021000-7fffd0000000 ---p 00000000 00:00 0 
7fffd24d8000-7fffd258d000 r-xp 00000000 00:35 12267184                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_max_tree.cpython-36m-x86_64-linux-gnu.so
7fffd258d000-7fffd278d000 ---p 000b5000 00:35 12267184                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_max_tree.cpython-36m-x86_64-linux-gnu.so
7fffd278d000-7fffd2794000 rw-p 000b5000 00:35 12267184                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_max_tree.cpython-36m-x86_64-linux-gnu.so
7fffd2794000-7fffd2796000 rw-p 00000000 00:00 0 
7fffd2796000-7fffd2799000 rw-p 000bc000 00:35 12267184                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_max_tree.cpython-36m-x86_64-linux-gnu.so
7fffd2799000-7fffd27e1000 r-xp 00000000 00:35 12267235                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_flood_fill_cy.cpython-36m-x86_64-linux-gnu.so
7fffd27e1000-7fffd29e0000 ---p 00048000 00:35 12267235                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_flood_fill_cy.cpython-36m-x86_64-linux-gnu.so
7fffd29e0000-7fffd29ea000 rw-p 00047000 00:35 12267235                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_flood_fill_cy.cpython-36m-x86_64-linux-gnu.so
7fffd29ea000-7fffd2a11000 r-xp 00000000 00:35 12267181                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_convex_hull.cpython-36m-x86_64-linux-gnu.so
7fffd2a11000-7fffd2c10000 ---p 00027000 00:35 12267181                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_convex_hull.cpython-36m-x86_64-linux-gnu.so
7fffd2c10000-7fffd2c13000 rw-p 00026000 00:35 12267181                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_convex_hull.cpython-36m-x86_64-linux-gnu.so
7fffd2c13000-7fffd2c14000 rw-p 00000000 00:00 0 
7fffd2c14000-7fffd2c17000 rw-p 0002a000 00:35 12267181                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_convex_hull.cpython-36m-x86_64-linux-gnu.so
7fffd2c17000-7fffd2c57000 rw-p 00000000 00:00 0 
7fffd2c57000-7fffd2c82000 r-xp 00000000 00:35 12267211                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_skeletonize_3d_cy.cpython-36m-x86_64-linux-gnu.so
7fffd2c82000-7fffd2e82000 ---p 0002b000 00:35 12267211                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_skeletonize_3d_cy.cpython-36m-x86_64-linux-gnu.so
7fffd2e82000-7fffd2e85000 rw-p 0002b000 00:35 12267211                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_skeletonize_3d_cy.cpython-36m-x86_64-linux-gnu.so
7fffd2e85000-7fffd2e86000 rw-p 00000000 00:00 0 
7fffd2e86000-7fffd2e8a000 rw-p 0002f000 00:35 12267211                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_skeletonize_3d_cy.cpython-36m-x86_64-linux-gnu.so
7fffd2e8a000-7fffd2eb6000 r-xp 00000000 00:35 12267209                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_skeletonize_cy.cpython-36m-x86_64-linux-gnu.so
7fffd2eb6000-7fffd30b5000 ---p 0002c000 00:35 12267209                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_skeletonize_cy.cpython-36m-x86_64-linux-gnu.so
7fffd30b5000-7fffd30b9000 rw-p 0002b000 00:35 12267209                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_skeletonize_cy.cpython-36m-x86_64-linux-gnu.so
7fffd30b9000-7fffd30ba000 rw-p 00000000 00:00 0 
7fffd30ba000-7fffd30bd000 rw-p 00030000 00:35 12267209                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_skeletonize_cy.cpython-36m-x86_64-linux-gnu.so
7fffd30bd000-7fffd30fa000 r-xp 00000000 00:35 12267199                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_extrema_cy.cpython-36m-x86_64-linux-gnu.so
7fffd30fa000-7fffd32fa000 ---p 0003d000 00:35 12267199                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_extrema_cy.cpython-36m-x86_64-linux-gnu.so
7fffd32fa000-7fffd32fe000 rw-p 0003d000 00:35 12267199                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_extrema_cy.cpython-36m-x86_64-linux-gnu.so
7fffd32fe000-7fffd32ff000 rw-p 00000000 00:00 0 
7fffd32ff000-7fffd3302000 rw-p 00042000 00:35 12267199                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/morphology/_extrema_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3302000-7fffd334e000 r-xp 00000000 00:35 12268741                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/bilateral_cy.cpython-36m-x86_64-linux-gnu.so
7fffd334e000-7fffd354e000 ---p 0004c000 00:35 12268741                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/bilateral_cy.cpython-36m-x86_64-linux-gnu.so
7fffd354e000-7fffd3552000 rw-p 0004c000 00:35 12268741                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/bilateral_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3552000-7fffd3553000 rw-p 00000000 00:00 0 
7fffd3553000-7fffd3556000 rw-p 00051000 00:35 12268741                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/bilateral_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3556000-7fffd35e1000 r-xp 00000000 00:35 12268737                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/percentile_cy.cpython-36m-x86_64-linux-gnu.so
7fffd35e1000-7fffd37e0000 ---p 0008b000 00:35 12268737                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/percentile_cy.cpython-36m-x86_64-linux-gnu.so
7fffd37e0000-7fffd37e6000 rw-p 0008a000 00:35 12268737                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/percentile_cy.cpython-36m-x86_64-linux-gnu.so
7fffd37e6000-7fffd37e7000 rw-p 00000000 00:00 0 
7fffd37e7000-7fffd37eb000 rw-p 00090000 00:35 12268737                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/percentile_cy.cpython-36m-x86_64-linux-gnu.so
7fffd37eb000-7fffd3837000 r-xp 00000000 00:35 12268735                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/core_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3837000-7fffd3a37000 ---p 0004c000 00:35 12268735                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/core_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3a37000-7fffd3a3a000 rw-p 0004c000 00:35 12268735                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/core_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3a3a000-7fffd3a3b000 rw-p 00000000 00:00 0 
7fffd3a3b000-7fffd3a3e000 rw-p 0004f000 00:35 12268735                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/core_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3a3e000-7fffd3b37000 r-xp 00000000 00:35 12268734                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/generic_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3b37000-7fffd3d37000 ---p 000f9000 00:35 12268734                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/generic_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3d37000-7fffd3d3e000 rw-p 000f9000 00:35 12268734                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/generic_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3d3e000-7fffd3d3f000 rw-p 00000000 00:00 0 
7fffd3d3f000-7fffd3d43000 rw-p 00101000 00:35 12268734                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/rank/generic_cy.cpython-36m-x86_64-linux-gnu.so
7fffd3d43000-7fffd3d6c000 r-xp 00000000 00:35 12268708                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/_multiotsu.cpython-36m-x86_64-linux-gnu.so
7fffd3d6c000-7fffd3f6b000 ---p 00029000 00:35 12268708                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/_multiotsu.cpython-36m-x86_64-linux-gnu.so
7fffd3f6b000-7fffd3f6e000 rw-p 00028000 00:35 12268708                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/_multiotsu.cpython-36m-x86_64-linux-gnu.so
7fffd3f6e000-7fffd3f6f000 rw-p 00000000 00:00 0 
7fffd3f6f000-7fffd3f72000 rw-p 0002c000 00:35 12268708                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/filters/_multiotsu.cpython-36m-x86_64-linux-gnu.so
7fffd3f72000-7fffd3fb2000 rw-p 00000000 00:00 0 
7fffd3fb2000-7fffd3fdd000 r-xp 00000000 00:35 12268517                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_radon_transform.cpython-36m-x86_64-linux-gnu.so
7fffd3fdd000-7fffd41dd000 ---p 0002b000 00:35 12268517                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_radon_transform.cpython-36m-x86_64-linux-gnu.so
7fffd41dd000-7fffd41e0000 rw-p 0002b000 00:35 12268517                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_radon_transform.cpython-36m-x86_64-linux-gnu.so
7fffd41e0000-7fffd41e1000 rw-p 00000000 00:00 0 
7fffd41e1000-7fffd41e4000 rw-p 0002f000 00:35 12268517                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_radon_transform.cpython-36m-x86_64-linux-gnu.so
7fffd41e4000-7fffd421b000 r-xp 00000000 00:35 12268508                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_warps_cy.cpython-36m-x86_64-linux-gnu.so
7fffd421b000-7fffd441a000 ---p 00037000 00:35 12268508                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_warps_cy.cpython-36m-x86_64-linux-gnu.so
7fffd441a000-7fffd441e000 rw-p 00036000 00:35 12268508                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_warps_cy.cpython-36m-x86_64-linux-gnu.so
7fffd441e000-7fffd441f000 rw-p 00000000 00:00 0 
7fffd441f000-7fffd4423000 rw-p 0003b000 00:35 12268508                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_warps_cy.cpython-36m-x86_64-linux-gnu.so
7fffd4423000-7fffd4462000 r-xp 00000000 00:35 12268502                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_hough_transform.cpython-36m-x86_64-linux-gnu.so
7fffd4462000-7fffd4662000 ---p 0003f000 00:35 12268502                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_hough_transform.cpython-36m-x86_64-linux-gnu.so
7fffd4662000-7fffd4668000 rw-p 0003f000 00:35 12268502                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_hough_transform.cpython-36m-x86_64-linux-gnu.so
7fffd4668000-7fffd4669000 rw-p 00000000 00:00 0 
7fffd4669000-7fffd466c000 rw-p 00045000 00:35 12268502                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/transform/_hough_transform.cpython-36m-x86_64-linux-gnu.so
7fffd466c000-7fffd46ac000 rw-p 00000000 00:00 0 
7fffd46ac000-7fffd46e8000 r-xp 00000000 00:35 12266706                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/restoration/_nl_means_denoising.cpython-36m-x86_64-linux-gnu.so
7fffd46e8000-7fffd48e8000 ---p 0003c000 00:35 12266706                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/restoration/_nl_means_denoising.cpython-36m-x86_64-linux-gnu.so
7fffd48e8000-7fffd48ed000 rw-p 0003c000 00:35 12266706                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/restoration/_nl_means_denoising.cpython-36m-x86_64-linux-gnu.so
7fffd48ed000-7fffd48ee000 rw-p 00000000 00:00 0 
7fffd48ee000-7fffd48f1000 rw-p 00041000 00:35 12266706                   /disk/home/ecarter/py3/lib/python3.6/site-packages/skimage/restoration/_nl_means_denoising.cpython-36m-x86_64-linux-gnu.so
7fffd48f1000-7fffd68f1000 rw-p 00000000 00:00 0 
7fffd68f1000-7fffd694e000 r-xp 00000000 00:35 12265124                   /disk/home/ecarter/py3/lib/python3.6/site-packages/pywt/_extensions/_swt.cpython-36m-x86_64-linux-gnu.so
7fffd694e000-7fffd6b4d000 ---p 0005d000 00:35 12265124                   /disk/home/ecarter/py3/lib/python3.6/site-packages/pywt/_extensions/_swt.cpython-36m-x86_64-linux-gnu.so
7fffd6b4d000-7fffd6b52000 rw-p 0005c000 00:35 12265124                   /disk/home/ecarter/py3/lib/python3.6/site-packages/pywt/_extensions/_swt.cpython-36m-x86_64-linux-gnu.so
7fffd6b52000-7fffd6b93000 rw-p 00000000 00:00 0 
7fffd6b93000-7fffd6bd8000 r-xp 00000000 00:35 12265126                   /disk/home/ecarter/py3/lib/python3.6/site-packages/pywt/_extensions/_cwt.cpython-36m-x86_64-linux-gnu.so
7fffd6bd8000-7fffd6dd7000 ---p 00045000 00:35 12265126                   /disk/home/ecarter/py3/lib/python3.6/site-packages/pywt/_extensions/_cwt.cpython-36m-x86_64-linux-gnu.so
7fffd6dd7000-7fffd6ddb000 rw-p 00044000 00:35 12265126                   /disk/home/ecarter/py3/lib/python3.6/site-packages/pywt/_extensions/_cwt.cpython-36m-x86_64-linux-gnu.so
Program received signal SIGABRT, Aborted.
0x00007ffff6bf6207 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x00007ffff6bf6207 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007ffff6bf78f8 in __GI_abort () at abort.c:90
#2  0x00007ffff6c38d27 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff6d4a678 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3  0x00007ffff6c41489 in malloc_printerr (ar_ptr=0x7ffff6f86760 <main_arena>, ptr=<optimized out>, str=0x7ffff6d4a710 "free(): invalid next size (normal)", action=3) at malloc.c:5004
#4  _int_free (av=0x7ffff6f86760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5  0x00007fffefdd082c in PyDataMem_FREE (ptr=0xa6fc70) at numpy/core/src/multiarray/alloc.c:273
#6  _npy_free_cache (dealloc=0x7fffefdd0a90 <PyDataMem_FREE>, cache=0x7ffff03691a0 <datacache>, msz=1024, nelem=<optimized out>, p=0xa6fc70) at numpy/core/src/multiarray/alloc.c:112
#7  npy_free_cache (p=0xa6fc70, sz=<optimized out>) at numpy/core/src/multiarray/alloc.c:147
#8  0x00007fffefdd3f97 in array_dealloc (self=0x7fffd2c26030) at numpy/core/src/multiarray/arrayobject.c:530
#9  0x00007ffff7a02544 in _PyEval_EvalFrameDefault () from /lib64/libpython3.6m.so.1.0
#10 0x00007ffff7a070da in PyEval_EvalCodeEx () from /lib64/libpython3.6m.so.1.0
#11 0x00007ffff7a07d7b in PyEval_EvalCode () from /lib64/libpython3.6m.so.1.0
#12 0x00007ffff7a8ef0e in run_mod () from /lib64/libpython3.6m.so.1.0
#13 0x00007ffff7a8f592 in PyRun_StringFlags () from /lib64/libpython3.6m.so.1.0
#14 0x00007ffff7939b27 in PyRun_SimpleStringFlags () from /lib64/libpython3.6m.so.1.0
#15 0x00007ffff7a951da in Py_Main () from /lib64/libpython3.6m.so.1.0
#16 0x0000000000400a99 in main ()
(gdb) 

Let me know if you need any more information.

[nkaretnikov]

So I’ve been trying to pinpoint this problem for a while. Unfortunately, I ran out of time allocated for this and cannot continue investigating.

However, I have a WIP branch here that detects the out of bounds in the repro and passes all the tests:

https://github.com/nkaretnikov/scipy/commits/wip-check-buffer-sizes3

I rewrote some macros as functions and added buffer/pointer size checks wherever I could. The latter required extending some functions to return buffer sizes.

That said, I’m against merging the above even after a potential clean up. There’s just too many changes for such a widely used project like SciPy. I don’t want us to break anything by accident. It would be better to understand the bug and fix just the bug itself. Having all those checks in numeric code that might be called quite a bit seems like a bad idea, for perf reasons, but also just for readability. We might also reject potentially valid inputs just by making a mistake in the checking logic, due to insufficient testing.

About the bug:

The bug triggers in CASE_ERODE_POINT2 (all names are from my branch) in NI_BinaryErosion2.

Specifically, this needs to be looked at. The returned value _to is added to an index pointing somewhere in the input array (not always the start of the array). After adding _to to it, it goes past the memory allocated for the input array.

npy_intp _to = _offsets[_oo + _hh];
...
if(*(_type *)(_pi + _to) == (_type)_true) {

(You can inline the macro to check.)

The functions that influence this are these, especially the last one where offsets is initialized:

NI_IteratorGoto
NI_FilterGoto
NI_InitFilterOffsets

I ran out of time before I could fully understand the offsets layout. It generates an array of size:

struct dims * struct footprint (set elements) * array elem size

E.g., for a 3x3 cross struct:

0 1 0
1 1 1
0 1 0

This would be:

3 * 3 * 5 * sizeof(npy_intp)

And then it stores offsets there, which is then indexed in NI_FilterGoto based on the filter (struct) array strides and input array coordinates:

*ptrf += itf->strides[ii] * jj;

I suspect this relationship between the struct, offsets, and the input array might be the problem. Bounds checks in the if-then-else there look suspicious too, but seem to work(?).

ptrf here is _oo in _offsets above.

IIUC, NI_BinaryErosion2 is an optimization where the code keeps track of the regions in the output that have True values, which are returned by NI_BinaryErosion (the first call, with iterations=1). I think the idea here is to only consider the coordinates that can affect the end result without iterating thru the entire input array.

So, I guess, a potential workaround could be just calling NI_BinaryErosion multiple times, making sure that input/output arrays don’t modify each other in the process, but it would impact performance. I’ve tried doing it where _nd_image.binary_erosion2 is called but it broke some tests, so perhaps it needs to be done even earlier. Maybe I made a mistake somewhere else. I tried exploring this for a bit and then ran out of time.

The repro seems to trigger an edge case somewhere in the offsets calculation, I assume. And it looks like the bug triggers only when the structure’s dimensions are larger than the input and the data has a certain pattern (since there’s a check on the Python side that looks at whether the center of the struct is 0 or 1 (cit = _center_is_true(structure, origin)), but there might be more code affected by this in the C functions I mention). For instance, it doesn’t trigger with all zeros and ones, so the placement of boolean values is important. There might be also some assumptions about the struct layout, e.g., I wonder if the code expects the struct to be similar to the ones generated by generate_binary_structure.

I also wonder if an easy fix would be just disallowing structs with larger dims than input. All the examples showcasing binary morphology that I’ve seen use a struct smaller than the input, so I’m not sure whether it’s even used by anyone in practice (in the repro above, the struct array is larger because the parameter to disk is the radius). But I might be wrong since I’m not working with morphology code myself. The original code attempts to support this, I think. There are checks comparing input array shape and struct shape. But as shown above, it doesn’t always work. Now that I wrote this, I wonder if we even have struct > input anywhere in the testsuite besides the repro. Perhaps that’s the bug? Maybe the checks that I’m referring to above are just to calculate the difference somewhere (e.g., when the struct partially overlays the input and goes OOB?) but the code always assumes that the input is larger(?).

For anyone interested in looking at this, I recommend starting by tracing the code using this test case, which doesn’t crash: test_binary_erosion27 (erosion and dilation are implemented by the same code). It uses multiple iterations and is relatively easy to understand because the inputs are small: you can validate that the operation works correctly on a sheet of paper too. Then you can move on to the repros above.

[rgommers]

This doesn’t crash for me on an M1 Mac, but it still does on 64-bit Arch Linux. @v0dro any Linux distro should work here to reproduce the issue I think, no need for CentOS specifically.