lzf compression filter crashes on macos with h5py3.0/HDF5 1.12.0

Tested on:

• macos 10.12
• python3.7
• HDF5 1.12.0
• h5py installed from azure pipeline wheel and built from current master with HDF51.12.0 built following ci/travis/get_hdf5_if_needed.sh

The following code crashes when calling File.close:

In [1]: import h5py, numpy                                                      
In [2]: f = h5py.File("test.h5", "w")                                           
In [3]: f.create_dataset("data", data=numpy.arange(100), compression='lzf')     
Out[3]: <HDF5 dataset "data": shape (100,), type "<i8">
In [4]: f.close()                                                               
Python(72206,0x7fffe44623c0) malloc: *** error for object 0x7f9f808d77f0: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

We have similar issue with hdf5plugin compression filters (see https://github.com/silx-kit/hdf5plugin/issues/92).

Crash information:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	0x00007fffdb660d42 __pthread_kill + 10
1   libsystem_pthread.dylib       	0x00007fffdb74e457 pthread_kill + 90
2   libsystem_c.dylib             	0x00007fffdb5c6420 abort + 129
3   libsystem_malloc.dylib        	0x00007fffdb6b607f free + 530
4   h5z.cpython-37m-darwin.so     	0x000000011977620f lzf_filter + 303 (lzf_filter.c:253)
5   libhdf5.200.dylib             	0x000000011084f6ea H5Z_pipeline + 2938
6   libhdf5.200.dylib             	0x00000001101e017a H5D__chunk_flush_entry + 2042
7   libhdf5.200.dylib             	0x00000001101d8360 H5D__chunk_flush + 544
8   libhdf5.200.dylib             	0x0000000110229641 H5D__flush_real + 625
9   libhdf5.200.dylib             	0x00000001102282d8 H5D_close + 520
10  libhdf5.200.dylib             	0x000000011082e29e H5VL__native_dataset_close + 398
11  libhdf5.200.dylib             	0x00000001107f4f92 H5VL__dataset_close + 722
12  libhdf5.200.dylib             	0x00000001107f4b73 H5VL_dataset_close + 867
13  libhdf5.200.dylib             	0x0000000110234e24 H5D__close_cb + 484
14  libhdf5.200.dylib             	0x000000011041d52e H5I_dec_ref + 414
15  libhdf5.200.dylib             	0x000000011041d1e6 H5I_dec_app_ref + 182
16  libhdf5.200.dylib             	0x000000011041d01e H5Idec_ref + 766
17  defs.cpython-37m-darwin.so    	0x00000001194d6ef4 __pyx_f_4h5py_4defs_H5Idec_ref + 36 (defs.c:20618)
18  h5i.cpython-37m-darwin.so     	0x00000001198b5bf0 __pyx_pw_4h5py_3h5i_13dec_ref + 32 (h5i.c:3895)
19  _objects.cpython-37m-darwin.so	0x0000000119532e91 __Pyx_PyObject_Call + 97 (_objects.c:14378)
20  _objects.cpython-37m-darwin.so	0x000000011953bd5d __pyx_pw_4h5py_8_objects_9with_phil_1wrapper + 621 (_objects.c:5442)
21  org.python.python             	0x000000010df93871 _PyObject_FastCallKeywords + 433
22  org.python.python             	0x000000010e052474 call_function + 420
23  org.python.python             	0x000000010e04f576 _PyEval_EvalFrameDefault + 25190
24  org.python.python             	0x000000010e0530d6 _PyEval_EvalCodeWithName + 2422
25  org.python.python             	0x000000010df93a21 _PyFunction_FastCallKeywords + 257
26  org.python.python             	0x000000010e0525b2 call_function + 738
[...]

Commenting the free here https://github.com/h5py/h5py/blob/master/lzf/lzf_filter.c#L252 avoids the crash and print the following assert when leaving python:

Assertion failed: (0 == H5MM_curr_alloc_bytes_s), function H5MM_final_sanity_check, file H5MM.c, line 232.
Abort trap: 6

This points to a change of behavior of libHDF5 regarding dataset chunk memory allocation that affects compression filters. I’m also opening a discussion on libhdf5.

[t20100]

I just found out that default build mode of libhdf5 1.12.0 is debug (it’s release for 1.10.6)… which is apparently the root cause of this issue. I’m testing with using production build mode, but that should be it.

[takluyver]

I see that on Mac it crashes when installing from a wheel, but not when building from source, and vice versa on Ubuntu. I suspect the difference is something like whether HDF5 is built in production or debug mode.