GEOS aborting with "Assertion failed" on OS X, discussion and work around

I’ve spent a bit of free time looking more deeply into the reports of GEOS aborting in the AbstractSTRtree.cpp module (either in query() or itemsTree()) on OS X when fiona is imported before shapely in a Python script.

Here are the conditions for a program that will unexpectedly abort:

1. Any version of Python on OS X.
2. Fiona and Shapely binary wheels for macosx downloaded from PyPI. These wheels include libgeos_c.dylib and libgeos-3.6.2.dylib files.
3. A script that imports fiona before shapely and which triggers the creation of a GEOS STR R-tree. Calling ops.unary_union with a number of lines or polygons will suffice.

Here’s an example program:

import fiona

from shapely.geometry import Point
from shapely.geos import geos_version
from shapely.ops import unary_union

print(geos_version)

SHAPES = [Point(i, i).buffer(1.5) for i in range(20)]
for i, s in enumerate(SHAPES):
    print(i, s.is_valid)
    print(i, s.wkt)
    print(i, s.area, s.length)
    print(i, s.intersects(Point(10, 10).buffer(8.0)))

union = unary_union(SHAPES)
print(union)

I’ve run the script under lldb with a modified version of GEOS 3.6.2 that prints some debugging output before calling assert().

...
19 7.057234103728362 9.420993470864257
19 False
Assertion failed: (!static_cast<bool>("should never be reached")), function itemsTree, file AbstractSTRtree.cpp, line 379.
dynamic_cast<AbstractNode*>: 0x0dynamic_cast<ItemBoundable*>: 0x0childBoundable: 0x100498510Process 17968 stopped
* thread #1: tid = 0x606ba9, 0x00007fff8d415f06 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00007fff8d415f06 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff8d415f06 <+10>: jae    0x7fff8d415f10            ; <+20>
    0x7fff8d415f08 <+12>: movq   %rax, %rdi
    0x7fff8d415f0b <+15>: jmp    0x7fff8d4107cd            ; cerror_nocancel
    0x7fff8d415f10 <+20>: retq

I’ve failed to get access to variables in the frame where assert() is called. I am frankly over my head in trying to debug a dlopen’ed C++ library.

Changing the intersects() call in the script to intersection() leads to another failed assertion and an aborted program the first time there is a non-empty intersection.

...
4 7.057234103728363 9.42099347086426
Assertion failed: (0), function query, file AbstractSTRtree.cpp, line 288.
Process 18002 stopped
* thread #1: tid = 0x6074fe, 0x00007fff8d415f06 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00007fff8d415f06 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff8d415f06 <+10>: jae    0x7fff8d415f10            ; <+20>
    0x7fff8d415f08 <+12>: movq   %rax, %rdi
    0x7fff8d415f0b <+15>: jmp    0x7fff8d4107cd            ; cerror_nocancel
    0x7fff8d415f10 <+20>: retq

The problem in query() appears to be the same as in itemsTree(): the dynamic casts return null pointers and the library fails by calling assert(0).

https://github.com/OSGeo/geos/blob/master/src/index/strtree/AbstractSTRtree.cpp#L277-L287

If fiona is imported after shapely, the programs (either with intersects() or intersection()) run without problems.

My hypothesis is that there is a race condition in initialization of the libgeos-3.6.2 library that leaves some objects in the AbstractSTRtree.cpp module in a partly or improperly initialized state when we dlopen the GEOS lib after it has already been loaded in the process via import of fiona. Use of GEOS as both a dependent library of fiona modules and as a library we dynamically load during import of shapely is a little weird, I grant, but doesn’t necessarily have to trigger crashing bugs. There’s no problem on Linux. I see some hints that the difference between ELF and Mach-O might be important.

Workaround 1

Install the GEOS library to one standard location on your macosx system using homebrew, fink, macports, or from source, do the same for GDAL, and then install fiona and shapely like so:

pip install --no-binary shapely,fiona shapely fiona

You can import fiona and shapely in any order if you do this.

Workaround 2

On OS X, if you’re using fiona and shapely wheels together, import shapely first. It’s annoying to have to do this, I know. Abstracting away the C/C++ details of GEOS and GDAL has been one of my main design principles for fiona and shapely, and I’m falling short.

Solution

I think there’s a race condition bug to be fixed in the GEOS library. You can see some indication of that at https://lists.osgeo.org/pipermail/geos-devel/2017-November/008157.html. Because this doesn’t affect Linux and PostGIS, it’s a very low-flying bug.

Fixing it will require more knowledge and experience with C++ and dynamic library building and loading on Mac OS X than I currently have, and so the fix isn’t going to be coming soon unless we can get some help.

I’m somewhat content to at least have a master ticket that I can point users at. It’s a start.

/cc @snorfalorpagus @perrygeo

[sgillies]

Update: in my experience, shapely 1.7a2 (released yesterday) solves this on OS X. The example program runs with no errors and no crashes. If you’d like to confirm, try pip install -U --pre shapely and let me know how it goes.

[arigesher]

Just to add to this thread, I’m seeing the same behavior in a virtualenv that meets the above descriptions.

Re-installing from source for Fiona and Shapely did not fix the problem.

I did some digging in my virtualenv and found the geos binary installed by rasterio:

.../lib/python3.5/site-packages/rasterio/.dylibs/libgeos-3.6.2.dylib
.../lib/python3.5/site-packages/rasterio/.dylibs/libgeos_c.1.dylib

This appears to have fixed it:

pip uninstall -y rasterio && pip install rasterio==1.0.11 --no-binary rasterio