Shapely can't find GEOS libraries

Expected behavior and actual behavior.

Expected behavior:

$ python -c 'import cartopy'

Actual behavior:

$ python -X faulthandler -c 'import cartopy'
Fatal Python error: Segmentation fault

Current thread 0x0000000112a59dc0 (most recent call first):
  File "/Users/Adam/.spack/darwin/.spack-env/view/lib/python3.7/site-packages/shapely/geometry/polygon.py", line 67 in _set_coords
  File "/Users/Adam/.spack/darwin/.spack-env/view/lib/python3.7/site-packages/shapely/geometry/polygon.py", line 52 in __init__
  File "/Users/Adam/.spack/darwin/.spack-env/view/lib/python3.7/site-packages/cartopy/crs.py", line 2375 in __init__
  File "/Users/Adam/.spack/darwin/.spack-env/view/lib/python3.7/site-packages/cartopy/crs.py", line 2403 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 728 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "/Users/Adam/.spack/darwin/.spack-env/view/lib/python3.7/site-packages/cartopy/__init__.py", line 107 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 728 in exec_module
  File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "<string>", line 1 in <module>
Segmentation fault: 11

It seems to segfault when Shapely calls some GEOS methods.

Steps to reproduce the problem.

import cartopy

Operating system

macOS 10.15.4

Shapely version and provenance

Shapely 1.7.0 and master, installed from source using the Spack package manager. I realize that you don’t want to debug a possible Spack issue, but I’m a Spack developer so I’m happy to help on that front. I’ve been debugging this myself for the last several days but am running out of ideas.

As far as I can tell, I don’t see anything wrong with the installation. Both cartopy and shapely are linked to the same GEOS library:

$ otool -L ./lib/python3.7/site-packages/shapely/speedups/_speedups.cpython-37m-darwin.so
./lib/python3.7/site-packages/shapely/speedups/_speedups.cpython-37m-darwin.so:
	/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.3-apple/geos-3.8.1-o7p27rfm2alnfpw6d44vagebr7ay5pdw/lib/libgeos_c.1.dylib (compatibility version 15.0.0, current version 15.3.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
$ otool -L ./lib/python3.7/site-packages/shapely/vectorized/_vectorized.cpython-37m-darwin.so
./lib/python3.7/site-packages/shapely/vectorized/_vectorized.cpython-37m-darwin.so:
	/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.3-apple/geos-3.8.1-o7p27rfm2alnfpw6d44vagebr7ay5pdw/lib/libgeos_c.1.dylib (compatibility version 15.0.0, current version 15.3.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
$ otool -L ./lib/python3.7/site-packages/cartopy/trace.cpython-37m-darwin.so
./lib/python3.7/site-packages/cartopy/trace.cpython-37m-darwin.so:
	/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.3-apple/proj-6.3.1-kfr7fwzmklwhqbnq7u5cxxgsti5wuhol/lib/libproj.15.dylib (compatibility version 19.0.0, current version 19.1.0)
	/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.3-apple/geos-3.8.1-o7p27rfm2alnfpw6d44vagebr7ay5pdw/lib/libgeos_c.1.dylib (compatibility version 15.0.0, current version 15.3.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

Note that the GEOS installation doesn’t seem to have any obvious bugs. It passed make check. It shouldn’t have trouble finding the library since the library has been RPATHed, but I have noticed this bug being transient when I mess with DYLD_LIBRARY_PATH.

[sgillies]

I’m sorry you got hung up on the need for DYLD_LIBRARY_PATH. We have an example of using it at https://github.com/Toblerity/Shapely/blob/master/.travis.yml#L100, but nowhere else.

In a future version of Shapely we will no longer use ctypes to dlopen the GEOS library. This is the main problem, and the one that rpath can’t touch.

I’m not inclined to adopt numpy’s site.cfg for shapely 1.7 or 1.8. It won’t be necessary with 2.0.

I’m going to close this issue. Modifying the library path is the solution.

[adamjstewart]

So it turns out that this issue is not as simple as I thought. macOS has this System Integrity Protection (SIP) feature which disables the use of DYLD_*LIBRARY_PATH when running system executables:

Any dynamic linker (dyld) environment variables, such as DYLD_LIBRARY_PATH, are purged when launching protected processes.

If I install my own Python (not in a system location), I can run:

$ python -c 'import cartopy'

but I can’t run:

$ /usr/bin/env python -c 'import cartopy'

This becomes problematic as the shebang in my code uses #!/usr/bin/env python. So I can run python foo.py but not ./foo.py.