Using shared lib Conan packages is broken on OSX

I’m not sure what version of OSX was in mind when the documentation was written, but on my OSX machine running El Capitan, the Getting Started tutorial does not work at the point of linking to the shared library versions of Poco and OpenSSL.

This is because the shared library install name, and RPATH approach talked about in the documentation will not generally work. The only way the example will work as currently set up, is if you execute it from the bin/ folder. This doesn’t work, for example:

$ ./bin/timer
dyld: Library not loaded: libPocoUtil.43.dylib
  Referenced from: /Users/johughes/conantest/build/./bin/timer
  Reason: image not found

Shared libraries on OSX have an “install name” that must match what is referenced in the binary.

Looking at libPocoCrypto:

otool -L bin/libPocoCrypto.43.dylib
bin/libPocoCrypto.43.dylib:
        libPocoCrypto.43.dylib (compatibility version 43.0.0, current version 43.0.0)
...

The install name is “libPocoCrypto.43.dylib” which means that the dynamic linker will ONLY ever be able to find it in the PWD of the parent process, regardless of where the actual binary is located.

[jondo2010]

@memsharded Yes, the dylibs have been copied to the local bin folder using [import] in the conanfile. Running cd bin && ./timer does work.

It’s simply that this setup is super less than ideal, to the point of being not useful. I have to cd to the bin folder to run my binary every time?

With regards to the docs, @lasote, at the bottom of http://docs.conan.io/en/latest/manage_deps/conanfile_txt.html where it talks about rpaths:

So, for OSX, conan requires dylibs to be built having an rpath with only the name of the required library (just the name, without path).

I would argue that a more robust approach is as follows:

1. When shared libs on OSX are installed in .conan, the install names are modified to the absolute location.
2. When shared libs on OSX are copied with [import], the install names are modified to be @rpath/libname.dylib.

This would give the consumer much better flexibility in how their binaries link to the shared libraries. For case 1, they would add the absolute path to the folder containing the library to the binaries’ RPATH (e.g. ~/.conan/data/Boost/..). For case 2, where the .dylib sits in the local bin folder, simply adding an RPATH of @executable_path/. will work. The bin folder is completely relocatable in this case.

The install_namess can be modified using https://pypi.python.org/pypi/macholib

As a further example, let’s take a look at how a homebrew-installed lib does it:

$ otool -L /usr/local/lib/libboost_log.dylib
/usr/local/lib/libboost_log.dylib:
        /usr/local/opt/boost/lib/libboost_log.dylib (compatibility version 0.0.0, current version 0.0.0)
        @loader_path/libboost_date_time.dylib (compatibility version 0.0.0, current version 0.0.0)
        @loader_path/libboost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
        @loader_path/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
        @loader_path/libboost_regex.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

libboost_log.dylib is given an install_name of the absolute path where it is located, and all other dependent libs from the same package are linked relative using @loader_path.

[jcar87]

I was trying to come up with a better explanation of how OSX works and turns out Kitware already did it for me some time ago: https://blog.kitware.com/upcoming-in-cmake-2-8-12-osx-rpath-support/

Also, one of the comments there summarizes the situation quite well. I’m quoting:

For install names, we have these options:

1. full path/filename
2. filename only
3. @rpath/filename
4. @loader_path/filename
5. @executable_path/filename

Option 1 is really only used by OSX system libraries, where they are guaranteed to be found at a certain location. Option 2 is used by any library where you expect it to be installed in one of the following directories: $(HOME)/lib, /usr/local/lib, /lib, /usr/lib (this is the default fallback library path, unless overridden by the environment variable DYLD_LIBRARY_PATH) Option 3 is what’s in style these days, and cmake usually handles these cases quite well, I think by default since a certain version.

Options 3 and 4 used to be common place before apple introduced the “@rpath” prefix in 10.5.

As for Conan, we have to take into account a few things.

• Currently the Conan documentation states that all dylib files are to be placed alongside the executable. But there is nothing in the documentation of dyld to suggest that this will actually work. However, the manpages for dlopen do suggest that the current working directory is in the search path and that’s maybe what is happening. Note that because of this limitation, any application linked with conan-delivered libraries with naked install names will only run if all files (executable and libraries) are in the same folder, and the libraries are in the working directory, or if those libraries are moved to any of the directories in the fall-back search path (or if DYLD_LIBRARY_PATH is set correctly, but this is not very mac friendly and can actually cause problems loading libraries-that-depend-on-other-libraries).

• Another very potential issue of using naked install names is that if I have download a package with, let’s say, “libz.dylib” from conan, and my executables are linked against it, unless I explicitly guarantee that the conan one can be found by the dynamic loader search path, it can very well be loading /usr/lib/libz.dylib from the fallback path. That file does exist and is installed by Apple, and it is potentially a different version that the conan one. It can get to the point where the application runs, loads the wrong library, and crashes inexplicably, and these scenarios are really hard to diagnose (unlike ldd on Linux, otool -L doesn’t tell you where it will be loading libraries from).

• CMake produces binaries with different install names / rpaths in two different scenarios: “make” and “make install”. This is what cmake calls the “build tree” and the “install tree”. The build tree is usually “polluted” with full, hardcoded paths to all library dependencies, to guarantee that developers will always be able to run their stuff. Build tree is not meant nor guaranteed to be relocatable. The install tree is cleaner, both on Linux (no paths to the local build machines), and on OSX. I’ve seen many conan recipes that are built using CMake but do not use the install tree (they don’t build the install target), and do not override CMAKE_SKIP_RPATH or CMAKE_BUILD_WITH_INSTALL_RPATH (the recipes that patch the root CMakeLists to call conan_basic_setup would be safe). It is possible that libraries deployed that way still keep references to hardcoded full paths of the local build machines. On Linux this isn’t a big issue, but on Mac this is bad. Even if the install name was fine (@rpath/libName.dylib), if there is any binary with a RPATH that points to a fixed location that eventually gets included in a signed OSX App bundle, it will fail the gatekeeper validation. And the only way to diagnose that problem is by reading the system log on the machine where it fails.

• The ideal way, which is option 2, is to prepend @rpath to the install names of libraries, but you will also need the executables to set their RPATH to the paths where those libraries will be found. So this places the ‘problem’ at the executable level. In CMake you can build like this: CMAKE_MACOSX_RPATH = ON (to prepend @rpath to the install name) CMAKE_INSTALL_RPATH = “/path/to/libs” The second will add that directory to the executables as well. An absolute path wouldn’t make sense for relocatable libraries, but a relative path would, e.g. @executable_path/../lib. Which should take care of it.

However, some libraries that use CMake already do this a different way (e.g. OpenCV, their install names on mac are /lib/opencv_module.dylib, which makes them impossible to use… !