Removal of libstdc++ reminants from the stack on Mac

So, there has been a lot of discussion and in some case frustration with segmentation faults caused by a mixture libstdc++ and libc++ being used. Basically, these libraries cannot pass C++ objects between each other. However, passing C objects is totally fine. Over the past few weeks @gillins and myself have been addressing this in the stack. Though it seems there is lots of confusion or simply lack of knowledge about this problem in general. As a result, I’m writing this up to discuss what is happening, why it is happening, what we are doing to fix, and how people can help. To start, let’s get some history about the choices we have made.

Why clang/clang++/libc++? First, on Mac the gcc/g++/libstdc++ went unmaintained some time ago and got converted to a frontend to clang/clang++. Second, the libstdc++ library seems to be missing on newer OS X’s (e.g. 10.11) from my experience, which causes issues using gcc/g++. Third, users often want to use new C++ features like C++11, which is only possible with clang++ and libc++. As a result, we accepted the use of clang/clang++/libc++ as our de facto toolchain.

However, it is hard to get software to pickup clang/clang++ and use libc++ for some reason. Instead things like to pick up the dated gcc and g++. So, we figured out some environment variables that mostly worked and copied them around. The suggestion at that time by @wesm and others was we should package this up. Personally I was feeling overwhelmed just staying on top of notifications related to building on Mac and this sounded like a good way to address that. So, we wrote the toolchain to try to set those automatically. The hope was this should make it easy for people to use and recommend, and it let me return to a lower volume of pings (but still remained quite busy).

Still there were many pieces of code that already entered the ecosystem that were built before we made the decision to use clang/clang++/libc++ and before the creation of the toolchain. Now this is a very serious problem as these pieces of software are using libstdc++. Why is this a problem? Because if C++ objects pass from a library using libc++ to libstdc++ or vice versa one will get segmentation faults. The only exception is exception objects, which can be thrown across the boundary. This causes people all sorts of issues see this thread, these recommendations, and this detail explanation of where this does or doesn’t cause problems. However, we really can’t be digging into code to see if might be ok (unless we know that code well) so we should be switching everything else to clang/clang++/libc++. This is what @gillins and I set out to do with the toolchain in hand.

While the toolchain does work ok in many cases, there is no guarantee that there won’t occasionally be software that needs special hand holding. @gillins and myself have already watched libtool gobble up some flags that we are passing around. We still don’t know why that is the case, but there are plenty of reports of this behavior like this SO question, this upstream patch (note the silence that follows), this Macports issue, this FreeBSD thread, etc. (really this list goes on and on). This problem is not exclusive to our libtool, but affects the system libtool. The workaround we have used is forcing -stdlib=libc++ in CC and CXX. It is not pretty, but it works with libtool. However, the workaround does not work (nor is it needed) with cmake. We could look at patching our libtool. Though it doesn’t seem the patch has been accepted upstream or whether it ever will. That being said, maybe it saves some pain. There may be other special cases I don’t yet know of please feel free to share them here.

cc @gillins @pelson @ocefpaf @msarahan @kmuehlbauer

[jakirkham]

Even if they seem committed, I see no harm in sending the email anyways. Worse thing that happens is they say no and continue with their current plan. Outcomes only improve from there.

[gillins]

Thanks for writing this up @jakirkham .

One other thing to consider before we start patching libtool is to use the config.site mechanism (see https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Site-Defaults.html). I’ve used this on Linux and Solaris with quite a bit of success before. Is this something you have used?

If I have a chance I’ll see if building gdal on OSX works in this way without breaking anything else.