Possibly wrong link order / devtoolset-8 libstdc++ unknown symbols

You are probably not going to like this one that much.

It seems that bazel + binutils + gcc + redhats devtoolset-8 gets angry regarding -lstdc++, this appears to be related to https://bugzilla.redhat.com/show_bug.cgi?id=1570853

Recording of the bug

The following is a (long) recording of the bug, and workaround, the following documents it outright

Bazel issues with devtoolset-8 regarding libstdc++

It appears that bazel is adding -lstdc++ into the linker arguments early in the command line for gcc. This freaks out oddball environments (such as the one provided by the software collections library).

SCL from RedHat provides a linker script in place of libstdc++ which does some trickery to fake out having an old libstdc++ installed on the system, but needing newer symbols for newer C++ features, versions and ABIs. This is seemingly done to allow for cross ABI compatibility

Reproducing

I have tested this in the environment we must compile for at $DAYJOB, I have not tested using stock RHEL or Centos but I suspect it might be an issue. It could also be a more general problem with link ordering.

Any bazel build using C++ will break including bootstrapping bazel itself

Setup a test env with docker

The following should give a test env that can reproduce the issue

docker run -it oraclelinux:slim-6

# Inside this env
yum install -y yum-security
yum updateinfo -y --verbose

yum-config-manager --enable -y ol6_latest
yum-config-manager --enable -y ol6_software_collections
yum updateinfo -y --verbose
yum install -y scl-utils

# Install some basics
yum install which curl wget unzip zip bzip2 patch xz

# Grab a compiler
yum install -y devtoolset-8-gcc devtoolset-8-gcc-c++ devtoolset-8-binutils devtoolset-8-toolchain

# Grab a python
yum install -y rh-python36-python

# Grab a jvm
rpm --import http://repos.azulsystems.com/RPM-GPG-KEY-azulsystems
curl -o /etc/yum.repos.d/zulu.repo http://repos.azulsystems.com/rhel/zulu.repo
yum install -y zulu-8

# Setup a environment faker to force the use of devtoolkit-8 as a compiler
cat > /opt/scl_enable <<EOF
#!/bin/bash
unset BASH_ENV PROMPT_COMMAND ENV
source scl_source enable devtoolset-8
source scl_source enable python27
EOF

chmod 755 /opt/scl_enable

# Grab bazel
mkdir /root/bazel_bootstrap
cd /root/bazel_bootstrap
wget https://github.com/bazelbuild/bazel/releases/download/1.2.1/bazel-1.2.1-dist.zip
unzip bazel-1.2.1-dist.zip

# The magical env is to force into SCL / devtoolkit
# It can be sourced, but that has historically proven to be unrealiable in our CI env
BASH_ENV=/opt/scl_enable ENV=/opt/scl_enable PROMPT_COMMAND=". /opt/scl_enable" ./compile.sh

… After a while bazel will fail to compile with some C++ linker error, normally complaining about some undefined symbol. The undef symbol will be something deep in libstdc++

What I think causes this

Looking at a failed link, we find that bazel produces linker args something like so

-o bazel-out/host/bin/external/com_google_protobuf/protoc
-Wl,-S
-fuse-ld=gold
-Wl,-no-as-needed 
-Wl,-z,relro,-z,now
-B/opt/rh/devtoolset-8/root/usr/bin
-pass-exit-codes
-lstdc++
-lm
-Wl,--gc-sections
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc/main.o                                                                     -Wl,--start-lib
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/code_generator.o                                                       
SNIP SNIP %<  ...
-Wl,--end-lib
-lpthread
-lm
-lpthread
-lm
-lpthread
-lm

Looking at the file libstdc++.so under devtoolkit-8 it looks like so

/* GNU ld script Use the shared library, but some functions are only in 
the static library, so try that secondarily.  */                       

OUTPUT_FORMAT(elf64-x86-64)
INPUT ( /usr/lib64/libstdc++.so.6 -lstdc++_nonshared )

Which is trickery to, as mentioned above, use ABI hacks, this confuses ld and I think removes the special handling for libstdc++

Moving libstdc++ to the bottom like so

-o bazel-out/host/bin/external/com_google_protobuf/protoc                                                                                  -Wl,-S
-fuse-ld=gold
-Wl,-no-as-needed
-Wl,-z,relro,-z,now
-B/opt/rh/devtoolset-8/root/usr/bin
-pass-exit-codes
-Wl,--gc-sections
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc/main.o
-Wl,--start-lib
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/code_generator.o
SNIP SNIP %<  ...
-Wl,--end-lib
-lpthread
-lm
-lpthread
-lm
-lpthread
-lm
-lstdc++
-lm

Allows for linking

Workaround

A workaround we have found is to export the following

BAZEL_LINKOPTS=-static-libstdc++ BAZEL_LINKLIBS=-l%:libstdc++.a

… technically this statically links libstdc++ and is probably not what some people want

[nukes]

solve this problem with: BAZEL_LINKOPTS=“” BAZEL_LINKLIBS=“-lstdc++” bazel build …

[stellaraccident]

Another egregious hack memorialized: https://github.com/stellaraccident/manylinux-bazel