Repeating macOS linking errors

Before closing this issue

Disable developer mode on mac-imac3-ky
Revoke dev access to mac-imac3-ky

We keep seeing these errors in public builds on macOS arm64 and x64 for a while. Rebuilds succeed.

FAILED: libVkLayer_khronos_validation.dylib libVkLayer_khronos_validation.dylib.TOC libVkLayer_khronos_validation.dylib.dSYM/Contents/Info.plist libVkLayer_khronos_validation.dylib.dSYM/Contents/Resources/DWARF/libVkLayer_khronos_validation.dylib

FAILED: libGLESv2.dylib libGLESv2.dylib.TOC libGLESv2.dylib.dSYM/Contents/Info.plist libGLESv2.dylib.dSYM/Contents/Resources/DWARF/libGLESv2.dylib

FAILED: character_data_generator character_data_generator.dSYM/Contents/Info.plist character_data_generator.dSYM/Contents/Resources/DWARF/character_data_generator

FAILED: v8_context_snapshot_generator v8_context_snapshot_generator.dSYM/Contents/Info.plist v8_context_snapshot_generator.dSYM/Contents/Resources/DWARF/v8_context_snapshot_generator

FAILED: chrome_crashpad_handler chrome_crashpad_handler.dSYM/Contents/Info.plist chrome_crashpad_handler.dSYM/Contents/Resources/DWARF/chrome_crashpad_handler

FAILED: torque torque.dSYM/Contents/Info.plist torque.dSYM/Contents/Resources/DWARF/torque

FAILED: obj/chrome/chrome_framework_shared_library/Brave Browser Nightly Framework obj/chrome/chrome_framework_shared_library/Brave Browser Nightly Framework.TOC Brave Browser Nightly Framework.dSYM/Contents/Info.plist Brave Browser Nightly Framework.dSYM/Contents/Resources/DWARF/Brave Browser Nightly Framework

FAILED: mksnapshot mksnapshot.dSYM/Contents/Info.plist mksnapshot.dSYM/Contents/Resources/DWARF/mksnapshot 
or
FAILED: clang_x64_v8_arm64/mksnapshot clang_x64_v8_arm64/mksnapshot.dSYM/Contents/Info.plist clang_x64_v8_arm64/mksnapshot.dSYM/Contents/Resources/DWARF/mksnapshot

FAILED: protoc protoc.dSYM/Contents/Info.plist protoc.dSYM/Contents/Resources/DWARF/protoc

FAILED: generate_colors_info generate_colors_info.dSYM/Contents/Info.plist generate_colors_info.dSYM/Contents/Resources/DWARF/generate_colors_info

FAILED: libvk_swiftshader.dylib libvk_swiftshader.dylib.TOC libvk_swiftshader.dylib.dSYM/Contents/Info.plist libvk_swiftshader.dylib.dSYM/Contents/Resources/DWARF/libvk_swiftshader.dylib

FAILED: app_mode_loader app_mode_loader.dSYM/Contents/Info.plist app_mode_loader.dSYM/Contents/Resources/DWARF/app_mode_loader

https://ci.brave.com/job/brave-browser-build-macos-x64/5403/execution/node/248/log/#142362631000f97e22-c981-4d1e-8d5b-6c0b7408d959

FAILED: ipc_plugin ipc_plugin.dSYM/Contents/Info.plist ipc_plugin.dSYM/Contents/Resources/DWARF/ipc_plugin

https://ci.brave.com/job/brave-browser-build-macos-x64/5492/execution/node/99/log/#3157176000f97e22-c981-4d1e-8d5b-6c0b7408d959

Issue Analytics

State:
Created a year ago
Comments:22 (21 by maintainers)

Top GitHub Comments

1reaction

goodovcommented, Sep 29, 2022

Some info after experiments on `mac-imac3-ky`.

The lld crash is pretty rare (around 1 in ~100 links) and only happens in lld Release/RelWithDebInfo configurations.
Rebuilding lld with ASan hides the crash, i.e. no sanitizer errors and no crash. Rebuilding lld in Debug configuration also hides the crash.
Rebuilding lld with UBSan revealed some unaligned memory access errors because of reinterpret_casts, but fixing them hasn’t solve the issue.
Rebuilding lld with TSan didn’t show any issues.

Properly symbolized crashes look like this:

ld64.lld(58975,0x70000f179000) malloc: *** error for object 0x6002a0484000: pointer being freed was not allocated
ld64.lld(58975,0x70000f179000) malloc: *** set a breakpoint in malloc_error_break to debug
0.	Running pass 'Function Pass Manager' on module 'Users/jenkins/jenkins/workspace/goodov-linking-errors-nightly/src/out/Release/obj/third_party/vulkan-deps/spirv-tools/src/libspvtools_opt.aamd_ext_to_khr.o6886248'.
1.	Running pass 'Debug Variable Analysis' on function '@_ZN8spvtools5utils14AppendToVectorERKNSt2Cr12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEPNS1_6vectorIjNS5_IjEEEE'
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  lld                      0x000000010d11fcee llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 46
1  lld                      0x000000010d11ee45 llvm::sys::RunSignalHandlers() + 85
2  lld                      0x000000010d120340 SignalHandler(int) + 272
3  libsystem_platform.dylib 0x00007ff809a3ae2d _sigtramp + 29
4  libsystem_platform.dylib 0x00000000000000e0 _sigtramp + 18446603370419213008
5  libsystem_c.dylib        0x00007ff809971d10 abort + 123
6  libsystem_malloc.dylib   0x00007ff80984f3e2 has_default_zone0 + 0
7  libsystem_malloc.dylib   0x00007ff8098525ed malloc_report + 151
8  lld                      0x000000010e0aafe8 llvm::LexicalScopes::~LexicalScopes() + 344
9  lld                      0x000000010e5b3183 (anonymous namespace)::LDVImpl::runOnMachineFunction(llvm::MachineFunction&, bool) + 15091
10 lld                      0x000000010e67205d llvm::MachineFunctionPass::runOnFunction(llvm::Function&) + 349
11 lld                      0x000000010f97be9f llvm::FPPassManager::runOnFunction(llvm::Function&) + 1055
12 lld                      0x000000010f982571 llvm::FPPassManager::runOnModule(llvm::Module&) + 65

0.	Running pass 'Function Pass Manager' on module 'Users/jenkins/jenkins/workspace/goodov-linking-errors-nightly/src/out/Release/obj/third_party/vulkan-deps/spirv-tools/src/libspvtools_opt.ainstrument_pass.o242484496'.
1.	Running pass 'Debug Variable Analysis' on function '@_ZN8spvtools3opt14InstrumentPass14GetVec4FloatIdEv'
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  lld                      0x000000011025316b llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 43
1  lld                      0x0000000110253534 SignalHandler(int) + 244
2  libsystem_platform.dylib 0x00007ff80d96be2d _sigtramp + 29
3  libsystem_malloc.dylib   0x00007ff80d7732a9 nanov2_size + 20
4  libsystem_c.dylib        0x00007ff80d8a2d10 abort + 123
5  libsystem_malloc.dylib   0x00007ff80d7803e2 has_default_zone0 + 0
6  libsystem_malloc.dylib   0x00007ff80d7835ed malloc_report + 151
7  lld                      0x000000010f692a95 llvm::LexicalScopes::~LexicalScopes() + 133
8  lld                      0x000000010f8a44eb (anonymous namespace)::LDVImpl::runOnMachineFunction(llvm::MachineFunction&, bool) + 2555
9  lld                      0x000000010f9255e4 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) + 244

The crash happens on exit from LDVImpl::computeIntervals() in LexicalScopes::~LexicalScopes destructor: some unordered_map items fail to deallocate.

Chromium has hit a similar bug recently on Windows, but it doesn’t look like our case - they’ve got a stable reproducible assert().

I did debug the failed lld instance in lldb, but at the point of failure it’s absolutely unclear what exactly and at what point has failed. The code needs to be profiled carefully to pinpoint the exact place when things go downhill, it requires a lot of time.

Next steps

Report this issue to llvm, but I don’t expect a quick or easy fix for that. https://github.com/llvm/llvm-project/issues/58056
If no one objects, I will add a temporary fix to restart the linker inside linker_driver.py once in a case of failure when lto-enabled link is detected.

The process of building llvm with sanitizers

Change src/tools/clang/scripts/build.py to enable sanitizer:

@@ -852,7 +854,7 @@ def main():
     # PGO needs libclang_rt.profile but none of the other compiler-rt stuff.
     bootstrap_args.extend([
         '-D' + f
-        for f in compiler_rt_cmake_flags(sanitizers=False, profile=args.pgo)
+        for f in compiler_rt_cmake_flags(sanitizers=True, profile=args.pgo)
     ])
     if sys.platform == 'darwin':
       bootstrap_args.extend([
@@ -1197,6 +1205,7 @@ def main():
         cmake_args.append('-DRUNTIMES_' + triple + '_' + arg)
   cmake_args.append('-DLLVM_BUILTIN_TARGETS=' + all_triples)
   cmake_args.append('-DLLVM_RUNTIME_TARGETS=' + all_triples)
+  cmake_args.append('-DLLVM_USE_SANITIZER=Address')

   if os.path.exists(LLVM_BUILD_DIR):
     RmTree(LLVM_BUILD_DIR)