Feature request: more powerful way to set C/C++ compile/link flags

Problem

Currently, Bazel does not have a way to set C/C++ compile/link flags in BUILD/.bzl files to all targets other than manually adding copts/linkotps manually to all cc_* rules.

Not being able to set copts/linkopts to all targets has caused a lot of problems in Tensorflow (e.g. compile error due to some targets not getting -DNOGDI flag in MSVC build). Other Google projects such as Protobuf and gRPC are also affected, but less severe than Tensorflow.

As far as I know, Abseil C++ is the only Google project that has the discipline to set copts to every cc_* with ABSL_DEFAULT_COPTS/ABSL_TEST_COPTS from copts.bzl manually, and even then accident can happen (uses “-fexceptions` directly instead of ABSL_EXCEPTIONS_FLAG, which break MSVC build).

Existing workarounds

• Tensorflow currently uses --copt in .bazelrc to set global compile flags.
  • Control of flags are passed over from BUILD and .bzl to configure.py. Need to implement compiler/platform detection yourself.
  • All Tensorflow’s dependencies are affected.
• Adding flags to CROSSTOOL.
  • Now all Bazel projects are affected. I violently oppose this workaround.
  • Need to maintain your own CROSSTOOL which is painful.
• Use macro to wrap around cc_library
  • Example: llvm_cc_library, tf_cc_library.
  • For some reason, Tensorflow team dislikes this and proposes the .bazelrc and CROSSTOOL workarounds as above, which in my opinion are even worse.
  • This is currently blocking my PR to make LLVM build with Bazel on Windows (because Tensorflow team don’t want llvm_cc_library macro).

What we need

A way to set compile/link flags to targets with custom visibility (current package only, current package with subpackages or whole @project//*).

Internal version of Blaze seems to have default_copts parameter in package rule.

What CMake does

In CMake, you can use add_compile_options and add_definitions to add compile/define flags to all targets under this current directory and sub-directories. (You can also directly add/replace/delete flags to CMAKE_C_FLAGS and CMAKE_CXX_FLAGS, but that will affect all directories which is very intrusive).

[rongjiecomputer]

Currently one thing that cannot be done is to enable a certain feature for all transitive dependencies of a particular target.

This seems to solve part of my problem. Can it work backward (i.e. a target can force everyone that depends on it to turn on/off certain feature, such no_asan if a target is known to not work with address sanitizer)?

but in the general case it’s the toolchain author who thinks about flags, and who has to have a way to specify which flags are applied when.

For me, Crosstool/Toolchain author’s responsibility is only to specify the least flags needed to use a toolchain (“least” is the keyword here). Project-specific flags should not appear in Crosstool.

Effect of my assumption:

• Crosstool should be reusable for most projects no matter what special requirement a project has.
• If two Crosstools support the same Clang, using either of them should not change the output.
• BUILD authors rarely have to communicate with Crosstool authors.

Crosstool is useful, but as far as I know, only one Crosstool file can be used in one build. If I have one project with 5 dependencies, all with one custom Crosstool just to set compiler/linker flags for its project, then which one should user chooses to use for the entire build?

In my ideal world, users don’t have to think about the compiler flags (or even the compiler used, or even the platform used, to some extent).

I don’t think this is possible in the world of C and C++. GCC, Clang and MSVC all have thousands of warning and optimization flags, no one can abstract that away. You can abstract away more general things such as no_exceptions, use_rtti, fully_static_link (and I wish Bazel will support more of such things), but beyond that, you can’t abstract away every single warning and optimization flag (and it is not going to be useful to anyone).

1. Different projects have different set of warning and optimization flags they want to disable (e.g. some projects might want to disable -Wnarrowing warning, some projects might not; some projects need -fdelete-null-pointer-checks optimization, some projects cannot use it).
2. Most Windows projects will define /DMINMAX to not define min and max macros, but there will be some Windows projects out there that do not want this flag.

My point is Crosstool should not be the one to force certain flags down these projects’ throat. Unless in Bazel’s ideal world, each project should have its own Crosstool. Note that in the above examples, those flags need to be set for all targets of a project or you might get some obscure compile/runtime errors.

I can sort of guess how Google uses Crosstool internally by looking at how Chromium uses GN. Chromium’s //src/build/config (which is kinda like Bazel’s Crosstool) is very gigantic. All Chromium’s dependencies will use this //src/build/config, which is feasible for Chromium because they have a team to deal with everything about “build” and owners for each third party dependency has the responsibility to maintain its custom BUILD.gn. In outside world however, people prefer the maintainer(s) of the dependencies to write build file so that they don’t have to deal with it.

//src/build/config is so tuned to Chromium’s usage (like turn on/off certain optimizations and adding defines such as SAFE_BROWSING_DB_REMOTE that only make sense to Chromium) that it is very difficult to modify it to use with GN outside Chromium.

Some other comments about Crosstool.

I fully agree that the legacy crosstool fields (compiler_flag, linker_flag …) are not sufficient for that. That’s why we introduced action_configs and features as a way to dynamically control flags.

I agree on this too. I have written a working CROSSTOOL for LLVM on Windows toolchain based on action_config. It can express more things than compiler_flag, linker_flag etc. alone.

Example: toolchain author creates a feature named fully_static_link, that will encapsulate all the flags needed to create a C++ binary that statically links libc. Users don’t have to know the details, they only need to enable this feature for their cc_binary:

That is also my dream. I wish I can just features = ["cpp14", "no_exceptions", "no_rtti"] in one cc_library, package, whole project (this one is currently impossible which I need heavily) or entire build (already possible with --features, can be use for thinlto, asan, ubsan …).

Issue tracking the migration in bazel will be #5187. Currently this effort is blocked by #4571 which is seeing good progress.

I have been tracking some Github issues about C++ Crosstool too. I have read Crosstool in Skylark since it was published for review, but I don’t fully understand what I read (not much example and have too much reference to Bazel’s Java implementation). I think it is already partially implemented? I can see action_config in Skylark macro form. I am very impressed about how the implementation moves closer to the goal of code-reusability and human-readability with Skylark macros while under-the-hood still produces proto text format for smoother migration.

I am willing to help with the Windows part when it is ready for external contributors to contribute.

Some more complaints about Crosstool

When can Bazel finally decouple operating system from --cpu? k8 (for Linux) and darwin (for Mac) are not even CPUs, and don’t let me start with x64_windows and x64_windows_msvc (cpu + os + compiler)! (Perhaps this should be in another Github issue?)

Bazel should be able to figure out cpu and os easily. Compiler can be selected based on --compiler or just from environment variables (#5186, I will add some comments in that issue). User should only need to specify cpu + os + compiler when cross-compiling with LLVM-style --target flag. (And that is for target_platform. Bazel should still figure out cpu + os for host_platform automatically).

Thanks for reading till the end.

[hlopko]

• Re: Backwards work: we don’t propagate features currently, but that would be a simple change. I would want to have multiple feature types then, because not all features are meant to be used transitively, and not all are meant to be used on targets directly.

• Re: “least amount of flags”: I see your point, and I actually agree. Writing a good crosstool (that has modules, thinlto, can verify headers, layering checks, uses linker effectively, has sanitizers) is hard. Individual projects should not be forced to be in the business of writing crosstools. I’d argue that features allow you to be decoupled from the crosstool more than copts/linkopts.

• Re: Warnings and fine tuning: If the project has good reasons to fine tune, great. But good crosstool that sets the standard for the modern C++ development (so most people don’t have to do “anything”) will have to be opinionated. I can also imagine a robust-enough way of modifying the crosstool for every project in BUILD files without the need to change the original (think adding a custom, project only features).

• Re: Google use of the Crosstool, you’re mostly accurate.

• Re: [“cpp14”, “no_exceptions”, “no_rtti”] - that is exactly what I think the good crosstool should contain.

There are 2 parallel efforts. One is changing crosstool format from text protobuf to Skylark code. That what Crosstool in Skylark is about. Second it making it easier to write cc_configure which is an autotools-like script that inspects the environment and autogenerates the crosstool. action_config macros you saw are for the latter.

We are migrating away from --cpu and --compiler options towards platforms. It’s not only about figuring it out (bazel already does that), but sometimes you need to cross compile, sometimes you need to cross compile on a different machine, etc. But yes, we’re working on it 😃

Thanks for taking the time to comment!