v0.3.0 build failure with CUDA support (CPU-only builds fine)

Please:

• Check for duplicate issues.
• Provide a complete example of how to reproduce the bug, wrapped in triple backticks like this:

Steps to reproduce:

• Clone https://github.com/NixOS/nixpkgs/pull/159099/commits/19c42cefd72befdbeab5b98c5534626d5905a8ce
• Update pkgs/development/python-modules/jaxlib/default.nix with these contents https://gist.github.com/samuela/931d7e29dd5e83ec5bf73de3bbc5b084.
• Run NIX_PATH=.. nix-build -E "with import <nixpkgs> {}; python3Packages.jaxlib.override { cudaSupport = true; }" inside your nixpkgs checkout.

Observe the failure:

ERROR: /build/output/external/org_tensorflow/tensorflow/compiler/xla/python/BUILD:206:11: Compiling tensorflow/compiler/xla/python/pprof_profile_builder.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF ... (remaining 184 arguments skipped)
In file included from external/com_google_absl/absl/strings/str_format.h:76,
                 from external/org_tensorflow/tensorflow/compiler/xla/util.h:35,
                 from external/org_tensorflow/tensorflow/compiler/xla/python/pprof_profile_builder.cc:21:
external/com_google_absl/absl/strings/internal/str_format/arg.h: In instantiation of ‘constexpr absl::lts_20211102::FormatConversionCharSet absl::lts_20211102::str_format_internal::ArgumentToConv() [with Arg = google::protobuf::stringpiece_internal::StringPiece]’:
external/com_google_absl/absl/strings/str_format.h:257:46:   required by substitution of ‘template<class ... Args> using FormatSpec = absl::lts_20211102::str_format_internal::FormatSpecTemplate<(ArgumentToConv<Args>)()...> [with Args = {google::protobuf::stringpiece_internal::StringPiece}]’
external/org_tensorflow/tensorflow/compiler/xla/util.h:214:8:   required by substitution of ‘template<class ... Args> tensorflow::Status xla::InvalidArgument(absl::lts_20211102::FormatSpec<Args ...>&, const Args& ...) [with Args = {google::protobuf::stringpiece_internal::StringPiece}]’
external/org_tensorflow/tensorflow/compiler/xla/python/pprof_profile_builder.cc:71:71:   required from here
external/com_google_absl/absl/strings/internal/str_format/arg.h:305:54: error: no matching function for call to ‘FormatConvertImpl(const google::protobuf::stringpiece_internal::StringPiece&, const absl::lts_20211102::str_format_internal::FormatConversionSpecImpl&, absl::lts_20211102::str_format_internal::FormatSinkImpl*)’
  305 |       decltype(str_format_internal::FormatConvertImpl(
      |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
  306 |           std::declval<const Arg&>(),
      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  307 |           std::declval<const FormatConversionSpecImpl&>(),
      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  308 |           std::declval<FormatSinkImpl*>())){});
      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/com_google_absl/absl/strings/internal/str_format/arg.h:60:6: note: candidate: ‘template<class T> decltype (absl::lts_20211102::str_format_internal::AbslFormatConvert(v, declval<const absl::lts_20211102::FormatConversionSpec&>(), declval<absl::lts_20211102::FormatSink*>())) absl::lts_20211102::str_format_internal::FormatConvertImpl(const T&, absl::lts_20211102::str_format_internal::FormatConversionSpecImpl, absl::lts_20211102::str_format_internal::FormatSinkImpl*)’
   60 | auto FormatConvertImpl(const T& v, FormatConversionSpecImpl conv,
      |      ^~~~~~~~~~~~~~~~~

I can confirm that this build “script” works on 0.1.75 and a number versions before that. Oddly enough the CPU-only build (run nix-build -A python3Packages.jaxlib) runs just fine.

• If applicable, include full error messages/tracebacks.

You can check out the unabridged build log here: https://gist.github.com/samuela/fca69ea5ad0eb20d77b8be060446db21.

[hawkinsp]

@samuela I’ll make that change upstream as well.

[samuela]

I found that protobuf 3.12 is the latest version that works. protobuf 3.13-3.19 all have this error.