Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Jaxlib 0.1.68 Fails to Build on Windows

See original GitHub issue

With Python 3.8, Cuda 11.2, Bazel 4.1, build run with the following command

python .\build\build.py --enable_cuda --cuda_path="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2" --cudnn_path="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2" --cuda_compute_capabilities="7.5" --cuda_version="11.2" --cudnn_version="8.1.0" --noenable_rocm --noenable_tpu

results in the following error.

ERROR: C:/users/adam/_bazel_adam/n5urqa67/external/org_tensorflow/tensorflow/compiler/xla/pjrt/BUILD:372:11: Compiling tensorflow/compiler/xla/pjrt/tracked_tfrt_cpu_device_buffer.cc failed: (Exit 2): python.exe failed: error executing command
  cd C:/users/adam/_bazel_adam/n5urqa67/execroot/__main__
  SET CUDA_TOOLKIT_PATH=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2
    SET CUDNN_INSTALL_PATH=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2
    SET INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\ATLMFC\include;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\include;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt
    SET LIB=C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\ATLMFC\lib\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\lib\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64
    SET PATH=C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\\Extensions\Microsoft\IntelliCode\CLI;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\bin\HostX64\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\VC\VCPackages;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Performance Tools\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Performance Tools;C:\Program Files (x86)\Microsoft Visual Studio\Shared\Common\VSPerfCollectionTools\vs2019\\x64;C:\Program Files (x86)\Microsoft Visual Studio\Shared\Common\VSPerfCollectionTools\vs2019\;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\Tools\devinit;C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x64;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\\MSBuild\Current\Bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\Tools\;;C:\WINDOWS\system32;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja
    SET PWD=/proc/self/cwd
    SET RUNFILES_MANIFEST_ONLY=1
    SET TEMP=C:\Users\Adam\AppData\Local\Temp
    SET TF_CUDA_COMPUTE_CAPABILITIES=7.5
    SET TF_CUDA_VERSION=11.2
    SET TF_CUDNN_VERSION=8.1.0
    SET TF_ROCM_AMDGPU_TARGETS=gfx803,gfx900,gfx906,gfx1010
    SET TMP=C:\Users\Adam\AppData\Local\Temp
  C:/Users/Adam/anaconda3/envs/tensorflow/python.exe -B external/local_config_cuda/crosstool/windows/msvc_wrapper_for_nvcc.py /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0600 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /D_SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS /bigobj /Zm500 /J /Gy /GF /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /Iexternal/org_tensorflow /Ibazel-out/x64_windows-opt/bin/external/org_tensorflow /Iexternal/eigen_archive /Ibazel-out/x64_windows-opt/bin/external/eigen_archive /Iexternal/com_google_absl /Ibazel-out/x64_windows-opt/bin/external/com_google_absl /Iexternal/nsync /Ibazel-out/x64_windows-opt/bin/external/nsync /Iexternal/gif /Ibazel-out/x64_windows-opt/bin/external/gif /Iexternal/libjpeg_turbo /Ibazel-out/x64_windows-opt/bin/external/libjpeg_turbo /Iexternal/com_google_protobuf /Ibazel-out/x64_windows-opt/bin/external/com_google_protobuf /Iexternal/com_googlesource_code_re2 /Ibazel-out/x64_windows-opt/bin/external/com_googlesource_code_re2 /Iexternal/farmhash_archive /Ibazel-out/x64_windows-opt/bin/external/farmhash_archive /Iexternal/fft2d /Ibazel-out/x64_windows-opt/bin/external/fft2d /Iexternal/highwayhash /Ibazel-out/x64_windows-opt/bin/external/highwayhash /Iexternal/zlib /Ibazel-out/x64_windows-opt/bin/external/zlib /Iexternal/double_conversion /Ibazel-out/x64_windows-opt/bin/external/double_conversion /Iexternal/snappy /Ibazel-out/x64_windows-opt/bin/external/snappy /Iexternal/tf_runtime /Ibazel-out/x64_windows-opt/bin/external/tf_runtime /Iexternal/llvm-project /Ibazel-out/x64_windows-opt/bin/external/llvm-project /Iexternal/org_tensorflow/third_party/eigen3/mkl_include /Ibazel-out/x64_windows-opt/bin/external/org_tensorflow/third_party/eigen3/mkl_include /Iexternal/eigen_archive /Ibazel-out/x64_windows-opt/bin/external/eigen_archive /Iexternal/nsync/public /Ibazel-out/x64_windows-opt/bin/external/nsync/public /Iexternal/gif /Ibazel-out/x64_windows-opt/bin/external/gif /Iexternal/gif/windows /Ibazel-out/x64_windows-opt/bin/external/gif/windows /Iexternal/com_google_protobuf/src /Ibazel-out/x64_windows-opt/bin/external/com_google_protobuf/src /Iexternal/farmhash_archive/src /Ibazel-out/x64_windows-opt/bin/external/farmhash_archive/src /Iexternal/zlib /Ibazel-out/x64_windows-opt/bin/external/zlib /Iexternal/double_conversion /Ibazel-out/x64_windows-opt/bin/external/double_conversion /Iexternal/tf_runtime/include /Ibazel-out/x64_windows-opt/bin/external/tf_runtime/include /Iexternal/llvm-project/llvm/include /Ibazel-out/x64_windows-opt/bin/external/llvm-project/llvm/include /Iexternal/tf_runtime/third_party/llvm_derived/include /Ibazel-out/x64_windows-opt/bin/external/tf_runtime/third_party/llvm_derived/include /DEIGEN_MPL2_ONLY /DEIGEN_MAX_ALIGN_BYTES=64 /DTF_USE_SNAPPY /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /D_CRT_NONSTDC_NO_DEPRECATE /D_CRT_NONSTDC_NO_WARNINGS /D_SCL_SECURE_NO_DEPRECATE /D_SCL_SECURE_NO_WARNINGS /DUNICODE /D_UNICODE /DLLVM_ENABLE_STATS /D__STDC_LIMIT_MACROS /D__STDC_CONSTANT_MACROS /D__STDC_FORMAT_MACROS /DLLVM_BUILD_GLOBAL_ISEL /showIncludes /MD /O2 /DNDEBUG /D_USE_MATH_DEFINES -DWIN32_LEAN_AND_MEAN -DNOGDI /Zc:preprocessor /std:c++14 /Fobazel-out/x64_windows-opt/bin/external/org_tensorflow/tensorflow/compiler/xla/pjrt/_objs/tracked_tfrt_cpu_device_buffer/tracked_tfrt_cpu_device_buffer.obj /c external/org_tensorflow/tensorflow/compiler/xla/pjrt/tracked_tfrt_cpu_device_buffer.cc
Execution platform: @local_execution_config_platform//:platform
Target //build:build_wheel failed to build
INFO: Elapsed time: 1710.423s, Critical Path: 331.72s
INFO: 6277 processes: 2542 internal, 3735 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
b"external/llvm-project/llvm/include\\llvm/Support/type_traits.h(79): warning C4624: 'llvm::detail::copy_construction_triviality_helper<T>': destructor was implicitly defined as deleted\r\n        with\r\n        [\r\n            T=std::string\r\n        ]\r\nC:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30037\\include\\type_traits(631): note: see reference to class template instantiation 'llvm::detail::copy_construction_triviality_helper<T>' being compiled\r\n        with\r\n        [\r\n            T=std::string\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(100): note: see reference to class template instantiation 'std::is_copy_constructible<llvm::detail::copy_construction_triviality_helper<T>>' being compiled\r\n        with\r\n        [\r\n            T=std::string\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/ADT/SmallVector.h(311): note: see reference to class template instantiation 'llvm::is_trivially_copy_constructible<T>' being compiled\r\n        with\r\n        [\r\n            T=std::string\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/ADT/SmallVector.h(1167): note: see reference to class template instantiation 'llvm::SmallVectorImpl<T>' being compiled\r\n        with\r\n        [\r\n            T=std::string\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/Support/Error.h(992): note: see reference to class template instantiation 'llvm::SmallVector<std::string,2>' being compiled\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(86): warning C4624: 'llvm::detail::move_construction_triviality_helper<T>': destructor was implicitly defined as deleted\r\n        with\r\n        [\r\n            T=std::string\r\n        ]\r\nC:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30037\\include\\type_traits(662): note: see reference to class template instantiation 'llvm::detail::move_construction_triviality_helper<T>' being compiled\r\n        with\r\n        [\r\n            T=std::string\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(111): note: see reference to class template instantiation 'std::is_move_constructible<llvm::detail::move_construction_triviality_helper<T>>' being compiled\r\n        with\r\n        [\r\n            T=std::string\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/ADT/SmallVector.h(311): note: see reference to class template instantiation 'llvm::is_trivially_move_constructible<T>' being compiled\r\n        with\r\n        [\r\n            T=std::string\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(79): warning C4624: 'llvm::detail::copy_construction_triviality_helper<T>': destructor was implicitly defined as deleted\r\n        with\r\n        [\r\n            T=tfrt::DebugInfo\r\n        ]\r\nC:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30037\\include\\type_traits(631): note: see reference to class template instantiation 'llvm::detail::copy_construction_triviality_helper<T>' being compiled\r\n        with\r\n        [\r\n            T=tfrt::DebugInfo\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(100): note: see reference to class template instantiation 'std::is_copy_constructible<llvm::detail::copy_construction_triviality_helper<T>>' being compiled\r\n        with\r\n        [\r\n            T=tfrt::DebugInfo\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/ADT/Optional.h(53): note: see reference to class template instantiation 'llvm::is_trivially_copy_constructible<T>' being compiled\r\n        with\r\n        [\r\n            T=tfrt::DebugInfo\r\n        ]\r\nexternal/tf_runtime/include\\tfrt/host_context/location.h(83): note: see reference to class template instantiation 'llvm::Optional<tfrt::DebugInfo>' being compiled\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(79): warning C4624: 'llvm::detail::copy_construction_triviality_helper<T>': destructor was implicitly defined as deleted\r\n        with\r\n        [\r\n            T=std::unique_ptr<tfrt::BaseTypedErrorInfo,std::default_delete<tfrt::BaseTypedErrorInfo>>\r\n        ]\r\nC:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30037\\include\\type_traits(631): note: see reference to class template instantiation 'llvm::detail::copy_construction_triviality_helper<T>' being compiled\r\n        with\r\n        [\r\n            T=std::unique_ptr<tfrt::BaseTypedErrorInfo,std::default_delete<tfrt::BaseTypedErrorInfo>>\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(100): note: see reference to class template instantiation 'std::is_copy_constructible<llvm::detail::copy_construction_triviality_helper<T>>' being compiled\r\n        with\r\n        [\r\n            T=std::unique_ptr<tfrt::BaseTypedErrorInfo,std::default_delete<tfrt::BaseTypedErrorInfo>>\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/ADT/SmallVector.h(311): note: see reference to class template instantiation 'llvm::is_trivially_copy_constructible<T>' being compiled\r\n        with\r\n        [\r\n            T=std::unique_ptr<tfrt::BaseTypedErrorInfo,std::default_delete<tfrt::BaseTypedErrorInfo>>\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/ADT/SmallVector.h(1167): note: see reference to class template instantiation 'llvm::SmallVectorImpl<T>' being compiled\r\n        with\r\n        [\r\n            T=std::unique_ptr<tfrt::BaseTypedErrorInfo,std::default_delete<tfrt::BaseTypedErrorInfo>>\r\n        ]\r\nexternal/tf_runtime/include\\tfrt/support/error_util.h(255): note: see reference to class template instantiation 'llvm::SmallVector<std::unique_ptr<tfrt::BaseTypedErrorInfo,std::default_delete<tfrt::BaseTypedErrorInfo>>,4>' being compiled\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(86): warning C4624: 'llvm::detail::move_construction_triviality_helper<T>': destructor was implicitly defined as deleted\r\n        with\r\n        [\r\n            T=std::unique_ptr<tfrt::BaseTypedErrorInfo,std::default_delete<tfrt::BaseTypedErrorInfo>>\r\n        ]\r\nC:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30037\\include\\type_traits(662): note: see reference to class template instantiation 'llvm::detail::move_construction_triviality_helper<T>' being compiled\r\n        with\r\n        [\r\n            T=std::unique_ptr<tfrt::BaseTypedErrorInfo,std::default_delete<tfrt::BaseTypedErrorInfo>>\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(111): note: see reference to class template instantiation 'std::is_move_constructible<llvm::detail::move_construction_triviality_helper<T>>' being compiled\r\n        with\r\n        [\r\n            T=std::unique_ptr<tfrt::BaseTypedErrorInfo,std::default_delete<tfrt::BaseTypedErrorInfo>>\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/ADT/SmallVector.h(311): note: see reference to class template instantiation 'llvm::is_trivially_move_constructible<T>' being compiled\r\n        with\r\n        [\r\n            T=std::unique_ptr<tfrt::BaseTypedErrorInfo,std::default_delete<tfrt::BaseTypedErrorInfo>>\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(79): warning C4624: 'llvm::detail::copy_construction_triviality_helper<T>': destructor was implicitly defined as deleted\r\n        with\r\n        [\r\n            T=tfrt::DecodedLocation\r\n        ]\r\nC:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30037\\include\\type_traits(631): note: see reference to class template instantiation 'llvm::detail::copy_construction_triviality_helper<T>' being compiled\r\n        with\r\n        [\r\n            T=tfrt::DecodedLocation\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/Support/type_traits.h(100): note: see reference to class template instantiation 'std::is_copy_constructible<llvm::detail::copy_construction_triviality_helper<T>>' being compiled\r\n        with\r\n        [\r\n            T=tfrt::DecodedLocation\r\n        ]\r\nexternal/llvm-project/llvm/include\\llvm/ADT/Optional.h(53): note: see reference to class template instantiation 'llvm::is_trivially_copy_constructible<T>' being compiled\r\n        with\r\n        [\r\n            T=tfrt::DecodedLocation\r\n        ]\r\nexternal/tf_runtime/include\\tfrt/host_context/diagnostic.h(48): note: see reference to class template instantiation 'llvm::Optional<tfrt::DecodedLocation>' being compiled\r\nexternal/tf_runtime/include\\tfrt/host_context/async_value.h(455): error C2338: Unexpected size for AsyncValue\r\n"
Traceback (most recent call last):
  File ".\build\build.py", line 564, in <module>
    main()
  File ".\build\build.py", line 559, in main
    shell(command)
  File ".\build\build.py", line 52, in shell
    output = subprocess.check_output(cmd)
  File "C:\Users\Adam\anaconda3\envs\tensorflow\lib\subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Users\Adam\anaconda3\envs\tensorflow\lib\subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\bazel\\bazel.EXE', 'run', '--verbose_failures=true', '--config=short_logs', '--config=mkl_open_source_only', '--config=cuda', '--define=xla_python_enable_gpu=true', ':build_wheel', '--', '--output_path=C:\\sdks\\jax-jaxlib-v0.1.68\\dist']' returned non-zero exit status 1.