question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

build against CUDA 10.0 fails - incorrect path to libcublas library

See original GitHub issue

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04.3
  • TensorFlow version and how it was installed (source or binary): binary (Anaconda package tensorflow-2.0.0-gpu_py37h768510d_0)
  • TensorFlow-Addons version and how it was installed (source or binary): source (a1a1e4c)
  • Python version: 3.7.6
  • Is GPU used? (yes/no): yes

Describe the bug

Attempting to build against CUDA 10.0 (which is installed in /usr/local/cuda) by entering “10.0” when prompted for the desired CUDA version by configure.sh and modifying requirements.txt to contain tensorflow>=2.0.0 results in the following exception when bazel build build_pip_pkg is run:

ERROR: Analysis of target '//:build_pip_pkg' failed; build aborted: no such package '@local_config_cuda//cuda': Traceback (most recent call last):
        File "/home/lebedov/tensorflow-addons/build_deps/toolchains/gpu/cuda_configure.bzl", line 1047
                _create_local_cuda_repository(<1 more arguments>)
        File "/home/lebedov/tensorflow-addons/build_deps/toolchains/gpu/cuda_configure.bzl", line 869, in _create_local_cuda_repository
                _find_libs(repository_ctx, <1 more arguments>)
        File "/home/lebedov/tensorflow-addons/build_deps/toolchains/gpu/cuda_configure.bzl", line 568, in _find_libs
                _find_cuda_lib("cublas", <4 more arguments>)
        File "/home/lebedov/tensorflow-addons/build_deps/toolchains/gpu/cuda_configure.bzl", line 529, in _find_cuda_lib
                find_lib(repository_ctx, <2 more arguments>)
        File "/home/lebedov/tensorflow-addons/build_deps/toolchains/gpu/cuda_configure.bzl", line 507, in find_lib
                auto_configure_fail(<1 more arguments>)
        File "/home/lebedov/tensorflow-addons/build_deps/toolchains/gpu/cuda_configure.bzl", line 261, in auto_configure_fail
                fail(<1 more arguments>)

Cuda Configuration Error: No library found under: /usr/local/cuda/lib64/libcublas.so.10

This is because CUDA 10.0 contains the lib symlinks /usr/local/cuda/lib64/libcublas.so and /usr/local/cuda/lib64/libcublas.so.10.0, but not /usr/local/cuda/lib64/libcublas.so.10. I’m specifically interested in using addons with tensorflow 2.0.0 because that is the latest version of the GPU-enabled packages provided by Anaconda as of today.

Code to reproduce the issue

See above.

Other info / logs

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
lebedovcommented, Jan 15, 2020

@seanpmorgan I was able to build 0.6.0 against CUDA 10.0 with bazel 2.0.0 successfully, although I had to tweak build_deps/toolchains/gpu/cuda_configure.bzl to suppress the following exception:

ERROR: Analysis of target '//:build_pip_pkg' failed; build aborted: no such package '@local_config_cuda//cuda': Traceback (most recent call last):
        File "/home/lebedov/tensorflow-addons/build_deps/toolchains/gpu/cuda_configure.bzl", line 1329
                _create_local_cuda_repository(<1 more arguments>)
        File "/home/lebedov/tensorflow-addons/build_deps/toolchains/gpu/cuda_configure.bzl", line 1231, in _create_local_cuda_repository
                _host_compiler_includes(<2 more arguments>)
        File "/home/lebedov/tensorflow-addons/build_deps/toolchains/gpu/cuda_configure.bzl", line 357, in _host_compiler_includes
                get_cxx_inc_directories(<2 more arguments>)
        File "/home/lebedov/tensorflow-addons/build_deps/toolchains/gpu/cuda_configure.bzl", line 336, in get_cxx_inc_directories
                inc not in includes_cpp_set
argument of type 'depset' is not iterable. in operator only works on lists, tuples, dicts and strings.

I’m specifically interested in using the seq2seq code - as the seq2seq unit tests in 0.6.0 seem to pass with my custom build, I’m hoping that whatever remaining tf 2.0 compatibility issues still exist will not be a concern.

0reactions
cecabertcommented, Apr 15, 2020

I add the exact same error while trying to build for older cuda version (i.e. 9.2). I solve the issue by replacing cuda_config.cuda_major_version to cuda_config.cuda_version on the lines 573, 580, 587, 594 in the build_deps/toolchains/gpu/cuda_configure.bzl file, because the cuda library are named <libname>.so.<Major>.<Minor> and not <libname>.so.<Major>.

Read more comments on GitHub >

github_iconTop Results From Across the Web

ImportError: libcublas.so.10.0: cannot open shared object file ...
I downloaded cuda 10.0 from the following link CUDA 10.0. Then I installed it using the following commands:
Read more >
cublas for 10.1 is missing - CUDA Setup and Installation
CUBLAS packaging changed in CUDA 10.1 to be outside of the toolkit installation path. ... Cuda Configuration Error: No library found under: ...
Read more >
FindCUDA — CMake 3.25.1 Documentation
Tools for building CUDA C files: libraries and build dependencies. This script locates the NVIDIA CUDA C tools. It should work on Linux,...
Read more >
How To Install CUDA 10 (together with 9.2) on Ubuntu 18.04 ...
This is the main problem with doing TensorFlow install as suggested by Google. Getting the proper CUDA toolkit in-place is a pain. As...
Read more >
NVIDIA - CUDA | onnxruntime
The CUDA Execution Provider enables hardware accelerated computation on Nvidia CUDA-enabled GPUs. Contents. Install; Requirements; Build; Configuration Options ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found