Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Warn if nvcc found in PATH does not match with `$CUDA_PATH/bin/nvcc`

See original GitHub issue
  • Conditions (you can just paste the output of python -c 'import cupy; cupy.show_config()')
    • CuPy version github repo 7f24cb5a5
    • OS/Platform CentOS 7.8.2003
    • CUDA version 11.0
    • cuDNN/NCCL version (if applicable)
  • Code to reproduce
export LC_ALL="en_US.UTF-8"
export CUDA_PATH=$LINUXBREWHOME/Cellar/cuda/11.0
export CC=$LINUXBREWHOME/opt/gcc@9/bin/gcc-9
export CXX=$LINUXBREWHOME/opt/gcc@9/bin/g++-9 
export LIBRARY_PATH="${CUDA_PATH}/lib"
export NVCC="nvcc --compiler-bindir $LINUXBREWHOME/opt/gcc@9/bin"
pip install . -vvvv
  • Error messages, stack traces, or logs
    building 'cupy.cuda.nvtx' extension
    //home/aznb/.pyenv/versions/3.8.3/include/python3.8 -c cupy/cuda/nvtx.cpp -o build/temp.linux-x86_64-3.8/cupy/cuda/nvtx.o -fopenmp
    building 'cupy_backends.cuda.libs.cutensor' extension
    //home/aznb/.pyenv/versions/3.8.3/include/python3.8 -c cupy_backends/cuda/libs/cutensor.cpp -o build/temp.linux-x86_64-3.8/cupy_backends/cuda/libs/cutensor.o -fopenmp
    building 'cupy.cuda.cub' extension
    //home/aznb/.pyenv/versions/3.8.3/include/python3.8 -c cupy/cuda/cub.cpp -o build/temp.linux-x86_64-3.8/cupy/cuda/cub.o -fopenmp
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/", line 169, in <module>
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/", line 145, in setup
        return distutils.core.setup(**attrs)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/", line 148, in setup
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/", line 966, in run_commands
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/", line 985, in run_command
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/command/", line 61, in run
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/", line 545, in run
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/", line 313, in run_command
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/", line 985, in run_command
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/", line 135, in run
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/", line 313, in run_command
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/", line 985, in run_command
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/", line 1012, in run
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/command/", line 84, in run
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/Cython/Distutils/", line 186, in run
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/", line 340, in run
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/Cython/Distutils/", line 195, in build_extensions
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/", line 449, in build_extensions
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/", line 474, in _build_extensions_serial
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/command/", line 205, in build_extension
        _build_ext.build_extension(self, ext)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/", line 528, in build_extension
        objects = self.compiler.compile(sources,
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/", line 574, in compile
        self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/", line 868, in _compile
        base_opts = build.get_compiler_base_options()
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/install/", line 232, in get_compiler_base_options
        _compiler_base_options = _get_compiler_base_options()
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/install/", line 270, in _get_compiler_base_options
    UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4-5: invalid continuation byte
    Running install for cupy: finished with status 'error'
ERROR: Command errored out with exit status 1: //home/aznb/.pyenv/versions/3.8.3/envs/loos/include/site/python3.8/cupy Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/req/", line 827, in install
    success = install_legacy(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/operations/install/", line 86, in install
    raise LegacyInstallFailure

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/cli/", line 188, in _main
    status =, args)
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/cli/", line 185, in wrapper
    return func(self, options, args)
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/commands/", line 398, in run
    installed = install_given_reqs(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/req/", line 67, in install_given_reqs
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/req/", line 845, in install
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_vendor/", line 703, in reraise
    raise value
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/operations/install/", line 74, in install
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/utils/", line 270, in runner
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/utils/", line 241, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: //home/aznb/.pyenv/versions/3.8.3/envs/loos/include/site/python3.8/cupy Check the logs for full command output.
Removed build tracker: '/tmp/160405612.1.noble-long.q/pip-req-tracker-rwqitlud'

Issue Analytics

  • State:open
  • Created 3 years ago
  • Comments:24 (14 by maintainers)

github_iconTop GitHub Comments

Char-Aznablecommented, Jul 15, 2020

What actually works export NVCC="${CUDA_PATH}/bin/nvcc --compiler-bindir $LINUXBREWHOME/opt/gcc@9/bin" which seems like a bug in the build config because ${CUDA_PATH}/bin is actually in my $PATH and the build system should just look up nvcc in ${CUDA_PATH}/bin anyway and should not risk allowing the user to set $NVCC to a different version of CUDA than ${CUDA_PATH}. Also, that nvcc is the only one I have on the system and I don’t see how it would complain about missing header given all the path set up the way in the OP

leofangcommented, Jul 24, 2020

btw @kmaehashi the title should be nvcc not nccl 😁

Read more comments on GitHub >

github_iconTop Results From Across the Web

ERROR: failed checking for nvcc. #46 - NVIDIA/cuda-samples
For people with problem with NVCC, try to check if CUDA is installed properly. Also, seems like nvcc is not in path, but...
Read more >
CUDA Compiler Driver NVCC - NVIDIA Documentation Center
The CUDA compilation trajectory separates the device functions from the host code, compiles the device functions using the proprietary NVIDIA compilers and ...
Read more >
Terminal trying to use a file that does not match which
What does type -a nvcc say? Possibly your shell has hashed a previous location of the command - see for example Pytest is...
Read more >
CUDA: nvcc cannot be detected though installed
Actually, it's the case that I set environment variable in a wrong way. Instead of using $ export PATH=/usr/local/cuda-7.0/bin:$PATH $ export ...
Read more >
Could not find a valid NVCC executable\n - Google Groups
The NVCC compiler 'C:\Program Files\MATLAB\R2019b\sys\cuda\win64\cuda\bin\nvcc.exe\bin\nvcc'. does not appear to be valid. Error in vl_compilenn (line 296).
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found