Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Warn if nvcc found in PATH does not match with `$CUDA_PATH/bin/nvcc`

See original GitHub issue

Conditions (you can just paste the output of python -c 'import cupy; cupy.show_config()')
- CuPy version github repo 7f24cb5a5
- OS/Platform CentOS 7.8.2003
- CUDA version 11.0
- cuDNN/NCCL version (if applicable)
Code to reproduce

export LC_ALL="en_US.UTF-8"
export CUDA_PATH=$LINUXBREWHOME/Cellar/cuda/11.0
export CC=$LINUXBREWHOME/opt/gcc@9/bin/gcc-9
export CXX=$LINUXBREWHOME/opt/gcc@9/bin/g++-9 
export LIBRARY_PATH="${CUDA_PATH}/lib"
export LD_LIBRARY_PATH="${CUDA_PATH}/lib:${LD_LIBRARY_PATH}"
export NVCC="nvcc --compiler-bindir $LINUXBREWHOME/opt/gcc@9/bin"
pip install . -vvvv

Error messages, stack traces, or logs

    building 'cupy.cuda.nvtx' extension
    //home/aznb/.pyenv/versions/3.8.3/include/python3.8 -c cupy/cuda/nvtx.cpp -o build/temp.linux-x86_64-3.8/cupy/cuda/nvtx.o -fopenmp
    //home/aznb/.linuxbrew/Cellar/cuda/11.0/lib
    building 'cupy_backends.cuda.libs.cutensor' extension
    //home/aznb/.pyenv/versions/3.8.3/include/python3.8 -c cupy_backends/cuda/libs/cutensor.cpp -o build/temp.linux-x86_64-3.8/cupy_backends/cuda/libs/cutensor.o -fopenmp
    //home/aznb/.linuxbrew/Cellar/cuda/11.0/lib
    building 'cupy.cuda.cub' extension
    //home/aznb/.pyenv/versions/3.8.3/include/python3.8 -c cupy/cuda/cub.cpp -o build/temp.linux-x86_64-3.8/cupy/cuda/cub.o -fopenmp
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/setup.py", line 169, in <module>
        setup(
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/cupy_setup_build.py", line 1012, in run
        build_ext.build_ext.run(self)
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run
        _build_ext.run(self)
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
        _build_ext.build_ext.build_extensions(self)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 205, in build_extension
        _build_ext.build_extension(self, ext)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
        objects = self.compiler.compile(sources,
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/ccompiler.py", line 574, in compile
        self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/cupy_setup_build.py", line 868, in _compile
        base_opts = build.get_compiler_base_options()
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/install/build.py", line 232, in get_compiler_base_options
        _compiler_base_options = _get_compiler_base_options()
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/install/build.py", line 270, in _get_compiler_base_options
        stderrdata.decode('utf8'))
    UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4-5: invalid continuation byte
    Running setup.py install for cupy: finished with status 'error'
ERROR: Command errored out with exit status 1: //home/aznb/.pyenv/versions/3.8.3/envs/loos/include/site/python3.8/cupy Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 827, in install
    success = install_legacy(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/operations/install/legacy.py", line 86, in install
    raise LegacyInstallFailure
pip._internal.operations.install.legacy.LegacyInstallFailure

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 398, in run
    installed = install_given_reqs(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 67, in install_given_reqs
    requirement.install(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 845, in install
    six.reraise(*exc.parent)
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_vendor/six.py", line 703, in reraise
    raise value
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/operations/install/legacy.py", line 74, in install
    runner(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/utils/subprocess.py", line 270, in runner
    call_subprocess(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/utils/subprocess.py", line 241, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: //home/aznb/.pyenv/versions/3.8.3/envs/loos/include/site/python3.8/cupy Check the logs for full command output.
Removed build tracker: '/tmp/160405612.1.noble-long.q/pip-req-tracker-rwqitlud'

Issue Analytics

State:
Created 3 years ago
Comments:24 (14 by maintainers)

Top GitHub Comments

3reactions

Char-Aznablecommented, Jul 15, 2020

What actually works export NVCC="${CUDA_PATH}/bin/nvcc --compiler-bindir $LINUXBREWHOME/opt/gcc@9/bin" which seems like a bug in the build config because ${CUDA_PATH}/bin is actually in my $PATH and the build system should just look up nvcc in ${CUDA_PATH}/bin anyway and should not risk allowing the user to set $NVCC to a different version of CUDA than ${CUDA_PATH}. Also, that nvcc is the only one I have on the system and I don’t see how it would complain about missing header given all the path set up the way in the OP

1reaction

leofangcommented, Jul 24, 2020

btw @kmaehashi the title should be nvcc not nccl 😁

Top Results From Across the Web

ERROR: failed checking for nvcc. #46 - NVIDIA/cuda-samples

For people with problem with NVCC, try to check if CUDA is installed properly. Also, seems like nvcc is not in path, but...

CUDA Compiler Driver NVCC - NVIDIA Documentation Center

The CUDA compilation trajectory separates the device functions from the host code, compiles the device functions using the proprietary NVIDIA compilers and ...

Terminal trying to use a file that does not match which

What does type -a nvcc say? Possibly your shell has hashed a previous location of the command - see for example Pytest is...

CUDA: nvcc cannot be detected though installed

Actually, it's the case that I set environment variable in a wrong way. Instead of using $ export PATH=/usr/local/cuda-7.0/bin:$PATH $ export ...

Could not find a valid NVCC executable\n - Google Groups

The NVCC compiler 'C:\Program Files\MATLAB\R2019b\sys\cuda\win64\cuda\bin\nvcc.exe\bin\nvcc'. does not appear to be valid. Error in vl_compilenn (line 296).