Development pip install issue

Hi I am trying to use the command

pip install -e .

to install a local copy of cupy for my feature #3569 in my conda env (called phox). As recently as a couple of weeks ago, this command worked. And I am currently able to install cupy using pip install cupy-cuda102==8.0.0b4.

But now I am getting the following error:

Complete output (54 lines):
Options: {'package_name': 'cupy', 'long_description': None, 'wheel_libs': [], 'wheel_includes': [], 'no_rpath': False, 'profile': False, 'linetrace': False, 'annotate': False, 'no_cuda': False, 'use_hip': False}
    **************************************************
    *** WARNING: nvcc path != CUDA_PATH
    *** WARNING: nvcc path: None
    *** WARNING: CUDA_PATH: /home/exx/miniconda3/envs/phox
    **************************************************

    -------- Configuring Module: cuda --------
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    /tmp/tmpsmnaadct/a.cpp:1:10: fatal error: cublas_v2.h: No such file or directory
     #include <cublas_v2.h>
              ^~~~~~~~~~~~~
    compilation terminated.
    command 'gcc' failed with exit status 1

    ************************************************************
    * CuPy Configuration Summary                               *
    ************************************************************

    Build Environment:
      Include directories: ['/home/exx/Documents/research/cupy/install/../cupy/core/include/cupy/cub', '/home/exx/Documents/research/cupy/install/../cupy/core/include', '/home/exx/miniconda3/envs/phox/include']
      Library directories: ['/home/exx/miniconda3/envs/phox/lib']
      nvcc command       : (not found)

    Environment Variables:
      CFLAGS          : (none)
      LDFLAGS         : (none)
      LIBRARY_PATH    : /opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64_lin/gcc4.7:/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64_lin
      CUDA_PATH       : /home/exx/miniconda3/envs/phox
      NVTOOLSEXT_PATH : (none)
      NVCC            : (none)
      ROCM_HOME       : (none)
      CUPY_CUB_PATH   : (none)

    Modules:
      cuda      : No
        -> Include files not found: ['cublas_v2.h', 'cuda.h', 'cuda_profiler_api.h', 'cuda_runtime.h', 'cufft.h', 'curand.h', 'cusparse.h', 'nvrtc.h']
        -> Check your CFLAGS environment variable.

    ERROR: CUDA could not be found on your system.
    Please refer to the Installation Guide for details:
    https://docs.cupy.dev/en/stable/install.html

    ************************************************************

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/exx/Documents/research/cupy/setup.py", line 142, in <module>
        ext_modules = cupy_setup_build.get_ext_modules()
      File "/home/exx/Documents/research/cupy/cupy_setup_build.py", line 765, in get_ext_modules
        extensions = make_extensions(arg_options, compiler, use_cython)
      File "/home/exx/Documents/research/cupy/cupy_setup_build.py", line 513, in make_extensions
        raise Exception('Your CUDA environment is invalid. '
    Exception: Your CUDA environment is invalid. Please check above error log.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Any reason for why this might be the case now (and not in the past?). I never went in and deleted any helper files. I did install pytorch with cudatoolkit just prior to this bug so perhaps that messed things up? Not sure…

[leofang]

I think previously something likely messed up your PATH and/or LD_LIBRARY_PATH and/or other compiler flags.

Another thing is cudatoolkit from conda-forge only comes with CUDA libraries (runtime, cuFFT, cuSPARSE, cuBLAS, etc). Neither nvcc nor CUDA headers (which are not distributable according to CUDA’s EULA, so none of the packages could provide them!) comes with cudatoolkit.

The third thing is if you ever install cupy from conda-forge, we provide the pre- and post-link scripts that point CUDA_PATH to $CONDA_PREFIX/lib. But in a few occasions I noticed after removing the cupy package, those scripts are not removed. Not sure if this depends on the conda version, though.

For myself, I routinely use conda, but for CuPy development I purposely do not use cudatoolkit or any CUDA related stuff from conda-forge, so as to ensure I always use my system’s CUDA.

[sunilkpai]

No worries, LDFLAGS was the main culprit, I think. The symbolic links and libraries were actually part of my cudatoolkit env installation so they were in my environment. I am still very confused as to why this worked before, but the silver lining is that I understand how to deal with these issues now.

My solution therefore was to run:

export LDFLAGS="-L<path-to-env>/lib/"

Generally I just don’t know when to use /usr/local/cuda-10.2/ and when to use env paths. I thought it would always be fine to use the env path (I assumed maybe incorrectly that cudatoolkit took care of this for me), but it seems like in my case nvcc wasn’t found there