setup.py seems to be expecting GCC

Description

When setting CC and CXX to non-gcc compilers – eg nvcc – the setup.py build breaks because it is passing GCC-only flags to CC. For example this works on NERSC’s Perlmutter (https://docs.nersc.gov/systems/perlmutter/) system:

blaschke@muller:login02:~> LDFLAGS="$CRAY_CUDATOOLKIT_POST_LINK_OPTS" NVCC="nvcc $CRAY_CUDATOOLKIT_INCLUDE_OPTS" CC=cc CXX=CC pip install cupy

when loading:

module load PrgEnv-gnu

as this will set cc=gcc and CC=g++. However loading:

module load PrgEnv-ndivida

will set cc=nvcc and CC=nvc++. And the command above will break because the -fwrapv flag is passed to cc (i.e. nvcc) which doesn’t know what to do with the flag:

    -------- Configuring Module: cuda --------
    nvc-Error-Unknown switch: -fwrapv
    command 'cc' failed with exit status 1
    nvc-Error-Unknown switch: -fwrapv
    **************************************************
    *** WARNING: Cannot check compute capability
    command 'cc' failed with exit status 1
    **************************************************

    ************************************************************
    * CuPy Configuration Summary                               *
    ************************************************************

    Build Environment:
      Include directories: ['/tmp/pip-install-io1hkbby/cupy_93b134b374a5423fa60a2e4e1263db36/cupy/_core/include/cupy/cub', '/tmp/pip-install-io1hkbby/cupy_93b134b374a5423fa60a2e4e1263db36/cupy/_core/include', '/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/compilers/include']
      Library directories: ['/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/compilers/lib']
      nvcc command       : ['nvc', '-I/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/include', '-I/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/nvvm/include', '-I/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/extras/Debugger/include', '-I/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/extras/CUPTI/include', '-I/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/math_libs/11.5/include']
      hipcc command      : (not found)

    Environment Variables:
      CFLAGS          : (none)
      LDFLAGS         : -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/nvvm/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/extras/Debugger/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/extras/CUPTI/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/math_libs/11.5/lib64 -Wl,--as-needed,-lcupti,-lcudart,--no-as-needed -lcuda
      LIBRARY_PATH    : (none)
      CUDA_PATH       : (none)
      NVTOOLSEXT_PATH : (none)
      NVCC            : nvc -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/nvvm/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/extras/Debugger/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/extras/CUPTI/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/math_libs/11.5/include
      HIPCC           : (none)
      ROCM_HOME       : (none)

    Modules:
      cuda      : No
        -> Include files not found: ['cublas_v2.h', 'cuda.h', 'cuda_profiler_api.h', 'cuda_runtime.h', 'cufft.h', 'curand.h', 'cusparse.h', 'nvrtc.h']
        -> Check your CFLAGS environment variable.

    ERROR: CUDA could not be found on your system.

    HINT: You are trying to build CuPy from source, which is NOT recommended for general use.
          Please consider using binary packages instead.

    Please refer to the Installation Guide for details:
    https://docs.cupy.dev/en/stable/install.html

    ************************************************************

To Reproduce

Log into NERSC’s Perlmutter system.

• This works:

user@perlmutter:login02:~> module load PrgEnv-gnu
user@perlmutter:login02:~> module load cudatoolkit
user@perlmutter:login02:~> module load python
user@perlmutter:login02:~> LDFLAGS="$CRAY_CUDATOOLKIT_POST_LINK_OPTS" NVCC="nvcc $CRAY_CUDATOOLKIT_INCLUDE_OPTS" CC=cc CXX=CC pip install cupy --user

• This doesn’t work:

user@perlmutter:login02:~> module load PrgEnv-nvidia
user@perlmutter:login02:~> module load cudatoolkit
user@perlmutter:login02:~> module load python
user@perlmutter:login02:~> LDFLAGS="$CRAY_CUDATOOLKIT_POST_LINK_OPTS" NVCC="nvcc $CRAY_CUDATOOLKIT_INCLUDE_OPTS" CC=cc CXX=CC pip install cupy --user

Installation

Source (pip install cupy)

Environment

N/A – this is a compile problem

Additional Information

No response

[emcastillo]

This is a problem of distutils where the flags used to compile python are unconditionally set to any UNIX compiler.

https://github.com/pypa/setuptools/blob/main/setuptools/_distutils/sysconfig.py#L219-L221

Also we call this from: https://github.com/cupy/cupy/blob/master/install/cupy_builder/cupy_setup_build.py#L487-L489

The problem here is that you won’t be able to override the CFLAGS passed to the compiler without calling mock or probably creating your own compiler definition for setuptools which I am not sure if it would be able to cleanly skip the sysconfig setting.

The only immediate way that comes to my mind that can cleanly solve this is to compile Python with nvc. However, I am not sure if that would even work or create other problems for users. But I think it is worth a try.

Thanks for the report! this is an interesting behavior that we were not fully aware. It would be great if we can find a nice fix for this so lets leave this open for discussion.

[JBlaschke]

@leofang For completeness:

Also, the Perlmutter instruction recommends to install CuPy wheel from PyPI: https://docs.nersc.gov/development/languages/python/using-python-perlmutter/#cupy

I like recommending that users try the wheels first, but it’s also important to me that we document how to build python dependencies using the vendor compilers wherever this is feasible. One situation where this might matter is when some python modules need to be compiled with specific vendor compilers, or linked against specific cuda versions. In this situation it is nice to know how all dependencies can be compiled using a single consistent toolchain.

Maybe opening a ticket with the admin for consultation? 🙂

HA! That would likely land on mine, or @dmargala 's or @lastephey 's desk 😜