Support CUDA 11.3

Tasks

• Read CUDA Release Notes.
• Update CUDA driver in Linux (Jenkins) / Windows (FlexCI) test infrastructure (ask @kmaehashi). -> ongoing
• Update chainer-test to test the CUDA version. https://github.com/chainer/chainer-test/pull/658
• Update compute capability support.
  • Check NVRTC Reference (–gpu-architecture option) and change the maximum value of supported compute capability defined in cupy/cuda/compiler.py (_nvrtc_max_compute_capability) if needed. -> no changes needed
  • Check NVCC Reference (–generate-code) and change the list of architectures in cupy_setup_build.py (_arch_list) if needed. -> no changes needed
  • Backport. -> not needed
• Update bundled FP16 header files.
  • Copy FP16 header files to cupy/core/include/cupy/_cuda/, then update cupy/core/core.pyx to use it. https://github.com/cupy/cupy/pull/5218
  • Backport. #5232
• Update library installer.
  • Fix cupyx/tools/install_library.py. https://github.com/cupy/cupy/pull/5219
  • Backport. #5235
• Wheel distribution -> https://github.com/cupy/cupy-release-tools/pull/111
  • Update cupy-release-tools. https://github.com/cupy/cupy-release-tools/pull/111
  • Ask PyPI team to increase the limit.
• Documentation
  • Add new wheel package to Installtion Guide and README.md. Update requirements in the installation guide. https://github.com/cupy/cupy/pull/5220
  • Backport.
  • Add new wheel package to the website. https://github.com/cupy/cupy.dev/pull/45
• Implement new features using the version, if any.

Blocked by https://github.com/cupy/cupy/issues/5105

[leofang]

FYI: Using driver from 11.2 and runtime from 11.3 is not going to work:

$ python -c "import cupy; cupy.show_config()"
OS                           : Linux-5.4.0-42-generic-x86_64-with-glibc2.10
CuPy Version                 : 9.0.0rc1
NumPy Version                : 1.20.2
SciPy Version                : 1.6.3
Cython Build Version         : 0.29.23
Cython Runtime Version       : 0.29.23
CUDA Root                    : /usr/local/cuda-11.3
CUDA Build Version           : 11030
CUDA Driver Version          : 11020
CUDA Runtime Version         : 11030
cuBLAS Version               : 11402
cuFFT Version                : 10402
cuRAND Version               : 10204
cuSOLVER Version             : (11, 1, 1)
cuSPARSE Version             : 11500
NVRTC Version                : (11, 3)
Thrust Version               : 101100
CUB Build Version            : 101100
Jitify Build Version         : 60e9e72
cuDNN Build Version          : None
cuDNN Version                : None
NCCL Build Version           : 2507
NCCL Runtime Version         : 2507
cuTENSOR Version             : None
cuSPARSELt Build Version     : None
Device 0 Name                : GeForce RTX 2080 Ti
Device 0 Compute Capability  : 75
Device 0 PCI Bus ID          : 0000:1A:00.0

Error 1:

$ pytest tests/cupy_tests/core_tests/test_raw.py -x
========================================================================= test session starts =========================================================================
platform linux -- Python 3.8.8, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/leofang/dev/cupy_cuda113, configfile: setup.cfg
collected 231 items                                                                                                                                                   

tests/cupy_tests/core_tests/test_raw.py F

============================================================================== FAILURES ===============================================================================
____________________________________________________ TestRaw_param_0_{backend='nvrtc', in_memory=False}.test_basic ____________________________________________________

self = <<cupy_tests.core_tests.test_raw.TestRaw_param_0_{backend='nvrtc', in_memory=False} testMethod=test_basic>  parameter: {'backend': 'nvrtc', 'in_memory': False}>

    def test_basic(self):
>       x1, x2, y = self._helper(self.kern, cupy.float32)

tests/cupy_tests/core_tests/test_raw.py:453: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/cupy_tests/core_tests/test_raw.py:446: in _helper
    x1 = cupy.arange(N**2, dtype=dtype).reshape(N, N)
cupy/_creation/ranges.py:57: in arange
    _arange_ufunc(typ(start), typ(step), ret, dtype=dtype)
cupy/_core/_kernel.pyx:1104: in cupy._core._kernel.ufunc.__call__
    kern = self._get_ufunc_kernel(dev_id, op, arginfos)
cupy/_core/_kernel.pyx:1127: in cupy._core._kernel.ufunc._get_ufunc_kernel
    kern = _get_ufunc_kernel(
cupy/_core/_kernel.pyx:901: in cupy._core._kernel._get_ufunc_kernel
    return _get_simple_elementwise_kernel(
cupy/_core/_kernel.pyx:61: in cupy._core._kernel._get_simple_elementwise_kernel
    module = compile_with_cache(module_code, options)
cupy/_core/core.pyx:1999: in cupy._core.core.compile_with_cache
    return cuda.compile_with_cache(
cupy/cuda/compiler.py:432: in compile_with_cache
    return _compile_with_cache_cuda(
cupy/cuda/compiler.py:553: in _compile_with_cache_cuda
    mod.load(cubin)
cupy/cuda/function.pyx:264: in cupy.cuda.function.Module.load
    cpdef load(self, bytes cubin):
cupy/cuda/function.pyx:266: in cupy.cuda.function.Module.load
    self.ptr = driver.moduleLoadData(cubin)
cupy_backends/cuda/api/driver.pyx:253: in cupy_backends.cuda.api.driver.moduleLoadData
    check_status(status)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   raise CUDADriverError(status)
E   cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_UNSUPPORTED_PTX_VERSION: the provided PTX was compiled with an unsupported toolchain.

cupy_backends/cuda/api/driver.pyx:125: CUDADriverError
======================================================================= short test summary info =======================================================================
FAILED tests/cupy_tests/core_tests/test_raw.py::TestRaw_param_0_{backend='nvrtc', in_memory=False}::test_basic - cupy_backends.cuda.api.driver.CUDADriverError: CUDA...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================================== 1 failed in 0.54s ==========================================================================

Error 2:

>>> import cupy as cp
>>> a = cp.random.random(10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/leofang/dev/cupy_cuda113/cupy/random/_sample.py", line 156, in random_sample
    return rs.random_sample(size=size, dtype=dtype)
  File "/home/leofang/dev/cupy_cuda113/cupy/random/_generator.py", line 618, in random_sample
    out = self._random_sample_raw(size, dtype)
  File "/home/leofang/dev/cupy_cuda113/cupy/random/_generator.py", line 605, in _random_sample_raw
    func(self._generator, out.data.ptr, out.size)
  File "cupy_backends/cuda/libs/curand.pyx", line 163, in cupy_backends.cuda.libs.curand.generateUniformDouble
    cpdef generateUniformDouble(size_t generator, size_t outputPtr, size_t num):
  File "cupy_backends/cuda/libs/curand.pyx", line 167, in cupy_backends.cuda.libs.curand.generateUniformDouble
    check_status(status)
  File "cupy_backends/cuda/libs/curand.pyx", line 86, in cupy_backends.cuda.libs.curand.check_status
    raise CURANDError(status)
cupy_backends.cuda.libs.curand.CURANDError: CURAND_STATUS_LAUNCH_FAILURE

[leofang]

I think we hard-wired CUDA_VERSION in a few places (say driver.get_build_version()). We may need to verify if building on CUDA 11.2 and running on 11.3 is OK (I guess so).