hip runtime compilation bug in simple cupy code

Hello, I’m trying to use cupy with AMD MI50 GPUs and I find that there is a bug in the hip runtime compilation code. Running this simple code piece will get these error messages:

import numpy as np
import cupy as cp

squared_diff = cp.ElementwiseKernel(
    'float32 x, float32 y',
    'float32 z',
    'z = (x - y) * (x - y)',
    'squared_diff')

x = cp.arange(10, dtype=np.float32).reshape(2, 5)
y = cp.arange(5, dtype=np.float32)
squared_diff(x, y)

Traceback (most recent call last):
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 512, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 108, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 122, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 58, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: HIPRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "111.py", line 10, in <module>
    x = cp.arange(10, dtype=np.float32).reshape(2, 5)
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/_creation/ranges.py", line 57, in arange
    _arange_ufunc(typ(start), typ(step), ret, dtype=dtype)
  File "cupy/core/_kernel.pyx", line 1084, in cupy.core._kernel.ufunc.__call__
  File "cupy/core/_kernel.pyx", line 1107, in cupy.core._kernel.ufunc._get_ufunc_kernel
  File "cupy/core/_kernel.pyx", line 881, in cupy.core._kernel._get_ufunc_kernel
  File "cupy/core/_kernel.pyx", line 60, in cupy.core._kernel._get_simple_elementwise_kernel
  File "cupy/core/core.pyx", line 1921, in cupy.core.core.compile_with_cache
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 324, in compile_with_cache
    return _compile_with_cache_hip(
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 662, in _compile_with_cache_hip
    binary = compile_using_nvrtc(source, options, arch, name + '.cu')
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 163, in compile_using_nvrtc
    return _compile(source, options, cu_path,
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 144, in _compile
    prog = _NVRTCProgram(source, cu_path,
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 523, in compile
    raise CompileException(log, self.src, self.name, options, 'nvrtc')
cupy.cuda.compiler.CompileException: /tmp/comgr-2f0b69/input/CompileSource:2:10: fatal error: 'cupy/complex.cuh' file not found
#include <cupy/complex.cuh>
         ^~~~~~~~~~~~~~~~~~
1 error generated when compiling for gfx906.
Error: Failed to compile opencl source (from CL or HIP source to LLVM IR).

I’ve already found that it’s the hiprtcCompileProgram API in cupy/cupy_backends/cuda/libs/cupy_hiprtc.h that cause this problem and I have submitted an issue to the HIP project https://github.com/ROCm-Developer-Tools/HIP/issues/2182#issue-731199084. I’m curious if anyone else encountered this problem.

[ipe-zhangyz]

Thank you! @leofang Now all of my test cases are passed.

[ipe-zhangyz]

Thank you @leofang ! The version I installed before is v8.0.0rc1. I installed it on a server with CentOS 7.6 and ROCM 3.5 from source in this steps:

$ cd cupy
$ export HCC_AMDGPU_TARGET=gfx906
$ export __HIP_PLATFORM_HCC__
$ export CUPY_INSTALL_USE_HIP=1
$ export ROCM_HOME=/opt/rocm/rocm-3.5.0
$ pip install .

I’ve switched to v8.0.0 and reinstalled it in the same steps. My test code can run now.