Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

hip runtime compilation bug in simple cupy code

See original GitHub issue

Hello, I’m trying to use cupy with AMD MI50 GPUs and I find that there is a bug in the hip runtime compilation code. Running this simple code piece will get these error messages:

import numpy as np
import cupy as cp

squared_diff = cp.ElementwiseKernel(
    'float32 x, float32 y',
    'float32 z',
    'z = (x - y) * (x - y)',

x = cp.arange(10, dtype=np.float32).reshape(2, 5)
y = cp.arange(5, dtype=np.float32)
squared_diff(x, y)

Traceback (most recent call last):
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/", line 512, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 108, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 122, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 58, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: HIPRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "", line 10, in <module>
    x = cp.arange(10, dtype=np.float32).reshape(2, 5)
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/_creation/", line 57, in arange
    _arange_ufunc(typ(start), typ(step), ret, dtype=dtype)
  File "cupy/core/_kernel.pyx", line 1084, in cupy.core._kernel.ufunc.__call__
  File "cupy/core/_kernel.pyx", line 1107, in cupy.core._kernel.ufunc._get_ufunc_kernel
  File "cupy/core/_kernel.pyx", line 881, in cupy.core._kernel._get_ufunc_kernel
  File "cupy/core/_kernel.pyx", line 60, in cupy.core._kernel._get_simple_elementwise_kernel
  File "cupy/core/core.pyx", line 1921, in cupy.core.core.compile_with_cache
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/", line 324, in compile_with_cache
    return _compile_with_cache_hip(
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/", line 662, in _compile_with_cache_hip
    binary = compile_using_nvrtc(source, options, arch, name + '.cu')
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/", line 163, in compile_using_nvrtc
    return _compile(source, options, cu_path,
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/", line 144, in _compile
    prog = _NVRTCProgram(source, cu_path,
  File "/home/.conda/envs/cupy/lib/python3.8/site-packages/cupy/cuda/", line 523, in compile
    raise CompileException(log, self.src,, options, 'nvrtc')
cupy.cuda.compiler.CompileException: /tmp/comgr-2f0b69/input/CompileSource:2:10: fatal error: 'cupy/complex.cuh' file not found
#include <cupy/complex.cuh>
1 error generated when compiling for gfx906.
Error: Failed to compile opencl source (from CL or HIP source to LLVM IR).

I’ve already found that it’s the hiprtcCompileProgram API in cupy/cupy_backends/cuda/libs/cupy_hiprtc.h that cause this problem and I have submitted an issue to the HIP project I’m curious if anyone else encountered this problem.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:9 (5 by maintainers)

github_iconTop GitHub Comments

ipe-zhangyzcommented, Nov 4, 2020

Thank you! @leofang Now all of my test cases are passed.

ipe-zhangyzcommented, Nov 3, 2020

Thank you @leofang ! The version I installed before is v8.0.0rc1. I installed it on a server with CentOS 7.6 and ROCM 3.5 from source in this steps:

$ cd cupy
$ export HCC_AMDGPU_TARGET=gfx906
$ export __HIP_PLATFORM_HCC__
$ export ROCM_HOME=/opt/rocm/rocm-3.5.0
$ pip install .

I’ve switched to v8.0.0 and reinstalled it in the same steps. My test code can run now.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Installation — CuPy 11.4.0 documentation
pip fails to install CuPy# · Installing cuDNN and NCCL# · Working with Custom CUDA Installation# · CuPy always raises cupy.cuda.compiler.CompileException # ·...
Read more >
CuPy Documentation - Read the Docs
CuPy is a NumPy/SciPy-compatible array library for GPU-accelerated computing with Python. CuPy acts as a drop-in.
Read more >
Programming with HIP - AMD Documentation - Portal
HIP supports runtime compilation (hipRTC), the usage of which provides the possibility of optimizations and performance improvement compared ...
Read more >
Question on MAGMA's gemm implementation - Google Groups
In CuPy, we have a special code path for integer GEMM in which we simply took MAGMA's ... The kernel compilation is done...
Read more >
Summit User Guide - OLCF User Documentation
The basic building block of Summit is the IBM Power System AC922 node. ... This is the place to write/edit/compile your code, manage...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found