Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Segfault when copying a non-contiguous CuPy array to host

See original GitHub issue

>>> import cupy as cp
>>>
>>> ptr = cp.cuda.alloc(2**27 * 8)
>>> a = cp.ndarray(shape=(2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2),\
...                strides=(128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072, 262144, 524288, 1048576, 2097152, 4194304, 8388608, 16777216, 33554432, 67108864, 134217728, 536870912, 268435456, 8, 16, 32, 64),\
...                memptr=ptr)
>>> 
>>> b = cp.asnumpy(a)
Segmentation fault (core dumped)

It either segfaults hard, or causes CUDA_ERROR_MISALIGNED_ADDRESS in a real workload:

Traceback (most recent call last):
  File "<string>", line 23, in <module>
  File "/home/leof/miniforge3/envs/opt_einsum_dev/lib/python3.8/site-packages/cupy/__init__.py", line 773, in asnumpy
    return a.get(stream=stream, order=order)
  File "cupy/_core/core.pyx", line 1567, in cupy._core.core.ndarray.get
  File "cupy/_core/core.pyx", line 1623, in cupy._core.core.ndarray.get
  File "cupy/_core/core.pyx", line 1625, in cupy._core.core.ndarray.get
  File "cupy/_core/core.pyx", line 2404, in cupy._core.core._internal_ascontiguousarray
  File "cupy/_core/_kernel.pyx", line 1185, in cupy._core._kernel.ufunc.__call__
  File "cupy/_core/_kernel.pyx", line 1212, in cupy._core._kernel.ufunc._get_ufunc_kernel
  File "cupy/_core/_kernel.pyx", line 952, in cupy._core._kernel._get_ufunc_kernel
  File "cupy/_core/_kernel.pyx", line 62, in cupy._core._kernel._get_simple_elementwise_kernel
  File "cupy/_core/core.pyx", line 2030, in cupy._core.core.compile_with_cache
  File "/home/leof/miniforge3/envs/opt_einsum_dev/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 460, in compile_with_cache
    return _compile_with_cache_cuda(
  File "/home/leof/miniforge3/envs/opt_einsum_dev/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 529, in _compile_with_cache_cuda
    mod.load(cubin)
  File "cupy/cuda/function.pyx", line 241, in cupy.cuda.function.Module.load
  File "cupy/cuda/function.pyx", line 243, in cupy.cuda.function.Module.load
  File "cupy_backends/cuda/api/driver.pyx", line 253, in cupy_backends.cuda.api.driver.moduleLoadData
  File "cupy_backends/cuda/api/driver.pyx", line 125, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_MISALIGNED_ADDRESS: misaligned address
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 260, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 125, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_MISALIGNED_ADDRESS: misaligned address
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
(... repeat the above traceback N times ...)

By the way, technically this array is illegal in CuPy (a.ndim=27 > MAX_NDIM=25, see #4193), but I am not sure if this is relevant.

Issue Analytics

State:
Created 2 years ago
Reactions:1
Comments:5 (5 by maintainers)

Top GitHub Comments

2reactions

leofangcommented, Sep 9, 2021

Hi @emcastillo, could you reopen and assign this issue to me (as I can’t reopen)? I’d like to add a safe check so that we don’t accidentally compile a copy kernel that can’t do its job correctly.

1reaction

emcastillocommented, Sep 9, 2021

Since this seems to be a duplicate, let’s close this one. Feel free to reopen if you think it is not 😃