Cannot determine the device from pointer using `cudaPointerGetAttributes` if device mismatch

It seems cudaPointerGetAttributes (and cuPointerGetAttributes) is no longer working in CUDA 10.1+ if the device the pointer resides and the current device do not match. It was working in CUDA 10.0.

import cupy
a = cupy.arange(10)  # array on GPU 0
cupy.cuda.Device(0).use()
cupy.cuda.runtime.pointerGetAttributes(a.data.ptr)
cupy.cuda.Device(1).use()
cupy.cuda.runtime.pointerGetAttributes(a.data.ptr) # raises CUDARuntimeError

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cupy_backends/cuda/api/runtime.pyx", line 601, in cupy_backends.cuda.api.runtime.pointerGetAttributes
    cpdef PointerAttributes pointerGetAttributes(intptr_t ptr):
  File "cupy_backends/cuda/api/runtime.pyx", line 604, in cupy_backends.cuda.api.runtime.pointerGetAttributes
    check_status(status)
  File "cupy_backends/cuda/api/runtime.pyx", line 246, in cupy_backends.cuda.api.runtime.check_status
    raise CUDARuntimeError(status)
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorInvalidValue: invalid argument

We rely on cudaPointerGetAttributes to determine the device for foreign pointers, e.g., __cuda_array_interface__ which does not contain the device number. https://github.com/cupy/cupy/blob/v8.0.0rc1/cupy/cuda/memory.pyx#L123

[anaruse]

This issue is related to timing of CUDA context creation. If you allocate memory on “current” GPU before calling cudaPointGetAttributes(), this issue can be avoided.

import cupy
a = cupy.arange(10)  # array on GPU 0
cupy.cuda.Device(0).use()
cupy.cuda.runtime.pointerGetAttributes(a.data.ptr)
cupy.cuda.Device(1).use()
b = cupy.arange(1)  # work-around
cupy.cuda.runtime.pointerGetAttributes(a.data.ptr)

[anaruse]

We’re checking with the CUDA team, so please wait for a while.