Support peer access

import cupy as cp


with cp.cuda.Device(0):
    d0 = cp.arange(10)
with cp.cuda.Device(1):
    # Enable dereferencing pointer of device 0 in the current device.
    # otherwise you will get `cudaErrorIllegalAddress: an illegal memory access was encountered`
    cp.cuda.runtime.deviceEnablePeerAccess(0)
    d1 = d0 * 2

print(d1)

Currently, this code fails with the error Array device must be same as the current device: array device = 0 while current = 1, but it should work. Maybe changing the error to the warning?

ref: https://stackoverflow.com/questions/68074985/read-write-a-cupy-array-on-multiple-gpus

Note: deviceEnablePeerAccess may not work with all combinations of GPUs.

[kmaehashi]

@MarwanAbdelatti I’m trying to get this into the next v10 release. The PR is not checked into master branch yet, but you can try my branch: https://github.com/cupy/cupy/pull/5496

[kmaehashi]

@MarwanAbdelatti Thanks for sharing the configuration in DGX! I’m thinking of:

• Check if peer access is available between two devices using cudaDeviceCanAccessPeer
• Enabling it only when peer access is necessary