Buffer Protocol Support in ChainerX ndarray

I want to use mpi4py allreduce in ChainerX ndarray in order to support use of MultiNodeEvaluator in ChainerX network model. However, ChainerX doesn’t support buffer protocol so mpi4py couldn’t handle ChainerX object.

(Actually, I found ChainerX.ndarray wants to support bufferprotocol (https://github.com/chainer/chainer/blob/master/chainerx_cc/chainerx/python/array.cc#L160), but there is no def_buffer)

You can try below simple script.

import chainerx as chx
import numpy as np

from mpi4py import MPI

def check_bp(array):
    try:
        memoryview(array)

        return True
    except BufferError:
        return False


if __name__ == '__main__':
    comm = MPI.COMM_WORLD
    print("MPI_WORLD_SIZE: ", comm.size)

    # NumPy
    rank_np = np.asarray([comm.rank])
    print("NumPy: ", rank_np)
    print("Buffer Protocol Support of NumPy: ", check_bp(rank_np))

    rank_np = comm.allreduce(rank_np)

    print("NumPy AllReduce: ", rank_np)

    # ChainerX
    rank_chx = chx.asarray([comm.rank])
    print("ChainerX: ", rank_chx)
    print("Buffer Protocol Support of ChainerX: ", check_bp(rank_chx))

    rank_chx = comm.allreduce(rank_chx)

    print("ChainerX AllReduce: ", rank_chx)

$ mpiexec -N 2 python test.py
MPI_WORLD_SIZE:  2
MPI_WORLD_SIZE:  2
NumPy:  [1]
Buffer Protocol Support of NumPy:  True
NumPy:  [0]
Buffer Protocol Support of NumPy:  True
NumPy AllReduce:  [1]
NumPy AllReduce:  [1]
ChainerX:  array([0], shape=(1,), dtype=int64, device='native:0')
ChainerX:  array([1], shape=(1,), dtype=int64, device='native:0')
Buffer Protocol Support of ChainerX:  False
Buffer Protocol Support of ChainerX:  False
Traceback (most recent call last):
  File "test.py", line 33, in <module>
    rank_chx = comm.allreduce(rank_chx)
  File "mpi4py/MPI/Comm.pyx", line 1289, in mpi4py.MPI.Comm.allreduce
  File "mpi4py/MPI/msgpickle.pxi", line 1216, in mpi4py.MPI.PyMPI_allreduce
  File "mpi4py/MPI/msgpickle.pxi", line 1168, in mpi4py.MPI.PyMPI_allreduce_intra
  File "mpi4py/MPI/msgpickle.pxi", line 1018, in mpi4py.MPI.PyMPI_reduce_p2p
  File "mpi4py/MPI/msgpickle.pxi", line 950, in mpi4py.MPI.PyMPI_copy
  File "mpi4py/MPI/msgpickle.pxi", line 104, in mpi4py.MPI.Pickle.dump
  File "mpi4py/MPI/msgpickle.pxi", line 91, in mpi4py.MPI.Pickle.cdumps
RuntimeError: return_value_policy = copy, but the object is non-copyable!
Traceback (most recent call last):
  File "test.py", line 33, in <module>
    rank_chx = comm.allreduce(rank_chx)
  File "mpi4py/MPI/Comm.pyx", line 1289, in mpi4py.MPI.Comm.allreduce
  File "mpi4py/MPI/msgpickle.pxi", line 1216, in mpi4py.MPI.PyMPI_allreduce
  File "mpi4py/MPI/msgpickle.pxi", line 1168, in mpi4py.MPI.PyMPI_allreduce_intra
  File "mpi4py/MPI/msgpickle.pxi", line 1018, in mpi4py.MPI.PyMPI_reduce_p2p
  File "mpi4py/MPI/msgpickle.pxi", line 950, in mpi4py.MPI.PyMPI_copy
  File "mpi4py/MPI/msgpickle.pxi", line 104, in mpi4py.MPI.Pickle.dump
  File "mpi4py/MPI/msgpickle.pxi", line 91, in mpi4py.MPI.Pickle.cdumps
RuntimeError: return_value_policy = copy, but the object is non-copyable!
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[3381,1],1]
  Exit code:    1
--------------------------------------------------------------------------

[keisukefukuda]

@leofang thanks for the info! We’ll consider using it.

[emcastillo]

Hello, We were just discussing this issue, we dropped BufferProtocol support due to conflicts with numpy array conversion. We thought that the most appropriate way is to work on the support for ChainerMN.

Thanks.