rpc_session.cc:151: Check failed: Can only TVMContext related to the same remote sesstion

When running a naive matrix multiplication demo based on CPU using cross-compilation, it occurs an error: rpc_session.cc:151: Check failed: dev_type / kRPCSessMask == rpc_sess_table_index_ + 1 (0 vs. 1) Can only TVMContext related to the same remote sesstion.

Env.

• local machine: Ubuntu 16.04 64bit
• remote device: Firefly RK3399, aarch64-linux-gnu

Code on Local Machine

from __future__ import absolute_import, print_function
import tvm 
import numpy as np
import numpy
from tvm.contrib import rpc, util

# The size of the square matrix
N = 1024
# The default tensor type in tvm
dtype = "float32"
# Random generated tensor for testing
a = tvm.nd.array(numpy.random.rand(N, N).astype(dtype), tvm.cpu(0))
b = tvm.nd.array(numpy.random.rand(N, N).astype(dtype), tvm.cpu(0))
c = tvm.nd.array(numpy.zeros(shape=[N,N], dtype=dtype), tvm.cpu(0))
# The expected answer
answer = numpy.dot(a.asnumpy(), b.asnumpy())

# Algorithm
k = tvm.reduce_axis((0, N), 'k')
A = tvm.placeholder((N, N), name = 'A')
B = tvm.placeholder((N, N), name = 'B')
C = tvm.compute(
           A.shape,
           lambda x, y: tvm.sum(A[x, k] * B[k, y], axis = k), 
           name = 'C')
# Default schedule

s = tvm.create_schedule(C.op)
func = tvm.build(s, [A, B, C], target='llvm -target=aarch64-linux-gnu -mcpu=cortex-a72 -mattr=+neon', name='mmult')
# save the lib at local temp folder
temp = util.tempdir()
path = temp.relpath('mylib.o')
func.save(path)

host = '192.168.1.56'
port = 9090
remote = rpc.connect(host, port)
remote.upload(path)
f = remote.load_module('mylib.o')

# create array on the remote device
ctx = remote.cpu(0)
a = tvm.nd.array(a.asnumpy().astype(dtype), ctx)
b = tvm.nd.array(b.asnumpy().astype(dtype), ctx)
c = tvm.nd.array(c.asnumpy().astype(dtype), ctx)
# the function will run on the remote device
f(a, b, c)
#np.testing.assert_equal(c.asnumpy(), answer)

assert f
evaluator = f.time_evaluator(func.entry_name, tvm.cpu(0), number = 1)
c = tvm.nd.array(numpy.zeros((N, N), dtype = dtype), tvm.cpu(0))
print('Baseline: %f' % evaluator(a, b, c).mean)

Logs on Local

yuens@Spark:~/Documents/charm/TVM_TRY$ python run_local.py 
[16:53:49] src/codegen/llvm/codegen_llvm.cc:76: set native vector to be 32 for target aarch64
[16:54:21] /home/yuens/Software/tvm/dmlc-core/include/dmlc/logging.h:308: [16:54:21] src/runtime/rpc/rpc_session.cc:151: Check failed: dev_type / kRPCSessMask == rpc_sess_table_index_ + 1 (0 vs. 1) Can only TVMContext related to the same remote sesstion

Stack trace returned 10 entries:
[bt] (0) /home/yuens/Software/tvm/lib/libtvm.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fc4b7c6296c]
[bt] (1) /home/yuens/Software/tvm/lib/libtvm.so(_ZN3tvm7runtime10RPCSession12EventHandler13SendPackedSeqEPK8TVMValuePKii+0xc26) [0x7fc4b8046ac6]
[bt] (2) /home/yuens/Software/tvm/lib/libtvm.so(+0xac79f3) [0x7fc4b80429f3]
[bt] (3) /home/yuens/Software/tvm/lib/libtvm.so(_ZN3tvm7runtime10RPCSession16GetTimeEvaluatorEPv9DLContexti+0xe6) [0x7fc4b8041176]
[bt] (4) /home/yuens/Software/tvm/lib/libtvm.so(+0xae3998) [0x7fc4b805e998]
[bt] (5) /home/yuens/Software/tvm/lib/libtvm.so(+0xae4924) [0x7fc4b805f924]
[bt] (6) /home/yuens/Software/tvm/lib/libtvm.so(TVMFuncCall+0x5e) [0x7fc4b80068ae]
[bt] (7) /usr/local/lib/anaconda2/lib/python2.7/lib-dynload/_ctypes.so(ffi_call_unix64+0x4c) [0x7fc4c299257c]
[bt] (8) /usr/local/lib/anaconda2/lib/python2.7/lib-dynload/_ctypes.so(ffi_call+0x1f5) [0x7fc4c2991cd5]
[bt] (9) /usr/local/lib/anaconda2/lib/python2.7/lib-dynload/_ctypes.so(_ctypes_callproc+0x3e6) [0x7fc4c2989376]

Traceback (most recent call last):
  File "run_local.py", line 54, in <module>
    evaluator = f.time_evaluator(func.entry_name, tvm.cpu(0), number = 1)
  File "/home/yuens/Software/tvm/python/tvm/module.py", line 133, in time_evaluator
    self, func_name, ctx.device_type, ctx.device_id, number)
  File "/home/yuens/Software/tvm/python/tvm/_ffi/function.py", line 255, in my_api_func
    return flocal(*args)
  File "/home/yuens/Software/tvm/python/tvm/_ffi/_ctypes/function.py", line 183, in __call__
    ctypes.byref(ret_val), ctypes.byref(ret_tcode)))
  File "/home/yuens/Software/tvm/python/tvm/_ffi/base.py", line 62, in check_call
    raise TVMError(py_str(_LIB.TVMGetLastError()))
tvm._ffi.base.TVMError: [16:54:21] src/runtime/rpc/rpc_session.cc:151: Check failed: dev_type / kRPCSessMask == rpc_sess_table_index_ + 1 (0 vs. 1) Can only TVMContext related to the same remote sesstion

Stack trace returned 10 entries:
[bt] (0) /home/yuens/Software/tvm/lib/libtvm.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fc4b7c6296c]
[bt] (1) /home/yuens/Software/tvm/lib/libtvm.so(_ZN3tvm7runtime10RPCSession12EventHandler13SendPackedSeqEPK8TVMValuePKii+0xc26) [0x7fc4b8046ac6]
[bt] (2) /home/yuens/Software/tvm/lib/libtvm.so(+0xac79f3) [0x7fc4b80429f3]
[bt] (3) /home/yuens/Software/tvm/lib/libtvm.so(_ZN3tvm7runtime10RPCSession16GetTimeEvaluatorEPv9DLContexti+0xe6) [0x7fc4b8041176]
[bt] (4) /home/yuens/Software/tvm/lib/libtvm.so(+0xae3998) [0x7fc4b805e998]
[bt] (5) /home/yuens/Software/tvm/lib/libtvm.so(+0xae4924) [0x7fc4b805f924]
[bt] (6) /home/yuens/Software/tvm/lib/libtvm.so(TVMFuncCall+0x5e) [0x7fc4b80068ae]
[bt] (7) /usr/local/lib/anaconda2/lib/python2.7/lib-dynload/_ctypes.so(ffi_call_unix64+0x4c) [0x7fc4c299257c]
[bt] (8) /usr/local/lib/anaconda2/lib/python2.7/lib-dynload/_ctypes.so(ffi_call+0x1f5) [0x7fc4c2991cd5]
[bt] (9) /usr/local/lib/anaconda2/lib/python2.7/lib-dynload/_ctypes.so(_ctypes_callproc+0x3e6) [0x7fc4c2989376]

Exception tvm._ffi.base.TVMError: TVMError('Except caught from RPC call: [08:54:21] include/tvm/runtime/./packed_func.h:190: Check failed: value_.v_int64 <= std::numeric_limits<int>::max() (12884901894 vs. 2147483647) \n\nStack trace returned 10 entries:\n[bt] (0) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN4dml
c15LogMessageFatalD1Ev+0x44) [0x7f811695b4]\n[bt] (1) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZNK3tvm7runtime12TVMPODValue_cviEv+0x968) [0x7f81179ed0]\n[bt] (2) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime19RPCGetTimeEvaluatorENS0_7TVMArgsEPNS0_11TVMRetValueE+0xa0) [0x7f811b9768]\n[bt] (3) /h
ome/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler11CallHandlerIPFvNS0_7TVMArgsEPNS0_11TVMRetValueEEEEvT_+0x84) [0x7f811bfcec]\n[bt] (4) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler16HandlePackedCallEv+0x570) [0x7f811ba2b8]\n[bt] (5) /home
/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler13SwitchToStateENS2_5StateE+0x2d4) [0x7f811c03ac]\n[bt] (6) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler22HandleRecvPackedSeqArgEv+0x5f4) [0x7f811c129c]\n[bt] (7) /home/yuanshuai/code/tvm/lib/
libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler15HandleNextEventEPNS0_11TVMRetValueEbPKNS0_10PackedFuncE+0x3a0) [0x7f811c1848]\n[bt] (8) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession22HandleUntilReturnEventEPNS0_11TVMRetValueEbPKNS0_10PackedFuncE+0x1f8) [0x7f811bb330]\n[bt] (9)
 /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession10ServerLoopEv+0x74) [0x7f811bb724]\n',) in <bound method NDArray.__del__ of <tvm.ndarray.NDArray object at 0x7fc4a28beb40>> ignored

Logs on Remote

yuanshuai@firefly:~/code/tvm-mobile-opencl/bandwidth$ ./start_server_on_device.sh 
Loading runtime library /home/yuanshuai/code/tvm/lib/libtvm_runtime.so... exec only
INFO:root:RPCServer: bind to 0.0.0.0:9090
INFO:root:RPCServer: connection from ('192.168.1.110', 35052)
INFO:root:Connection from ('192.168.1.110', 35052)
[08:53:49] src/runtime/rpc/rpc_server_env.cc:23: Upload /tmp/tmpP4n3co/mylib.o... nbytes=7912
INFO:root:Create shared library based on /tmp/tmpP4n3co/mylib.o
INFO:root:load_module /tmp/tmpP4n3co/mylib.o.so
[08:54:21] /home/yuanshuai/code/tvm/dmlc-core/include/dmlc/./logging.h:308: [08:54:21] include/tvm/runtime/./packed_func.h:190: Check failed: value_.v_int64 <= std::numeric_limits<int>::max() (12884901894 vs. 2147483647) 

Stack trace returned 10 entries:
[bt] (0) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN4dmlc15LogMessageFatalD1Ev+0x44) [0x7f811695b4]
[bt] (1) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZNK3tvm7runtime12TVMPODValue_cviEv+0x968) [0x7f81179ed0]
[bt] (2) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime19RPCGetTimeEvaluatorENS0_7TVMArgsEPNS0_11TVMRetValueE+0xa0) [0x7f811b9768]
[bt] (3) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler11CallHandlerIPFvNS0_7TVMArgsEPNS0_11TVMRetValueEEEEvT_+0x84) [0x7f811bfcec]
[bt] (4) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler16HandlePackedCallEv+0x570) [0x7f811ba2b8]
[bt] (5) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler13SwitchToStateENS2_5StateE+0x2d4) [0x7f811c03ac]
[bt] (6) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler22HandleRecvPackedSeqArgEv+0x5f4) [0x7f811c129c]
[bt] (7) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession12EventHandler15HandleNextEventEPNS0_11TVMRetValueEbPKNS0_10PackedFuncE+0x3a0) [0x7f811c1848]
[bt] (8) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession22HandleUntilReturnEventEPNS0_11TVMRetValueEbPKNS0_10PackedFuncE+0x1f8) [0x7f811bb330]
[bt] (9) /home/yuanshuai/code/tvm/lib/libtvm_runtime.so(_ZN3tvm7runtime10RPCSession10ServerLoopEv+0x74) [0x7f811bb724]

config.mk on Local

DEBUG = 1
USE_OPENCL = 1
USE_RPC = 1

LLVM_CONFIG = llvm-config

config.mk on Device

runtime configure as below:

DEBUG = 1
USE_OPENCL = 1
USE_RPC = 1

LLVM_CONFIG = llvm-config

[tqchen]

You passed a local ctx to a remote cpu in creating and running of time evaluator , pass remote.cpu instead

[ysh329]

fixed. So careless I am. 🤣

c = tvm.nd.array(numpy.zeros((N, N), dtype = dtype), remote.cpu(0))

[15:24:42] src/codegen/llvm/codegen_llvm.cc:76: set native vector to be 32 for target aarch64
Baseline: 34.950410