AttributeError: 'NoneType' object has no attribute 'record'

CuPy Version : 7.3.0 CUDA Root : /usr/local/cuda CUDA Build Version : 10020 CUDA Driver Version : 10020 CUDA Runtime Version : 10020 cuBLAS Version : 10202 cuFFT Version : 10102 cuRAND Version : 10102 cuSOLVER Version : (10, 3, 0) cuSPARSE Version : 10301 NVRTC Version : (10, 2) cuDNN Build Version : 7605 cuDNN Version : 7605 NCCL Build Version : 2406 NCCL Runtime Version : 2507

If I try to run the following command, I get the following error. cp.asarray(z[cp.newaxis, :, :])

Traceback (most recent call last):
  File "kalman.py", line 125, in <module>
    z = cp.asarray(z[cp.newaxis, :, :])
  File "/home/belt/anaconda3/envs/cusignal/lib/python3.8/site-packages/cupy/creation/from_data.py", line 68, in asarray
    return core.array(a, dtype, False, order)
  File "cupy/core/core.pyx", line 1785, in cupy.core.core.array
  File "cupy/core/core.pyx", line 1862, in cupy.core.core.array
  File "cupy/core/core.pyx", line 1950, in cupy.core.core._send_object_to_gpu
AttributeError: 'NoneType' object has no attribute 'record'

What’s interesting is that it only happens after a run a CuPy kernel. One which doesn’t even access variable z.

[takagi]

I mean CuPy’s Stream objects at the following lines:

https://github.com/mnicely/cusignal/blob/a366bb66fb56357d98e77276ba9d031c347febb8/python/cusignal/estimation/filters.py#L150

https://github.com/mnicely/cusignal/blob/a366bb66fb56357d98e77276ba9d031c347febb8/python/cusignal/estimation/filters.py#L160

but, sorry, I misunderstood that these objects are actually referenced by _cupy_predict_wrapper and _cupy_update_wrapper so it’s not the cause.

The actual cause is that, in a previous loop, a KalmanFilter instance’s CuPy Stream is left being used and, in the next loop, it is collected when another KalmanFilter instance is assigned, eventually the CuPy current stream becomes None with a dead weak reference.

I’ve confirmed that with statement for temporary Stream use eliminates the error:

diff --git a/python/cusignal/estimation/_filters.py b/python/cusignal/estimation/_filters.py
index 3686d68..0fbad8b 100644
--- a/python/cusignal/estimation/_filters.py
+++ b/python/cusignal/estimation/_filters.py
@@ -676,8 +676,8 @@ class _cupy_predict_wrapper(object):

         kernel_args = (x.shape[0], alpha_sq, x, F, P, Q)

-        self.stream.use()
-        self.kernel(self.grid, self.block, kernel_args)
+        with self.stream:
+            self.kernel(self.grid, self.block, kernel_args)


 class _cupy_update_wrapper(object):
@@ -696,8 +696,8 @@ class _cupy_update_wrapper(object):

         kernel_args = (x.shape[0], x, z, H, P, R)

-        self.stream.use()
-        self.kernel(self.grid, self.block, kernel_args)
+        with self.stream:
+            self.kernel(self.grid, self.block, kernel_args)


 def _get_backend_kernel(dtype, grid, block, smem, stream, use_numba, k_type):

[mnicely]

@takagi I believe I made the appropriate changes. Can you try now?