CUB-related test failures

• Conditions CuPy Version : 7.0.0rc1 CUDA Root : /usr/local/cuda CUDA Build Version : 10010 CUDA Driver Version : 10010 CUDA Runtime Version : 10010 cuDNN Build Version : 7402 cuDNN Version : 7402 NCCL Build Version : None NCCL Runtime Version : None

• Code to reproduce run pytest with / without CUB enabled

• Error messages, stack traces, or logs

The following test cases give me an error when running pytest with CUB_DISABALED=0, but pass with CUB_DISABLED=1. I haven’t looked into specific cases, but based on the test names I would guess it is a combination of a few possible things:

1.) reduction in the presence of NaN values 2.) reduction of zero-sized arrays 3.) errors in some axis-specific reduction cases

 - tests/cupy_tests/core_tests/test_ndarray_reduction.py:39 TestArrayReduction.test_max_nan
 - tests/cupy_tests/core_tests/test_ndarray_reduction.py:51 TestArrayReduction.test_max_nan_imag
 - tests/cupy_tests/core_tests/test_ndarray_reduction.py:45 TestArrayReduction.test_max_nan_real
 - tests/cupy_tests/core_tests/test_ndarray_reduction.py:87 TestArrayReduction.test_min_nan
 - tests/cupy_tests/core_tests/test_ndarray_reduction.py:99 TestArrayReduction.test_min_nan_imag
 - tests/cupy_tests/core_tests/test_ndarray_reduction.py:93 TestArrayReduction.test_min_nan_real
 - tests/cupy_tests/linalg_tests/test_einsum.py:279 TestEinSumUnaryOperation_param_20.test_einsum_unary
 - tests/cupy_tests/linalg_tests/test_einsum.py:297 TestEinSumUnaryOperation_param_20.test_einsum_unary_dtype
 - tests/cupy_tests/linalg_tests/test_einsum.py:279 TestEinSumUnaryOperation_param_49.test_einsum_unary
 - tests/cupy_tests/linalg_tests/test_einsum.py:297 TestEinSumUnaryOperation_param_49.test_einsum_unary_dtype
 - tests/cupy_tests/linalg_tests/test_einsum.py:279 TestEinSumUnaryOperation_param_136.test_einsum_unary
 - tests/cupy_tests/linalg_tests/test_einsum.py:297 TestEinSumUnaryOperation_param_136.test_einsum_unary_dtype
 - tests/cupy_tests/linalg_tests/test_einsum.py:289 TestEinSumUnaryOperation_param_136.test_einsum_unary_views
 - tests/cupy_tests/linalg_tests/test_norms.py:18 TestTrace.test_external_trace
 - tests/cupy_tests/linalg_tests/test_norms.py:12 TestTrace.test_trace
 - tests/cupy_tests/padding_tests/test_pad.py:21 TestPadDefault_param_9.test_pad_default
 - tests/cupy_tests/padding_tests/test_pad.py:21 TestPadDefault_param_10.test_pad_default
 - tests/cupy_tests/padding_tests/test_pad.py:21 TestPadDefault_param_11.test_pad_default
 - tests/cupy_tests/padding_tests/test_pad.py:21 TestPadDefault_param_12.test_pad_default
 - tests/cupy_tests/padding_tests/test_pad.py:21 TestPadDefault_param_13.test_pad_default
 - tests/cupy_tests/padding_tests/test_pad.py:21 TestPadDefault_param_14.test_pad_default
 - tests/cupy_tests/padding_tests/test_pad.py:133 TestPad_param_9.test_pad
 - tests/cupy_tests/padding_tests/test_pad.py:133 TestPad_param_10.test_pad
 - tests/cupy_tests/padding_tests/test_pad.py:133 TestPad_param_11.test_pad
 - tests/cupy_tests/padding_tests/test_pad.py:133 TestPad_param_12.test_pad
 - tests/cupy_tests/padding_tests/test_pad.py:133 TestPad_param_13.test_pad
 - tests/cupy_tests/padding_tests/test_pad.py:133 TestPad_param_14.test_pad
 - tests/cupy_tests/padding_tests/test_pad.py:133 TestPad_param_15.test_pad
 - tests/cupy_tests/padding_tests/test_pad.py:133 TestPad_param_16.test_pad
 - tests/cupy_tests/sorting_tests/test_search.py:23 TestSearch.test_argmax_nan
 - tests/cupy_tests/sorting_tests/test_search.py:65 TestSearch.test_argmax_zero_size
 - tests/cupy_tests/sorting_tests/test_search.py:89 TestSearch.test_argmin_nan
 - tests/cupy_tests/sorting_tests/test_search.py:137 TestSearch.test_argmin_zero_size
 - tests/cupy_tests/sorting_tests/test_search.py:327 TestNanArgMin.test_nanargmin_zero_size
 - tests/cupy_tests/sorting_tests/test_search.py:417 TestNanArgMax.test_nanargmax_zero_size

cc @leofang and @anaruse as fellow CUB contributors here

ref: #2579 which should help avoid similar issues in the future

[leofang]

Found a way to treat NaNs without touching CUB source code! For example, this works for fixing Max():

diff --git a/cupy/cuda/cupy_cub.cu b/cupy/cuda/cupy_cub.cu
index ad78a89..cd73383 100644
--- a/cupy/cuda/cupy_cub.cu
+++ b/cupy/cuda/cupy_cub.cu
@@ -6,6 +6,31 @@
 
 using namespace cub;
 
+// 1. The function is appended "cupy_" to avoid confusion and name collision
+// 2. std::isnan() cannot be used here as <cmath> cannot be included
+template <typename T>
+__host__ __device__ __forceinline__ bool cupy_isnan(const T &a) { return a != a; }
+
+// specialization for float for handling NaNs
+// NumPy behavior: NaN is always chosen!
+template <>
+__host__ __device__ __forceinline__ float Max::operator()(const float &a, const float &b) const
+{
+    if (cupy_isnan(a)) {return a;}
+    else if (cupy_isnan(b)) {return b;}
+    else {return CUB_MAX(a, b);}
+}
+
+// specialization for double for handling NaNs
+// NumPy behavior: NaN is always chosen!
+template <>
+__host__ __device__ __forceinline__ double Max::operator()(const double &a, const double &b) const
+{
+    if (cupy_isnan(a)) {return a;}
+    else if (cupy_isnan(b)) {return b;}
+    else {return CUB_MAX(a, b);}
+}
+
 // Minimum boilerplate to support complex numbers in sum(), min(), and max():
 // - This works only because all data fields in the *Traits struct are not
 //   used in <cub/device/device_reduce.cuh>.

Will add and expand this to #2636 later today.

[kmaehashi]

Hmm, I’m trying to reproduce but currently unable to build CuPy master with CUB 1.8.0, maybe I’m missing something… cupy/core/include/cupy/complex/arithmetic.h:262:80: error: ‘double thrust::abs(const thrust::complex<double>&)’ is not declared in ‘thrust’

For record: Using gcc-7 instead of gcc (v5.4.0) by specifying --compiler-bindir gcc-7 to nvcc (export NVCC="nvcc --compiler-bindir gcc-7") solved the issue on Ubuntu 16.04.

cc/ @anaruse