Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

`test_tsqr_uncertain` failing on gpuCI

See original GitHub issue

dask/array/tests/test_cupy_linalg.py::test_tsqr_uncertain failed on gpuCI with TypeError: Implicit conversion to a NumPy array is not allowed. Please use `.get()` to construct a NumPy array

11:34:33 =================================== FAILURES ===================================
11:34:33 ______________ test_tsqr_uncertain[10-5-chunks0-True-False-None] _______________
11:34:33 [gw3] linux -- Python 3.8.10 /opt/conda/envs/dask/bin/python
11:34:33 
11:34:33 m_min = 10, n_max = 5, chunks = (10, 5), vary_rows = True, vary_cols = False
11:34:33 error_type = None
11:34:33 
11:34:33     @pytest.mark.parametrize(
11:34:33         "m_min,n_max,chunks,vary_rows,vary_cols,error_type",
11:34:33         [
11:34:33             (10, 5, (10, 5), True, False, None),  # single block tall
11:34:33             (10, 5, (10, 5), False, True, None),  # single block tall
11:34:33             (10, 5, (10, 5), True, True, None),  # single block tall
11:34:33             (40, 5, (10, 5), True, False, None),  # multiple blocks tall
11:34:33             (40, 5, (10, 5), False, True, None),  # multiple blocks tall
11:34:33             (40, 5, (10, 5), True, True, None),  # multiple blocks tall
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (40, 10),
11:34:33                 True,
11:34:33                 False,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=2
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (30, 10),
11:34:33                 True,
11:34:33                 False,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=3
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (20, 10),
11:34:33                 True,
11:34:33                 False,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=4
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (40, 10),
11:34:33                 False,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=2
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (30, 10),
11:34:33                 False,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=3
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (20, 10),
11:34:33                 False,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=4
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (40, 10),
11:34:33                 True,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=2
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (30, 10),
11:34:33                 True,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=3
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (20, 10),
11:34:33                 True,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=4
11:34:33         ],
11:34:33     )
11:34:33     def test_tsqr_uncertain(m_min, n_max, chunks, vary_rows, vary_cols, error_type):
11:34:33         mat = cupy.random.rand(m_min * 2, n_max)
11:34:33         m, n = m_min * 2, n_max
11:34:33         mat[0:m_min, 0] += 1
11:34:33         _c0 = mat[:, 0]
11:34:33         _r0 = mat[0, :]
11:34:33         c0 = da.from_array(_c0, chunks=m_min, name="c", asarray=False)
11:34:33         r0 = da.from_array(_r0, chunks=n_max, name="r", asarray=False)
11:34:33         data = da.from_array(mat, chunks=chunks, name="A", asarray=False)
11:34:33         if vary_rows:
11:34:33             data = data[c0 > 0.5, :]
11:34:33             mat = mat[_c0 > 0.5, :]
11:34:33             m = mat.shape[0]
11:34:33         if vary_cols:
11:34:33             data = data[:, r0 > 0.5]
11:34:33             mat = mat[:, _r0 > 0.5]
11:34:33             n = mat.shape[1]
11:34:33     
11:34:33         # qr
11:34:33         m_q = m
11:34:33         n_q = min(m, n)
11:34:33         m_r = n_q
11:34:33         n_r = n
11:34:33     
11:34:33         # svd
11:34:33         m_u = m
11:34:33         n_u = min(m, n)
11:34:33         n_s = n_q
11:34:33         m_vh = n_q
11:34:33         n_vh = n
11:34:33         d_vh = max(m_vh, n_vh)  # full matrix returned
11:34:33     
11:34:33         if error_type is None:
11:34:33             # test QR
11:34:33             q, r = da.linalg.tsqr(data)
11:34:33 >           q = q.compute()  # because uncertainty
11:34:33 
11:34:33 dask/array/tests/test_cupy_linalg.py:229: 
11:34:33 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
11:34:33 dask/base.py:286: in compute
11:34:33     (result,) = compute(self, traverse=False, **kwargs)
11:34:33 dask/base.py:568: in compute
11:34:33     results = schedule(dsk, keys, **kwargs)
11:34:33 dask/threaded.py:79: in get
11:34:33     results = get_async(
11:34:33 dask/local.py:517: in get_async
11:34:33     raise_exception(exc, tb)
11:34:33 dask/local.py:325: in reraise
11:34:33     raise exc
11:34:33 dask/local.py:223: in execute_task
11:34:33     result = _execute_task(task, data)
11:34:33 dask/core.py:121: in _execute_task
11:34:33     return func(*(_execute_task(a, cache) for a in args))
11:34:33 dask/optimization.py:969: in __call__
11:34:33     return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
11:34:33 dask/core.py:151: in get
11:34:33     result = _execute_task(task, cache)
11:34:33 dask/core.py:121: in _execute_task
11:34:33     return func(*(_execute_task(a, cache) for a in args))
11:34:33 <__array_function__ internals>:5: in dot
11:34:33     ???
11:34:33 cupy/_core/core.pyx:1513: in cupy._core.core.ndarray.__array_function__
11:34:33     ???
11:34:33 /opt/conda/envs/dask/lib/python3.8/site-packages/cupy/linalg/_product.py:65: in dot
11:34:33     return a.dot(b, out)
11:34:33 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
11:34:33 
11:34:33 >   ???
11:34:33 E   TypeError: Implicit conversion to a NumPy array is not allowed. Please use `.get()` to construct a NumPy array explicitly.
11:34:33 
11:34:33 cupy/_core/core.pyx:1298: TypeError

cc @dask/gpu

Issue Analytics

State:
Created 2 years ago
Comments:12 (12 by maintainers)

Top GitHub Comments

2reactions

jakirkhamcommented, Aug 30, 2021

Guessing you are already familiar with pytest’s ability to run a debugger during testing, might be useful for exploring the stack and finding out where things go off the rails.

1reaction

pentschevcommented, Sep 6, 2021

#8122 should fix this, no more than 2-3 hours were spent debugging that one-liner fix. 🙂

Thanks everyone for jumping in trying to resolve this.

Top Results From Across the Web

rapidsai/gpuci-build-environment

Common build environment used by gpuCI for building RAPIDS - GitHub - rapidsai/gpuci-build-environment: Common ... Failed to load latest commit information.