Test failures and segfaults on big endian archs (s390x, ppc64)

Describe the bug Debian recently upgraded from sparse 0.2.0 to 0.11.2 (we missed the repo move to pydata).

Since the introduction of numba (250d7c6effb5438614400a004d9469f234e4f5a0) lots of tests have failed on s390x. I can reproduce the issue on ppc64, another big-endian arch. Not sure if this is a sparse bug or a numba bug.

Also, not sure if you care about this at all. 😃

To Reproduce Run the test suite on a big endian machine.

I was using: python3 -m pytest --pyargs sparse -c /dev/null -v

Expected behavior Tests should pass.

System

• OS and version: Debian unstable
• python version: 3.9.1
• sparse version: 0.11.2. Reproduced on 250d7c6effb5438614400a004d9469f234e4f5a0 through 440809fc1f82e1e6f681cb64bfecce67f75d9988.
• NumPy version: 1.19.5.
• Numba version: 0.52.0.

Also reproduced on:

• OS and version: Debian stable
• python version: 3.7.3
• sparse version: 250d7c6effb5438614400a004d9469f234e4f5a0 through 939f3ff1b2d4dd69853e5720942c5525e369ee63 (compatible with the numpy & numba available)
• NumPy version: 1.16.2.
• Numba version: 0.42.0.

Additional context

Too many tests segfault for me to give you a full list of passing and failing tests, but here are some hints:

==================================== test session starts =====================================
platform linux -- Python 3.9.1+, pytest-6.0.2, py-1.10.0, pluggy-0.13.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/stefanor/sparse/sparse, configfile: ../../../../dev/null
plugins: cov-2.10.1
collected 4990 items                                                                         

sparse/_common.py::sparse._common._memoize_dtype PASSED                                [  0%]
sparse/_common.py::sparse._common.eye PASSED                                           [  0%]
sparse/_common.py::sparse._common.full PASSED                                          [  0%]
sparse/_common.py::sparse._common.full_like PASSED                                     [  0%]
sparse/_common.py::sparse._common.moveaxis PASSED                                      [  0%]
sparse/_common.py::sparse._common.ones PASSED                                          [  0%]
sparse/_common.py::sparse._common.ones_like PASSED                                     [  0%]
sparse/_common.py::sparse._common.outer FAILED                                         [  0%]

========================================== FAILURES ==========================================
_______________________________ [doctest] sparse._common.outer _______________________________
1560         The input arrays.
1561     out : sparse.SparseArray
1562         The output array.
1563 
1564     Examples
1565     --------
1566     >>> import numpy as np
1567     >>> import sparse
1568     >>> a = sparse.COO(np.arange(4))
1569     >>> o = sparse.outer(a, a)
UNEXPECTED EXCEPTION: SystemError('CPUDispatcher(<function _compute_mask at 0x3ff782605e0>) returned a result with an error set')
IndexError: tuple assignment index out of range
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/lib/python3.9/doctest.py", line 1336, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest sparse._common.outer[3]>", line 1, in <module>
  File "/home/stefanor/sparse/sparse/_common.py", line 1582, in outer
    return np.multiply.outer(a.flatten(), b.flatten(), out=out)
  File "/home/stefanor/sparse/sparse/_sparse_array.py", line 299, in __array_ufunc__
    inputs_transformed.append(inp[(Ellipsis,) + (None,) * cum_ndim])
  File "/home/stefanor/sparse/sparse/_coo/indexing.py", line 77, in getitem
    mask, adv_idx = _mask(x.coords, index, x.shape)
  File "/home/stefanor/sparse/sparse/_coo/indexing.py", line 180, in _mask
    mask, is_slice = _compute_mask(coords, _ind_ar_from_indices(indices))
SystemError: CPUDispatcher(<function _compute_mask at 0x3ff782605e0>) returned a result with an error set
/home/stefanor/sparse/sparse/_common.py:1569: UnexpectedException
================================== short test summary info ===================================
FAILED sparse/_common.py::sparse._common.outer
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================ 1 failed, 7 passed in 2.09s =================================

There are many failures, and many segfaults

sparse/_coo/indexing.py::sparse._coo.indexing._get_mask_pairs Fatal Python error: Segmentation fault

Current thread 0x000003ff85775710 (most recent call first):
  File "<doctest sparse._coo.indexing._get_mask_pairs[4]>", line 1 in <module>
  File "/usr/lib/python3.9/doctest.py", line 1336 in __run
  File "/usr/lib/python3.9/doctest.py", line 1483 in run
  File "/usr/lib/python3.9/doctest.py", line 1844 in run
  File "/usr/lib/python3/dist-packages/_pytest/doctest.py", line 287 in runtest
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 153 in pytest_runtest_call
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 83 in <lambda>
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 92 in _hookexec
  File "/usr/lib/python3/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 247 in <lambda>
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 294 in from_call
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 246 in call_runtest_hook
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 207 in call_and_report
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 117 in runtestprotocol
  File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 100 in pytest_runtest_protocol
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 83 in <lambda>
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 92 in _hookexec
  File "/usr/lib/python3/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3/dist-packages/_pytest/main.py", line 321 in pytest_runtestloop
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 83 in <lambda>
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 92 in _hookexec
  File "/usr/lib/python3/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3/dist-packages/_pytest/main.py", line 296 in _main
  File "/usr/lib/python3/dist-packages/_pytest/main.py", line 240 in wrap_session
  File "/usr/lib/python3/dist-packages/_pytest/main.py", line 289 in pytest_cmdline_main
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 83 in <lambda>
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 92 in _hookexec
  File "/usr/lib/python3/dist-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3/dist-packages/_pytest/config/__init__.py", line 157 in main
  File "/usr/lib/python3/dist-packages/_pytest/config/__init__.py", line 180 in console_main
  File "/usr/lib/python3/dist-packages/pytest/__main__.py", line 7 in <module>
  File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
  File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
Segmentation fault

[stuartarchibald]

This is almost certainly due to Numba not supporting the named architectures.

Supported architectures/OS are listed here: https://numba.readthedocs.io/en/latest/user/installing.html

@stefanor If these architectures are important to you then please do open an issue on the Numba issue tracker to discuss. Many thanks

@hameerabbasi I think it’d be safe to close this, pretty sure that this is not a sparse bug/problem.

[stuartarchibald]

Thanks, @stuartarchibald!

No problem 😃