ValueError: Does not understand character buffer dtype format string ('?')

I’m trying to build and test SciPy in Colab.

I hit an issue with boolean numpy arrays, this happens in ~100 tests

_______________________________ TestCSR.test_ne ________________________________
scipy/sparse/tests/test_base.py:369: in test_ne
    check(dtype)
/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py:2279: in new_func
    return func(*args, **kwargs)
/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py:2279: in new_func
    return func(*args, **kwargs)
scipy/sparse/tests/test_base.py:355: in check
    assert_array_equal_dtype(dat != dat2, (datlil != datsp2).todense())
scipy/sparse/base.py:369: in __ne__
    return self.tocsr().__ne__(other)
scipy/sparse/lil.py:478: in tocsr
    _csparsetools.lil_flatten_to_array(self.data, data)
_csparsetools.pyx:135: in scipy.sparse._csparsetools.lil_flatten_to_array
    ???
_csparsetools.pyx:138: in scipy.sparse._csparsetools._lil_flatten_to_array_bool_
    ???
E   ValueError: Does not understand character buffer dtype format string ('?')

_lil_flatten_to_array_bool_ is being generated in _csparsetools.pyx.in

There are some threads on StackOverflow about boolean arrays in cython so I assume that’s the problem. it’s weird I’m the first person to hit this (below you’ll see I have a very recent cython version maybe that’s it) and the CSR to lil code is fairly new (Feb 28th)

it’s possible the solution is to replace the valuetype with uint8 for flatten_to_array_bool_

Scipy/Numpy/Python version information:

Building from head of the git repo

Numpy: 1.18.2 Python 3.6.9 Cython: ‘0.29.16’ (seems to be less than a week old)

[sethtroisi]

This seems to be fixed in Colab by running pip install --force-reinstall -U cython

[sethtroisi]

@smola I can reproduce in Colab I suspect it’s some environmental setting Colab sets.

SciPy: 1.6.0.dev0+c38df7b
NumPy: 1.18.4

Python 3.6.9
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
GNU Fortran (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Cython version 0.29.12

This was the relevant data I saw in build / build_ext

lapack_mkl_info:
customize UnixCCompiler
  FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/usr/local/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/usr/local/include', '/usr/include']

  FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/usr/local/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/usr/local/include', '/usr/include']

/usr/local/lib/python3.6/dist-packages/numpy/distutils/system_info.py:820: UserWarning: Specified path /usr/local/include/python3.6m is invalid.
  return self.get_paths(self.section, key)
blas_opt_info:
blas_mkl_info:
  FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/usr/local/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/usr/local/include', '/usr/include']

  FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/usr/local/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/usr/local/include', '/usr/include']

extending extension 'scipy.sparse.linalg.dsolve._superlu' defined_macros with [('USE_VENDOR_BLAS', 1)]