numpy.dot cannot handle arrays with >2**31 elements without support in the underlying BLAS

[The bug here is that some BLAS libraries will crash if you pass in > 16 GiB arrays, because they use 32-bit indices internally. The most obvious solution is to break the dot call into multiple calls to dgemm, though there are also other possibilities. Original report follows:]

---

On python 2.7.8, numpy 1.9.1, on Mac OS X:

import numpy
numpy.random.seed(1)
X = numpy.random.random((50000,100))
numpy.dot(X, X.T)

Results in:

Segmentation fault: 11

Segfault doesn’t occur on smaller arrays (e.g. 30000x100 is fine). In case it’s useful, some linkage info:

>>> numpy.__config__.show()
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    define_macros = [('NO_ATLAS_INFO', 3)]
atlas_blas_threads_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
lapack_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH']
    define_macros = [('NO_ATLAS_INFO', 3)]
openblas_lapack_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

[sturlamolden]

Rasmus, that is indeed what I suggest. I would propbably prefer to do the blocking in C, at least to implement the @ operator, but Cython or Python blocking code piggy-backed on np.dot would suffice as proof-of-concept.

The overhead would not be large. In most cases the matrices will be sufficiently small, and the extra overhead just amounts to one conditional branch. The CPUs branch prediction can even eliminate this tiny overhead. Performance-wise we can do the blocking with impunity.

[pv]

Sane 64-bit blas/lapack implementations change the symbol names, the the adoption of which has progressed in the recent years — e.g. Openblas does — and there is no ABI conflict. MKL maybe not, but for pypi binaries we don’t care.

The integer size of course matters for integer arrays, but that’s not an issue for dot in particular.