Complex numbers (in ak.from_iter and elsewhere)

Probably low priority, as awkward1.from_numpy does.

>>> import awkward1 as ak
>>> import numpy as np

>>> ak.from_iter([0j])

~/venv/awkward/lib/python3.6/site-packages/awkward1/operations/convert.py in from_iter(iterable, highlevel, behavior, allow_record, initial, resize)
    359     out = awkward1.layout.ArrayBuilder(initial=initial, resize=resize)
    360     for x in iterable:
--> 361         out.fromiter(x)
    362     layout = out.snapshot()
    363     if highlevel:

ValueError: cannot convert 0j (type complex) to an array element

>>> ak.from_numpy(np.asarray([0j]))
 <Array [0j] type='1 * complex128'>

[jpivarski]

In that case, I can give you some pointers, so that you can evaluate how big of a project it will be.

For the complex types, there are fortunately stubs everywhere: they throw std::runtime_errors, but that makes it convenient to find all the places where something needs to be inserted.

% fgrep -r complex128 include src tests
include/awkward/util.h:        complex128,
src/python/forms.cpp:        case ak::util::dtype::complex128:
src/awkward1/operations/structure.py:    numpy.dtype(numpy.complex128): "complex128",
src/libawkward/Content.cpp:                 util::dtype_to_format(util::dtype::complex128),
src/libawkward/Content.cpp:                 util::dtype::complex128);
src/libawkward/util.cpp:      else if (name == "complex128") {
src/libawkward/util.cpp:        return util::dtype::complex128;
src/libawkward/util.cpp:      case util::dtype::complex128:
src/libawkward/util.cpp:        return "complex128";
src/libawkward/util.cpp:        return dtype::complex128;
src/libawkward/util.cpp:      case dtype::complex128:
src/libawkward/util.cpp:      case dtype::complex128:
src/libawkward/util.cpp:      case dtype::complex128:
src/libawkward/array/NumpyArray.cpp:        case util::dtype::complex128:
src/libawkward/array/NumpyArray.cpp:          throw std::runtime_error("FIXME: complex128 to JSON");
src/libawkward/array/NumpyArray.cpp:            dtype_ == util::dtype::complex128  ||
src/libawkward/array/NumpyArray.cpp:            rawother->dtype() == util::dtype::complex128  ||
src/libawkward/array/NumpyArray.cpp:      else if (dtype_ == util::dtype::complex128  ||
src/libawkward/array/NumpyArray.cpp:               rawother->dtype() == util::dtype::complex128) {
src/libawkward/array/NumpyArray.cpp:        dtype = util::dtype::complex128;
src/libawkward/array/NumpyArray.cpp:        dtype = util::dtype::complex128;
src/libawkward/array/NumpyArray.cpp:      // to complex128
src/libawkward/array/NumpyArray.cpp:      case util::dtype::complex128:
src/libawkward/array/NumpyArray.cpp:        throw std::runtime_error("FIXME: merge to complex128 not implemented");
src/libawkward/array/NumpyArray.cpp:      case util::dtype::complex128:
src/libawkward/array/NumpyArray.cpp:        throw std::runtime_error("FIXME: reducers on complex128");
src/libawkward/array/NumpyArray.cpp:      case util::dtype::complex128:
src/libawkward/array/NumpyArray.cpp:        throw std::runtime_error("FIXME: sort for complex128 not implemented");
src/libawkward/array/NumpyArray.cpp:      case util::dtype::complex128:
src/libawkward/array/NumpyArray.cpp:        throw std::runtime_error("FIXME: argsort for complex128 not implemented");
src/libawkward/array/NumpyArray.cpp:      case util::dtype::complex128:
src/libawkward/array/NumpyArray.cpp:        throw std::runtime_error("FIXME: numbers_to_type for complex128 not implemented");
src/libawkward/array/NumpyArray.cpp:    case util::dtype::complex128:
src/libawkward/array/NumpyArray.cpp:      throw std::runtime_error("FIXME: as_type for complex128 not implemented");

I just looked at these search results in context and none of them need any work except the ones in NumpyArray.cpp. NumpyArray.cpp needs:

• a conversion of complex numbers to and from JSON: what would that be? Is there a standard, like {"real": #, "imag": #} or maybe {"r": #, "i": #}? Do other libraries converge on some convention?
• both NumpyArray::merge and NumpyArray::numbers_to_type call kernel::NumpyArray_fill<X, Y> (the latter does so indirectly), and this needs to be specialized to include complex cases in the X and Y. That goes into kernel functions named awkward_NumpyArray_fill_toY_fromX, which are described in more detail below.
• the reducers need complex cases, which is a straightforward extrapolation of the floating point ones: count, count_nonzero (implementations don’t depend on the type; it’s just a function signature), sum, prod (add the function signatures and the compiler does the work), any, all (have to determine whether a complex number is zero or not, but otherwise formulaic), min, max, argmin, argmax (can raise std::invalid_argument("don't do that!") because complex numbers are unordered).
• there’s a stub for sorting and arg-sorting complex numbers, but this can raise std::invalid_argument("don't do that!") because complex numbers are unordered.

Kernel functions separate code that actually touches the arrays from all the rest of the codebase. They’re a separate layer in the architecture:

and are, in fact, a separate shared library that can be dynamically loaded (libawkward-cpu-kernels.so and libawkward-cuda-kernels.so). The actual implementations of these functions (filling arrays and counting, summing, multiplying) are very simple and are templated by numerical type:

https://github.com/scikit-hep/awkward-1.0/blob/029ae3991849d391733043db5e87389e333638fc/src/cpu-kernels/operations.cpp#L1136-L1146

However, that extern "C" interface between the kernels layer and the C++ layer means that all of the kernel cases have to be given explicit names that dynamic library loaders can use to identify them. That means there’s a lot of boilerplate, linking named functions to template specializations:

https://github.com/scikit-hep/awkward-1.0/blob/029ae3991849d391733043db5e87389e333638fc/src/cpu-kernels/operations.cpp#L1147-L1162

On the other side of the extern "C" interface, we curb this insanity by gathering them back up again into template instantiations that are easier to use in the C++ codebase:

https://github.com/scikit-hep/awkward-1.0/blob/029ae3991849d391733043db5e87389e333638fc/src/libawkward/kernel.cpp#L7073-L7095

(so the number of differently named functions explodes only at the interface). Why do we need an extern "C" interface if it’s causing all this boilerplate? First of all, because of that ptr_lib == kernel::lib::cuda case in the above code; it lets us swap CPU-bound implementations for CUDA implementations at runtime, based on whether we get the array from NumPy or from CuPy. (The project we’re looking at here is just the CPU-bound implementation—we’re trying to auto-generate CUDA kernels from the CPU ones, and whatever you write would be part of that automatic translation.) Beyond swapping out implementations below the extern "C" interface, it also lets us swap out implementations above it, if someone’s interested in building a Julia/Rust/Swift/whatever implementation of Awkward Array.

The only extra issue that complex numbers introduces is that std::complex is not extern "C" compatible, so you’d have to reinterpret_cast a std::complex<double> array of length n as a double array of length n * 2 before passing it through the interface. C++ types like std::complex can be used on both sides of the extern "C" interface, just not through it. Thus, you don’t need to define type conversion or complex addition/multiplication inside the cpu-kernels, you can reinterpret_cast that double array of length n * 2 as a std::complex<double> array of length n and then the compiler takes over.

But I apologize in advance about the boilerplate. I’ve been generating code with Emacs keyboard macros…

[jpivarski]

This should have been linked to #652: it’s done now!