`cfunc` unable to compile UDF with record dtype that accesses a record attribute

Reporting a bug

I want to use numba to compile functions that operate on custom numpy dtypes (including record dtypes) to use with cffi. To do this, I use numba.cfunc that takes in pointers of the output and input arguments.

Here is a working example that doesn’t access a record attribute:

import numba
import numpy as np

record_dtype = np.dtype([("x", np.float64), ("y", np.float64)], align=True)
numba_dtype = numba.typeof(record_dtype)
sample = np.zeros(1, dtype=record_dtype)[0]

##############
# This works #
##############

@numba.njit
def identity(val):
    return val

assert identity(sample) == sample

wrapper_signature = numba.types.void(
    numba.types.CPointer(numba_dtype),
    numba.types.CPointer(numba_dtype),
)

@numba.cfunc(wrapper_signature, nopython=True)
def wrapper(result_ptr, val_ptr):
    result_ptr[0] = identity(val_ptr[0])

This fails:

@numba.njit
def get0(val):
    return val[0]

assert get0(sample) == sample[0]

wrapper_signature = numba.types.void(
    numba.types.CPointer(numba.types.float64),
    numba.types.CPointer(numba_dtype),
)

@numba.cfunc(wrapper_signature, nopython=True)  # <-- fails here
def wrapper(result_ptr, val_ptr):
    result_ptr[0] = get0(val_ptr[0])

and this fails:

@numba.njit
def getx(val):
    return val["x"]

assert getx(sample) == sample["x"]

wrapper_signature = numba.types.void(
    numba.types.CPointer(numba.types.float64),
    numba.types.CPointer(numba_dtype),
)

@numba.cfunc(wrapper_signature, nopython=True)  # <-- fails here
def wrapper(result_ptr, val_ptr):
    result_ptr[0] = getx(val_ptr[0])

Here is a traceback:

Traceback (most recent call last):
  File "/home/erik/git/grblas/run8.py", line 44, in <module>
    def wrapper(result_ptr, val_ptr):
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/decorators.py", line 282, in wrapper
    res.compile()
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/ccallback.py", line 67, in compile
    cres = self._compile_uncached()
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/ccallback.py", line 81, in _compile_uncached
    return self._compiler.compile(sig.args, sig.return_type)
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/dispatcher.py", line 129, in compile
    raise retval
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/dispatcher.py", line 139, in _compile_cached
    retval = self._compile_core(args, return_type)
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/dispatcher.py", line 152, in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler.py", line 693, in compile_extra
    return pipeline.compile_extra(func)
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler.py", line 429, in compile_extra
    return self._compile_bytecode()
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler.py", line 497, in _compile_bytecode
    return self._compile_core()
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler.py", line 476, in _compile_core
    raise e
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler.py", line 463, in _compile_core
    pm.run(self.state)
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 353, in run
    raise patched_exception
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 341, in run
    self._runPass(idx, pass_inst, state)
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 296, in _runPass
    mutated |= check(pss.run_pass, internal_state)
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 269, in check
    mangled = func(compiler_state)
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/typed_passes.py", line 105, in run_pass
    typemap, return_type, calltypes, errs = type_inference_stage(
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/typed_passes.py", line 83, in type_inference_stage
    errs = infer.propagate(raise_errors=raise_errors)
  File "/home/erik/miniconda3/envs/grblas/lib/python3.9/site-packages/numba/core/typeinfer.py", line 1086, in propagate
    raise errors[0]
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<built-in function getitem>) found for signature:

 >>> getitem(dtype(Record(x[type=float64;offset=0],y[type=float64;offset=8];16;True)), Literal[int](0))

There are 22 candidate implementations:
  - Of which 22 did not match due to:
  Overload of function 'getitem': File: <numerous>: Line N/A.
    With argument(s): '(dtype(Record(x[type=float64;offset=0],y[type=float64;offset=8];16;True)), int64)':
   No match.

During: typing of intrinsic-call at /home/erik/git/grblas/run8.py (34)
During: typing of static-get-item at /home/erik/git/grblas/run8.py (34)

File "run8.py", line 34:
def get0(val):
    return val[0]
    ^

During: resolving callee type: type(CPUDispatcher(<function get0 at 0x7f8c300fbd30>))
During: typing of call at /home/erik/git/grblas/run8.py (45)

During: resolving callee type: type(CPUDispatcher(<function get0 at 0x7f8c300fbd30>))
During: typing of call at /home/erik/git/grblas/run8.py (45)


File "run8.py", line 45:
def wrapper(result_ptr, val_ptr):
    result_ptr[0] = get0(val_ptr[0])
    ^

And I’m curious: what’s the difference between numba_dtype and numba_dtype.dtype? I get different errors when I use numba_dtype.dtype above in wrapper_signature .

In [4]: numba_dtype
Out[4]: dtype(Record(x[type=float64;offset=0],y[type=float64;offset=8];16;True))

In [5]: numba_dtype.dtype
Out[5]: Record(x[type=float64;offset=0],y[type=float64;offset=8];16;True)

[sklam]

I still need to check why without the carray call it is producing the error Type of #3 arg mismatch: [16 x i8]* != [16 x i8]

[github-actions[bot]]

This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.