question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Support duck arrays in numba gufuncs

See original GitHub issue

There might be some room for improvement in handling numpy-like arrays in numba-generated gufuncs

I was quite happy to see this work well:

In [1]: from numba import vectorize, float64
   ...: 
   ...: @vectorize([float64(float64, float64)])
   ...: def f(x, y):
   ...:     return x + y
   ...: 

In [2]: import dask.array as da

In [3]: x = da.arange(10, chunks=(5,))

In [4]: f(x, x)  # hooray!
Out[4]: dask.array<f, shape=(10,), dtype=float64, chunksize=(5,)>

Unfortunately this didn’t fare as well. Am I using vectorize incorrectly or is this a feature request?

In [5]: @vectorize
   ...: def f(x, y):
   ...:     return x + y
   ...: 

In [6]: f(x, x)  # oh no!
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-4438cca8adef> in <module>()
----> 1 f(x, x)  # oh no!

~/Software/anaconda/lib/python3.6/site-packages/numba/npyufunc/dufunc.py in _compile_for_args(self, *args, **kws)
    162                 argtys.append(numpy_support.map_arrayscalar_type(arg))
    163             else:
--> 164                 argty = typeof(arg)
    165                 if isinstance(argty, types.Array):
    166                     argty = argty.dtype

~/Software/anaconda/lib/python3.6/site-packages/numba/typing/typeof.py in typeof(val, purpose)
     29     if ty is None:
     30         msg = "cannot determine Numba type of %r" % (type(val),)
---> 31         raise ValueError(msg)
     32     return ty
     33 

ValueError: cannot determine Numba type of <class 'dask.array.core.Array'>

Issue Analytics

  • State:open
  • Created 5 years ago
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
mrocklincommented, May 27, 2018

Just to put a bit of motivation behind this feature request, I’d like to write a blogpost that shows that numba and dask.array are able to inter-operate over the gufunc protocol without explicit coordination.

0reactions
sklamcommented, Jun 4, 2018

@mrocklin, yes, your comment in https://github.com/sklam/numba/commit/34f9d48b10603948b1b345091b4d8175217614bc is exactly right. I have only done the minimal to see how to make it work and to see what code needed to be changed. If someone wanted to take on the work, they would need to:

  • implement the ufunc protocol properly as per your comment. they can continue from where I made the changes.
  • add test (presumably w/o using dask.array; using some made up type with __array_ufunc__).
  • and docs.
Read more comments on GitHub >

github_iconTop Results From Across the Web

Numpy Support in numba — numba 0.16.0 documentation
Access to Numpy arrays is very efficient, as indexing is lowered to memory accessing when possible. * numba is able to generate ufuncs/gufuncs....
Read more >
stable PDF - Numba Documentation
Numba is a just-in-time compiler for Python that works best on code that uses NumPy arrays and functions, and loops.
Read more >
Numba for CUDA Programmers - GitHub
What is Numba? A Just-in-time (JIT) compiler for Python functions. ▻ Opt-in: Numba only compiles the functions you specify. ▻ Focused on array-oriented ......
Read more >
Expansion of generalized universal function signatures - NumPy
Possibly missing dimensions. This part is almost entirely driven by the wish to wrap matmul in a gufunc. matmul stands for matrix multiplication ......
Read more >
Applying unvectorized functions with apply_ufunc - Xarray
The function we will apply is np.interp which expects 1D numpy arrays. ... 797 "applied function returned data with unexpected " 798 f"number...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found