Support List as a value type in njit dictionaries

It doesn’t appear to be possible to pass a list as a value inside a dictionary. Here is a simple reproducer.

import numba

@numba.njit
def f():
    d = {}
    for i in range(10):
        d[str(i)] = [i]
    return d

print(f())

Produces: numba.core.errors.TypingError: list(int64)<iv=None> as value is forbidden

[esc]

It seems a little weird to me, that while typed.List is allowed by the compiler as value as per @esc’s example

@numba.njit
def f():
    d = {}
    for i in range(10):
        d[str(i)] = numba.typed.List((1,))
    return d
print(f())

and seems to work just fine in my application

The same code fails when assigning a set as value.

@numba.njit
def f2():
    d = {}
    for i in range(10):
        d[str(i)] = {1}
    return d
print(f2())

I know that the documentation states that neither List nor Set are supported. It still makes me wonder why one works and the other doesn’t, especially because you can easily put typed.Dict in typed.Dict as value

@numba.njit
def f3():
    d = {}
    for i in range(10):
       tmp = numba.typed.Dict()
       tmp[i] = True
        d[str(i)] = tmp
    return d
print(f3())

Not sure whether this is the right place or I should put it in a different issue 😉

I suppose it’s fine to ask here, thank you! The reasons are quite technical, but I will try to shed some light on this:

It is important to note that Numba has two list implementations, a reflected list and a typed list. See also: https://numba.readthedocs.io/en/stable/reference/pysupported.html#list The reflected list can not be used inside a typed dictionary whereas the typed List can (perhaps the docs need an update here?).

The two list variants – reflected and typed – differ in how they are implemented and consequently differ in what features they currently support. The reflected list is a type of proxy object for a Python list. When a regular Python list transitions the jit-boundary, for example by being used as an argument to a @njit dec. function, Numba will setup a reflected list as a sort-of-copy of the original Python list. This reflected list then serves as a buffer to hold all the operations to be done on the list inside the function. Lastly, when the function completes, the reflected list will then “reflect” all the changes to the original Python list. As a consequence, a reflected list can’t really exist outside of an @njit decorated function. The typed list on the other hand is implemented differently and can exist outside of a @njit dec. function and be operated on as first-class object in Python.

Now the typed dictionary is similar to a typed List, which means it can also exist as a first-class object and additionally, the values must also support this. For example, if you created a typed dict inside a @njit dec. function and added a reflected list, how should that list behave, when the dictionary is returned from the function as a Python object?

Lastly the only set implementation currently available in Numba is the reflected set implementation and as a result it’s not possible to have sets as values in typed dicts. In order to support sets in dicts in such a context a proper implementation of a typed Set (similar to typed List and typed Dict) would be the best solution. As a rule of thumb I usually remember: reflected: no – typed: yes.

I hope this clarifies some of the nuances involved.

[esc]

I do hope that at some point in the not too distant future [] in JIT-ed code will result in a typed List, instead of a reflected one.