Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Nested dict-of-jitclass throws pickling error

See original GitHub issue

Numba 0.52.0, 64-bit Python 3.7.3 on Windows Discourse discussion is here

The code below doesn’t work, but seems like it ought to

from numba import types, typed, njit
from numba.experimental import jitclass

@jitclass([
    ("x", types.int64),
])
class Level1:
    def __init__(self):
        self.x = 12

level1 = Level1()  # Okay, I can create a Level1 instance
level1_instance_type = Level1.class_type.instance_type
level1_kv = (types.int64, level1_instance_type)

@jitclass([
    ("level1s", types.DictType(*level1_kv)),
])
class Level2(object):
    def __init__(self):
        self.level1s = typed.Dict.empty(*level1_kv)
        self.level1s[0] = Level1()


level2 = Level2()  # Okay, I can create a Level2 instance
level2_instance_type = Level2.class_type.instance_type
level2_kv = (types.int64, level2_instance_type)

@jitclass([
    ("level2s", types.DictType(*level2_kv)),
])
class Level3(object):
    def __init__(self):
        self.level2s = typed.Dict.empty(*level2_kv)

# _pickle.PicklingError: Can't pickle <class 'numba.experimental.jitclass.base.Level1'>: it's not found as numba.experimental.jitclass.base.Level1
# level3 = Level3()

Issue Analytics

State:
Created 3 years ago
Comments:10 (7 by maintainers)

Top GitHub Comments

1reaction

esccommented, Jun 14, 2021

@tombh thanks I can confirm I am seeing this on master (5f6859908e7b73f7e1baf30381bac9e863640210) too.

1reaction

esccommented, May 31, 2021

This still reproduces, on OSX even with current master (cbb4301852d0950f15eef0a2b8c0478db5d57201):

 💣 zsh» python issu6640.py                                                                                                                                         :(
Traceback (most recent call last):
  File "/Users/vhaenel/git/numba/numba/core/pythonapi.py", line 1371, in serialize_object
    gv = self.module.__serialized[obj]
KeyError: DictType[int64,instance.jitclass.Level2#7fcb5b38c610<level1s:DictType[int64,instance.jitclass.Level1#7fcb54ac9fd0<x:int64>]<iv=None>>]<iv=None>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "issu6640.py", line 41, in <module>
    level3 = Level3()
  File "/Users/vhaenel/git/numba/numba/experimental/jitclass/base.py", line 124, in __call__
    return cls._ctor(*bind.args[1:], **bind.kwargs)
  File "/Users/vhaenel/git/numba/numba/core/dispatcher.py", line 440, in _compile_for_args
    raise e
  File "/Users/vhaenel/git/numba/numba/core/dispatcher.py", line 373, in _compile_for_args
    return_val = self.compile(tuple(argtypes))
  File "/Users/vhaenel/git/numba/numba/core/dispatcher.py", line 911, in compile
    cres = self._compiler.compile(args, return_type)
  File "/Users/vhaenel/git/numba/numba/core/dispatcher.py", line 80, in compile
    status, retval = self._compile_cached(args, return_type)
  File "/Users/vhaenel/git/numba/numba/core/dispatcher.py", line 94, in _compile_cached
    retval = self._compile_core(args, return_type)
  File "/Users/vhaenel/git/numba/numba/core/dispatcher.py", line 107, in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
  File "/Users/vhaenel/git/numba/numba/core/compiler.py", line 675, in compile_extra
    return pipeline.compile_extra(func)
  File "/Users/vhaenel/git/numba/numba/core/compiler.py", line 419, in compile_extra
    return self._compile_bytecode()
  File "/Users/vhaenel/git/numba/numba/core/compiler.py", line 483, in _compile_bytecode
    return self._compile_core()
  File "/Users/vhaenel/git/numba/numba/core/compiler.py", line 462, in _compile_core
    raise e
  File "/Users/vhaenel/git/numba/numba/core/compiler.py", line 453, in _compile_core
    pm.run(self.state)
  File "/Users/vhaenel/git/numba/numba/core/compiler_machinery.py", line 343, in run
    raise patched_exception
  File "/Users/vhaenel/git/numba/numba/core/compiler_machinery.py", line 334, in run
    self._runPass(idx, pass_inst, state)
  File "/Users/vhaenel/git/numba/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/Users/vhaenel/git/numba/numba/core/compiler_machinery.py", line 289, in _runPass
    mutated |= check(pss.run_pass, internal_state)
  File "/Users/vhaenel/git/numba/numba/core/compiler_machinery.py", line 262, in check
    mangled = func(compiler_state)
  File "/Users/vhaenel/git/numba/numba/core/typed_passes.py", line 398, in run_pass
    lower.create_cpython_wrapper(flags.release_gil)
  File "/Users/vhaenel/git/numba/numba/core/lowering.py", line 242, in create_cpython_wrapper
    self.context.create_cpython_wrapper(self.library, self.fndesc,
  File "/Users/vhaenel/git/numba/numba/core/cpu.py", line 183, in create_cpython_wrapper
    builder.build()
  File "/Users/vhaenel/git/numba/numba/core/callwrapper.py", line 123, in build
    self.build_wrapper(api, builder, closure, args, kws)
  File "/Users/vhaenel/git/numba/numba/core/callwrapper.py", line 188, in build_wrapper
    obj = api.from_native_return(retty, retval, env_manager)
  File "/Users/vhaenel/git/numba/numba/core/pythonapi.py", line 1398, in from_native_return
    out = self.from_native_value(typ, val, env_manager)
  File "/Users/vhaenel/git/numba/numba/core/pythonapi.py", line 1412, in from_native_value
    return impl(typ, val, c)
  File "/Users/vhaenel/git/numba/numba/experimental/jitclass/boxing.py", line 139, in _box_class_instance
    box_subclassed = _specialize_box(typ)
  File "/Users/vhaenel/git/numba/numba/experimental/jitclass/boxing.py", line 122, in _specialize_box
    fast_fget = fget.compile((typ,))
  File "/Users/vhaenel/git/numba/numba/core/dispatcher.py", line 911, in compile
    cres = self._compiler.compile(args, return_type)
  File "/Users/vhaenel/git/numba/numba/core/dispatcher.py", line 80, in compile
    status, retval = self._compile_cached(args, return_type)
  File "/Users/vhaenel/git/numba/numba/core/dispatcher.py", line 94, in _compile_cached
    retval = self._compile_core(args, return_type)
  File "/Users/vhaenel/git/numba/numba/core/dispatcher.py", line 107, in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
  File "/Users/vhaenel/git/numba/numba/core/compiler.py", line 675, in compile_extra
    return pipeline.compile_extra(func)
  File "/Users/vhaenel/git/numba/numba/core/compiler.py", line 419, in compile_extra
    return self._compile_bytecode()
  File "/Users/vhaenel/git/numba/numba/core/compiler.py", line 483, in _compile_bytecode
    return self._compile_core()
  File "/Users/vhaenel/git/numba/numba/core/compiler.py", line 462, in _compile_core
    raise e
  File "/Users/vhaenel/git/numba/numba/core/compiler.py", line 453, in _compile_core
    pm.run(self.state)
  File "/Users/vhaenel/git/numba/numba/core/compiler_machinery.py", line 343, in run
    raise patched_exception
  File "/Users/vhaenel/git/numba/numba/core/compiler_machinery.py", line 334, in run
    self._runPass(idx, pass_inst, state)
  File "/Users/vhaenel/git/numba/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/Users/vhaenel/git/numba/numba/core/compiler_machinery.py", line 289, in _runPass
    mutated |= check(pss.run_pass, internal_state)
  File "/Users/vhaenel/git/numba/numba/core/compiler_machinery.py", line 262, in check
    mangled = func(compiler_state)
  File "/Users/vhaenel/git/numba/numba/core/typed_passes.py", line 398, in run_pass
    lower.create_cpython_wrapper(flags.release_gil)
  File "/Users/vhaenel/git/numba/numba/core/lowering.py", line 242, in create_cpython_wrapper
    self.context.create_cpython_wrapper(self.library, self.fndesc,
  File "/Users/vhaenel/git/numba/numba/core/cpu.py", line 183, in create_cpython_wrapper
    builder.build()
  File "/Users/vhaenel/git/numba/numba/core/callwrapper.py", line 123, in build
    self.build_wrapper(api, builder, closure, args, kws)
  File "/Users/vhaenel/git/numba/numba/core/callwrapper.py", line 188, in build_wrapper
    obj = api.from_native_return(retty, retval, env_manager)
  File "/Users/vhaenel/git/numba/numba/core/pythonapi.py", line 1398, in from_native_return
    out = self.from_native_value(typ, val, env_manager)
  File "/Users/vhaenel/git/numba/numba/core/pythonapi.py", line 1412, in from_native_value
    return impl(typ, val, c)
  File "/Users/vhaenel/git/numba/numba/typed/typeddict.py", line 242, in box_dicttype
    dicttype_obj = c.pyapi.unserialize(c.pyapi.serialize_object(typ))
  File "/Users/vhaenel/git/numba/numba/core/pythonapi.py", line 1373, in serialize_object
    struct = self.serialize_uncached(obj)
  File "/Users/vhaenel/git/numba/numba/core/pythonapi.py", line 1344, in serialize_uncached
    data = serialize.dumps(obj)
  File "/Users/vhaenel/git/numba/numba/core/serialize.py", line 57, in dumps
    p.dump(obj)
  File "/Users/vhaenel/git/numba/numba/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/Users/vhaenel/git/numba/numba/core/serialize.py", line 185, in reducer_override
    _no_pickle(obj)  # noreturn
  File "/Users/vhaenel/git/numba/numba/core/serialize.py", line 164, in _no_pickle
    raise pickle.PicklingError(f"Pickling of {type(obj)} is unsupported")
_pickle.PicklingError: Failed in nopython mode pipeline (step: native lowering)
Failed in nopython mode pipeline (step: native lowering)
Pickling of <class 'numba.experimental.jitclass.base.JitClassType'> is unsupported

Top Results From Across the Web

Python multiprocessing PicklingError: Can't pickle <type ...

Update: The function I pickle is defined at the top level of the module. Though it calls a function that contains a nested...

Mailman 3 Pickling instances of nested classes - Python-Dev

Currently instances of nested classes can't be pickled. For old style classes unpickling fails to find the ... PicklingError: Can't pickle <class '__main__....

Python Pickle: Serialize Your Objects [With Examples]

Pickle a Nested Dictionary Object. Let's find out if a Python nested dictionary can be serialized and deserialized using the Pickle module.

PicklingError in pyspark (PicklingError: Can't pickle <class ...

Possible answer from here. The problem is that you're trying to pickle an object from the module where it's defined. If you move...

Fixing pickle for nested classes — Sage 9.3.beta9 Reference ...

So there is no pickling problem for nested classes in Python 3, and the two utilities are not really necessary. However, NestedClassMetaclass is...