Nuitka: memory leak in dataclasses_json library (type(name, bases, dict) changes __module__ value after init()).
See original GitHub issue-
Nuitka version 0.6.19.4 Commercial: None Python: 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] Flavor: Unknown Executable: C:\Temp\type3ex.venv\Scripts\python.exe OS: Windows Arch: x86_64 WindowsRelease: 10
-
How did you install Nuitka and Python pip install nuitka
-
The specific PyPI names and versions python -m pip freeze dataclasses-json==0.5.6 marshmallow==3.14.1 marshmallow-enum==1.5.1 mypy-extensions==0.4.3 Nuitka==0.6.19.4 typing-extensions==4.0.1 typing-inspect==0.7.1
-
Provide in your issue the Nuitka options used nuitka --standalone type3ex.py
-
Not a regression - previous versions seems have the issue
-
Short example type3ex.py :
import gc
from dataclasses import dataclass
from dataclasses_json import dataclass_json, Undefined
from marshmallow import class_registry
@dataclass_json(undefined=Undefined.EXCLUDE)
@dataclass
class Test:
name: str
def _init_(self):
print(self.__module__)
super().__init__()
for i in range(0, 5):
Test.schema()
gc.collect()
print(len(gc.get_objects())) # constantly increasing value in nuitka
print(i, [each.__module__ for each in getattr(class_registry, '_registry')['TestSchema']]) # "leaking" refs
Running example:
- by Python:
(.venv) C:\Temp\type3ex>python type3ex.py
14220
0 ['marshmallow.schema']
14249
1 ['marshmallow.schema']
14249
2 ['marshmallow.schema']
14249
3 ['marshmallow.schema']
14249
- Packed by nuitka:
(.venv) C:\Temp\type3ex>type3ex.dist\type3ex.exe
14773
0 ['dataclasses_json.mm']
14801
1 ['dataclasses_json.mm', 'dataclasses_json.mm']
14827
2 ['dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm']
14853
3 ['dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm']
14879
4 ['dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm']
Investigation shows that there is possible issue in nuitka/build/static_src/HelpersBuiltin.c BUILTIN_TYPE3() method
PyObject *BUILTIN_TYPE3(PyObject *module_name, PyObject *name, PyObject *bases, PyObject *dict) {
PyObject *pos_args = PyTuple_New(3);
PyTuple_SET_ITEM(pos_args, 0, name);
Py_INCREF(name);
PyTuple_SET_ITEM(pos_args, 1, bases);
Py_INCREF(bases);
PyTuple_SET_ITEM(pos_args, 2, dict);
Py_INCREF(dict);
PyObject *result = PyType_Type.tp_new(&PyType_Type, pos_args, NULL);
if (unlikely(result == NULL)) {
Py_DECREF(pos_args);
return NULL;
}
PyTypeObject *type = Py_TYPE(result);
if (likely(PyType_IsSubtype(type, &PyType_Type))) {
if (NuitkaType_HasFeatureClass(type) && type->tp_init != NULL) {
int res = type->tp_init(result, pos_args, NULL);
if (unlikely(res < 0)) {
Py_DECREF(pos_args);
Py_DECREF(result);
return NULL;
}
}
}
Py_DECREF(pos_args);
int res = PyObject_SetAttr(result, const_str_plain___module__, module_name);
if (res < 0) {
return NULL;
}
return result;
}
Method PyObject_SetAttr(result, const_str_plain___module__, module_name) called after type->tp_init(result, pos_args, NULL)
so in init() method of parent object __module__
value can differ from __module__
value after object creation.
And thus every code that uses type(name, bases, dict)
and in init() function rely on __module__
value may encounter
unexpected behavior (In this case it’s memory leak).
Moreover - the final __module__
value in nuitka differ from such in python.
Issue Analytics
- State:
- Created 2 years ago
- Reactions:2
- Comments:5 (4 by maintainers)
The memory leak might come from not releasing the result when it cannot set the
__module__
value, which though it seems it does do, but in that instance aPy_DECREF
was missing. With this code it works and does not leak:This will not overwrite existing values, and also release in case of an error. Thanks for your report. This will be in the next hotfix of Nuitka and on factory shortly.
Good news! Thanks for the quick solution.