Nuitka: memory leak in dataclasses_json library (type(name, bases, dict) changes __module__ value after init()).

• Nuitka version 0.6.19.4 Commercial: None Python: 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] Flavor: Unknown Executable: C:\Temp\type3ex.venv\Scripts\python.exe OS: Windows Arch: x86_64 WindowsRelease: 10

• How did you install Nuitka and Python pip install nuitka

• The specific PyPI names and versions python -m pip freeze dataclasses-json==0.5.6 marshmallow==3.14.1 marshmallow-enum==1.5.1 mypy-extensions==0.4.3 Nuitka==0.6.19.4 typing-extensions==4.0.1 typing-inspect==0.7.1

• Provide in your issue the Nuitka options used nuitka --standalone type3ex.py

• Not a regression - previous versions seems have the issue

• Short example type3ex.py :

import gc
from dataclasses import dataclass
from dataclasses_json import dataclass_json, Undefined
from marshmallow import class_registry

@dataclass_json(undefined=Undefined.EXCLUDE)
@dataclass
class Test:
    name: str

    def _init_(self):
        print(self.__module__)
        super().__init__()

for i in range(0, 5):
    Test.schema()

    gc.collect()
    print(len(gc.get_objects()))  # constantly increasing value in nuitka

    print(i, [each.__module__ for each in getattr(class_registry, '_registry')['TestSchema']])  # "leaking" refs

Running example:

• by Python:

(.venv) C:\Temp\type3ex>python type3ex.py
14220
0 ['marshmallow.schema']
14249
1 ['marshmallow.schema']
14249
2 ['marshmallow.schema']
14249
3 ['marshmallow.schema']
14249

• Packed by nuitka:

(.venv) C:\Temp\type3ex>type3ex.dist\type3ex.exe       
14773
0 ['dataclasses_json.mm']
14801
1 ['dataclasses_json.mm', 'dataclasses_json.mm']
14827
2 ['dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm']
14853
3 ['dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm']
14879
4 ['dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm', 'dataclasses_json.mm']

Investigation shows that there is possible issue in nuitka/build/static_src/HelpersBuiltin.c BUILTIN_TYPE3() method

PyObject *BUILTIN_TYPE3(PyObject *module_name, PyObject *name, PyObject *bases, PyObject *dict) {
    PyObject *pos_args = PyTuple_New(3);
    PyTuple_SET_ITEM(pos_args, 0, name);
    Py_INCREF(name);
    PyTuple_SET_ITEM(pos_args, 1, bases);
    Py_INCREF(bases);
    PyTuple_SET_ITEM(pos_args, 2, dict);
    Py_INCREF(dict);

    PyObject *result = PyType_Type.tp_new(&PyType_Type, pos_args, NULL);

    if (unlikely(result == NULL)) {
        Py_DECREF(pos_args);
        return NULL;
    }

    PyTypeObject *type = Py_TYPE(result);

    if (likely(PyType_IsSubtype(type, &PyType_Type))) {
        if (NuitkaType_HasFeatureClass(type) && type->tp_init != NULL) {
            int res = type->tp_init(result, pos_args, NULL);

            if (unlikely(res < 0)) {
                Py_DECREF(pos_args);
                Py_DECREF(result);
                return NULL;
            }
        }
    }

    Py_DECREF(pos_args);

    int res = PyObject_SetAttr(result, const_str_plain___module__, module_name);

    if (res < 0) {
        return NULL;
    }

    return result;
}

Method PyObject_SetAttr(result, const_str_plain___module__, module_name) called after type->tp_init(result, pos_args, NULL) so in init() method of parent object __module__ value can differ from __module__ value after object creation. And thus every code that uses type(name, bases, dict) and in init() function rely on __module__ value may encounter unexpected behavior (In this case it’s memory leak). Moreover - the final __module__ value in nuitka differ from such in python.

[kayhayen]

The memory leak might come from not releasing the result when it cannot set the __module__ value, which though it seems it does do, but in that instance a Py_DECREF was missing. With this code it works and does not leak:

    if (HAS_ATTR_BOOL(result, const_str_plain___module__) == false) {
        int res = SET_ATTRIBUTE(result, const_str_plain___module__, module_name);

        if (res < 0) {
            Py_DECREF(result);
            return NULL;
        }
    }

This will not overwrite existing values, and also release in case of an error. Thanks for your report. This will be in the next hotfix of Nuitka and on factory shortly.

[VyacheslavVlasenko]

Good news! Thanks for the quick solution.