Memory growth fast when schema is defined in a function

When I use the following code, I observe that memory growth is very fast, but when I define the schema outside the function, there is no problem.

def test_schema():
    class FooSchema(Schema):
        foo = fields.String()
    # not do anything
    return "xxx"
if __name__ == '__main__':
    while True:
        print test_schema()```

[jtillman]

Hey @justanr, thanks for the reasoning and details explanation here which I appreciate. I too have hit this issue which causes high memory issues in my production environment. Now that I am aware of it I would like to know the suggested solution, in what I see as a valid use case here.

I typically use inheritance to generate classes to give standard packaging.

class ResponseObject(Schema):
    @classmethod
    def load_paged_result(cls, data):
        class ResponsePageObject(Schema):
            page_number = fields.Integer()
            total = fields.Integer()
            items = fields.List(fields.Nested(cls))
        return ResponsePageObject.loads(data)

class UserResponse(ResponseObject):
    name = fields.String()

This way I can easily standardize my response formats across my schemas.

UserResponse.load_paged_result(response.content)

Is there a suggested solution here. I was thinking about manually checking things in spots, but I fear its hard to train others users to follow suit and avoid the memory leaks.

[sloria]

@jtillman You can prevent marshmallow from storing your generated classes in its internal registry of schemas by generating the classes without a name.

class ResponseObject(Schema):
    @classmethod
    def load_paged_result(cls, data):
        fields = {
            'page_number': fields.Integer(),
            'total': fields.Integer(),
            'items': fields.List(fields.Nested(cls)),
        }
        attrs = dict(fields, Meta=Meta)
        # Create a nameless class to prevent storing the class in
        # marshmallow's in-memory registry
        ResponsePageObject = type(str(''), (ma.Schema,), attrs)
        return ResponsePageObject(strict=True).loads(data)

class UserResponse(ResponseObject):
    name = fields.String()

I’ve also proposed #660 , which would expose a more straightforward way to bypass the registry. In the meantime, the above workaround should get you most of the way.