What's the proper way of serializing a dynamically extended model for use in another process?

I can hack my way out of this situation, but it seems a common enough pattern that there might be a conical way of doing it that I’m not able to Google.

I want to be able to:

1. Dynamically add fields to some basic model, then save a bunch of data to a db.
2. In a second python process, I want to be able to query the data in the db I just created, using the dynamically extended class

However, if I try serializing the class with pickle, the dynamically added fields are missing from the model. Here’s some toy code to demonstrate:

import peewee
import pickle

DB = peewee.SqliteDatabase("test1.db")

class BasicModel(peewee.Model):
    class Meta:
        database = DB
    
    header = peewee.CharField()

user_defined_field_name = "my_field"
BasicModel._meta.add_field(user_defined_field_name, peewee.CharField())

row1 = BasicModel(header="header", my_field="data")
row1.save()

# This part runs, but doesn't work as intended
pickle.dump(BasicModel, open("BasicModel.pkl", "wb"))

So great, I have a db that has the data I want in it. Now, in a second process, I’d like to be able to query that data:

import pickle

BasicModel = pickle.load(open("BasicModel.pkl", "rb"))
example = BasicModel.select().limit(1)[0]
print(example.my_field)

Unfortunately, this results in:

AttributeError: 'VariantRecord' object has no attribute 'my_field'

A couple of sanity checks:

1. In the first process, I’m able to do print(dir(BasicModel)) and see my_field
2. In the second process, I’m able to do print(example.header)

It seems pretty clear that pickle isn’t serializing anything from add_field, but I’m not sure what to do about it. I don’t think it would be that hard to pickle enough information to re-execute the add_field code at runtime of the second process, but it’s still a hassle I’d rather avoid if possible.

Any suggestions? Thanks!

[coleifer]

The database schema is always the ultimate source of truth, regardless of the model class – that’s why introspection / playhouse.reflection is probably the best bet.

[Jessime]

Ah, I should have used a more descriptive word than just “valid”. I meant valid not so much in the technical sense as in a “this is a normal pattern we support in peewee” sense. In other words, maybe there could/should be a canonical serialization method, perhaps wrapped in Model.serialize()?

Regardless of whether that’s a reasonable suggestion:

1. I played around with pwiz and it get’s me where I need to go
2. I’ve actually used dataset a ton and love its simplicity and ease-of-use
3. Thanks for all the time and effort you’ve put into this library!