Unknown field error on foreign key : cannot validate the marshmallow schemas of two SQLAlchemy in a Many-to-One relationship

I have got two SQLAlchemy objects as follow:

• an object Person with a string name that acts as a primary key.
• an object Submission with a simple id primary key and a field requester_name

In other words one person can make many submission, this is a many-to-one relationship, that I have set up exactly like in the sqlalchemy doc.

I can manipulate the code SQLAlchemy side without any problems, and when I dump the two schemas throughs marshmallow, I get the expected result.

But My issue is an unknown_field error on the foreign key when I try to validate the schema of the parent object.

Here is the code:

# tests
from pprint import pprint


from sqlalchemy import Column, ForeignKey, Integer, String, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship, sessionmaker


from marshmallow import ValidationError
from marshmallow_sqlalchemy import SQLAlchemyAutoSchema

# sqlalchemy

Base = declarative_base()
engine = create_engine("sqlite:///:memory:")
session = sessionmaker(bind=engine)()


class Person(Base):
    __tablename__ = "persons"
    name = Column(String(64), primary_key=True)


# MANY-TO-ONE. One Person can make many Submission
class Submission(Base):
    __tablename__ = "submissions"
    id = Column(Integer, primary_key=True)
    requester_name = Column(String(64), ForeignKey("persons.name"))
    requester = relationship("Person")


Base.metadata.create_all(engine)

# marshmallow
class PersonSchema(SQLAlchemyAutoSchema):
    class Meta:
        model = Person
        load_instance = True


class SubmissionSchema(SQLAlchemyAutoSchema):
    class Meta:
        model = Submission
        load_instance = True
    requester = fields.Nested(PersonSchema)



def test_person():#SUCCESS
    try:
        PersonSchema().load({"name": "Maria"}, session=session)
        assert True
    except ValidationError as err:
        pprint(err.messages)
        assert False


def test_submission():#FAILURE
    try:
        SubmissionSchema().load({"requester_name": "Maria"}, session=session)
        assert True
    except ValidationError as err:
        pprint(err.messages)
        assert False

Here is the raised ValidationError:

marshmallow.exceptions.ValidationError: {'requester_name': ['Unknown field.']}

Obviously the field is there (shows in the console, if I insert the object in the database, it’s there, etc.) and I don’t understand why thre’s an error like that.

I would appreciate any help.

[peterschutt]

By default the auto-generated schemas exclude foreign key fields:

https://github.com/marshmallow-code/marshmallow-sqlalchemy/blob/3d02bbb15d43e2e4c65039694faa8a73278c4006/src/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py#L67-L72

If you add include_fk = True to SubmissionSchema.Meta your example will run.

[peterschutt]

They do different things. One serializes a column property, the other serializes a relationship property. I’ll try to demonstrate why they should be independent of eachother.

Given SubmissionSchema defined like this:

class SubmissionSchema(SQLAlchemyAutoSchema):
    class Meta:
        model = Submission
        load_instance = True
        include_fk = True
        include_relationships = True
        
        # removed requester = fields.Nested(...)

Serializing a submission would look like this:

{'requester': 'A Name', 'id': 1, 'requester_name': 'A Name'}

… where the key requester represents the relationship, requester_name the foreign key and they both present the same information.

From the Person side, say we add a submissions relationship to Person:

class Person(Base):
    __tablename__ = "persons"
    name = Column(String(64), primary_key=True)
    submissions = relationship("Submission")

Adding include_fk=True to PersonSchema in this case will do nothing as there are no FKs, however adding include_relationships=True we get:

{'submissions': [1], 'name': 'A Name'}

I can’t see it mentioned in the docs anywhere but include_relationships=True creates a Related field to represent the relationship and then wraps it in a RelatedList if it is a “many” side. The default behavior of Related is to use the primary key in a “flattened” representation of the related instances.

If your models looked like this:

class Person(Base):
    __tablename__ = "persons"
    id = Column(Integer, primary_key=True, autoincrement=True)  # add surrogate PK
    name = Column(String(64), nullable=False)  # no longer PK field


# MANY-TO-ONE. One Person can make many Submission
class Submission(Base):
    __tablename__ = "submissions"
    id = Column(Integer, primary_key=True)
    requester_id = Column(Integer, ForeignKey("persons.id"))  # FK to Person.id
    requester = relationship("Person")

And the SubmissionSchema:

class SubmissionSchema(SQLAlchemyAutoSchema):
    class Meta:
        model = Submission
        load_instance = True
        include_fk = True
        include_relationships = True

Then serializing the submission:

{'id': 1, 'requester': 1, 'requester_id': 1}

Not a very helpful representation, so I’ll change SubmissionSchema to present more info about the requester:

class SubmissionSchema(SQLAlchemyAutoSchema):
    class Meta:
        model = Submission
        load_instance = True
        include_fk = True
        # don't need include_relationships=True as I've explicitly included it below

    requester = Related(["id", "name"])

Now serializing the submission returns:

{'requester': {'id': 1, 'name': 'A Name'}, 'id': 1, 'requester_id': 1}

Now that I’ve got a nested object representation of the requester, I might decide that it is redundant to also have the requester_id in the enclosing scope, so I’ll remove include_fk=True from the schema as well which then serializes the submission to:

{'requester': {'id': 1, 'name': 'A Name'}, 'id': 1}