Unknown Excluded in sublists

Hi,

I am quite new to the marshmallow ecosystem. I am considering the use of desert as a way to write python API clients by explicitly describing their data structure to ease the deserialisation.

In the process of writing an API Client for a specific purpose one could be confronted to a lot of unnecessary data in API responses that he does not intend to use. Exploring this data and writing up the schemas while continuously testing, a way to build such code could be to default to marshmallow EXCLUDE for unknown fields. That way, continuous TDD with “deserialisation of real life JSONS” tests would be possible without ValidationError while not having to describe unused chunks of the data.

Furthermore, continuing to use EXCLUDE after this initial stage of development could be acceptable, if the policy of the API itself is to authorize adding of values for minor changes. One could want to make its API client future-proof by not breaking when such a change happens on the other side that he does not necessarily controls.

But this behavior is broken when deserialising unknown fields in a List:

from dataclasses import dataclass
from typing import List

import desert
import marshmallow


@dataclass
class Ambiguous:
    firstname: str
    lastname: str


@dataclass
class AmbiguousList:
    id: str
    not_yet_known: List[Ambiguous]


AmbiguousListSchema = desert.schema_class(AmbiguousList, meta={"unknown": marshmallow.EXCLUDE})()

AmbiguousListSchema.loads("""
{
    "id":"789456132",
    "not_yet_known": [
        {
            "firstname": "John",
            "lastname": "Smith"
        },
        {
            "firstname": "Henry",
            "lastname": "Smith",
        }
    ]
}
""")
# Ok

AmbiguousListSchema.loads("""
{
    "id":"789456132",
    "shiny_new_arg": "uninteresting info",
    "not_yet_known": [
        {
            "firstname": "John",
            "lastname": "Smith"
        },
        {
            "firstname": "Henry",
            "lastname": "Smith"
        }
    ]
}
""")
# Still Ok

AmbiguousListSchema.loads("""
{
    "id":"789456132",
    "shiny_new_arg": "uninteresting info",
    "not_yet_known": [
        {
            "firstname": "John",
            "lastname": "Smith"
        },
        {
            "firstname": "Henry",
            "lastname": "Smith",
            "shiny_new_arg": "uninteresting info"
        }
    ]
}
""")
# marshmallow.exceptions.ValidationError: {'not_yet_known': {1: {'shiny_new_arg': ['Unknown field.']}}}

I am not sure what is the library that is responsible. It seems that marshmallow_dataclass shares the same error.

[Ouradze]

Hi, if it can help someone, I used this to make it work with desert and still keeping it not too convoluted.

@dataclass
class Child:
    name: str

@dataclass
class Parent:
    name: str
    dummy_list: List[str]
    children: List[Child] = field(
        default_factory=list,
        metadata=desert.metadata(
            m_fields.Nested(
                desert.schema(Child, meta={"unknown": EXCLUDE}, many=True), required=True
            )
        ),
    )

[wanderrful]

There’s a similar discussion to this in the marshmallow-code repo: https://github.com/marshmallow-code/marshmallow/issues/1490