Nested field with many=True and required=True gives unexpected error for missing data

Say I have following schema definition

from marshmallow import Schema, fields


class ObjectSchema(Schema):
    id = fields.Integer(required=True)
    name = fields.String(required=True)


class CollectionSchema(Schema):
    data = fields.Nested(ObjectSchema, many=True, required=True)

Here’s how this schema works for several invalid inputs

In [6]: CollectionSchema().load({})
Out[6]: UnmarshalResult(data={}, errors={'data': {0: {'name': ['Missing data for required field.'], 'id': ['Missing data for required field.']}}})

In [7]: CollectionSchema().load({'data': []})
Out[7]: UnmarshalResult(data={'data': []}, errors={})

In [8]: CollectionSchema().load({'data': [{}]})
Out[8]: UnmarshalResult(data={'data': [{}]}, errors={'data': {0: {'name': ['Missing data for required field.'], 'id': ['Missing data for required field.']}}})

IMO Out[8] is the only correct validation error from the examples - it correctly describes the problem - all required fields are missing in nested item. For other examples I actually expect something like:

In [6]: CollectionSchema().load({})
Out[6]: UnmarshalResult(data={}, errors={'data': ['Missing data for required field.']})

shows that ‘data’ field is required and missing. I’m missing this one badly.

In [7]: CollectionSchema().load({'data': []})
Out[7]: UnmarshalResult(data={'data': []}, errors={'data': ['Field may not be empty list.']})

shows that field contains empty list - for my usecase it’s the same as missing field. Well, this one is arguable, maybe Nested field requires another parameter to enforce validation errors in case of empty nested value.

So currently marshmallow cannot distinguish between these three cases which complicates automated error handling in JSON api I’m working on now - as I said errors do not describe the problem properly. What do you think - is it an issue or I’m missing something and there’s better approach to this problem?

[lafrech]

I was just caught by this while working on https://github.com/marshmallow-code/marshmallow/issues/378 and I also think current behaviour is surprising and this change should be reverted.

I disagree with the rationale presented in https://github.com/marshmallow-code/marshmallow/issues/319#issuecomment-166770421. There are ways to document an API, for instance using apispec to generate an OpenAPI doc. I can’t really picture my clients using this trial and error approach to learn what data is required. (OK, in fact, I’m pretty sure that’s what they do, we all do… But this is wrong.) The error message is not meant to document the API. We don’t expect the clients to send garbage just to get an error message specifying what the server expects. Besides, for those using the trial and error approach, reverting this change will just make it a few more steps.

I’d like to change this in v3.0. Any objection?

To be explicit, I’m talking about case 1 in OP. Regarding case 2, I agree with @sloria’s answer, this is the right behaviour.

[martinstein]

Quick comment from my side, since this issue currently has the status “Feedback welcome”: I agree with @chekunkov and @maximkulkin that the current functionality is very surprising/weird. I was expecting mm.fields.Nested(SomeSchema, many=True, required=True) to give me strict feedback when that field was missing completely. Instead I get error messages about missing data within the list, even though nothing was provided at all for that field.

I would strongly prefer if the change from #235 was reverted back to the old, simple behavior (or at least that should be the default behavior). Marshmallow shouldn’t try to be too clever in what it does. If people want this advanced functionality, it should be on an opt-in basis.