Schema fields with dict attribute names return dict attributes when keys missing

Whenever a Schema is defined that has fields that shadow dict attributes (e.g. “items”, “keys”, “values”, etc), passing a dict with those keys not present to Schema.dump() results in dict attributes being serialized:

from marshmallow import Schema, fields

class A(Schema):
    a = fields.Str()
    keys = fields.Str(default='k')
    values = fields.Str(missing='v')
    items = fields.Str()

# No problem on load.
loaded = A().load({'a': 'x'})
print(loaded.data)
# {'a': 'x', 'values': 'v'}

# No problem when fields present.
dumped = A().dump({'a': 'x', 'keys': 'k', 'values': 'v', 'items': 'i'})
print(dumped.data)
# {'a': 'x', 'keys': 'k', 'items': 'i', 'values': 'v'}

# Undesired attribute access when fields not present.
dumped = A().dump({'a': 'x'})
print(dumped.data)

# on marshmallow==2.13.5
# {'a': 'x',
#  'keys': "dict_keys(['a'])",
#  'values': "dict_values(['x'])",
#  'items': "dict_items([('a', 'x')])"}

# on marshmallow==3.0.0b2
# {'a': 'x',
#  'keys': '<built-in method keys of dict object at 0x7f4e206bc948>'}
#  'values': '<built-in method values of dict object at 0x7f4e206bc948>',
#  'items': '<built-in method items of dict object at 0x7f4e206bc948>',

I found #30 which indicates that this is intended behavior for dumping dict objects, but I was wondering if 3.0.0 may be open to changing it (for dict classes only) or providing an option to disable dict-attribute access when dumping.

I’m aware that I could change the attribute name in the Schema definition and utilize load_from/dump_to but I wanted to avoid aliasing to keep the Schema attributes the same as the de-/serialization format.

In my own application, I’ve overridden Schema.get_attribute with a custom implementation so that dict attributes aren’t accessed at all but it would be nice to not to have to do this in each project.

If overriding Schema.get_attribute is the preferred method for implementing this functionality (instead of having a first-class option/setting), then I’ll keep doing that. However, it may be worth adding a warning or note in the documentation about this “gotcha” when dumping a dict object with a Schema that mirrors dict attributes. The section on Specifying Attribute Names does mention object attribute access but might be helpful to call out the potential field/dict-attribute conflicts that can result when dumping (unless I’m missing something else in the docs that covers this scenario).

Thoughts?

[lafrech]

I tried to refactor the get_value method and couldn’t find a way to support all cases. It’s no surprise, this function has been thought and re-thought a few times already.

def _get_value_for_key(obj, key, default):
    try:
        return obj[key]
    except KeyError:
        return default
    except (IndexError, TypeError, AttributeError):
        pass
    return getattr(obj, key, default)

This breaks a use case: a mapping object from which we want to extract both keys and attributes.

class PointDict(dict):
    def __init__(self, x, y):
        super().__init__({"x": x})
        self.y = y

Of course we can’t fix the OP and support that use case.

Now I’m wondering. This looks like quite an edge case, while the pure dict case should be common. And grabbing arbitrary attributes from an object meant to be a mapping is a bit risky. Especially since Python, for instance, could add new attributes to mapping classes anytime (unlikely, and that would probably be underscored attributes, but still).

Perhaps we could change the behaviour in marshmallow 4. It would be nice to provide a way (be it an override in user land documented in the docs) to get back the old behaviour, though.

Meanwhile, I’m afraid we can’t do much better than

• document the limitation
• list the forbidden attribute names
• perhaps raise a warning on fields with forbidden names

[semiga]

_values = Dict(String(), Dict(), data_key="values") might be worth included data_key="values" as an option for people who have stumbled into this issue