typed containers

Not sure if this should be classified as a bug or as a feature request.

• OS: Linux
• Python version: 3.7.3
• Pydantic version: 0.24

from typing import Dict, List
from pydantic import BaseModel, ValidationError


class C(BaseModel):
    class Config:
        validate_assignment = True

    f1: List[Dict[int, str]] = []


try:
    # for some reason mypy doesn't catch this one
    o = C(f1=[1])
    print("FAIL")
except ValidationError:
    print("PASS")

o = C()

try:
    # mypy: List item 0 has incompatible type "int"; expected "Dict[int, str]"
    o.f1 = [1]
    print("FAIL")
except ValidationError:
    print("PASS")

try:
    # mypy: Argument 1 to "append" of "list" has incompatible type "int"; expected "Dict[int, str]"
    o.f1.append(1)
    print("FAIL")
except ValidationError:
    print("PASS")

try:
    # mypy: Dict entry 0 has incompatible type "str": "int"; expected "int": "str"
    o.f1.append({"": 1})
    print("FAIL")
except ValidationError:
    print("PASS")

o.f1.append({})

try:
    # mypy: Incompatible types in assignment (expression has type "int", target has type "str")
    o.f1[-1]["x"] = 2
    print("FAIL")
except ValidationError:
    print("PASS")

results in:

PASS
PASS
FAIL
FAIL
FAIL
types <class 'list'> <class 'dict'>

I am well aware of all the philosophical aspects of Python’s typing, but in my use case, where user uses Python to gradually build complex configuration model (including incremental adding of items to containers), it would be a much more productive workflow to be able to fail immediately at the location of the offending model modification rather than do a model validation afterwards, making user wonder “hey, where in my hundreds of lines of model I have done this wrong assignment/modification”. It is my understanding, that to be able to achieve this, it would be necessary for the fields of container types to be of custom container types performing all the necessary validations on all operations but from my experience, this is perfectly feasible.

P.S. The mypy has been executed as:

mypy --python-executable venv-pydantic/bin/python --strict pydantic1.py

but this issue is not about mypy.

[samuelcolvin]

We’re not talking about lots of people, so probably easier via comments with people saying they want it.

If 5 separate people say they’re interested in the feature I’ll accept a PR.

I think that seems fair?

[imtiazmangerah]

Having this feature baked into pydantic would be useful. I am currently working on a typed container for my specific use case. Here is what I have so far:

# typedlist.py
from collections.abc import MutableSequence
from typing import TypeVar, Generic, get_args

from pydantic import BaseModel
from pydantic.error_wrappers import ErrorWrapper, ValidationError

ListType = TypeVar('ListType')


class TypedList(Generic[ListType], MutableSequence):

    def __init__(self, lst = None):
        self._list_type = get_args(self.__orig_bases__[0])[0]
        self.list = list()

        if lst is not None:
            # This will validate each element in the list
            self.extend(lst)

    def __len__(self):
        return len(self.list)

    def __getitem__(self, i):
        return self.list[i]

    def __delitem__(self, i):
        del self.list[i]

    def __setitem__(self, i, v):
        self.list[i] = self._validate(v)

    def insert(self, i, v):
        self.list.insert(i, self._validate(v))

    def __str__(self):
        return str(self.list)

    def __repr__(self):
        return self.__str__()

    def _validate(self, v):
        """
        Run validation on an element in the list. If the bounded type is a custom type with a __get_validators__
        method, use that to perform the validation - else attempt to initialize the value using the list's type

        Fakes raising a pydantic ValidationError. The loc and Model is not known, so these values are meaningless.
        """
        validators = getattr(self._list_type, '__get_validators__', [])
        try:
            if validators:
                for validator in validators():
                    v = validator(v)
                return v

            return v if isinstance(v, self._list_type) else self._list_type(v)

        except (ValueError, TypeError, AssertionError) as exc:
            raise ValidationError([ErrorWrapper(exc, loc='TypedList')], BaseModel)

    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v):
        if isinstance(v, cls):
            return v

        return cls(v)

It makes use of __orig_bases__ from PEP 560, so this works for Python 3.7+. The way it currently runs validation and raises the ValidationError is hacky and fragile. It also requires a subclass of TypedList bound to the specific type of the container before use in a pydantic model.

Example usage of this container:

import pytest
from pydantic import BaseModel, ValidationError
from typedlist import TypedList 


class IntList(TypedList[int]): ...


class Test(BaseModel):
    numbers: IntList


def test_model():
    test_1 = Test(numbers=[1, 2, 3])
    assert list(test_1.numbers) == list([1, 2, 3])

    test_1.numbers.append(4)
    assert list(test_1.numbers) == list([1, 2, 3, 4])

    test_1.numbers.append("5")
    assert list(test_1.numbers) == list([1, 2, 3, 4, 5])

    with pytest.raises(ValidationError):
        test_1.numbers.append("not an int")

I currently do not have much knowledge of the internals of pydantic, and I am certain that this only works for my use case with the type of validators I am using. Any suggestions on how this can be improved upon, or concerns with the approach that I might not be aware of?