Support Generic Container Types

Feature Request

We want to define our own generic container-types (similar to List[T]), but we want to do the type-checking on our own:

class Array(numpy.ndarray, Collection[T]):
    @classmethod
    def get_validators(cls):
        yield cls.validate_type

    @classmethod
    def validate_type(cls, val):
        return np.array(val, dtype=cls.__args__[0])

(We read in very large lists from a json-file, and pydantic's validation logic is a real bottleneck).

class MyModel:
    values: Array[float]

Problem

For now, pydantic assumes that generic types are from a selected set (Tuple, List, Mapping, plus a few others) and therefore this assertion error is thrown here:

assert issubclass(origin, Mapping)

Possible solutions

Allowing this, could generally be done in two ways:

• Treat custom generic types not differently than other types. The type-parameter is just ignored by pydantic.
• Have a notion of general generic types in pydantic, and provide a validation-interface which allows users to interact with the type-parameters.

[kylebarron]

Note that as of Numpy 1.20, Numpy has a typing module that exposes its own Generic NDArray and DTypeLike classes. This means that numpy arrays can be typed now as:

import numpy as np
from numpy.typing import NDArray
arr: NDArray[np.float64] = np.array([1, 2, 3, 4], dtype=np.float64)

It would be great to be able to reuse that same syntax directly in Pydantic models:

from pydantic import BaseModel

class Model(BaseModel):
	arr_float64: NDArray[np.float64]
	arr_floating: NDArray[np.floating]
	arr_bool: NDArray[np.bool_]

I still get the Fields of type "<class 'numpy.ndarray'>" are not supported message when trying to assign this to a model. I looked briefly at trying to extend the pydantic.validators._VALIDATORS list to support NDArray but hit a wall and don’t have too much time at this moment to explore this.

I wouldn’t be surprised if there’s no appetite for official support for this in Pydantic itself, because arrays can’t be serialized efficiently to JSON, but it might be worthwhile enough for a plugin to implement this.

[danielhfrank]

Hello current and future pydantic users - this topic is near and dear to my heart, and I chose to run a bit with the solutions presented above. Here’s something that I came up with, using typing.Generic instead of custom metaclasses - the latter method wasn’t passing type checks in my codebase. I didn’t get around to shapes because those tend to be a bit more dynamic for my usecase. Hope it’s useful to someone!

DType = TypeVar('DType')


class TypedArray(np.ndarray, Generic[DType]):
    """Wrapper class for numpy arrays that stores and validates type information.
    This can be used in place of a numpy array, but when used in a pydantic BaseModel
    or with pydantic.validate_arguments, its dtype will be *coerced* at runtime to the
    declared type.
    """

    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, val, field: ModelField):
        dtype_field = field.sub_fields[0]
        actual_dtype = dtype_field.type_.__args__[0]
        # If numpy cannot create an array with the request dtype, an error will be raised
        # and correctly bubbled up.
        np_array = np.array(val, dtype=actual_dtype)
        return np_array

Full gist with tests to demonstrate functionality: https://gist.github.com/danielhfrank/00e6b8556eed73fb4053450e602d2434