[Feature Request] Recursive type aliases

Description

I’m trying to describe a top level union that allows arbitrary shapes of list, set or dict containing either primitives {bool, int , float, bytes, str} or collections when permissible (i.e. lists can contain anything, sets can only contain primitives and dicts can only have primitives as keys). This (example below) fails both in mypy & beartype and I’m wondering if there is an alternative path I could take or if support for this use case is in the roadmap.

Comparison with static type checkers

• beartype: error: beartype.roar.BeartypeCallHintForwardRefException: Forward reference "__main__.Response" referent typing.Union[int, bool, float, str, bytes, list['Response'], set[typing.Union[int, bool, float, str, bytes]], dict[typing.Union[int, bool, float, str, bytes], 'Response']] not class.
• mypy: ~Fails to deal with this and raises an error: error: Cannot resolve name "Response" (possible cyclic definition). A related long standing discussion at python/mypy#731~. Tested to work with version >= 0.981 with the enable_recursive_aliases option set.
• pyright: gets this right to the extent I was able to test (demonstrated in the example below)

Example

from __future__ import annotations
from typing import Union
from typing_extensions import TypeGuard

import beartype

Primitive = Union[int, bool, float, str, bytes]
Response = Union[
    Primitive, list["Response"], set[Primitive], dict[Primitive, "Response"]
]


def valid_keys(
    dct: Union[
        dict[Primitive, Response],
        dict[
            Response, Response
        ],  # needed as this is what the typeguard is filtering out.
    ]
) -> TypeGuard[dict[Primitive, Response]]:
    return all(isinstance(k, (int, bool, float, str, bytes)) for k in dct.keys())


@beartype.beartype
def whynot(x: Response) -> dict[Primitive, Response]:
    resp = {}

    if isinstance(x, list):
        it = iter(x)
        resp = dict(zip(it, it))
    elif isinstance(x, dict):
        resp = x
    else:
        raise ValueError()

    if valid_keys(resp):
        return resp
    raise ValueError()


print(whynot({"bar": "baz"}))  # {"bar": "baz"}
print(whynot(["bar", "baz"]))  # {"bar": "baz"}
print(whynot({"bar": 1, b"baz": [2, 3]}))  # {"bar": 1, b"baz": [2,3]}
print(whynot(["bar", 1, b"baz", [2, 3]]))  # {"bar": 1, b"baz": [2,3]}
print(whynot({"bar": [1, {2, 3}]}))  # {"bar": [1, {2,3}]}
print(whynot(["bar", [1, {2, 3}]]))  # {"bar": [1, {2,3}]}
print(whynot(["bar", [1, {2, 3}]]))  # {"bar": [1, {2,3}]}
print(
    whynot({frozenset({"bar"}): [1, {2, 3}]})
)  # Runtime ValueError due to typeguard, and pyright error at callsite: Argument of type "dict[frozenset[str], list[int | set[int]]]" cannot be assigned....

In case you’re wondering why I can’t find a better use of my time

If you squint a bit the structure starts looking like something you’d get from a serialization protocol for a database that supports types such as dicts, lists and sets and a handful of primitives that mostly map to python built-ins and returns various combinations of those for different operations. Let’s say the database has two versions of said protocol, one which does not speak maps and sets and flattens those into arrays for the client to structure if they choose to and another that provides a hint when the response collection is a map or set instead of just an array. Now … you decide to write a seriously type safe client for said database … coredis

p.s. Thank you for beartype. It is phenomenal!

[alisaifee]

First up, thank you for the thoughtful and comprehensive response. It’s so refreshing and allows me to remain hopeful about the kind of collaborative community I had imagined open source to be when I was young and didn’t need to worry about things like posture and standing desks…

Your recommendation to conditionally lower the type strictness based on typing.TYPE_CHECKING is indeed what I had arrived at - though I’m now left with thoughts of an even more disturbing nested conditional to change the type of Response depending on whether pyright or mypy is being used. Since coredis is a client library the real benefit of type annotations is for the consumers of the library and I’m not the kind of guy that tells you how you should statically lint your code (but I apparently have no problem recommending enabling beartype in production 😁 reference). This is not an urgent requirement though since the recursive datastructures are almost never exposed by coredis’ public API and are mostly munged internally with copious tomfoolery until a more concrete type can be exposed.

My rationale for trying to use beartype for said recursive datastructures returned by redis though is because it is just not possible for me to write sufficient tests to exercise all the variants of responses from redis across different server versions and use cases and I hope consumers of the library will turn on runtime checking in development environments and if there are de-serialization issues they can be documented in the destructured type definitions from the top level recursive definition.

Somewhat seriously though:

1. How can I help?
2. How can coredis help (not necessarily the same as question 1)? At the moment I enable beartype for the public redis API when running tests in CI which are run across redis {5,6,7} with pythons 3.{8,9,10,11}. Soon enough I’ll be enabling it at the de-serialization substrate (which in my local testing has already found a bunch of bugs).

For the record, I didn’t create coredis (well, I created the name, and I suck at naming things, so I created the least impressive part of the project), it is a fork of aredis which went unmaintained over the last couple of years.

[leycec]

Gah! It’s odd that mypy chose not to enable alias recursion by default, but I suppose we’ll all take what we can get. Still, that leaves @beartype in the unenviable “odd man out” position of being the only actively maintained type-checker to not support alias recursion. It hurts.

Thanks for the heads up, @alisaifee. I’ll reprioritize support for at least self-recursive module-scoped type aliases in beartype 0.12.0, our next stable release. Cue that dancing bear!