3.9+ PEP 585 - type hints for builtin types

In case you haven’t seen this: https://www.python.org/dev/peps/pep-0585/

Relevant:

>>> isinstance([1, 2, 3], list[str])
TypeError: isinstance() arg 2 cannot be a parameterized generic

[leycec]

Give @amogorkon a lever and he will move the Python world. Never would I consider riskily monkey-patching a GenericAlias-based __class_getitem__() implementation into Python’s core C types. You’ve just shown me a darkness beyond all prior conception of darkness – and I love it.

Does that cauldron of insanity generalize to everything else deprecated by PEP 585 too, like:

• Abstract base classes published by collections.abc and contextlib?
• Concrete classes published by collections and re?

If so, I’m all in and will gleefully add this eldritch horror “clean resolution” to our public knowledge base on circumventing PEP 585 deprecations.

Unrelatedly, justuse looks amazing. Between one-off imports (e.g., use("pprint").pprint(some_dict) ← _wut) and auto-installing imports (e.g., np = use("numpy", version="1.21.0rc2", hashes={"3c90b0bb77615bda5e007cfa4c53eb6097ecc82e247726e0eb138fcda769b45d"}, modes=use.auto_install) ← _{wtf is even happening there}), I’m both disturbed and impressed – and that’s hard to accomplish these days. Like most wilderness cabin-dwellers, I thrive on a daily diet of being jaded, bitter, and oppressively cynical.

Have you considered posting justuse to /r/Python? Actually, I’m considering doing that for you. All your sweet Reddit karma will be mine. Oh, yes. 👍

[leycec]

28 days ago, I promised I would do a thing. Today, I complete that promise by doing a thing. Introducing…

Beartype 0.3.0, dangerously released into the radioactive GitHub and PyPI badlands but moments ago. This release reveals the pure win of O(1) runtime type-checking through on onslaught of quasi-religious revelations, including…

Deep Type-checking of Nested Sequences in O(1) Time: Oh, It’s Deep

@beartype now type-checks both the shallow types and deep contents of parameters and return values annotated with typing-based sequences in guaranteed constant time. To do so, we exploit a variant of the well-known coupon collector’s problem by randomly type-checking one item at each nesting level of each sequence on each call of each decorated callable. This then enables us to statistically predict the number of calls required to fully type-check all items of an arbitrary container on average. ^{Math spoiler alert: it’s O(nlogn) calls.}

Currently supported typing-based sequences include:

• typing.List.
• typing.MutableSequence.
• typing.Sequence.

Since beartype 0.2.0 previously implemented deep type-checking for typing.Optional and typing.Union, we now deeply type-check all possible permutations of these type hints: e.g.,

from beartype import beartype
from typing import List, MutableSequence, Sequence, Union

@beartype
def hissing_nest_of_vipers(scary_vipers: Union[
    bool, str, Sequence[List[MutableSequence[float]]]) -> float:
    return my_floaty_tuple[0]

# These all totally work. Would beartype lie?
hissing_nest_of_vipers(True)
hissing_nest_of_vipers('They really taste great.')
hissing_nest_of_vipers(([[10.23, 9.24,], [8.25, 43.62,],], [[42.63,], [41.18,],]))

# This raises an exception. Beartype never lies! Never!
hissing_nest_of_vipers(([['Like, maybe don't put in your hand in there.'],],))

typing.Tuple and typing.NamedTuple are conspicuously absent with respect to deep type-checking, as their cray-cray syntax requires special-purposed handling that merits its own stable release (e.g., typing.Tuple[wut, even, is, happening, here, man]). More on that later.

Of course, tuples and named tuples are both immutable sequences. This means that you can already deeply type-check tuples and named tuples in beartype 0.3.0 by simply annotating parameters and return values as typing.Sequence instead: e.g.,

from beartype import beartype
from typing import Sequence

@beartype
def accept_my_floaty_tuple(my_floaty_tuple: Sequence[float]) -> float:
     return my_floaty_tuple[0]

# This totally works -- even without suspicious squinting, though that helps.
accept_my_floaty_tuple((0.1, 3.6, 2.7, 13.20, 12.21, 11.22,))

Optimized for His and Her Computational Pleasure under Python ≥ 3.8

Let’s talk about speed, 'cause everyone likes it fast.

Under Python ≥ 3.8, @beartype generates optimal code when deeply type-checking items contained in nested subsequences (e.g., List[List[List[str]]]). Of course, we use the unclean dirty magic of PEP 572-style assignment expressions to do so. Yes, that’s right, friends! PEP 572, the controversial syntactic change that prompted Guido to voluntarily abdicate his ancestral leadership role as BDFL, viciously lashes out again. Thanks to beartype 0.3.0, assignment expressions have finally proven their real-world utility. ^{Suck it, python-ideas!}

I’ve profiled this eye-, soul-, and career-destroying optimization to yield speedups of at least 200% by compare to the equivalent unoptimized code generated under Python < 3.8. For doubly- and triply-nested sequences (e.g., List[List[str]] and List[List[List[str]), it’s about 200% – but the performance gap only increases as the nesting level increases. A doubling in performance is only the minimum improvement under Python ≥ 3.8. Sequences more than triply-nested (e.g., List[List[List[List[List[str]]]]) should expect a comparably dramatic speedup, though it does seem a tad doubtful that anyone in the history of our plucky species would ever commit that kind of anarchic typing monstrosity. Still, it’s there. That’s nice.

Deep Human-readable Exception Messages: They’re Super-deep

@beartype now raises human-readable exceptions exhibiting the exact cause(s) of deep type-checking failures, including failures deeply nested in subcontainers: e.g.,

@beartyped pep_hinted() parameter pep_hinted_param=[([37],)] violates PEP type hint typing.List[typing.Sequence[typing.MutableSequence[str]]], as list item 0 tuple item 0 list item 0 value “37” not str.

Yeah. That’s deep like my old soul. Currently supported typing type hints that now generate deep exception messages include:

• typing.List.
• typing.MutableSequence.
• typing.Optional.
• typing.Sequence.
• typing.Union.

A win for readable typing is a win for all mankind.

The Dark Road Ahead

As America stands on the precipice of a renewed national unity and good tidings of brotherly love, so too do we chart the unmarked trail forwards to a brighter beartyped future:

Beartype 0.4.0: It’s Tuples All the Way Down

To maybe be released by late October, Beartype 0.4.0 will probably implement support for deeply type-checking typing.Tuple and typing.NamedTuple, mostly because that’s the minimal effort feature request that everyone (…which basically just means me and you, @albanie) wants.

Beartype 0.5.0: NumPy Arrays, Eh?

To maybe be released by late November, Beartype 0.5.0 will probably implement support for shallowly type-checking NumPy arrays via a newly introduced beartype.cave.NumpyArrayType thingy, because we desperately need this in both BETSE and our data science and machine learning consultancy. Of course, NumPy arrays deeply type-check themselves by definition. So, there’s no need for that here. What is there a need for then?

Shallow type-checking of NumPy array metadata, including:

• Dtype. Some dtypes like structured dtypes and the object dtype dramatically alter usage patterns. So, it only makes sense for callables to want to constrain the dtypes of the arrays they accept and/or return.
• Shape. The sizes of each dimension of a NumPy array (i.e., the number of items in that dimension) are usually treated as unconstrained free variables, but the number of these dimensions is usually constrained at callable definition time. As example, a callable expecting a two-dimensional NumPy array usually doesn’t care how many rows or columns that array has – only that array has exactly two dimensions. No more. No less.

So, I’m cogitating preliminary syntax like this:

from beartype import beartype
from beartype.cave import NumpyArrayType

# This constrains the passed "funbag_of_fun" parameter to be a two-dimensional
# NumPy array with any float dtype (e.g., "np.float32", "np.float64").
@beartype
def catch_the_funbag(funbag_of_fun: NumpyArrayType[float, 2]) -> float:
     return funbag_of_fun[0, 0]

Of course, there currently exists no typing standard for annotating NumPy arrays – but since when did a lack of accepted standards, community debate, or common sense ever stop us? Never, of course! Since NumPy is a third-party package (albeit a basically mandatory third-party package in 2020), it kinda seems doubtful there ever will exist a typing standard for annotating NumPy arrays.

That’s where we come in. Like, us. You and me. It’s lonely in here, but a sample size of 2 is still a sample size.

Beartype 0.6.0: The Mappings Are Not the Territory

To maybe be released by late December, Beartype 0.6.0 will probably implement support for deeply type-checking typing.Dict, typing.Mapping, and typing.MutableMapping. Those are the last big-ticket items we need, but they’re also the least trivial. Although the C-based CPython implementation almost certainly stores dictionary keys and values as hash bucket sequences, it doesn’t expose those sequences to the Python layer. That means we have no efficient random access to arbitrary dictionary keys or values from within @beartype.

Does that complicate O(1) runtime type-checking of dictionaries? Yes. Yes, it does. Which is why we’re saving dictionaries for last. I do have a number of risky ideas here, most of which revolve around internal caches of key and value iterators (i.e., the memory views returned by the dict.keys() and dict.values() methods). I don’t want to blow anything up, so this requires care, forethought, and a rusty blood-flecked scalpel.

Beartype 0.7.0: Who Can Tell the Future, Controls the Future

Beartype 0.7.0 is currently open to creative interpretation. Whatever else BETSE needs, beartype 0.7.0 gets.

Beyond that, the future is a smoking ruin of lost causes, dead dreams, and misspent middle-aged-hood. If you have any typing requests, speak now or forever hold your beartype feature list. 🗣️

^{don’t say “TypeVar” don’t say “TypeVar”}