How to use beartype with boto3?

Recently discovered beartype and have been loving it, especially love the humor and lighthearted nature of the project!

I’m working with boto3 (the AWS Python SDK) and disocvered that it uses a factory to dynamically create the various class types at runtime, meaning that by default, you cannot annotate them. I discovered the boto3-stubs package exists as a way to provide both types to annotate with, as well as allowing various IDE auto-complete features to work.

However…(you might already see where this is going), because the types provided by boto3-stubs and the actual type that boto3 creates are not identical (even though they represent the same object/schema), the type validation fails when decorating a function that accepts/returns boto3 types as arguments with beartype. Below is a simple example:

from beartype import beartype
import boto3
from mypy_boto3_s3.service_resource import S3ServiceResource, Bucket

@beartype
def example(s3: S3ServiceResource) -> Bucket:
    return s3.Bucket('example')

if __name__ == "__main__":
    s3 = boto3.resource('s3')
    bucket = example(s3)

Traceback (most recent call last):
  File "example.py", line 11, in <module>
    bucket = example(s3)
  File "<string>", line 23, in example
  File "/home/paul/.cache/pypoetry/virtualenvs/beartype-boto3-V5MWr3_P-py3.8/lib/python3.8/site-packages/beartype/_decor/_error/errormain.py", line 301, in raise_pep_call_exception
    raise exception_cls(  # type: ignore[misc]
beartype.roar.BeartypeCallHintPepParamException: @beartyped example() parameter s3="s3.ServiceResource()" violates type hint <class 'mypy_boto3_s3.service_resource.S3ServiceResource'>, as "s3.ServiceResource()" not instance of <class "mypy_boto3_s3.service_resource.S3ServiceResource">.

I’ve also made a small example project for this so that people can play around with it/reproduce it.

Now poking around, I discovered validators and I assume this is the appropriate way to handle situations like this. However, after playing around a bit, while I can create a custom type (see example below) that works with beartype, I lose the other features that the boto3-stubs library provides such as the autocomplete/suggest features. Granted, those are not the most important things in the world, but they certainly help, especially when working in a larger project.

from typing import Annotated
from beartype.vale import Is

S3Bucket = Annotated[object, lambda obj: obj.__class__.__name__ == "s3.Bucket"] 

What I’m really after I suppose is a way to “map” types, or basically tell beartype, “hey, these types are the same” or something along those lines. Is it possible to have my salmon and eat it too?

[leycec]

Eureka! Ignore everything above, because it was bad. I have something better – nay, something ingenious – nay, something cruel and malicious that eats at the very beating heart of static type-checking like Lyme disease in PEP-compliant form.

So. There’s something awful and amazing called typing.TYPE_CHECKING. Ever heard of it? No? Great. Let’s continue. typing.TYPE_CHECKING is wonderful, because it officially allows us to lie straight to the face of static type checkers and be rewarded for it. The idea here is that we create two variants of each Boto3 type hint via PEP 484 type aliases, conditionally assigned depending on whether we’re currently performing static or runtime type-checking:

import boto3
from beartype import beartype
from typing import TYPE_CHECKING

# If performing static type-checking (e.g., mypy, pyright),
# conditionally import Boto3 type hint stubs that are
# safely usable *ONLY* at static type-checking time.
if TYPE_CHECKING:
    from mypy_boto3_s3.service_resource import S3ServiceResource, Bucket
# Else, @beartime-based runtime type-checking is
# being performed. In this case, alias the same
# Boto3 type hint stub names imported above to their
# semantically equivalent beartype validators that are
# safely usable *ONLY* at runtime type-checking time.
else:
    from typing import Annotated
    from beartype.vale import Is

    # Suck it, intellisense IDE. Suck it.
    S3ServiceResource = Annotated[object, Is[
        lambda obj: obj.__class__.__name__ == "s3.ServiceResourceOrSmthng"]]
    Bucket = Annotated[object, Is[
        lambda obj: obj.__class__.__name__ == "s3.Bucket"]]

# Do this for the good of the gross domestic product, @beartype.
@beartype
def example(s3: S3ServiceResource) -> Bucket:
    return s3.Bucket('example')

if __name__ == "__main__":
    s3 = boto3.resource('s3')
    bucket = example(s3)

Lastly, if you care about efficiency, ^{…you do care about efficiency, don’t you} you can optimize away the additional stack frame imposed by each lambda function defined above with semantically equivalent but much faster IsAttr[...] and IsEqual[...] beartype validators: e.g.,

...
else:
    from typing import Annotated
    from beartype.vale import IsAttr, IsEqual

    # Suck it even faster, intellisense IDE. Suck it.
    S3ServiceResource = Annotated[object,
        IsAttr['__class__', IsAttr['__name__', IsEqual["s3.ServiceResourceOrSmthng"]]]]
    Bucket = Annotated[object,
        IsAttr['__class__', IsAttr['__name__', IsEqual["s3.Bucket"]]]]

...

If that doesn’t work, I will graffittii my next GitHub comment with 10,000 soul-destroying repetitions of the sparkly heart emoji. We pray it never comes to that. 💖 💖 💖 💖 💖 💖 💖 💖 💖 💖 💖 💖 💖 💖 💖 💖

[leycec]

Awwwwwww. You so nice, Paul. As a supremely skilled sloth that rocks shades, I’m kinda concerned about hurting you if I emoji you too fast over here. Oh, God – here goes! 🤗

I can make a PR to add this to the documentation so that others who may come across in the future won’t have to scratch their heads.

Bwa-hah. I accidentally got there first ahead of you. Let me know how that reads for you.

Must be rough being a sloth. I sympathize. My beautiful wifey ^{←you just know she’s gonna see this} calls me a sloth all the time. Metaphorically speaking, she’s not wrong. This issue is everything I did tonight. 😅

In all seriousness, how may I express my thanks to you and the project for helping solve the missing piece of this puzzle?

Why, I’m so glad you asked. The answer is money. Just kidding! Yikes. Tough crowd tonight. Seriously, though. If you find any available volunteer time for open source, ^{yup so never} I’m certain we’d all love a package from Paul that is either:

• bearboto3 (…do it, do it), publishing Boto3 types safely usable by both @beartype and static type checkers via the TYPE_CHECKING Uncertainty Principle. GitHub stars will flow like wine.
• midinow (…do it, do it), packaging midi-nn into a full-blown thing. Machine-generated music is this fascinating thing that seeks to obsolete the last remaining human MIDI composer alive. When I was young and had this even more fascinating thing nebulously called “free time,” I made a 90’s-era online metal MIDI repository called “Mysidia.” Yup. Metal. MIDI. Synth don’t get more hardcore. </ahem> It’d be fascinating to hear machine intelligence tackle classically unrepresented genres. So, anything other than classical is what we’re saying. Does such a thing exist? For the love of the poorly sampled fretless bass, make such a thing exist.

Thanks again for all this voluminous interest! Don’t hesitate to drop me a line anywhere you need a helping hand with @beartype and/or Python type-checking in general.