can't register atexit after shutdown thrown in Python 3.9

Describe the bug

When attempting to create an s3 session resource running on python 3.9 function call throws with message can't register atexit after shutdown

Behaves as expected in python 3.8

Expected Behavior

When running in python 3.9 the following code (assuming valid values) results in s3 being a resource

session = boto3.Session(
        aws_access_key_id=public_key,
        aws_secret_access_key=secret_key,
    )

s3 = session.resource('s3', region_name=aws_region)

Current Behavior

When running in python 3.9 the following code (assuming valid values) results in an Exception

session = boto3.Session(
        aws_access_key_id=public_key,
        aws_secret_access_key=secret_key,
    )

s3 = session.resource('s3', region_name=aws_region)

Reproduction Steps

Run the following in python 3.9 with public_key, secret_key, and aws_region set to valid values

def x():
        session = boto3.Session(
                aws_access_key_id=public_key,
                aws_secret_access_key=secret_key,
        )

        s3 = session.resource('s3', region_name=aws_region)

Timer(0.5, x).start()

Possible Solution

Run with python 3.8? Error does not occur.

Additional Information/Context

No response

SDK version used

1.21.38

Environment details (OS name and version, etc.)

Mac OS 12.3.1, Python 3.9

[nateprewitt]

Hi @relativityboy, thanks for bringing this to our attention. This appears to be an issue specific to the S3 resource. I wasn’t able to reproduce it on other resources (ec2, dynamodb, etc). There was a change in Python 3.9 to prevent daemon threads in concurrent.futures due to unexpected behaviors with subprocesses.

We’ll need more investigation but from first glance this is likely an issue with the transfer module or s3transfer itself. It’ll need a deep dive to understand what’s conflicting.

[medley56]

I have another example of this that is occurring with a highly upvoted SO answer for uploading logs to S3 using atexit.register to register the uploader function during python shutdown. I suspect this registered hook is being cloned into some sort of multiprocessing under the hood and it’s attempting to call the hook for every subprocess (thread?).

Here is the SO answer that appears to now be a broken pattern: https://stackoverflow.com/a/51070892/2970906

Code sample below to reproduce:

import io
import logging
import os

import boto3

os.environ['AWS_PROFILE'] = 'my-aws-profile'


def write_logs(body, bucket, key):
    s3 = boto3.client("s3")
    s3.put_object(Body=body.getvalue(), Bucket=bucket, Key=key)


log = logging.getLogger("some_log_name")
log_stringio = io.StringIO()
handler = logging.StreamHandler(log_stringio)
handler.setLevel('DEBUG')
log.addHandler(handler)

atexit.register(write_logs, body=log_stringio, bucket="2022-04-28-log-bucket1", key="key_name")

log.warning("Hello S3")

Python 3.9.9 boto 1.21.42