botocore vendored AWS cert bundle does not include us-iso root CA or use system trusted CA roots

Describe the bug

At the moment botocore vendors its own copy of cacert.pem which does not include the CA root for the us-iso partition. While we have trusted it in our systems CA root, botocore does not pick this up unless we explicitly pass the CA root to the tooling or set the ca_bundle flag in the config.

This does not seem to be an issue with the official Go SDK as that trusts the system root store and only if there is a custom ca_bundle set will it attempt to use it instead.

Steps to reproduce

Attempt to use the aws tooling/botocore through Ansible on us-iso partition.

Expected behavior

If the CA root is added to the system store, the connection should succeed.

Related

This was previously brought up in 2017: https://github.com/boto/botocore/issues/1332 when the landscape for the system managed CA roots was vastly different from today.

[archoversight]

The issue is that the default aws tooling does not support contacting the us-iso endpoints with the included cacert.pem since that does not include the CA root used for those endpoints (AFAIK they do not use commercial CA root).

This is not something that my AWS account manager or AWS support is able to help with.

We have been provided the appropriate CA certs and roots, and added them to our systems CA roots, however the aws utility and botocore does not consider those roots thereby causing failures and having to manually set AWS_CA_BUNDLE or setting ca_bundle in the AWS configuration to the appropriate CA roots. This is not necessary for Go tooling which uses the official AWS SDK since the Go AWS SDK does not bundle its own cacert.pem.

I am asking for this change in botocore to fix the issue in the underlying library. If you believe an active AWS support case or reaching out to my TAM will help expedite fixes in botocore for this issue, I am happy to pursue that avenue.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.