New region ap-east-1 causes issues with multiple services

Opening a new issue (formerly https://github.com/boto/boto3/issues/1943) as the new region has an impact on multiple services.

The following calls are failing:

• CloudFormation ListStacks
• CloudWatch DescribeAlarms
• CloutTrail DescribeTrails
• Config DescribeConfigRules
• Config DescribeConfigurationRecorderStatus
• Config DescribeConfigurationRecorders
• DynamoDB ListTables
• EC2 DescribeFlowLogs
• EC2 DescribeImages
• EC2 DescribeSnapshots
• EC2 DescribeVolumes
• EC2 DescribeVpcPeeringConnections
• EC2 DescribeVpcs
• ELB DescribeLoadBalancers
• EMR ListClusters
• ElastiCache DescribeCacheSecurityGroups
• Elasticache DescribeCacheParameterGroups
• KMS ListKeys
• Lambda ListFunctions
• RDS DBParameterGroups
• RDS DescribeDBSecurityGroups
• Redshift DescribeClusterParameterGroups
• Redshift DescribeClusterSecurityGroups
• SNS ListTopics
• SQS ListQueues

This is caused by the ap-east-1 region being returned by get_available_regions, even though it is not enabled.

There may be other instances, these are just the ones failing in https://github.com/nccgroup/ScoutSuite.

[kyleknap]

@j4v I agree we need to add a workaround for users that want to programmatically make API calls to all available regions but have not opted into every region (e.g. ap-east-1). I think it would make sense to have an opt-out flag in the get_available_regions method call that removes any opt-in regions from the returned list. So something like:

import boto3.session
# Does not list ap-east-1 because it is opt-in
boto3.session.Session().get_available_regions('s3', allow_opt_in_regions=False)
# However if you do not specify allow_opt_in_regions, you get all of the regions
boto3.session.Session().get_available_regions('s3')

I think this direction makes sense because:

1. We cannot know if a specific account has opted into a region. The get_available_regions() does not make any HTTP request and there is not even an API call to list all of the regions that is available to a user. So we cannot add the functionality of excluding a region that is opt-in without a new parameter or new method on the session.

2. There is already a allow_non_regional parameter to exclude regions that are just special endpoints. So there is a precedence for having a filter parameter in the method.

3. We would make the allow_opt_in_regions default to True because it would be a breaking change for users that have already opted into ap-east-1 and expect the get_available_regions() calls to return it to start filtering it out.

The only issue is that we do not have the upstream metadata yet in our endpoints.json to programmatically account for each new opt-in region. So we will have to go get that added in order to programmatically account for any new opt-in region.

Let us know what you think.

[SpenGietz]

@kyleknap I like that idea a lot, it would be really helpful for my use case.

It’s probably on the AWS side of development and not the boto side, but it would be nice if we could determine whether opt-in regions were enabled or not programmatically as well without the need for new APIs or permissions. Maybe something like STS GetCallerIdentity to a disabled opt-in region returns “region disabled” or something, rather than a somewhat-cryptic error message. I’m not sure the best route for that, more-so just throwing out ideas.