GetBucketLocation api failure for cross region buckets

Describe the bug

With the new release, looks like there’s a problem with the GetBucketLocation api call such that if the bucket is in a different region, the api call gives back an AccessDenied response. Here’s an event that shows the behavior. bucket-xyz is in us-east-1 and the S3 host endpoint is s3.us-west-2.amazonaws.com. This behavior was seen across all cross region api invocations.

    "eventSource": "s3.amazonaws.com",
    "eventName": "GetBucketLocation",
    "awsRegion": "us-west-2",
    "sourceIPAddress": "X.X.X.X",
    "userAgent": "[CloudCustodian/0.9.18 Python/3.9.13 Linux/4.14.255-276-224.499.amzn2.x86_64 exec-env/AWS_Lambda_python3.9 Botocore/1.23.32]",
    "errorCode": "AccessDenied",
    "errorMessage": "Access Denied",
    "requestParameters": {
        "bucketName": "bucket-xyz",
        "location": "",
        "Host": "s3.us-west-2.amazonaws.com"
    },

Seems to be related to this PR: https://github.com/cloud-custodian/cloud-custodian/pull/7524 The lambda fails pre execution. I suspect that’s happening since we’re trying to create a session as part of initializer method

What did you expect to happen?

I suspect this could be an issue upstream where cross region s3 endpoints return AccessDenied if the bucket is a different region.

Cloud Provider

Amazon Web Services (AWS)

Cloud Custodian version and dependency information

No response

Policy

No response

Relevant log/traceback output

[ERROR] ClientError: An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
Traceback (most recent call last):
  File "/var/task/custodian_policy.py", line 4, in run
    return handler.dispatch_event(event, context)
  File "/var/task/c7n/handler.py", line 165, in dispatch_event
    p.push(event, context)
  File "/var/task/c7n/policy.py", line 1246, in push
    return mode.run(event, lambda_ctx)
  File "/var/task/c7n/policy.py", line 466, in run
    return self.run_resource_set(event, resources)
  File "/var/task/c7n/policy.py", line 478, in run_resource_set
    with self.policy.ctx as ctx:
  File "/var/task/c7n/ctx.py", line 74, in __enter__
    self.initialize()
  File "/var/task/c7n/ctx.py", line 48, in initialize
    self.output = blob_outputs.select(self.options.output_dir, self)
  File "/var/task/c7n/output.py", line 54, in select
    return self[k](ctx, parse_url_config(selector))
  File "/var/task/c7n/resources/aws.py", line 539, in __init__
    region = s3_client.get_bucket_location(Bucket=self.bucket)['LocationConstraint']
  File "/var/runtime/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)

Extra information or context

No response

[ajkerrigan]

As of now, for us I can report that it’s happening with cross region output buckets. Let me double check other policies [update] aws.s3 policies seem to be working well

Cool, sounded that way from the description, good to confirm 😅

[PratMis]

As of now, for us I can report that it’s happening with cross region output buckets. Let me double check other policies [update] aws.s3 policies seem to be working well