Connecting to SQS in docker after assume role/kubernetes IAM role not working

Please fill out the sections below to help us address your issue.

What issue did you see ? logs-from-kubernetes.txt when inside docker, can’t access role assumed on computer/iam role on kubernetes from my computer it works fine, it finds the credential and config files. when creating s3 client all works fine. this happens only in sqs client…

Steps to reproduce If you have a runnable example, please include it as a snippet or link to a repository/gist for larger code examples. simple python (3.7.4) code, boto3 (1.14.2), just creating a client for sqs. if __name__ == '__main__': boto3.set_stream_logger('') sqs = boto3.client('sqs')

Debug logs Full stack trace by adding boto3.set_stream_logger('') to your code. here is local docker, and attached kubernetes logs file

2020-07-02 07:05:24,593 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2020-07-02 07:05:24,597 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2020-07-02 07:05:24,598 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2020-07-02 07:05:24,602 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2020-07-02 07:05:24,602 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2020-07-02 07:05:24,604 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2020-07-02 07:05:24,605 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2020-07-02 07:05:24,612 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2020-07-02 07:05:24,613 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2020-07-02 07:05:24,613 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2020-07-02 07:05:24,613 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: env
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2020-07-02 07:05:24,632 botocore.credentials [DEBUG] Looking for credentials via: sso
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: config-file
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2020-07-02 07:05:24,633 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2020-07-02 07:05:24,634 botocore.credentials [DEBUG] Looking for credentials via: container-role
2020-07-02 07:05:24,634 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2020-07-02 07:05:24,635 urllib3.connectionpool [DEBUG] Starting new HTTP connection (1): 169.254.169.254:80
2020-07-02 07:05:25,646 urllib3.connectionpool [DEBUG] Starting new HTTP connection (2): 169.254.169.254:80
2020-07-02 07:05:26,660 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1336, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 379, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 428, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: AWSHTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 342, in _get_request
    response = self._session.send(request.prepare())
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2020-07-02 07:05:26,669 botocore.utils [DEBUG] Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2020-07-02 07:05:26,671 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/endpoints.json
2020-07-02 07:05:26,681 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f503ec53b00>
2020-07-02 07:05:26,696 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/sqs/2012-11-05/service-2.json
2020-07-02 07:05:26,701 botocore.hooks [DEBUG] Event creating-client-class.sqs: calling handler <function add_generate_presigned_url at 0x7f503eca0f80>
Traceback (most recent call last):
  File "EnrichmentWorkerService.py", line 88, in <module>
    sqs = boto3.client('sqs')
  File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 91, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 835, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 85, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 287, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 73, in get_client_args
    endpoint_url, is_secure, scoped_config)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 153, in compute_client_args
    s3_config=s3_config,
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 218, in _compute_endpoint_config
    return self._resolve_endpoint(**resolve_endpoint_kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 301, in _resolve_endpoint
    service_name, region_name, endpoint_url, is_secure)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 361, in resolve
    service_name, region_name)
  File "/usr/local/lib/python3.7/site-packages/botocore/regions.py", line 134, in construct_endpoint
    partition, service_name, region_name)
  File "/usr/local/lib/python3.7/site-packages/botocore/regions.py", line 148, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

[kdaily]

Hi @eldarnegrinperion,

I took your Dockerfile and can reproduce your error. I’ve determined the error is indeed caused by missing AWS configuration and credentials - they are not in the right place. The location of the AWS config and credentials file will change depending on the Docker container image you’re using, so they’ll need to be moved to the home directory of the user who is running the command. When using the base container image python:3.7.4, the container is run as root, and the home directory is /root/. This means your config and credentials files must be in /root/.aws/config and /root/.aws/credentials. I verified this by adding in the following to the Dockerfile, assuming your config and credential files are present in the current local directory (CAUTION: I would NOT recommend this method going forward, only to illustrate that this is the correct location):

## Add AWS config and credentials
RUN mkdir /root/.aws/
COPY config /root/.aws/
COPY credentials /root/.aws/

If I do not copy them, or copy them to /.aws/, I receive the same error as you, NoRegionError. I could also provide them by mounting the directory containing the config and credential files when running docker run via the -v flag.

There was some confusion as to why you seemed to get different behavior depending on which service client you were using - S3 or SQS. You can instantiate an S3 client without a region because it assumes you would be using the global region, so you will not receive a NoRegionError. However, you would eventually get a NoCredentialsError if you tried to run an S3 client command if your configuration file is missing (or, you haven’t set a region using another method). Other clients, like SQS, require a region when constructing the client; if it cannot find a region using any of the methods available (IMDS, environment variable, parameter, or config file) you’ll receive the NoRegionError.

I’ve ruled out any reason why boto3 would not be working, and this is an issue with your specific environment. We aren’t able to assist on the details related to Kubernetes - you would be better off asking in a forum like repost.aws or Stack Overflow.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.