Boto3 Session cannot find credentials in the environment, throws exception

Here is my code:

import os

import boto3

print os.environ
session = boto3.Session(region_name='us-east-1')

Here’s the contents of os.environ, which get printed to the screen (with some variables stripped). Note that AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are both present.

{
    'CIRCLE_BUILD_NUM': '93',
    'SSH_ASKPASS': 'false',
    'HOME': '/home/ubuntu',
    'CI_PULL_REQUESTS': '',
    'NVM_CD_FLAGS': '',
    'rvm_trust_rvmrcs_flag': '1',
    'CIRCLE_TEST_REPORTS': '/tmp/circle-junit.Kx9Hk38',
    'rvm_path': '/home/ubuntu/.rvm',
    'CIRCLE_USERNAME': '(stripped)',
    'RUBY_VERSION': 'ruby-1.9.3-p448',
    'SSH_CLIENT': '10.16.130.250 40127 22',
    'CIRCLE_REPOSITORY_URL': '(stripped)',
    'USER': 'ubuntu',
    'NVM_DIR': '/home/ubuntu/nvm',
    'M2': '/home/ubuntu/.m2/apache-maven-3.2.5/bin',
    'PATH': '/home/ubuntu/webhooks/venv/bin:/usr/local/heroku/bin:/opt/google-cloud-sdk/bin:/home/ubuntu/webhooks/venv/bin:/home/ubuntu/.pyenv/shims:/home/ubuntu/.pyenv/bin:/home/ubuntu/.local/bin:/home/ubuntu/.go_workspace/bin:/usr/local/go_workspace/bin:/usr/local/go/bin:/opt/ghc/7.6.3/bin:/opt/happy/1.19.3/bin:/opt/alex/3.1.3/bin:/opt/cabal/1.22/bin:/opt/google-cloud-sdk/bin:/home/ubuntu/.m2/apache-maven-3.2.5/bin:/home/ubuntu/nvm/v0.10.33/bin:/home/ubuntu/.phpenv/shims:/home/ubuntu/.phpenv/bin:/home/ubuntu/.rvm/gems/ruby-1.9.3-p448/bin:/home/ubuntu/.rvm/gems/ruby-1.9.3-p448@global/bin:/home/ubuntu/.rvm/rubies/ruby-1.9.3-p448/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/android-sdk-linux/tools:/usr/local/android-sdk-linux/platform-tools:/home/ubuntu/.rvm/bin:/home/ubuntu/.composer/vendor/bin:/usr/local/gradle-1.10/bin:/usr/local/heroku/bin:/home/ubuntu/.rvm/bin',
    'CIRCLE_PROJECT_USERNAME': '(stripped)',
    'PS1': '(venv)',
    'DISPLAY': ':99',
    'MAKEFLAGS': '',
    'CIRCLE_COMPARE_URL': '(stripped)',
    'NVM_BIN': '/home/ubuntu/nvm/v0.10.33/bin',
    'TERM': 'vt100',
    'SHELL': '/bin/bash',
    'CIRCLECI': 'true',
    'XDG_SESSION_COOKIE': '(stripped)',
    'PYENV_SHELL': 'bash',
    'RBENV_SHELL': 'bash',
    'SHLVL': '2',
    '_system_arch': 'x86_64',
    'rvm_gemset_create_on_use_flag': '1',
    'rvm_version': '1.26.10 (1.26.10)',
    '_system_name': 'Ubuntu',
    'MAVEN_OPTS': '-Xmx2048m',
    'NVM_PATH': '/home/ubuntu/nvm/v0.10.33/lib/node',
    'GIT_ASKPASS': 'echo',
    'CIRCLE_NODE_INDEX': '0',
    'CIRCLE_NODE_TOTAL': '1',
    'MANPATH': '/home/ubuntu/nvm/v0.10.33/share/man:/home/ubuntu/.rvm/rubies/ruby-1.9.3-p448/share/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/home/ubuntu/.rvm/man',
    'GOPATH': '/home/ubuntu/.go_workspace:/usr/local/go_workspace',
    'CIRCLE_PREVIOUS_BUILD_NUM': '92',
    'M2_HOME': '/home/ubuntu/.m2/apache-maven-3.2.5',
    'JAVA_HOME': '/usr/lib/jvm/jdk1.7.0',
    'MFLAGS': '',
    'GEM_PATH': '/home/ubuntu/.rvm/gems/ruby-1.9.3-p448:/home/ubuntu/.rvm/gems/ruby-1.9.3-p448@global',
    'rvm_bin_path': '/home/ubuntu/.rvm/bin',
    'LANG': 'en_US.UTF-8',
    'CI': 'true',
    '_system_version': '12.04',
    'rvm_prefix': '/home/ubuntu',
    'LOGNAME': 'ubuntu',
    'CI_REPORTS': '/tmp/circle-junit.Kx9Hk38',
    'CIRCLE_SHA1': 'bd4cbd0943d1cde45abf6cacb580a3a389cc897a',
    'IRBRC': '/home/ubuntu/.rvm/rubies/ruby-1.9.3-p448/.irbrc',
    '_system_type': 'Linux',
    'MY_RUBY_HOME': '/home/ubuntu/.rvm/rubies/ruby-1.9.3-p448',
    'CI_PULL_REQUEST': '',
    'ANDROID_HOME': '/usr/local/android-sdk-linux',
    'AWS_ACCESS_KEY_ID': 'AKIA(stripped)',
    '_': '/home/ubuntu/webhooks/venv/bin/python',
    'ADB_INSTALL_TIMEOUT': '10',
    'CIRCLE_BUILD_URL': '(private)',
    'VIRTUAL_ENV': '/home/ubuntu/webhooks/venv',
    'NVM_IOJS_ORG_MIRROR': 'https://iojs.org/dist',
    'SSH_TTY': '/dev/pts/5',
    'OLDPWD': '/home/ubuntu/webhooks',
    'CIRCLE_PROJECT_REPONAME': 'webhooks',
    'ANDROID_NDK': '/usr/local/android-ndk',
    'CIRCLE_BRANCH': 'develop',
    'MAKELEVEL': '1',
    'PWD': '/home/ubuntu/webhooks/src',
    'CIRCLE_ARTIFACTS': '/tmp/circle-artifacts.Hk67jto',
    'AWS_SECRET_ACCESS_KEY': '(secret)',
    'MAIL': '/var/mail/ubuntu',
    'SSH_CONNECTION': '10.16.130.250 40127 10.0.3.193 22',
    'GEM_HOME': '/home/ubuntu/.rvm/gems/ruby-1.9.3-p448',
    'NVM_NODEJS_ORG_MIRROR': 'https://nodejs.org/dist'
}

However, the next line yields the following stack trace:

Traceback (most recent call last):
  File "deploy.py", line 11, in main
    session = boto3.Session(region_name='us-east-1')
  File "/home/ubuntu/webhooks/src/boto3/boto3/session.py", line 80, in __init__
    self._setup_loader()
  File "/home/ubuntu/webhooks/src/boto3/boto3/session.py", line 112, in _setup_loader
    self._loader = self._session.get_component('data_loader')
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 687, in get_component
    return self._components.get_component(name)
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 868, in get_component
    self._components[name] = factory()
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 176, in <lambda>
    lambda:  create_loader(self.get_config_variable('data_path')))
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 256, in get_config_variable
    elif self._found_in_config_file(methods, var_config):
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 277, in _found_in_config_file
    return var_config[0] in self.get_scoped_config()
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 339, in get_scoped_config
    profile_map = self._build_profile_map()
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 211, in _build_profile_map
    self._profile_map = self.full_config['profiles']
  File "/home/ubuntu/webhooks/src/botocore/session.py", line 366, in full_config
    self._config = botocore.configloader.load_config(config_file)
  File "/home/ubuntu/webhooks/src/botocore/botocore/configloader.py", line 105, in load_config
    parsed = raw_config_parse(config_filename)
  File "/home/ubuntu/webhooks/src/botocore/botocore/configloader.py", line 126, in raw_config_parse
    raise botocore.exceptions.ConfigNotFound(path=path)
botocore.exceptions.ConfigNotFound: The specified config file (/home/ubuntu/.aws/config) could not be found.

I expect that boto3/botocore will be able to find my credentials in the environment. The documentation states they should be load-able from the environment:

Boto3 will check these environment variables for credentials:

AWS_ACCESS_KEY_ID The access key for your AWS account. AWS_SECRET_ACCESS_KEY The secret key for your AWS account. AWS_SESSION_TOKEN The session key for your AWS account. This is only needed when you are using temporary credentials. > The AWS_SECURITY_TOKEN environment variable can also be used, but is only supported for backwards compatibility purposes. AWS_SESSION_TOKEN is supported by multiple AWS SDKs besides python.

I’m confused why this is occurring.

[smstec]

I still have this problem quite often. I’m using IAM roles, so I have no credentials file (only a .aws/config). I see it most often during runs of long multiprocessing jobs.

[kristijorgji]

2021 and still this issue is present. even with aws_ssm plugin which uses boto

Example

- name: lookup ssm parameter store in the current region
  debug: msg="{{ lookup('aws_ssm', '/somekey/subkey') }}"

This will crash due to unauthorised/no permission

Even though I have exported AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to environment as mentioned https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-environment-variables

Only way to work is ugly and specify credentials for every ssm plugin call like:

- name: lookup ssm parameter store in the current region
  debug: msg="{{ lookup('aws_ssm', '/mysuperparam', region=eu-central-1, aws_access_key='verysecretid', aws_secret_key='secreterkey'  ) }}"

Please fix boto 3 so it can read enviornment variables and save us so much boilerplate code