Exceptions coming from boto3/botocore when running boto3.client('sts') too many times simultaneously

import boto3, threading
for i in range(50):
    threading.Thread(target=lambda: boto3.client('sts')).start()

And you get, tested on my Windows 10 machine (boto3 version 1.5.31, botocore version 1.8.45) and also on an Amazon Linux EC2, a bunch of exceptions like this:

Exception in thread Thread-20:
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "<stdin>", line 2, in <lambda>
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\__init__.py", line 83, in client
    return _get_default_session().client(*args, **kwargs)
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 850, in create_client
    credentials = self.get_credentials()
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 474, in get_credentials
    'credential_provider').load_credentials()
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 926, in get_component
    del self._deferred[name]
KeyError: 'credential_provider'

or:

Exception in thread Thread-24:
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "<stdin>", line 2, in <lambda>
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\__init__.py", line 83, in client
    return _get_default_session().client(*args, **kwargs)
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 851, in create_client
    endpoint_resolver = self.get_component('endpoint_resolver')
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 726, in get_component
    return self._components.get_component(name)
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 926, in get_component
    del self._deferred[name]
KeyError: 'endpoint_resolver'

Normally seems to be several credential_provider ones followed by several endpoint_resolver ones. The chance of getting these exceptions seems to increase with the number of threads.

[zgoda-mobica]

So finally sessions and resources can be shared across threads or not? Immediately after I start using them as described in the docs I hit KeyError: 'credential_provider' or similar. My threads do upload only so I guess I’m on safe side but I keep finding statements contradicting the docs.

[azec-pdx]

So finally sessions and resources can be shared across threads or not? Immediately after I start using them as described in the docs I hit KeyError: 'credential_provider' or similar. My threads do upload only so I guess I’m on safe side but I keep finding statements contradicting the docs.

@zgoda-mobica , Same issue here. I had ThreadPoolExecutor and each thread was invoking boto3.client('lambda').invoke() without using Session() which throws KeyError: 'credential_provider' error.

So now using following in thread invoked method…

session = boto3.session.Session()
lambda_client = session.client('lambda')
lambda_response = lambda_client.invoke(...)
....

and it seems to be ok.