Exceptions coming from boto3/botocore when running boto3.client('sts') too many times simultaneously
See original GitHub issueimport boto3, threading
for i in range(50):
threading.Thread(target=lambda: boto3.client('sts')).start()
And you get, tested on my Windows 10 machine (boto3 version 1.5.31, botocore version 1.8.45) and also on an Amazon Linux EC2, a bunch of exceptions like this:
Exception in thread Thread-20:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 914, in _bootstrap_inner
self.run()
File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "<stdin>", line 2, in <lambda>
File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\__init__.py", line 83, in client
return _get_default_session().client(*args, **kwargs)
File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 850, in create_client
credentials = self.get_credentials()
File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 474, in get_credentials
'credential_provider').load_credentials()
File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 926, in get_component
del self._deferred[name]
KeyError: 'credential_provider'
or:
Exception in thread Thread-24:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 914, in _bootstrap_inner
self.run()
File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "<stdin>", line 2, in <lambda>
File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\__init__.py", line 83, in client
return _get_default_session().client(*args, **kwargs)
File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 851, in create_client
endpoint_resolver = self.get_component('endpoint_resolver')
File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 726, in get_component
return self._components.get_component(name)
File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 926, in get_component
del self._deferred[name]
KeyError: 'endpoint_resolver'
Normally seems to be several credential_provider ones followed by several endpoint_resolver ones. The chance of getting these exceptions seems to increase with the number of threads.
Issue Analytics
- State:
- Created 5 years ago
- Reactions:12
- Comments:12 (4 by maintainers)
Top Results From Across the Web
Error handling — Boto3 Docs 1.26.32 documentation - AWS
Exceptions that you might encounter when using Boto3 will come from one of two sources: botocore or the AWS services your client is...
Read more >Handling Errors in Boto3 & Botocore - Trek10
This is one of the more common exceptions: a botocore ClientError is bubbling up from the API call layer (botocore) up to your...
Read more >Is boto3 client thread-safe - python - Stack Overflow
I recently tried using the single boto client instance using concurrent.futures.ThreadPoolExecutor . I run into exceptions coming from boto.
Read more >boto3 Sessions, and Why You Should Use Them | by Ben Kehoe
APPENDIX: Why is the AWS Python SDK called “boto3”? As so often happens, an AWS customer had to write something because AWS hadn't...
Read more >Botocore exceptions - miocittadino.it
If the proxy URL contained user information, this may have been included in logs where exceptions are recorded. s3 = boto3. 5, it...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
So finally sessions and resources can be shared across threads or not? Immediately after I start using them as described in the docs I hit
KeyError: 'credential_provider'
or similar. My threads do upload only so I guess I’m on safe side but I keep finding statements contradicting the docs.@zgoda-mobica , Same issue here. I had
ThreadPoolExecutor
and each thread was invokingboto3.client('lambda').invoke()
without usingSession()
which throwsKeyError: 'credential_provider'
error.So now using following in thread invoked method…
and it seems to be ok.