question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ElasticSearch Streaming Function - You have exceeded the number of permissible concurrent requests with unique IAM Identities

See original GitHub issue

Before opening, please confirm:

  • I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
  • I have searched for duplicate or closed issues.
  • I have read the guide for submitting bug reports.
  • I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
  • I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v10.23.2

Amplify CLI Version

6.3.1

What operating system are you using?

Linux

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No

Amplify Categories

Not applicable

Amplify Commands

Not applicable

Describe the bug

Firstly, let me say that I know we are on an old version of the Amplify CLI, and thus an old version of the python Lambda function that streams data from DynamoDB to ElasticSearch. We are of course also on ElasticSearch 6.2. We started to look at a migration to the V2 Transformer and getting onto the latest version, but ran into a problem that prevented moving forward (#9123) .

That aside, we have found a problem with the DdbToEsFn-* function as shown below:

[ERROR] 2021-12-21T05:55:54.3Z 990a0f9b-c075-419c-81e9-86b105006d24 Traceback (most recent call last):
File "/var/task/python_streaming_function.py", line 261, in lambda_handler
return _lambda_handler(event, context)
File "/var/task/python_streaming_function.py", line 255, in _lambda_handler
post_to_es(es_payload) # Post to ES with exponential backoff
File "/var/task/python_streaming_function.py", line 102, in post_to_es
payload, es_region, creds, es_endpoint, '/_bulk')
File "/var/task/python_streaming_function.py", line 76, in post_data_to_es
raise ES_Exception(res.status_code, res._content)
python_streaming_function.ES_Exception: ES_Exception: status_code=400, payload=b'
{ "Message": "You have exceeded the number of permissible concurrent requests with unique IAM Identities. Please retry." }
'

In order to get the streaming function to work in a timely fashion (#4695) I have set “BatchSize”:100 & “MaximumBatchingWindowInSeconds”:2 - wondering if these settings might have an impact or if there are other configuration settings I should set.

Secondly, if this is an issue that can only be fixed with an update to the python Lambda function itself, can you confirm the latest V2 OpenSearch lambda function has no issue like this - that it retries if there is a failure like this.

Expected behavior

Any update in DynamoDB streams to ElasticSearch in a fail-safe manner.

Reproduction steps

See description

GraphQL schema(s)

# Put schemas below this line


Log output

# Put your logs below this line


Additional information

No response

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:9 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
sacramptoncommented, Feb 14, 2022

Hi @cjihrig - thanks for pointing me in the right direction.

I logged an issue with AWS OpenSearch Support and got this response. I’m including it here in case others run into this problem.

"…The above error happens when an Amazon OpenSearch node receives signed requests for authentication. Every node which received signed requests will interact with our internal service to validate an IAM access key and secret key included in the signature. The request rate is limited and throttled at each node level. This is done so that the internal service is not overloaded.

Unique IAM Identities mean unique access keys and secret keys. With the current setup, you might be using lot of unique credentials to sign the request that would be causing the problem. IE you may have many different clients sending signed requests simultaneously to this cluster to index or search data and all of those clients are using different IAM roles or credentials which have different secret and access keys.

To resolve the issue, the workarounds will be:

[A] Reducing variety of credentials (the number of IAM entities sending signed requests to the cluster) should help. To deal with this, you need to use the same credentials for your request for some time so that they won’t get throttled. Only unique IAM credentials matter. Requests with the same credentials won’t be throttled.

[B] The threshold for throttling is per node based. That means adding nodes to the cluster can increase limit.

[C] Retry the failed requests after some interval.

[D] Gradually increase the requests. Please try hitting the cluster with requests that have same signed requests or not more unique credentials in the requests so that the limit is not reached rather than hitting the cluster in a bulk. (Load testing)…"

0reactions
sacramptoncommented, Apr 20, 2022

Hi @cjihrig - just putting this here in case someone else runs into the same issue. I followed the above advice from OpenSearch support, but it made no differences (doubling the number of nodes).

Eventually OpenSearch Support said "We will increase the cache if confirm that you want it to be increased, so we need a confirmation to proceed. As stated by the internal team : “Increasing cache size may or may not improve cache hit ratio and thus reduce throttling”. Since the support team increased the cache limit on the OpenSearch cluster this issue has not reoccurred.

Read more comments on GitHub >

github_iconTop Results From Across the Web

You have exceeded the number of permissible concurrent ...
I am trying to query AWS ElasticSearch with 100,000 requests per minute. But after few mins i am getting the error as "You...
Read more >
Concurrent search request to elasticsearch - Elastic Discuss
I am able to perform only 5 parallel search requests in my application with default settings. how can we improve the scalability of...
Read more >
Amazon OpenSearch Service quotas
Your AWS account has default quotas, formerly referred to as limits, for each AWS service. Unless otherwise noted, each quota is Region-specific.
Read more >
xssfox parody account (@xssfox@cloudisland.nz) on ... - Twitter
"You have exceeded the number of permissible concurrent requests with unique IAM Identities. Please retry." This seems a very me problem.
Read more >
Amazon Elasticsearch Service - Noise
In this blog post, we will show you how you can use Amazon OpenSearch Service ... IAM Role ARN: arn:aws:iam::<111111111111>:role/aes-siem-sns-role.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found