Excessive Log Parser Lambda executions causing rate limit problems

Describe the bug I have reported this to the support team and I’m in ongoing conversations with them which you may be aware of, but they mentioned I could post here to receive updates too.

We migrate to WAF v2 recently with the latest version of Security Automations, and since doing so we’re having constant problems with rate limit errors, both via the API and in the console. This has now been tracked down to the Log Parser Lambda function being run excessively, meaning it’s using up the quota for updates to IP lists, causing us to be unable to make changes ourselves. The knock-on effect of this is that our costs have noticeably increased since migrating.

To Reproduce Migrate to WAF v2 and set up Security Automations by following the instructions. Then try to update an IP set, for example delete an IP. You then get one of two errors: ThrottlingException: Rate exceeded or WAFOptimisticLockException: AWS WAF couldn’t save your changes because someone changed the resource after you started to edit it. Reapply your changes.

Expected behavior No rate limiting errors from regular usage. No increase in costs from WAF Classic with previous version of Security Automations.

Please complete the following information about the solution:

• Version: 3.0.0

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, “(SO0021) - Video On Demand workflow with AWS Step Functions, MediaConvert, MediaPackage, S3, CloudFront and DynamoDB. Version v5.0.0”. If the description does not contain the version information, you can look at the mappings section of the template:

Mappings:
  SourceCode:
    General:
      S3Bucket: "solutions"
      KeyPrefix: "video-on-demand-on-aws/v5.0.0"

• Region: us-east-1
• Was the solution modified from the version published on this repository? No
• If the answer to the previous question was yes, are the changes available on GitHub?
• Have you checked your service quotas for the sevices this solution uses?
• Were there any errors in the CloudWatch Logs?

Screenshots If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context Add any other context about the problem here.

[suthar105]

I am facing the same problem, infect I have noticed that due to “WAFOptimisticLockException: AWS WAF couldn’t save your changes because someone changed the resource after you started to edit it. Reapply your changes.” error from updateipset action of logparsar lambda function it is doing same ip update attempts many times. Below is example event of update ip set with error… {"eventVersion":"1.05","userIdentity":{"type":"AssumedRole","principalId":"xxxx:xxxx-LogParser-xxxx","arn":"arn:aws:sts::xxxx:assumed-role/xxxx-LambdaRoleLogParser-xxxx/xxxx-LogParser-xxxx","accountId":"xxxx","accessKeyId":"xxxx","sessionContext":{"sessionIssuer":{"type":"Role","principalId":"xxxx","arn":"arn:aws:iam::xxxx:role/xxxx-LambdaRoleLogParser-xxxx","accountId":"xxxxx","userName":"xxxx-LambdaRoleLogParser-xxxx"},"webIdFederationData":{},"attributes":{"mfaAuthenticated":"false","creationDate":"2020-09-23T01:29:23Z"}}},"eventTime":"2020-09-23T01:45:22Z","eventSource":"wafv2.amazonaws.com","eventName":"UpdateIPSet","awsRegion":"ap-northeast-1","sourceIPAddress":"x.x.x.x","userAgent":"Boto3/1.14.48 Python/3.8.5 Linux/4.14.177-104.253.amzn2.x86_64 exec-env/AWS_Lambda_python3.8 Botocore/1.17.48","errorCode":"WAFOptimisticLockException","errorMessage":"AWS WAF couldn’t save your changes because someone changed the resource after you started to edit it. Reapply your changes.","requestParameters":{"name":"ScannersProbesSetIPV4","scope":"REGIONAL","id":"eaca0359-0200-4b8f-ac26-383a09fa89af","description":"Block Scanners/Probes IPV4 addresses","addresses":["59.127.234.253/32"],"lockToken":"058953b5-e66a-4da3-ae3f-109bc1087add"},"responseElements":null,"requestID":"6794773d-3290-4a56-9f46-3d7bb7b60ba3","eventID":"9f72ab5b-0aa8-4bc3-a2f7-2287730149f4","eventType":"AwsApiCall","apiVersion":"2019-04-23","recipientAccountId":"xxx"}

[dscpinheiro]

Hi!

We just released v3.2.0 of the solution, and this issue has been fixed.