add_athena_partitions.py should fire more often, or at the top of every hour exactly.

Describe the bug When using Athena-Log-Parser options for HTTP Flood, the add_athena_partitions.py is set to once per hour via CloudWatch events. If the CFN Stack is kicked off midway through the hour, then the partitions in AWS Glue pointing to the correct S3 hour of logs isn’t updated until midway through the hour. This creates a condition where Athena queries do not scan the correct S3 hour key until the Lambda kicks off updating the AWS Glue partitions.

relevent template snippet:

  LambdaAddAthenaPartitionsEventsRule:
    Type: 'AWS::Events::Rule'
    Condition: AthenaLogParser
    Properties:
      Description: Security Automations - Add partitions to Athena table
      ScheduleExpression: rate(1 hour)

To Reproduce Change QueryScheduledRunTime to 1 (line 294). Change line 1196 to ScheduleExpression: !Join ['', ['rate(', !FindInMap ["Solution", "Athena", "QueryScheduledRunTime"], ' minute)']]

Run the CFN template midway through any hour of the day with these params (making sure it completes by :45 after the hour or so):

ActivateAWSManagedRulesParam	no	
ActivateBadBotProtectionParam	no	
ActivateCrossSiteScriptingProtectionParam	yes	
ActivateHttpFloodProtectionParam	yes  Amazon Athena log parser	
ActivateReputationListsProtectionParam	yes	
ActivateScannersProbesProtectionParam	yes  Amazon Athena log parser	
ActivateSqlInjectionProtectionParam	yes	
AppAccessLogBucket	truncated	
EndpointType	ALB	
ErrorThreshold	50	
KeepDataInOriginalS3Location	No	
RequestThreshold	100	
WAFBlockPeriod	5

Modify Kinesis Firehose hints to 60s and 1MB via console.

This works, until the hour changes. Now, the AWS Glue Partition is not updated to point at the next hour in the S3 WAF Logs bucket until the LambdaAddAthenaPartitionsEventsRule fires which all depends on the minute within the hour that the LambdaAddAthenaPartitionsEventsRule resource was created.

To reproduce, Associate WAF with ALB. Send requests meeting threshold for flood. Wait until hour changes to next hour. Repeat and watch the flood rule not engage until the next time the LambdaAddAthenaPartitionsEventsRule fires.

Expected behavior Expect HTTP Flood to add IP to Blacklist after approximately 2 minutes (when threshold is achieved) and revert it back after 5 minutes (when requests stop) for every minute of every hour. This is the expectation with the parameters and modifications described above.

Please complete the following information about the solution:

• Version: [v3.1.0]
• Region: [us-east-1]
• [Yes as described above] Was the solution modified from the version published on this repository?
• [No ] If the answer to the previous question was yes, are the changes available on GitHub?
• [No ] Have you checked your service quotas for the sevices this solution uses?
• [ No] Were there any errors in the CloudWatch Logs?

Screenshots If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context To fix the problem,

  LambdaAddAthenaPartitionsEventsRule:
    Type: 'AWS::Events::Rule'
    Condition: AthenaLogParser
    Properties:
      Description: Security Automations - Add partitions to Athena table
      ScheduleExpression: rate(1 hour)

should be set to CRON at the top of every hour or to run every minute.

[aijunpeng]

Thanks for reporting the issue and providing detailed information. You are correct that the job should run at the top of every hour. We have added your request to the backlog and it will be looked into in future solution releases.

[dscpinheiro]

Hi!

We just released v3.2.0 of the solution, and this issue has been fixed.