Unable to load up certain aws.service-quota Service Codes with Cloud Custodian

Describe the bug

In my org, I’m using Cloud Custodian to check the AWS service quotas. I have a script that goes through all the aws.service-quota servicecodes by updating a yml file then a runs the custodian run -s log -c <yml file> -r us-<region> -v for each of the service codes. Each loop of the script it runs one servicecode at a time.

The issue I’m facing that that, there are some service codes that still have this error:

Error: botocore.errorfactory.InvalidParameterCombinationException: An error occurred (InvalidParameterCombination) when calling the GetMetricStatistics operation: You have requested up to 86400 datapoints, which exceeds the limit of 1440. You may reduce the datapoints requested by increasing Period, or decreasing the time range.

The service codes are:

• braket
• cognito-idp
• ecr
• elasticmapreduce
• kms
• monitoring
• rds
• rekognition
• servicequotas

Is there a way to have these service codes be able to be checked?

What did you expect to happen?

These and the rest of the service codes to run and email with services that hot the AWS limits

Cloud Provider

Amazon Web Services (AWS)

Cloud Custodian version and dependency information

Please copy/paste the following info along with any bug reports:

Custodian:   0.9.13
Python:      3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:39:15) 
             [Clang 6.0 (clang-600.0.57)]
Platform:    posix.uname_result(sysname='Darwin', nodename='C02DW2AVMD6M', release='19.6.0', version='Darwin Kernel Version 19.6.0: Thu Sep 16 20:58:47 PDT 2021; root:xnu-6153.141.40.1~1/RELEASE_X86_64', machine='x86_64')
Using venv:  True
Docker: False
Installed: 

argcomplete==1.12.3
attrs==21.2.0
boto3==1.17.102
botocore==1.20.102
importlib-metadata==4.6.0
jmespath==0.10.0
jsonschema==3.2.0
pyrsistent==0.18.0
python-dateutil==2.8.1
pyyaml==5.4.1
s3transfer==0.4.2
setuptools==57.0.0
six==1.16.0
tabulate==0.8.9
typing-extensions==3.10.0.0
urllib3==1.26.6
zipp==3.4.1
(nct-cloud-custodian) giannattasiog@C02DW2AVMD6M nct-cloud-custodian %

Policy

No response

Relevant log/traceback output

Error: 
botocore.errorfactory.InvalidParameterCombinationException: An error occurred (InvalidParameterCombination) when calling the GetMetricStatistics operation: You have requested up to 86400 datapoints, which exceeds the limit of 1440. You may reduce the datapoints requested by increasing Period, or decreasing the time range.

Extra information or context

Yml file

policies:
  - name: service-quota-usage-limit
    description: |
      find any services that have usage stats of over 80%
    resource: aws.service-quota
    filters:
      - type: value
        key: ServiceCode
        op: eq
        value: braket
      - UsageMetric: present
      - type: usage-metric
        limit: 5 # persentage, should be 80 according to the requirement
    actions:
      - type: notify
        to:
          - <my email>
        subject: "Service Quota Limit - [custodian <my AWS account> - <The AWS region>]"
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/<my AWS account>/CustodianNotifyQueue
          region: <The AWS region>

Python Script

#!/usr/bin/env python
import subprocess
import fileinput 
import os
import time

number = 0 
file = <yml file name>
path = <yml file path>
temp_word = 'value: TextChange'
temp_word_verify = 'value: TextChange'
last_word = 'TextChange'
serviceCode = [ last_word, "AWSCloudMap", "a4b", "access-analyzer", "account", "acm", "acm-pca", "airflow", "amplify", "apigateway", "appconfig", "appflow", "application-autoscaling", "application-cost-profiler", "appmesh", "apprunner", "appstream2", "appsync", "aps", "athena", "auditmanager", "autoscaling", "autoscaling-plans", "backup", "batch", "bugbust", "cassandra", "ce", "chatbot", "chime", "cloud9", "cloudformation", "cloud-front", "cloudhsm", "cloudsearch", "cloudshell", "cloudtrail", "codeartifact", "codebuild", "codecommit", "codedeploy", "codeguru-profiler", "codeguru-reviewer", "codepipeline", "cognito-identity", "cognito-sync", "comprehend", "comprehendmedical", "compute-optimizer", "connect", "crowdscale-usagelimitservice", "databrew", "dataexchange", "datapipeline", "datasync", "dax", "deeplens", "deepracer", "directconnect", "discovery", "dlm", "dms", "docdb", "ds", "dynamodb", "ebs", "ec2", "ecs", "eks", "elastic-inference", "elasticache", "elasticbeanstalk", "elasticfilesystem", "elasticloadbalancing", "elastictranscoder", "es", "events", "fargate", "finspace", "firehose", "fis", "fms", "forecast", "frauddetector", "fsx", "gamelift", "geo", "glacier", "globalaccelerator", "glue", "grafana", "greengrass", "guardduty", "iam", "imagebuilder","inspector", "inspector2", "iot", "iot1click", "iotanalytics", "iotcore", "iotdeviceadvisor", "iotevents", "iotfleethub", "iotsitewise", "iotthingsgraph", "iotwireless", "ivs", "kafka", "kendra", "kinesis", "kinesisanalytics","kinesisvideo", "lakeformation", "lambda", "launchwizard", "lex", "license-manager", "lightsail", "logs", "lookoutequipment", "lookoutmetrics", "lookoutvision",  "machinelearning", "macie", "macie2", "managedblockchain", "mediaconnect", "mediaconvert", "medialive", "mediapackage", "mediastore", "mediatailor", "mgn", "migrationhubstrategy",  "monitron", "mq", "neptune", "network-firewall", "networkinsights", "networkmanager", "nimble","opsworks", "opsworks-cm", "organizations", "panorama", "personalize", "pinpoint", "polly", "profile", "proton", "qldb", "quicksight", "ram", "redshift", "resource-groups", "robomaker", "route53", "route53resolver", "s3", "s3-outposts", "schemas", "secretsmanager", "securityhub", "serverlessrepo", "servicecatalog", "ses", "shield", "signer", "sms", "snow-device-management", "snowball", "sns", "sqs", "ssm", "ssm-contacts", "ssm-incidents", "sso", "states", "storagegateway", "sumerian", "support", "swf", "textract", "timestream","transcribe", "transfer", "translate", "vmimportexport", "vpc", "waf", "waf-regional", "wafv2", "wam", "workspaces", "xray", last_word]
serviceCodelen = len(serviceCode)


os.chdir(path)
open(file, "r+")
open_file = open(file, "r+")
filedata = fileinput.FileInput(file)


# Runs custodian for each service code. Each pass modifies the yml file
for code in serviceCode:
    with open(file, "r+") as f:
        content = f.read()
        f.seek(0)
        f.truncate()
        content = content.replace(str(temp_word), 'value: '+str(code))
        f.write(content)
        temp_word = 'value: '+code
        number += 1 
        print('--------------')
        print('--------------')
        print('Processing Service code '+ str(code) + ' : '+ str(number - 1) +' of '+str(serviceCodelen - 2))
        print('--------------')
        print('--------------')
        f.close()      
        subprocess.check_call(['custodian run -s log -c <yml file name> -r <The AWS region> -v '], shell=True)

[CarloGiannattasio]

Sorry for the late response but yes

[kentnsw]

FYI, created a new PR https://github.com/cloud-custodian/cloud-custodian/pull/7140 to clear the CLA