OpenSearch InstanceType for create_domain not accepting m6g (Graviton)

Describe the bug I receive a ValidationException when calling create_domain for the opensearch client

Steps to reproduce If you have a runnable example, please include it as a snippet or link to a repository/gist for larger code examples.

boto3.version ‘1.18.63’

import boto3


simplified_cluster_config = {
    'InstanceCount': 1,
    'InstanceType': 'm6g.large.search',
    'DedicatedMasterEnabled': False,
    'ZoneAwarenessEnabled': False,
}


ebs_options = {
    'EBSEnabled': True,
    'VolumeType': 'gp2',
    'VolumeSize': 100,
}


advanced_security_options = {
    'Enabled': True,
    'InternalUserDatabaseEnabled': True,
    'MasterUserOptions': {
        'MasterUserName': '<user>',
        'MasterUserPassword': '<password>'
    }
}


root_ap = '''
{{
    "Version": "2012-10-17",
    "Statement": [
        {{
            "Effect": "Allow",
            "Principal": {{ "AWS": "*" }},
            "Action": ["es:*"],
            "Resource": "arn:aws:es:{region}:{ACCOUNT}:domain/{domain_name}/*"
        }}
    ]
}}'''


region = 'eu-west-1'
aes = boto3.client('opensearch', region)

domain_name = 'b3test'
ap = root_ap.format(domain_name=domain_name, region=region, ACCOUNT=ACCOUNT)
aes.create_domain(
    DomainName=domain_name,
    EngineVersion=f'OpenSearch_1.0',
    ClusterConfig=simplified_cluster_config,
    AccessPolicies='',
    EBSOptions=ebs_options,
    AdvancedSecurityOptions=advanced_security_options,
    NodeToNodeEncryptionOptions={'Enabled': True},
    EncryptionAtRestOptions={'Enabled': True},
    DomainEndpointOptions={'EnforceHTTPS': True}
)

Expected behavior This should create an OpenSearch Service domain with m6.large.search data nodes.

Debug logs Full stack trace by adding boto3.set_stream_logger('') to your code.

2021-10-16 19:57:25,694 botocore.hooks [DEBUG] Event before-parameter-build.opensearch.CreateDomain: calling handler <function generate_idempotent_uuid at 0x7f8dedf995f0>
2021-10-16 19:57:25,695 botocore.hooks [DEBUG] Event before-call.opensearch.CreateDomain: calling handler <function inject_api_version_header_if_needed at 0x7f8dedf9de60>
2021-10-16 19:57:25,695 botocore.endpoint [DEBUG] Making request for OperationModel(name=CreateDomain) with params: {'url_path': '/2021-01-01/opensearch/domain', 'query_string': {}, 'method': 'POST', 'headers': {'User-Agent': 'Boto3/1.18.63 Python/3.7.10 Linux/4.14.243-185.433.amzn2.x86_64 Botocore/1.21.63'}, 'body': b'{"DomainName": "b3test", "EngineVersion": "OpenSearch_1.0", "ClusterConfig": {"InstanceCount": 1, "InstanceType": "m6g.large.search", "DedicatedMasterEnabled": false, "ZoneAwarenessEnabled": false}, "AccessPolicies": "", "EBSOptions": {"EBSEnabled": true, "VolumeType": "gp2", "VolumeSize": 100}, "AdvancedSecurityOptions": {"Enabled": true, "InternalUserDatabaseEnabled": true, "MasterUserOptions": {"MasterUserName": "jon", "MasterUserPassword": "Kibana123!"}}, "NodeToNodeEncryptionOptions": {"Enabled": true}, "EncryptionAtRestOptions": {"Enabled": true}, "DomainEndpointOptions": {"EnforceHTTPS": true}}', 'url': 'https://es.eu-west-1.amazonaws.com/2021-01-01/opensearch/domain', 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x7f8ded6c08d0>, 'has_streaming_input': False, 'auth_type': None}}
2021-10-16 19:57:25,695 botocore.hooks [DEBUG] Event request-created.opensearch.CreateDomain: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f8ded6c0890>>
2021-10-16 19:57:25,696 botocore.hooks [DEBUG] Event choose-signer.opensearch.CreateDomain: calling handler <function set_operation_specific_signer at 0x7f8dedf994d0>
2021-10-16 19:57:25,696 botocore.auth [DEBUG] Calculating signature using v4 auth.
2021-10-16 19:57:25,697 botocore.auth [DEBUG] CanonicalRequest:
POST
/2021-01-01/opensearch/domain

host:es.eu-west-1.amazonaws.com
x-amz-date:20211016T195725Z
x-amz-security-token:IQoJb3JpZ2luX2VjEPP//////////wEaCXVzLWVhc3QtMSJHMEUCIBB2G4zVFX29gQzoFmX4VUzf+hfn5umoXHIhJYJt8UB9AiEAycmw6HuWkkn2ZdbQ+QVOckzNIU/Jxxs9nqrMtug+dEkqnQIIfBAAGgwxNzgxNjExOTc3OTEiDB7DP9MySnnujnNRuSr6AUwn2r4wGRrQe4kD7MT64/bOsgPO0fi4Le+IeLWnzry0oWJlr8aCXw3zA1KTeHIpET04gauKHChoWya4QXdDz7aIPFKls9Tv7RuPui6fXqjdrClilJ27w81HJ8Mu2lKhkhXPOORdzfPfM9kaGknGrETmRGfxG77QhK4GqHiomusvDbymr5yBG6RVLU8IvGOtIgo/IQyOOp4G2buFf8mAL+edO0mBXGcVRtv9vyTU/RvOoVN4JlS/y1ogiZXpQ67UyIP5HcmGCOS5Rt4pZDksHW9bJbjbIkGWYJRxCCwapvcQ/nry48V+10rUbJCry8Gf6MfxlvXGka/CJJkw8cGsiwY6nQGLobtLwirj7W2F/6zkdwGpbR1VR/DsYGPJ96N2Fi+oNZrjTGeXhcUoeaQd4cZZF7PuzKFfh39DCZF2uBuBfsPnxLOfsJz9bcg7Qct18NbJcUH+VGDvHJubF5y5vHcpgjHBuQh2LGYRpmgi7K/QCYvIvk6TTdBgX4puAFHep5lxqHV83ts2+IUDBk/km/2Fpw/5YS4KKKj5Ej9m4/BF

host;x-amz-date;x-amz-security-token
55a751ef1d16ad8a8ed22e28ea98ef433701582abda2284ff0d6bc369419b24f
2021-10-16 19:57:25,697 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20211016T195725Z
20211016/eu-west-1/es/aws4_request
81f9f5efacb4de5bad140e0e9f8776c97b5cfdabb566c6591cb46889173b2872
2021-10-16 19:57:25,697 botocore.auth [DEBUG] Signature:
dc9d26ccc8547b5dc5c2070c0a92b9a2b83494fbe993d9a72985acd82322510b
2021-10-16 19:57:25,697 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://es.eu-west-1.amazonaws.com/2021-01-01/opensearch/domain, headers={'User-Agent': b'Boto3/1.18.63 Python/3.7.10 Linux/4.14.243-185.433.amzn2.x86_64 Botocore/1.21.63', 'X-Amz-Date': b'20211016T195725Z', 'X-Amz-Security-Token': b'IQoJb3JpZ2luX2VjEPP//////////wEaCXVzLWVhc3QtMSJHMEUCIBB2G4zVFX29gQzoFmX4VUzf+hfn5umoXHIhJYJt8UB9AiEAycmw6HuWkkn2ZdbQ+QVOckzNIU/Jxxs9nqrMtug+dEkqnQIIfBAAGgwxNzgxNjExOTc3OTEiDB7DP9MySnnujnNRuSr6AUwn2r4wGRrQe4kD7MT64/bOsgPO0fi4Le+IeLWnzry0oWJlr8aCXw3zA1KTeHIpET04gauKHChoWya4QXdDz7aIPFKls9Tv7RuPui6fXqjdrClilJ27w81HJ8Mu2lKhkhXPOORdzfPfM9kaGknGrETmRGfxG77QhK4GqHiomusvDbymr5yBG6RVLU8IvGOtIgo/IQyOOp4G2buFf8mAL+edO0mBXGcVRtv9vyTU/RvOoVN4JlS/y1ogiZXpQ67UyIP5HcmGCOS5Rt4pZDksHW9bJbjbIkGWYJRxCCwapvcQ/nry48V+10rUbJCry8Gf6MfxlvXGka/CJJkw8cGsiwY6nQGLobtLwirj7W2F/6zkdwGpbR1VR/DsYGPJ96N2Fi+oNZrjTGeXhcUoeaQd4cZZF7PuzKFfh39DCZF2uBuBfsPnxLOfsJz9bcg7Qct18NbJcUH+VGDvHJubF5y5vHcpgjHBuQh2LGYRpmgi7K/QCYvIvk6TTdBgX4puAFHep5lxqHV83ts2+IUDBk/km/2Fpw/5YS4KKKj5Ej9m4/BF', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIASS6Z4ELPYIIX3V6P/20211016/eu-west-1/es/aws4_request, SignedHeaders=host;x-amz-date;x-amz-security-token, Signature=dc9d26ccc8547b5dc5c2070c0a92b9a2b83494fbe993d9a72985acd82322510b', 'Content-Length': '608'}>
2021-10-16 19:57:25,698 botocore.httpsession [DEBUG] Certificate path: /home/ec2-user/.local/lib/python3.7/site-packages/certifi/cacert.pem
2021-10-16 19:57:25,698 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): es.eu-west-1.amazonaws.com:443
2021-10-16 19:57:25,746 urllib3.connectionpool [DEBUG] https://es.eu-west-1.amazonaws.com:443 "POST /2021-01-01/opensearch/domain HTTP/1.1" 409 53
2021-10-16 19:57:25,747 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'b96ffd5a-18c8-435e-a512-cba983ff539e', 'x-amzn-ErrorType': 'InvalidTypeException', 'Content-Type': 'application/json', 'Content-Length': '53', 'Date': 'Sat, 16 Oct 2021 19:57:25 GMT'}
2021-10-16 19:57:25,747 botocore.parsers [DEBUG] Response body:
b'{"message":"Invalid instance type: m6g.large.search"}'
2021-10-16 19:57:25,748 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'b96ffd5a-18c8-435e-a512-cba983ff539e', 'x-amzn-ErrorType': 'InvalidTypeException', 'Content-Type': 'application/json', 'Content-Length': '53', 'Date': 'Sat, 16 Oct 2021 19:57:25 GMT'}
2021-10-16 19:57:25,748 botocore.parsers [DEBUG] Response body:
b'{"message":"Invalid instance type: m6g.large.search"}'
2021-10-16 19:57:25,749 botocore.hooks [DEBUG] Event needs-retry.opensearch.CreateDomain: calling handler <botocore.retryhandler.RetryHandler object at 0x7f8ded6c0c90>
2021-10-16 19:57:25,749 botocore.retryhandler [DEBUG] No retry needed.
Traceback (most recent call last):
  File "m6g.py", line 62, in <module>
    DomainEndpointOptions={'EnforceHTTPS': True}
  File "/home/ec2-user/.local/lib/python3.7/site-packages/botocore/client.py", line 388, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/botocore/client.py", line 708, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidTypeException: An error occurred (InvalidTypeException) when calling the CreateDomain operation: Invalid instance type: m6g.large.search

[Jon-AtAWS]

Sure! Thanks for the response.

The access policy problem was unrelated. My string had newlines in it and that was causing the parse error. As an improvement, consider replacing newlines embedded in access policies with spaces.

[stobrien89]

Hi @Jon-AtAWS,

Thanks for the update— Feel free to reach out if anything else comes up!