AWS IdentityStore client - cannot call `list_groups` without specifying a filter

Describe the bug I’m trying to use the boto3 binding for AWS IdentityStore, which is very new API. The API documentation says that when calling list_groups the Filter is optional, however it’s enforced as mandatory in the boto3 library.

Also, calls to list_groups with an empty list as the filter input or with a wildcard don’t appear to work etiher.

Steps to reproduce

client = boto3.client('identitystore')
# Fails with validation error
client.list_groups(IdentityStoreId=idst_id)

# Fails with validation error
client.list_groups(IdentityStoreId=idst_id, Filter=[])

# Fails with validation error
client.list_groups(IdentityStoreId=idst_id, Filter=[{}])

# Fails with validation error
client.list_groups(IdentityStoreId=idst_id, Filter=[{"AttributePath": "DisplayName", "AttributeValue": ""}])

# Returns a list with zero groups
client.list_groups(IdentityStoreId=idst_id, Filter=[{"AttributePath": "DisplayName", "AttributeValue": "*"}])

# Returns a list containing only the "sso-developer" group
client.list_groups(IdentityStoreId=idst_id, Filter=[{"AttributePath": "DisplayName", "AttributeValue": "sso-developer"}])

Expected behavior

I would like to call the API without a filter to iterate all groups in the account - it says on the API documentation that this is possible.

https://docs.aws.amazon.com/singlesignon/latest/IdentityStoreAPIReference/API_ListGroups.html#API_ListGroups_RequestSyntax

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/identitystore.html#IdentityStore.Client.list_groups

Debug logs Full stack trace by adding boto3.set_stream_logger('') to your code.

boto3.set_stream_logger('')
None
idst.list_groups(IdentityStoreId=idst_id)
2020-12-02 18:06:37,564 botocore.hooks [DEBUG] Event before-parameter-build.identitystore.ListGroups: calling handler <function generate_idempotent_uuid at 0x7f5f43e0db70>
2020-12-02 18:06:37,565 botocore.hooks [DEBUG] Event before-call.identitystore.ListGroups: calling handler <function inject_api_version_header_if_needed at 0x7f5f43e15378>
2020-12-02 18:06:37,565 botocore.endpoint [DEBUG] Making request for OperationModel(name=ListGroups) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AWSIdentityStore.ListGroups', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.16.28 Python/3.6.9 Linux/4.4.0-18362-Microsoft Botocore/1.19.28'}, 'body': b'{"IdentityStoreId": "d-REDACTED"}', 'url': 'https://identitystore.eu-west-1.amazonaws.com/', 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x7f5f42c355c0>, 'has_streaming_input': False, 'auth_type': None}}
2020-12-02 18:06:37,565 botocore.hooks [DEBUG] Event request-created.identitystore.ListGroups: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5f42c35c50>>
2020-12-02 18:06:37,565 botocore.hooks [DEBUG] Event choose-signer.identitystore.ListGroups: calling handler <function set_operation_specific_signer at 0x7f5f43e0da60>
2020-12-02 18:06:37,567 botocore.auth [DEBUG] Calculating signature using v4 auth.
2020-12-02 18:06:37,567 botocore.auth [DEBUG] CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:identitystore.eu-west-1.amazonaws.com
x-amz-date:20201202T180637Z
x-amz-security-token:REDACTED
x-amz-target:AWSIdentityStore.ListGroups

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
REDACTED
2020-12-02 18:06:37,568 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20201202T180637Z
20201202/eu-west-1/identitystore/aws4_request
REDACTED
2020-12-02 18:06:37,568 botocore.auth [DEBUG] Signature:
REDACTED
2020-12-02 18:06:37,568 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://identitystore.eu-west-1.amazonaws.com/, headers={'X-Amz-Target': b'AWSIdentityStore.ListGroups', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.16.28 Python/3.6.9 Linux/4.4.0-18362-Microsoft Botocore/1.19.28', 'X-Amz-Date': b'20201202T180637Z', 'X-Amz-Security-Token': b'REDACTED', 'Authorization': b'AWS4-HMAC-SHA256 Credential=REDACTED/20201202/eu-west-1/identitystore/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=REDACTED', 'Content-Length': '35'}>
2020-12-02 18:06:37,570 urllib3.connectionpool [DEBUG] Resetting dropped connection: identitystore.eu-west-1.amazonaws.com
2020-12-02 18:06:37,744 urllib3.connectionpool [DEBUG] https://identitystore.eu-west-1.amazonaws.com:443 "POST / HTTP/1.1" 400 63
2020-12-02 18:06:37,745 botocore.parsers [DEBUG] Response headers: {'Date': 'Wed, 02 Dec 2020 18:06:37 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '63', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'df9089b0-fdb2-4f48-9905-065440f17a2e'}
2020-12-02 18:06:37,745 botocore.parsers [DEBUG] Response body:
b'{"__type":"ValidationException","Message":"Filter is required"}'
2020-12-02 18:06:37,745 botocore.parsers [DEBUG] Response headers: {'Date': 'Wed, 02 Dec 2020 18:06:37 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '63', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'df9089b0-fdb2-4f48-9905-065440f17a2e'}
2020-12-02 18:06:37,745 botocore.parsers [DEBUG] Response body:
b'{"__type":"ValidationException","Message":"Filter is required"}'
2020-12-02 18:06:37,745 botocore.hooks [DEBUG] Event needs-retry.identitystore.ListGroups: calling handler <botocore.retryhandler.RetryHandler object at 0x7f5f42c35ef0>
2020-12-02 18:06:37,746 botocore.retryhandler [DEBUG] No retry needed.
2020-12-02 18:06:37,746 botocore.hooks [DEBUG] Event before-parameter-build.identitystore.ListGroups: calling handler <function generate_idempotent_uuid at 0x7f5f43e0db70>
2020-12-02 18:06:37,747 botocore.hooks [DEBUG] Event before-call.identitystore.ListGroups: calling handler <function inject_api_version_header_if_needed at 0x7f5f43e15378>
2020-12-02 18:06:37,747 botocore.endpoint [DEBUG] Making request for OperationModel(name=ListGroups) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AWSIdentityStore.ListGroups', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.16.28 Python/3.6.9 Linux/4.4.0-18362-Microsoft Botocore/1.19.28'}, 'body': b'{"IdentityStoreId": "d-936702bdcb"}', 'url': 'https://identitystore.eu-west-1.amazonaws.com/', 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x7f5f42c355c0>, 'has_streaming_input': False, 'auth_type': None}}
2020-12-02 18:06:37,747 botocore.hooks [DEBUG] Event request-created.identitystore.ListGroups: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5f42c35c50>>
2020-12-02 18:06:37,747 botocore.hooks [DEBUG] Event choose-signer.identitystore.ListGroups: calling handler <function set_operation_specific_signer at 0x7f5f43e0da60>
2020-12-02 18:06:37,748 botocore.auth [DEBUG] Calculating signature using v4 auth.
2020-12-02 18:06:37,748 botocore.auth [DEBUG] CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:identitystore.eu-west-1.amazonaws.com
x-amz-date:20201202T180637Z
x-amz-security-token:REDACTED
x-amz-target:AWSIdentityStore.ListGroups

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
REDACTED
2020-12-02 18:06:37,748 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20201202T180637Z
20201202/eu-west-1/identitystore/aws4_request
52f2c96132ef16b911f9a954c477672cb626ec21825ac2b009e8de8d9da81002
2020-12-02 18:06:37,749 botocore.auth [DEBUG] Signature:
REDACTED
2020-12-02 18:06:37,749 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://identitystore.eu-west-1.amazonaws.com/, headers={'X-Amz-Target': b'AWSIdentityStore.ListGroups', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.16.28 Python/3.6.9 Linux/4.4.0-18362-Microsoft Botocore/1.19.28', 'X-Amz-Date': b'20201202T180637Z', 'X-Amz-Security-Token': b'REDACTED', 'Authorization': b'AWS4-HMAC-SHA256 Credential=REDACTED/20201202/eu-west-1/identitystore/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=REDACTED', 'Content-Length': '35'}>
2020-12-02 18:06:37,851 urllib3.connectionpool [DEBUG] https://identitystore.eu-west-1.amazonaws.com:443 "POST / HTTP/1.1" 400 63
2020-12-02 18:06:37,852 botocore.parsers [DEBUG] Response headers: {'Date': 'Wed, 02 Dec 2020 18:06:37 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '63', 'Connection': 'keep-alive', 'x-amzn-RequestId': '5f120830-448a-4581-bb11-811ad417df99'}
2020-12-02 18:06:37,852 botocore.parsers [DEBUG] Response body:
b'{"__type":"ValidationException","Message":"Filter is required"}'
2020-12-02 18:06:37,852 botocore.parsers [DEBUG] Response headers: {'Date': 'Wed, 02 Dec 2020 18:06:37 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '63', 'Connection': 'keep-alive', 'x-amzn-RequestId': '5f120830-448a-4581-bb11-811ad417df99'}
2020-12-02 18:06:37,853 botocore.parsers [DEBUG] Response body:
b'{"__type":"ValidationException","Message":"Filter is required"}'
2020-12-02 18:06:37,853 botocore.hooks [DEBUG] Event needs-retry.identitystore.ListGroups: calling handler <botocore.retryhandler.RetryHandler object at 0x7f5f42c35ef0>
2020-12-02 18:06:37,854 botocore.retryhandler [DEBUG] No retry needed.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/luke/.virtualenvs/aws-deployments/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/luke/.virtualenvs/aws-deployments/lib/python3.6/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the ListGroups operation: Filter is required

[kdaily]

@adrienbuffet and others, I’m investigating where the service team is on this. Just to reiterate what was noted above, the API documentation currently states that filter is required:

We only support DisplayName as a valid filter attribute path currently, and filter is required.

As well as:

Filters

A list of Filter objects, which is used in the ListUsers and ListGroups request.

Type: Array of Filter objects

Required: Yes

It is incorrectly modeled in the API that the Python SDK uses as not required. This is what I’ll investigate.

[adrienbuffet]

Hello guys. Is there any news on this topic ? 😃