Contract Test Decoding Issue

I was writing the contract tests for AWS::Redshift::ClusterSubnetGroup https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-redshift My handlers passed the inte test, but I failed on almost all of the contract test. And the error at all about decoding

I wrote the override file

{
  "CREATE": {
    "/SubnetIds": ["XXXXXXXX"],
    "/Description": "cfn test"
  }
}

The command I run

cfn test --enforce-timeout 180 -- -k contract_create_delete

I got the result below

==================================================================== FAILURES ====================================================================
_____________________________________________________________ contract_create_delete _____________________________________________________________

resource_client = <rpdk.core.contract.resource_client.ResourceClient object at 0x1100b89d0>

    @pytest.mark.create
    @pytest.mark.delete
    def contract_create_delete(resource_client):
        requested_model = delete_model = resource_client.generate_create_example()
        try:
            response = test_create_success(resource_client, requested_model)
            # check response here
            delete_model = response["resourceModel"]
        finally:
>           test_delete_success(resource_client, delete_model)

/usr/local/lib/python3.7/site-packages/rpdk/core/contract/suite/handler_create.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.7/site-packages/rpdk/core/contract/suite/handler_commons.py:108: in test_delete_success
    Action.DELETE, OperationStatus.SUCCESS, current_resource_model
/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rpdk/core/contract/resource_client.py:346: in call_and_assert
    status, response = self.call(action, request)
/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rpdk/core/contract/resource_client.py:357: in call
    response = self._call(payload)
/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rpdk/core/contract/resource_client.py:335: in _call
    payload = json.load(result["Payload"])
/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py:296: in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py:348: in loads
    return _default_decoder.decode(s)
/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py:337: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <json.decoder.JSONDecoder object at 0x10e9002d0>, s = '', idx = 0

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` beginning with
        a JSON document) and return a 2-tuple of the Python
        representation and the index in ``s`` where the document ended.
    
        This can be used to decode a JSON document from a string that may
        have extraneous data at the end.
    
        """
        try:
            obj, end = self.scan_once(s, idx)
        except StopIteration as err:
>           raise JSONDecodeError("Expecting value", s, err.value) from None
E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py:355: JSONDecodeError

rpdk.log below

[2020-07-23T00:29:17Z] DEBUG    - Logging set up successfully
[2020-07-23T00:29:17Z] DEBUG    - Running test: Namespace(cloudformation_endpoint_url=None, command=<function test at 0x10ff6e7a0>, endpoint='http://127.0.0.1:3001', enforce_timeout='180', function_name='TestEntrypoint', passed_to_pytest=['-k', 'contract_create_delete'], region='us-east-1', role_arn=None, subparser_name='test', verbose=0, version=False)
[2020-07-23T00:29:17Z] DEBUG    - Root directory: /Volumes/unix/workplace/resource-providers-redshift/clustersubnetgroup
[2020-07-23T00:29:17Z] DEBUG    - Loading project file '/Volumes/unix/workplace/resource-providers-redshift/clustersubnetgroup/.rpdk-config'
[2020-07-23T00:29:17Z] INFO     - Validating your resource specification...
[2020-07-23T00:29:17Z] DEBUG    - Resolved '#/definitions/Tag' to 'file:///Volumes/unix/workplace/resource-providers-redshift/clustersubnetgroup/aws-redshift-clustersubnetgroup.json#/definitions/Tag'
[2020-07-23T00:29:17Z] DEBUG    - Parsed 'file:///Volumes/unix/workplace/resource-providers-redshift/clustersubnetgroup/aws-redshift-clustersubnetgroup.json#/definitions/Tag' to '(<BASE>, 'definitions', 'Tag')'
[2020-07-23T00:29:17Z] DEBUG    - Edge from '(<BASE>, 'properties', 'Tags', 'items')' to '(<BASE>, 'definitions', 'Tag')'
[2020-07-23T00:29:17Z] DEBUG    - Rewriting refs in '<BASE>' (file:///Volumes/unix/workplace/resource-providers-redshift/clustersubnetgroup/aws-redshift-clustersubnetgroup.json)
[2020-07-23T00:29:17Z] DEBUG    -   '#/definitions/Tag' -> '#/definitions/Tag'
[2020-07-23T00:29:18Z] DEBUG    - temporary pytest.ini path: /var/folders/s2/f832hxln3vdg08m4hsvm7yy949s0c_/T/pytest_qvodv4p7.ini
[2020-07-23T00:29:18Z] DEBUG    - extra args: ['-k', 'contract_create_delete']
[2020-07-23T00:29:18Z] DEBUG    - pytest args: ['-c', '/private/var/folders/s2/f832hxln3vdg08m4hsvm7yy949s0c_/T/pytest_qvodv4p7.ini', '-m', '', '-k', 'contract_create_delete']
[2020-07-23T00:29:18Z] DEBUG    - Sending request
{
  "credentials": {
    "accessKeyId": "XXX",
    "secretAccessKey": "XXX",
    "sessionToken": "XXX"
  },
  "action": "CREATE",
  "request": {
    "clientRequestToken": "XXX",
    "desiredResourceState": {
      "Description": "cfn test",
      "SubnetGroupName": "",
      "SubnetIds": [
        "XXX"
      ],
      "Tags": []
    },
    "previousResourceState": null,
    "logicalResourceIdentifier": null
  },
  "callbackContext": null
}
[2020-07-23T00:30:49Z] DEBUG    - Sending request
{
  "credentials": {
    "accessKeyId": "XXX",
    "secretAccessKey": "XXX",
    "sessionToken": "XXX"
  },
  "action": "DELETE",
  "request": {
    "clientRequestToken": "XXX",
    "desiredResourceState": {
      "Description": "cfn test",
      "SubnetGroupName": "",
      "SubnetIds": [
        "XXX"
      ],
      "Tags": []
    },
    "previousResourceState": null,
    "logicalResourceIdentifier": null
  },
  "callbackContext": null
}
[2020-07-23T00:32:20Z] DEBUG    - Caught exit recommendation
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rpdk/core/cli.py", line 98, in main
    args.command(args)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rpdk/core/test.py", line 192, in test
    invoke_test(args, project, overrides, None)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rpdk/core/test.py", line 217, in invoke_test
    raise SysExitRecommendedError("One or more contract tests failed")
rpdk.core.exceptions.SysExitRecommendedError: One or more contract tests failed
[2020-07-23T00:32:20Z] CRITICAL - One or more contract tests failed

The output on sam docker:

START RequestId: b2dafb3e-9858-1dc9-f647-3718a1f260df Version: $LATEST
[DELETE] invoking handler...java.lang.OutOfMemoryError: Metaspace
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.resolveSyncHttpClient(SdkDefaultClientBuilder.java:243)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.finalizeSyncConfiguration(SdkDefaultClientBuilder.java:210)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:148)
	at software.amazon.awssdk.services.redshift.DefaultRedshiftClientBuilder.buildClient(DefaultRedshiftClientBuilder.java:28)
	at software.amazon.awssdk.services.redshift.DefaultRedshiftClientBuilder.buildClient(DefaultRedshiftClientBuilder.java:22)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:119)
	at software.amazon.redshift.clustersubnetgroup.ClientBuilder.getClient(ClientBuilder.java:10)
	at software.amazon.redshift.clustersubnetgroup.BaseHandlerStd$$Lambda$57/314337396.get(Unknown Source)
	at software.amazon.cloudformation.proxy.AmazonWebServicesClientProxy$1.client(AmazonWebServicesClientProxy.java:147)
	at software.amazon.cloudformation.proxy.AmazonWebServicesClientProxy$CallContext$1$1.done(AmazonWebServicesClientProxy.java:365)
	at software.amazon.cloudformation.proxy.CallChain$Completed.success(CallChain.java:393)
	at software.amazon.redshift.clustersubnetgroup.DeleteHandler.lambda$handleRequest$0(DeleteHandler.java:35)
	at software.amazon.redshift.clustersubnetgroup.DeleteHandler$$Lambda$58/1729199940.apply(Unknown Source)
	at software.amazon.cloudformation.proxy.ProgressEvent.then(ProgressEvent.java:175)
	at software.amazon.redshift.clustersubnetgroup.DeleteHandler.handleRequest(DeleteHandler.java:31)
	at software.amazon.redshift.clustersubnetgroup.BaseHandlerStd.handleRequest(BaseHandlerStd.java:18)
	at software.amazon.redshift.clustersubnetgroup.BaseHandlerStd.handleRequest(BaseHandlerStd.java:11)
	at software.amazon.redshift.clustersubnetgroup.HandlerWrapper.invokeHandler(HandlerWrapper.java:76)
	at software.amazon.redshift.clustersubnetgroup.HandlerWrapper.testEntrypoint(HandlerWrapper.java:100)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Metaspace: java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: Metaspace

Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Metaspace
	at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:65)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:119)
Caused by: java.lang.OutOfMemoryError: Metaspace
Function 'TestEntrypoint' timed out after 240 seconds

I can only see the the log that I sent the create and delete request. But I did not see any response. Maybe the root cause is that the contract tests are not receiving any response before timeout. I increased the timeout value to 240 in local repo as this issue. Still see the same problem. Can we root cause why we did not receive any response? Possible solution:

1. All of my contract test failed. Maybe it is configuration issue.
2. This resource is similar to https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-rds/tree/master/aws-rds-dbsubnetgroup. Although this is a simple resource. I still need to add stabilization as dbSubnetGroup did in create handler.

[angranamzn]

Increasing MemorySize also helped me:

Globals:
  Function:
    Timeout: 180  # docker start-up times can be long for SAM CLI
    MemorySize: 256

[xaoyuli]

some updates on my side. One issue I have is I cannot receive any response. The root cause is my Lambda function hit the limit size. I increase the size at my local repo. (Add property “MemorySize: 256”in template.yml) And problem fixed