contract_create_list_success fails when there are writeOnlyProperties

(I could also be misunderstanding the contract, see later. Related: the contract could probably use some extra documentation)

I currently have the following schema and list handler:

{
...
    "properties": {
        "KeyName": { ... },
        "PublicKey": { ... },
        "Fingerprint": { ... }
    },
    "required": [
        "KeyName",
        "PublicKey"
    ],
    "additionalIdentifiers": [
        [
            "/properties/Fingerprint"
        ]
    ],
    "readOnlyProperties": [
        "/properties/Fingerprint"
    ],
    "writeOnlyProperties": [
        "/properties/PublicKey"
    ],
    "createOnlyProperties": [
        "/properties/PublicKey",
        "/properties/KeyName"
    ],
    "primaryIdentifier": [
        "/properties/KeyName"
    ],
    "additionalProperties": false,
...
}

@resource.handler(Action.LIST)
def list_handler(
    session: Optional[SessionProxy],
    request: ResourceHandlerRequest,
    callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
    ec2 = session.client("ec2")  # type: botostubs.EC2
    keypairs = ec2.describe_key_pairs()["KeyPairs"]

    return ProgressEvent(
        status=OperationStatus.SUCCESS,
        resourceModels=[_create_model(x) for x in keypairs],
    )


def _create_model(o: Mapping) -> ResourceModel:
    return ResourceModel(
        KeyName=o["KeyName"],
        Fingerprint=o["KeyFingerprint"],
        # There is no way to get the PublicKey from the EC2 api.
        # That's why it's defined as a writeOnlyProperty in the resource spec:
        PublicKey=None,
    )

If I run cfn test (both with the version from pypi as with the current master branch, commit 06d5fa9dbd9e42016e8b3f8c4fda09be062646b4) I always get a failure of the contract_create_list_success test:

======================================================================= FAILURES =======================================================================
_____________________________________________________________ contract_create_list_success _____________________________________________________________

created_resource = ({'Fingerprint': '51:9a:ab:e6:9d:1c:f4:b9:c7:59:35:f5:0c:4c:06:8e', 'KeyName': '-', 'PublicKey': 'ssh-rsa AAAAB3NzaC1y...x0MGuK7VLppPvoozQVIi3zmGFcfjgou/WhkwUQy0GOo7RSeEQl20zluqn/7/uwkqapM3utXl1AFYxce7eA12whV2G0ByJLVZEKs40tNX Ben@Cloudar'})
resource_client = <rpdk.core.contract.resource_client.ResourceClient object at 0x10476f750>

    @pytest.mark.create
    @pytest.mark.list
    def contract_create_list_success(created_resource, resource_client):
        created_model, _request = created_resource
        models = test_list_success(resource_client, created_model)
>       assert created_model in models
E       AssertionError: assert {'Fingerprint': '51:9a:ab:e6:9d:1c:f4:b9:c7:59:35:f5:0c:4c:06:8e', 'KeyName': '-', 'PublicKey': 'ssh-rsa AAAAB3NzaC1yc...ux0MGuK7VLppPvoozQVIi3zmGFcfjgou/WhkwUQy0GOo7RSeEQl20zluqn/7/uwkqapM3utXl1AFYxce7eA12whV2G0ByJLVZEKs40tNX Ben@Cloudar'} in [{'Fingerprint': '51:9a:ab:e6:9d:1c:f4:b9:c7:59:35:f5:0c:4c:06:8e', 'KeyName': '-', 'PublicKey': None}]

/Users/ben/.local/share/virtualenvs/cloudformation-samples-riS8wOWn/lib/python3.7/site-packages/rpdk/core/contract/suite/handler_create.py:64: AssertionError

I would expect the test / contract to ignore all writeOnlyProperties when doing the list command.

(this is the part where I may be misunderstanding the contract, because:)

If I get the PublicKey (= a writeOnlyProperty) from the model, I can make the test succeed, but that feels wrong. I can’t always guarantee that this property has not been changed outside of CloudFormation (in this example I could do that by looking at the Fingerprint, but for other resources I might not have that kind of relationship available).

@resource.handler(Action.LIST)
def list_handler(
    session: Optional[SessionProxy],
    request: ResourceHandlerRequest,
    callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
    model = request.desiredResourceState
    ec2 = session.client("ec2")  # type: botostubs.EC2
    keypairs = ec2.describe_key_pairs()["KeyPairs"]

    return ProgressEvent(
        status=OperationStatus.SUCCESS,
        resourceModels=[_create_model(x, model) for x in keypairs],
    )


def _create_model(o: Mapping, model: ResourceModel) -> ResourceModel:

    if model.KeyName == o["KeyName"]:
        public_key = model.PublicKey
    else:
        public_key = None
    return ResourceModel(
        KeyName=o["KeyName"],
        Fingerprint=o["KeyFingerprint"],
        # There is no way to get the PublicKey from the EC2 api.
        # That's why it's defined as a writeOnlyProperty in the resource spec:
        PublicKey=public_key,
    )

is this a bug in the test, or am I not understanding the expected behaviour (related: is there more documentation about the contract than the source code of the tests)?

[jotompki]

Yes @ikben we are addressing some issues with writeOnly properties in the above PR. However, In the above test contract_create_list_success, the returned model from a successful create is taken and we ensure that the same model is returned in the list call. I suspect that the issue might be related to some sort of mismatch between the model returned by your create handler and the ones returned by the list handler. Does your model from create return writeOnlyProperties?

Also agree we need to add some more documentation around the contract of these handlers

[rjlohan]

Yeah we are working on fixing the handling of writeOnlyProperties in the contract tests. @jotompki posted a PR against the schema to start with: https://github.com/aws-cloudformation/aws-cloudformation-resource-schema/pull/64