Not working with boto3 RDS

I’ve built a tiny unittest using boto3 and moto, hoping for success (per https://github.com/spulec/moto/issues/348):

import os
from unittest import TestCase

import boto3
import moto


class Tests(TestCase):

    @moto.mock_rds2
    def setUp(self):
        os.environ['AWS_DEFAULT_REGION'] = 'us-west-2'
        self.src_name = 'super-prod-clone-5000'
        self.src_type = 'db.r3.large'

        self.rds = boto3.client('rds')
        _create_db_instance(self.rds, self.src_name, self.src_type)

    @moto.mock_rds2
    def test_existing_destination(self):
        pass


def _create_db_instance(rds, name, instance_type):
    return rds.create_db_instance(
        DBName=name,
        DBInstanceIdentifier=name,
        AllocatedStorage=10,
        DBInstanceClass=instance_type,
        Engine='postgres',
        MasterUsername='root',
        MasterUserPassword='sosecretz',
    )

Unfortunately, running the testcase throws the following exception:

======================================================================
ERROR: test_existing_destination (tests.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/job/counsyl/rds_cp/venv/lib/python3.4/site-packages/moto/core/models.py", line 71, in wrapper
    result = func(*args, **kwargs)
  File "/home/job/counsyl/rds_cp/test/tests.py", line 20, in setUp
    _create_db_instance(self.rds, self.src_name, self.src_type)
  File "/home/job/counsyl/rds_cp/test/tests.py", line 35, in _create_db_instance
    MasterUserPassword='sosecretz',
  File "/home/job/counsyl/rds_cp/venv/lib/python3.4/site-packages/botocore/client.py", line 269, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/job/counsyl/rds_cp/venv/lib/python3.4/site-packages/botocore/client.py", line 312, in _make_api_call
    operation_model, request_dict)
  File "/home/job/counsyl/rds_cp/venv/lib/python3.4/site-packages/botocore/endpoint.py", line 110, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/job/counsyl/rds_cp/venv/lib/python3.4/site-packages/botocore/endpoint.py", line 137, in _send_request
    request, operation_model, attempts)
  File "/home/job/counsyl/rds_cp/venv/lib/python3.4/site-packages/botocore/endpoint.py", line 190, in _get_response
    operation_model.output_shape)),
  File "/home/job/counsyl/rds_cp/venv/lib/python3.4/site-packages/botocore/parsers.py", line 208, in parse
    parsed = self._do_parse(response, shape)
  File "/home/job/counsyl/rds_cp/venv/lib/python3.4/site-packages/botocore/parsers.py", line 405, in _do_parse
    root = self._parse_xml_string_to_dom(xml_contents)
  File "/home/job/counsyl/rds_cp/venv/lib/python3.4/site-packages/botocore/parsers.py", line 341, in _parse_xml_string_to_dom
    "invalid XML received:\n%s" % (e, xml_string))
nose.proxy.ResponseParserError: Unable to parse response (not well-formed (invalid token): line 1, column 0), invalid XML received:
b'{\n  "CreateDBInstanceResponse": {\n    "CreateDBInstanceResult": {\n      "DBInstance": {\n        "AllocatedStorage": 10,\n        "AutoMinorVersionUpgrade": "True",\n        "AvailabilityZone": "None",\n        "BackupRetentionPeriod": "1",\n        "CharacterSetName": null,\n        "DBInstanceClass": "db.r3.large",\n        "DBInstanceIdentifier": "super-prod-clone-5000",\n        "DBInstanceStatus": "available",\n        "DBName":"super-prod-clone-5000",\n        "DBParameterGroups": {\n            "DBParameterGroup": {\n            "ParameterApplyStatus": "in-sync",\n            "DBParameterGroupName": "default.postgres9.3"\n          }\n        },\n        "DBSecurityGroups": [\n          \n        ],\n        "Engine": "postgres",\n        "EngineVersion": "9.3.3",\n        "LatestRestorableTime": null,\n        "LicenseModel": "general-public-license",\n        "MasterUsername": "root",\n        "MultiAZ": "None",\n        "OptionGroupMemberships": [{\n          "OptionGroupMembership": {\n            "OptionGroupName": "default.postgres9.3",\n            "Status": "in-sync"\n          }\n        }],\n        "PendingModifiedValues": { "MasterUserPassword": "****" },\n        "PreferredBackupWindow": "13:14-13:44",\n        "PreferredMaintenanceWindow": "wed:06:38-wed:07:08",\n        "PubliclyAccessible": "True",\n        "AllocatedStorage": "10",\n        "Endpoint": {\n            "Address": "super-prod-clone-5000.aaaaaaaaaa.us-west-2.rds.amazonaws.com",\n            "Port": "None"\n        },\n        "InstanceCreateTime": null,\n        "Iops": null,\n        "ReadReplicaDBInstanceIdentifiers": [],"ReadReplicaSourceDBInstanceIdentifier": null,"SecondaryAvailabilityZone": null,\n        "StatusInfos": null,\n        "VpcSecurityGroups": [\n            {\n                "Status": "active",\n                "VpcSecurityGroupId": "sg-123456"\n            }\n        ]\n      }\n    },\n    "ResponseMetadata": { "RequestId": "523e3218-afc7-11c3-90f5-f90431260ab4" }\n  }\n}'

Here’s some additional logging:

botocore.credentials: DEBUG: Looking for credentials via: env
botocore.credentials: DEBUG: Looking for credentials via: shared-credentials-file
botocore.credentials: INFO: Found credentials in shared credentials file: ~/.aws/credentials
botocore.client: DEBUG: Registering retry handlers for service: rds
botocore.hooks: DEBUG: Event creating-client-class.rds: calling handler <function add_generate_presigned_url at 0x7fd87c0f9bf8>
botocore.endpoint: DEBUG: Making request for <botocore.model.OperationModel object at 0x7fd87b6a2828> (verify_ssl=True) with params: {'body': {'Version': '2014-10-31', 'DBInstanceIdentifier': 'super-prod-clone-5000', 'MasterUserPassword': 'sosecretz', 'MasterUsername': 'root', 'Action': 'CreateDBInstance', 'DBName': 'super-prod-clone-5000', 'AllocatedStorage': 10, 'DBInstanceClass': 'db.r3.large', 'Engine': 'postgres'}, 'url_path': '/', 'query_string': '', 'url': 'https://rds.us-west-2.amazonaws.com/', 'method': 'POST', 'headers': {'User-Agent': 'Boto3/1.1.1 Python/3.4.0 Linux/3.16.0-44-generic Botocore/1.1.7'}}
botocore.hooks: DEBUG: Event request-created.rds.CreateDBInstance: calling handler <bound method RDS._sign_request of <botocore.client.RDS object at 0x7fd87b42b0b8>>
botocore.auth: DEBUG: Calculating signature using v4 auth.
botocore.auth: DEBUG: CanonicalRequest:
POST
/

host:rds.us-west-2.amazonaws.com
user-agent:Boto3/1.1.1 Python/3.4.0 Linux/3.16.0-44-generic Botocore/1.1.7
x-amz-date:20150806T232740Z

host;user-agent;x-amz-date
ed72c98e5a7536fb64dd3ac85c5bd29762e597ae21ad53cfd428e3befabbbac1
botocore.auth: DEBUG: StringToSign:
AWS4-HMAC-SHA256
20150806T232740Z
20150806/us-west-2/rds/aws4_request
e1f9fc2382a625c599b93afac23e7519376e9cb588bb86dcf15778575c356f00
botocore.auth: DEBUG: Signature:
9d6b1949e82b1686448afbcde0bbc430c32c9466b3706860ecd7b10da85f0454
botocore.endpoint: DEBUG: Sending http request: <PreparedRequest [POST]>
botocore.vendored.requests.packages.urllib3.connectionpool: INFO: Starting new HTTPS connection (1): rds.us-west-2.amazonaws.com
botocore.vendored.requests.packages.urllib3.connectionpool: DEBUG: "POST / HTTP/1.1" 200 1960
botocore.parsers: DEBUG: Response headers: {'content-length': '1960', 'content-type': 'text/plain; charset=utf-8', 'date': 'Thu, 06 Aug 2015 23:27:40 GMT', 'server': 'Python/HTTPretty', 'status': '200', 'connection': 'close'}
botocore.parsers: DEBUG: Response body:
b'{\n  "CreateDBInstanceResponse": {\n    "CreateDBInstanceResult": {\n      "DBInstance": {\n        "AllocatedStorage": 10,\n        "AutoMinorVersionUpgrade": "True",\n        "AvailabilityZone": "None",\n        "BackupRetentionPeriod": "1",\n        "CharacterSetName": null,\n        "DBInstanceClass": "db.r3.large",\n        "DBInstanceIdentifier": "super-prod-clone-5000",\n        "DBInstanceStatus": "available",\n        "DBName":"super-prod-clone-5000",\n        "DBParameterGroups": {\n            "DBParameterGroup": {\n            "ParameterApplyStatus": "in-sync",\n            "DBParameterGroupName": "default.postgres9.3"\n          }\n        },\n        "DBSecurityGroups": [\n          \n        ],\n        "Engine": "postgres",\n        "EngineVersion": "9.3.3",\n        "LatestRestorableTime": null,\n        "LicenseModel": "general-public-license",\n        "MasterUsername": "root",\n        "MultiAZ": "None",\n        "OptionGroupMemberships": [{\n          "OptionGroupMembership": {\n            "OptionGroupName": "default.postgres9.3",\n            "Status": "in-sync"\n          }\n        }],\n        "PendingModifiedValues": { "MasterUserPassword": "****" },\n        "PreferredBackupWindow": "13:14-13:44",\n        "PreferredMaintenanceWindow": "wed:06:38-wed:07:08",\n        "PubliclyAccessible": "True",\n        "AllocatedStorage": "10",\n        "Endpoint": {\n            "Address": "super-prod-clone-5000.aaaaaaaaaa.us-west-2.rds.amazonaws.com",\n            "Port": "None"\n        },\n        "InstanceCreateTime": null,\n        "Iops": null,\n        "ReadReplicaDBInstanceIdentifiers": [],"ReadReplicaSourceDBInstanceIdentifier": null,"SecondaryAvailabilityZone": null,\n        "StatusInfos": null,\n        "VpcSecurityGroups": [\n            {\n                "Status": "active",\n                "VpcSecurityGroupId": "sg-123456"\n            }\n        ]\n      }\n    },\n    "ResponseMetadata": { "RequestId": "523e3218-afc7-11c3-90f5-f90431260ab4" }\n  }\n}'

Any help would be appreciated; would really like to be able to use moto to test boto3 code!

[spulec]

Yes, very similar. For an example of the CreateInstance response, see the XML at the bottom of here (the other endpoints usually have examples as well).

It looks like boto passes a Content-Type header so we should be able to vary responses based on that. I would think that it could be a pretty easy project (though you never really know until you get into it).

[spulec]

Yes, this should be fixed with #789. If not, feel free to reopen.