Unable to delete hosted zone with DNS verified certificate

When I try to delete a stack that has a hosted zone with a certificate created through certificate manager, it has a CNAME entry created by certificate manager that prevents the hosted zone being deleted. Error report attached below.

My case is as follows though it’s possible you don’t need separate accounts, I haven’t checked. I have two accounts, one that has the DNS hosted zone and name servers for my existing domain name (say Account A) and a separate one that I’m creating a DNS hosted zone underneath (say Account B). For example, account A has the Route53 records for my domain example.com and I want to be able to create a subdomain test.example.com in account B that has a valid certificate.

I’ve got 3 CDK stacks do this, one to create the hosted zone in Account B, one to add the nameservers from that account to Account A and finally one to create the certificate in Account B (see the code samples below). This all works really well for creation but for deletion the certificate manager stack has left a CNAME entry similar to the following in the hosted zone preventing deletion:

_72fb0....bc94c.test.example.com. | CNAME | _c1c3d...ebd76.vhzmpjdqfx.acm-validations.aws. |  
-- | -- | -- | --

I would have expected the certificate manager that created this to remove it as well.

Reproduction Steps

Release the following stacks in order and delete them in reverse order, you’ll need to pass in appropriate environment variables.

Stack 1:

import os

from aws_cdk import aws_route53 as route53, core


class DnsHostedZoneStack(core.Stack):
    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        domain_url = os.environ["domain_url"]

        route53.PublicHostedZone(self, "DomainEnvironment", zone_name=domain_url)

Stack 2:

import json
import os

from aws_cdk import aws_route53 as route53, core


class DnsLinkHostedZoneStack(core.Stack):
    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        root_domain_url = os.environ["root_domain_url"]
        environment_subdomain = os.environ["environment_subdomain"]
        hosted_zone_id = os.environ["hosted_zone_id"]
        nameservers = json.loads(os.environ["nameservers"])

        full_domain_url = environment_subdomain + "." + root_domain_url

        main_hosted_zone = route53.HostedZone.from_hosted_zone_attributes(
            self,
            "MainHostLookup",
            zone_name=root_domain_url,
            hosted_zone_id=hosted_zone_id,
        )

        route53.RecordSet(
            self,
            "NameServerLink",
            record_name=full_domain_url,
            record_type=route53.RecordType.NS,
            zone=main_hosted_zone,
            target=route53.RecordTarget.from_values(*nameservers),
        )

Stack 3:

import os

from aws_cdk import (
    aws_route53 as route53,
    aws_certificatemanager as certificatemanager,
    core,
)


class CertificateStack(core.Stack):
    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        hosted_domain_url = os.environ["hosted_domain_url"]
        hosted_zone_id = os.environ["hosted_zone_id"]
        hosted_zone = route53.HostedZone.from_hosted_zone_attributes(
            self,
            "MainHostLookup",
            zone_name=hosted_domain_url,
            hosted_zone_id=hosted_zone_id,
        )

        self.certificate = certificatemanager.DnsValidatedCertificate(
            self,
            "EnvironmentCertificate",
            hosted_zone=hosted_zone,
            domain_name=hosted_domain_url,
            region="us-east-1",
            subject_alternative_names=[f"*.{hosted_domain_url}"],
            validation_method=certificatemanager.ValidationMethod.DNS,
        )

Error Log

   0 | 5:07:47 PM | DELETE_IN_PROGRESS   | AWS::CDK::Metadata       | CDKMetadata
   1 | 5:07:48 PM | DELETE_FAILED        | AWS::Route53::HostedZone | DomainEnvironment (DomainEnvironment7F28B06E) The specified hosted zone contains non-required resource record sets  and so cannot be deleted. (Service: AmazonRoute53; Status Code: 400; Error Code: HostedZoneNotEmpty; Request ID: 33c379cc-bede-48e8-8083-205a44fed498)
        new HostedZone (/private/tmp/jsii-kernel-HYp2Da/node_modules/@aws-cdk/aws-route53/lib/hosted-zone.js:16:26)
        \_ new PublicHostedZone (/private/tmp/jsii-kernel-HYp2Da/node_modules/@aws-cdk/aws-route53/lib/hosted-zone.js:116:9)
        \_ /Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7838:49
        \_ Kernel._wrapSandboxCode (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:8298:20)
        \_ Kernel._create (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7838:26)
        \_ Kernel.create (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7585:21)
        \_ KernelHost.processRequest (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7372:28)
        \_ KernelHost.run (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7312:14)
        \_ Immediate._onImmediate (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7315:37)
        \_ processImmediate (internal/timers.js:439:21)
   2 | 5:07:48 PM | DELETE_COMPLETE      | AWS::CDK::Metadata       | CDKMetadata
   3 | 5:07:48 PM | DELETE_FAILED        | AWS::CloudFormation::Stack | dns-hosted-zone The following resource(s) failed to delete: [DomainEnvironment7F28B06E].

 ❌  dns-hosted-zone: destroy failed Error: The stack named dns-hosted-zone is in a failed state: DELETE_FAILED (The following resource(s) failed to delete: [DomainEnvironment7F28B06E]. )
    at /Users/<snip>/node_modules/aws-cdk/lib/api/util/cloudformation.ts:165:13
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at waitFor (/Users/<snip>/node_modules/aws-cdk/lib/api/util/cloudformation.ts:76:20)
    at Object.destroyStack (/Users/<snip>/node_modules/aws-cdk/lib/api/deploy-stack.ts:261:26)
    at CdkToolkit.destroy (/Users/<snip>/node_modules/aws-cdk/lib/cdk-toolkit.ts:211:9)
    at main (/Users/<snip>/node_modules/aws-cdk/bin/cdk.ts:245:16)
    at initCommandLine (/Users/<snip>/node_modules/aws-cdk/bin/cdk.ts:172:9)
The stack named dns-hosted-zone is in a failed state: DELETE_FAILED (The following resource(s) failed to delete: [DomainEnvironment7F28B06E]. )

Environment

• CLI Version : 1.31
• Framework Version: 1.31 (guessing this is the Python library versions)
• OS : Mac
• Language : Python

Other

---

This is 🐛 Bug Report

[moltar]

This is extremely frustrating in CI, because the cleanup cannot happen automatically.

[skkrail-amzn]

Thanks @runtooctober! I was using aws-certificatemanager.Certificate with the validation prop set to CertificateValidation.fromDns(publicHostedZone). Switching to aws-certificatemanager.DnsValidatedCertificate like you mentioned gave me access to this feature.