[Request] DD log forwarder: Allow CF stack to be deleted/destroyed

Describe what happened: Datadog forwarder is set up in our AWS account to forward logs from lambdas, using these steps. The forwarder uses DD’s CF stack, which is an IaC module in our infra. When we destroy our infra, CF stack fails to delete because S3 resource “ForwarderBucket” has an object (cache.json) in it. This causes our destroy pipelines to fail.

Describe what you expected: DD’s CF stack deletes seamlessly. Forwarder bucket should have force-delete enabled.

Steps to reproduce the issue:

1. Deploy this stack with terraform apply
2. terraform destroy yields: Error: error waiting for CloudFormation Stack deletion: failed to delete CloudFormation stack (DELETE_FAILED): ["The following resource(s) failed to delete: [ForwarderBucket]. " "The bucket you tried to delete is not empty (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty

[jpogo]

@agocs Would it be possible to export the name/arn of the S3 bucket as a CF output? That way it could be cleaned up manually in a deterministic way as a part of CI/CD pipelines

[agocs]

Hi @vkakde , this is going to be a longer-running feature request. Unfortunately, the Lambda Forwarder is written as a Cloudformation Stack, and there is no native force-delete on Cloudformation stacks. There is DeletionPolicy: Delete, but that puts us back into the same position where it requires the S3 bucket to be empty first.

The supported method for deploying the Lambda Forwarder as a Terraform stack is simply wrapping the Cloudformation template in Terraform, and that does not give us access to aws_s3_bucket’s Force Destroy option. We would have to reimplement the Lambda Forwarder deployment in native terraform. I’m not against that, but it’s an engineering effort.

There’s also the option in which we implement a bucket cleanup lambda function as a custom resource. It’s not the worst solution, but it needs thorough testing.

I’m going to document these possibilities in our internal feature request so that our PMs can prioritize it accurately, but I’ll close this issue for now. Please feel free to reopen or reach out to [e-mail] in the mean time.