Invalid MessageAttributes on SNS

Describe the bug After we upgraded to 2.32.0 our subscription filters on AWS SNS unfortunately broke, so messages were not forwarded to SQS queues. Looking into AWS CloudWatch metrics we saw NumberOfNotificationsFilteredOut-InvalidAttributes had started to happen.

The NumberOfNotificationsFilteredOut-InvalidAttributes metric shows messages that your filter policy rejected because the message attributes weren’t in a valid format.

https://repost.aws/knowledge-center/sns-subscription-filter-policy-issues

After downgrading to 2.31.0 we stopped getting NumberOfNotificationsFilteredOut-InvalidAttributes.

Here’s the MessageAttributes of one of our messages, but nothing in there immediately looks suspicious to me.

"MessageAttributes" : {
  "ourKey" : {"Type":"String","Value":"OurData"},
  "_datadog" : {"Type":"String","Value":"{\"traceparent\":\"00-000000000000000023b62f1bd3212848-1fe678cc9b22cbd2-01\",\"tracestate\":\"dd=s:1;t.dm:-1\",\"x-datadog-trace-id\":\"2573296033641605192\",\"x-datadog-parent-id\":\"2298657479990627282\",\"x-datadog-sampling-priority\":\"1\",\"x-datadog-tags\":\"_dd.p.dm=-1\"}"}
}

To Reproduce

Expected behavior MessageAttributes should contain valid data, parsable by AWS SNS subscription filters.

Runtime environment (please complete the following information):

• Instrumentation mode: automatic
• Tracer version: 2.32.0
• OS: Windows Server 2019 (AWS EB Windows Server)
• CLR: .NET 6.0.18
• AWSSDK.SimpleNotificationService: 3.3.101.11

Additional context

[zARODz11z]

If you’d like to continue using Tracer version: 2.32.0 you can disable the sns integration using DD_TRACE_AwsSns_ENABLED=0 I take responsibility for this issue as well, apologies for any inconvenience. The release is coming soon.

[pierotibou]

Fix released in https://github.com/DataDog/dd-trace-dotnet/releases/tag/v2.33.0