[BUG] `IgnoredHeaders` are not excluded from result comparison

Library name and version

Azure.Core.TestFramework

Describe the bug

According to the guide, adding headers through IgnoredHeaders.Add("XXX") should exclude those headers from result comparison.

However, I have a PR which will add repeatable request headers in some SDKs. Those headers should be excluded, because their value is automatically set by SDK and its value will change each time. So I add those headers as ignored headers. But the test still fails.

Here is a failed case:

• RevokeTest failed because extra header is found. Here is the error log in pipeline (can be reproduced on local).
• The original test case is here.
• The invoked API client method is https://github.com/Azure/azure-sdk-for-net/blob/8ff2023034367ab4936bf285372a0c8aae625baf/sdk/purview/Azure.Analytics.Purview.Share/src/Generated/AcceptedSentSharesClient.cs#L293
• In the PR, the repeatabilityRequestId parameter is removed from the API client method. That value is automatically set in the SDK at runtime.
• Previously in the test case, null value is set for repeatabilityRequestId. So in existing test recording, there is no repeatabilityRequestId.
• I’ve added those headers into IgnoredHeaders (see here), and the error still exist.

Expected behavior

IgnoredHeaders should be excluded from result comparison.

Actual behavior

IgnoredHeaders is not excluded from result comparison.

Reproduction Steps

See the description.

Environment

No response

[JoshLove-msft]

There is a back compat property that has the behavior that is being requested here - https://github.com/Azure/azure-sdk-for-net/blob/8ff2023034367ab4936bf285372a0c8aae625baf/sdk/core/Azure.Core.TestFramework/src/RecordedTestBase.cs#L173

We intentionally hid this property for all the reasons that @m-nash mentioned.

[archerzz]

Lets address bullet 3 independently as that is separate from the repeatability use case here.

I do understand the challenges with bullet 1 and 2, how many tests would we need to re-record here?

In dataplane where we have reliable live test I would be fine with simply modifying the recordings to have a new header and no need to re-record.

In mgmtplane one of the reasons I would push for re-recording here is that we can’t rely on live testing for the same reasons you list. There is some regression danger in relying on the same recordings for too long since the service can change its behavior over time and we wouldn’t necessarily know.

Periodically re-recording is a necessary thing until we have the time to get our live testing solution in place.

@m-nash for the repeatability request header changes, 8 SDKs are impacted. And 4 of them require re-recording. 2 of 4 have good test cases so that re-recording is done easily. The remaining 2 SDKs have their test cases tightly bound to service teams’ own testing environments.

• Azure.Communication.CallingServer: 18 of 155 test cases failed
• Azure.Analytics.Purview.Share: 6 of 36 test cases failed

I’ve modified the test recordings to make the tests pass. I agree that for such number of impacted test cases, modifying test recordings is a better option.

Closing this issue.