TSA upload errors causing SDL checks in arcade and arcade-validation to break the official build

• This issue is blocking This blocks both arcade and arcade-validation from producing official builds, which means we cannot flow new versions of Arcade until it’s resolved.

• This issue is causing unreasonable pain

SDL runs in arcade and arcade-validation are failing with the following error in the Run SDL tool job:

Arcade: https://dnceng.visualstudio.com/internal/_build/results?buildId=746674&view=results

Break:
[Error]     s/src/Microsoft.DotNet.Arcade.Sdk/tools/Licenses/MicrosoftDotNetLibrary.txt(116,0): Error 79570: Term: country. TermClass: Geopolitical. Action: Action : Context = When used in a legal, international standards or similar context, such as in license terms (EULAs).; ActionRecommendation = <b>Leave term unchanged</b>; Context = When used in any other context.; ActionRecommendation = Ask your Microsoft contact for guidance.; Context = When used in legacy code to label a country/region list or similar resource.; ActionRecommendation = <b>Leave term unchanged</b>; Context = When used to label a country/region list or similar resource.; ActionRecommendation = <b>REPLACE WITH</b> "country/region"; : Term: 'country'. Term class: Geopolitical. Term table: English-9. Position: Line: 116. Context(term at 26): 'the laws of your state or country. Separate and apart from your relationship'.
[Error]     s/src/Microsoft.DotNet.Arcade.Sdk/tools/Licenses/MicrosoftDotNetLibrary.txt(119,0): Error 79570: Term: country. TermClass: Geopolitical. Action: Action : Context = When used in a legal, international standards or similar context, such as in license terms (EULAs).; ActionRecommendation = <b>Leave term unchanged</b>; Context = When used in any other context.; ActionRecommendation = Ask your Microsoft contact for guidance.; Context = When used in legacy code to label a country/region list or similar resource.; ActionRecommendation = <b>Leave term unchanged</b>; Context = When used to label a country/region list or similar resource.; ActionRecommendation = <b>REPLACE WITH</b> "country/region"; : Term: 'country'. Term class: Geopolitical. Term table: English-9. Position: Line: 119. Context(term at 36): 'rights if the laws of your state or country do not permit it to do so.'.
[Error]     s/src/Microsoft.DotNet.Arcade.Sdk/tools/Licenses/MicrosoftDotNetLibrary.txt(121,0): Error 79570: Term: country. TermClass: Geopolitical. Action: Action : Context = When used in a legal, international standards or similar context, such as in license terms (EULAs).; ActionRecommendation = <b>Leave term unchanged</b>; Context = When used in any other context.; ActionRecommendation = Ask your Microsoft contact for guidance.; Context = When used in legacy code to label a country/region list or similar resource.; ActionRecommendation = <b>Leave term unchanged</b>; Context = When used to label a country/region list or similar resource.; ActionRecommendation = <b>REPLACE WITH</b> "country/region"; : Term: 'country'. Term class: Geopolitical. Term table: English-9. Position: Line: 121. Context(term at 10): 'mandatory country law applies, then the following provisions apply to you:'.
[Error]     s/src/Microsoft.DotNet.Arcade.Sdk/tools/Licenses/MicrosoftDotNetLibrary.txt(166,0): Error 79570: Term: country. TermClass: Geopolitical. Action: Action : Context = When used in a legal, international standards or similar context, such as in license terms (EULAs).; ActionRecommendation = <b>Leave term unchanged</b>; Context = When used in any other context.; ActionRecommendation = Ask your Microsoft contact for guidance.; Context = When used in legacy code to label a country/region list or similar resource.; ActionRecommendation = <b>Leave term unchanged</b>; Context = When used to label a country/region list or similar resource.; ActionRecommendation = <b>REPLACE WITH</b> "country/region"; : Term: 'country'. Term class: Geopolitical. Term table: English-9. Position: Line: 166. Context(term at 14): 'your state or country may not allow the exclusion or limitation of incidental,'.
[Error]     s/src/Microsoft.DotNet.Arcade.Sdk/tools/Licenses/VisualStudioExtensions.txt(42,0): Error 79570: Term: country. TermClass: Geopolitical. Action: Action : Context = When used in a legal, international standards or similar context, such as in license terms (EULAs).; ActionRecommendation = <b>Leave term unchanged</b>; Context = When used in any other context.; ActionRecommendation = Ask your Microsoft contact for guidance.; Context = When used in legacy code to label a country/region list or similar resource.; ActionRecommendation = <b>Leave term unchanged</b>; Context = When used to label a country/region list or similar resource.; ActionRecommendation = <b>REPLACE WITH</b> "country/region"; : Term: 'country'. Term class: Geopolitical. Term table: English-9. Position: Line: 42. Context(term at 96): 't to do so. For example, if you acquired the software in one of the below regions, or mandatory country law applies, then the following provisions apply to you:'.
    Nothing was specified with --update-baseline-file. The baseline will be created in the default location: D:\a\1\.gdn\r\user.gdnbaselines
D:\a\1\s\.packages\Microsoft.Guardian.Cli.win10-x64.0.20.1\tools\guardian.cmd tsa-publish --all-tools --repository-name "Arcade" --branch-name "refs/heads/MergeManifestsTest" --build-number "20200725.1" --codebase-name "Arcade" --notification-alias "[e-mail]" --codebase-admin "REDMOND\dn-bot" --instance-url "https://devdiv.visualstudio.com/" --project-name "DEVDIV" --area-path "DevDiv\NET Fundamentals\Infrastructure\Arcade\SDL" --iteration-path "DevDiv" --working-directory D:\a\1 --logger-level Standard
TSA Upload:
  TSA Onboard:
    Saving TsaCodebaseXml file: D:\a\1\.gdn\.r\tsa-onboard.xml
    Onboarding or updating TSA codebase: Arcade
[Error]     TsaOnboardCodebaseFailedException: An exception occured while attempting to onboard or update a TSA codebase: Arcade
[Error]     TSA Response: {"Message":"TSA Error Code: 200 Error message: Invalid TFS Settings for Instance: [https://devdiv.visualstudio.com/] , Project [DevDiv] , Template [TFSDEVDIV]\r\n                                                  were passed to TSA: Encountered Exception [Microsoft.TeamFoundation.TeamFoundationServerUnauthorizedException: TF30063: You are not authorized to access https://devdiv.visualstudio.com/.\r\n   at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.EnsureTokenProvider(HttpWebResponse webResponse)\r\n   at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.SendRequest()\r\n   at Microsoft.TeamFoundation.Client.Channels.TfsHttpRequestChannel.Request(TfsMessage message, TimeSpan timeout)\r\n   at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.Invoke(TfsClientOperation operation, Object[] parameters, TimeSpan timeout, Object[]& outputs)\r\n   at Microsoft.TeamFoundation.Framework.Client.LocationWebService.Connect(Int32 connectOptions, Int32 lastChangeId, Int32 features)\r\n   at Microsoft.TeamFoundation.Framework.Client.FrameworkServerDataProvider.Connect(ConnectOptions connectOptions)\r\n   at Microsoft.TeamFoundation.Framework.Client.FrameworkServerDataProvider.EnsureConnected(ConnectOptions optionsNeeded)\r\n   at Microsoft.TeamFoundation.Framework.Client.FrameworkServerDataProvider.get_InstanceId()\r\n   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.GetInstanceIdentifier()\r\n   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.InitializeInternal()\r\n   at Microsoft.TeamFoundation.Client.TfsTeamProjectCollection.InitializeTeamFoundationObject(String fullName, Object instance)\r\n   at Microsoft.TeamFoundation.Client.TfsConnection.CreateServiceInstance(Assembly assembly, String fullName)\r\n   at Microsoft.TeamFoundation.Client.TfsConnection.GetService(Type serviceType)\r\n   at Microsoft.TeamFoundation.Client.TfsConnection.GetService[T]()\r\n   at TSA.Common.TFSUtility.<>c__DisplayClass6_0.<GetServiceWorkItem>b__0() in d:\\a\\1\\s\\TSA\\TSACommon\\TFS\\TFSUtility.cs:line 91\r\n   at TSA.Common.Retry.ExecuteInternal(Action action, Int32 retries, Int32 delay, CancellationToken cancellationToken, String[] acceptMessages) in d:\\a\\1\\s\\TSA\\TSACommon\\Executions\\Retry.cs:line 107\r\n   at TSA.Common.Retry.Execute(Action action, Int32 retries, Int32 delay, CancellationToken cancellationToken, String[] acceptMessages) in d:\\a\\1\\s\\TSA\\TSACommon\\Executions\\Retry.cs:line 46\r\n   at TSA.Common.TFSUtility.GetServiceWorkItem(TfsTeamProjectCollection projectCollection, Int32 retries) in d:\\a\\1\\s\\TSA\\TSACommon\\TFS\\TFSUtility.cs:line 90\r\n   at TSA.Core.TFS.ThrowIfInvalid(Boolean validateBugTemplate) in d:\\a\\1\\s\\TSA\\TSACore\\Cores\\TFS\\TFS.cs:line 421], Correlation ID: 6b7648e9-4d0e-4737-b6f5-f42fd8988015"}
[Error]     TsaWebException: TSA web call returned a non-successful status code: PreconditionFailed
##[error]PowerShell exited with code '1'.

Arcade-Validation: https://dev.azure.com/dnceng/internal/_build/results?buildId=745967&view=results

Break:
    Nothing was specified with --update-baseline-file. The baseline will be created in the default location: D:\a\1\.gdn\r\user.gdnbaselines
    Found no breaking results.
D:\a\1\s\.packages\Microsoft.Guardian.Cli.win10-x64.0.20.1\tools\guardian.cmd tsa-publish --all-tools --repository-name "Arcade-Validation" --branch-name "refs/heads/master" --build-number "20200725.1" --codebase-name "Arcade-Validation" --notification-alias "[e-mail]" --codebase-admin "REDMOND\dn-bot" --instance-url "https://devdiv.visualstudio.com/" --project-name "DEVDIV" --area-path "DevDiv\NET Fundamentals\Infrastructure\Arcade\SDL" --iteration-path "DevDiv" --working-directory D:\a\1 --logger-level Standard
TSA Upload:
  TSA Onboard:
    Saving TsaCodebaseXml file: D:\a\1\.gdn\.r\tsa-onboard.xml
    Onboarding or updating TSA codebase: Arcade-Validation
[Error]     TsaOnboardCodebaseFailedException: An exception occured while attempting to onboard or update a TSA codebase: Arcade-Validation
[Error]     TSA Response: {"Message":"TSA Error Code: 200 Error message: Invalid TFS Settings for Instance: [https://devdiv.visualstudio.com/] , Project [DevDiv] , Template [TFSDEVDIV]\r\n                                                  were passed to TSA: Encountered Exception [Microsoft.TeamFoundation.TeamFoundationServerUnauthorizedException: TF30063: You are not authorized to access https://devdiv.visualstudio.com/.\r\n   at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.EnsureTokenProvider(HttpWebResponse webResponse)\r\n   at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.SendRequest()\r\n   at Microsoft.TeamFoundation.Client.Channels.TfsHttpRequestChannel.Request(TfsMessage message, TimeSpan timeout)\r\n   at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.Invoke(TfsClientOperation operation, Object[] parameters, TimeSpan timeout, Object[]& outputs)\r\n   at Microsoft.TeamFoundation.Framework.Client.LocationWebService.Connect(Int32 connectOptions, Int32 lastChangeId, Int32 features)\r\n   at Microsoft.TeamFoundation.Framework.Client.FrameworkServerDataProvider.Connect(ConnectOptions connectOptions)\r\n   at Microsoft.TeamFoundation.Framework.Client.FrameworkServerDataProvider.EnsureConnected(ConnectOptions optionsNeeded)\r\n   at Microsoft.TeamFoundation.Framework.Client.FrameworkServerDataProvider.get_InstanceId()\r\n   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.GetInstanceIdentifier()\r\n   at Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.InitializeInternal()\r\n   at Microsoft.TeamFoundation.Client.TfsTeamProjectCollection.InitializeTeamFoundationObject(String fullName, Object instance)\r\n   at Microsoft.TeamFoundation.Client.TfsConnection.CreateServiceInstance(Assembly assembly, String fullName)\r\n   at Microsoft.TeamFoundation.Client.TfsConnection.GetService(Type serviceType)\r\n   at Microsoft.TeamFoundation.Client.TfsConnection.GetService[T]()\r\n   at TSA.Common.TFSUtility.<>c__DisplayClass6_0.<GetServiceWorkItem>b__0() in d:\\a\\1\\s\\TSA\\TSACommon\\TFS\\TFSUtility.cs:line 91\r\n   at TSA.Common.Retry.ExecuteInternal(Action action, Int32 retries, Int32 delay, CancellationToken cancellationToken, String[] acceptMessages) in d:\\a\\1\\s\\TSA\\TSACommon\\Executions\\Retry.cs:line 107\r\n   at TSA.Common.Retry.Execute(Action action, Int32 retries, Int32 delay, CancellationToken cancellationToken, String[] acceptMessages) in d:\\a\\1\\s\\TSA\\TSACommon\\Executions\\Retry.cs:line 46\r\n   at TSA.Common.TFSUtility.GetServiceWorkItem(TfsTeamProjectCollection projectCollection, Int32 retries) in d:\\a\\1\\s\\TSA\\TSACommon\\TFS\\TFSUtility.cs:line 90\r\n   at TSA.Core.TFS.ThrowIfInvalid(Boolean validateBugTemplate) in d:\\a\\1\\s\\TSA\\TSACore\\Cores\\TFS\\TFS.cs:line 421], Correlation ID: 416bf94e-c1bc-435f-bffa-fb56fb213a65"}
[Error]     TsaWebException: TSA web call returned a non-successful status code: PreconditionFailed
##[error]PowerShell exited with code '1'.

This looks very similar to https://github.com/dotnet/arcade/issues/4170. Did area paths in devdiv change again?

CC @jonfortescue @sunandabalu

[alexperovich]

This was an issue in guardian, and it has been fixed.

[MattGal]

Created CLI area, reran, looking good.