Arcade-validation validate publishing job failing

• This issue is blocking Arcade-validation official build is failing, so promotion of new arcade versions is blocked.
• This issue is causing unreasonable pain

https://dnceng.visualstudio.com/internal/_build/results?buildId=764450 is failing as of this change: https://github.com/dotnet/arcade/pull/5899

darc add-build-to-channel is failing to run the build promotion pipeline correctly due to what appear to be missing properties in the asset manifest: https://dnceng.visualstudio.com/internal/_build/results?buildId=764508&view=results

D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error : Could not parse asset manifest file: D:\a\1\a\AssetManifests\Windows_NT-AnyCPU.xml
##[error].packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Could not parse asset manifest file: D:\a\1\a\AssetManifests\Windows_NT-AnyCPU.xml
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error : Required attribute(s) missing: Extension
##[error].packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Required attribute(s) missing: Extension
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error : Object reference not set to an instance of an object.
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error :    at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifest.WhichPublishingTask(String manifestFullPath) in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifest.cs:line 217
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error :    at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifest.<ExecuteAsync>b__97_0(ITaskItem manifestParam) in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifest.cs:line 153
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error :    at System.Linq.Enumerable.SelectArrayIterator`2.ToArray()
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error :    at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error :    at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifest.ExecuteAsync()
##[error].packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20403.5\tools\SdkTasks\PublishArtifactsInManifest.proj(97,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Object reference not set to an instance of an object.
   at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifest.WhichPublishingTask(String manifestFullPath) in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifest.cs:line 217
   at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifest.<ExecuteAsync>b__97_0(ITaskItem manifestParam) in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifest.cs:line 153
   at System.Linq.Enumerable.SelectArrayIterator`2.ToArray()
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifest.ExecuteAsync()

CC @jonfortescue @jcagme

[riarenas]

This had been working out of sheer luck. @jonfortescue’s change was just the first unfortunate change that makes publishing with version N-1 incompatible with manifests produced by version N.

[riarenas]

We got the bottom of this, and the conclusion is that the validate publishing step in arcade-validation is not doing the right thing:

• The job uses darc add-build-to-channel to add the build of arcade-validation that is currently running to the general testing channel: https://github.com/dotnet/arcade-validation/blob/9c0e24432401773ccbac41edb2248aa5f37999db/eng/validation/test-publishing.ps1#L45
• If a --source-branch parameter is not provided to darc, it will default to using the branch + commit used to produce the Arcade SDK which is under validation. This SHA will always be using an older version of the Arcade SDK than the one we’re attempting to validate.
• This causes a conflict. Arcade-validation is validating version N, but it will attempt to publish using the latest promoted arcade build, which is always at least version N - 1.

In order to fix this, I think we should make it so the validate-publishing step:

1. Creates a new branch in Arcade, based off master.
2. Updates dependencies to the arcade build that is being validated
3. When doing add-build-to-channel, pass a source-branch parameter pointing to the branch created in step 1.