Maestro Build Promotion 403 when calling Blob storage for Winforms-Designer builds

• This issue is blocking
• This issue is causing unreasonable pain

For Winforms-Designer builds, Publish Using Darc is throwing errors: https://dev.azure.com/dnceng/internal/_build/results?buildId=864547&view=results

The cause is a 403 when Maestro Build Promotion pipeline reaches out to Blob storage. https://dnceng.visualstudio.com/internal/_build/results?buildId=864601&view=results

D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : Unable to upload to assets/manifests/dotnet-winforms-designer/0.9.0-prerelease-preview2.20523.4/MergedManifest.xml in Azure Storage account dotnetclimsrc/dotnet due to Azure.RequestFailedException: Service request failed.
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : Status: 403 (Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.)
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : ErrorCode: AuthenticationFailed
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : Headers:
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : Transfer-Encoding: chunked
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : Server: Microsoft-HTTPAPI/2.0
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : x-ms-request-id: 2934a709-901e-004f-698d-a9f57b000000
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : x-ms-error-code: AuthenticationFailed
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : Date: Fri, 23 Oct 2020 22:40:39 GMT
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : 
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error :    at Azure.Storage.Blobs.BlobRestClient.Blob.GetPropertiesAsync_CreateResponse(ClientDiagnostics clientDiagnostics, Response response)
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error :    at Azure.Storage.Blobs.BlobRestClient.Blob.GetPropertiesAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, String version, String snapshot, Nullable`1 timeout, String leaseId, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, Nullable`1 ifMatch, Nullable`1 ifNoneMatch, String requestId, Boolean async, String operationName, CancellationToken cancellationToken)
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error :    at Azure.Storage.Blobs.Specialized.BlobBaseClient.ExistsInternal(Boolean async, CancellationToken cancellationToken)
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error :    at Azure.Storage.Blobs.Specialized.BlobBaseClient.ExistsAsync(CancellationToken cancellationToken)
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error :    at Microsoft.DotNet.Build.CloudTestTasks.AzureStorageUtils.CheckIfBlobExistsAsync(String blobPath) in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/common/AzureStorageUtils.cs:line 197
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error :    at Microsoft.DotNet.Build.Tasks.Feed.BlobFeedAction.UploadAssetAsync(ITaskItem item, SemaphoreSlim clientThrottle, PushOptions options) in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/BlobFeedAction.cs:line 175.
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishArtifactsInManifest.proj(99,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Unable to upload to assets/manifests/dotnet-winforms-designer/0.9.0-prerelease-preview2.20523.4/MergedManifest.xml in Azure Storage account dotnetclimsrc/dotnet due to Azure.RequestFailedException: Service request failed.
Status: 403 (Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.)
ErrorCode: AuthenticationFailed

[riarenas]

@mmitche provided this via email:

there is an issue with the internal builds right now. It looks like v3 publishing is not configured properly to switch over to using the internal PATs for internal channel configurations:

• Starting with the v3 publishing template used in the promotion job, this line will use the context of the promotion job, not the repo being promoted (since arcade is always the repo being checked out): https://github.com/dotnet/arcade/blob/master/eng/publishing/v3/publish-assets.yml#L62
• Secondly, the sdk task project that does the publishing appears to use an incorrect variable name here: https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Arcade.Sdk/tools/SdkTasks/PublishArtifactsInManifest.proj#L86. “InternalBuild” isn’t really seen anywhere

This means that we send the dotnetcli keys when we are trying to publish to dotnetclimsrc.

What I think needs to happen is that the V3 publishing task should get all of the possible keys that could be used: https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestV3.cs#L31-L35, then the target channel configs https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Build.Tasks.Feed/src/model/PublishingConstants.cs should specify the keys for each of the feeds they want to publish to. Because v3 attempts to publish to all target channels at the same time, this would be better than just switching the task between public and internal feed keys, because theoretically you could have a build that publishes blobs to two different storage accounts.

[epananth]

PR : https://github.com/dotnet/arcade/pull/6705