[Arcade-Services] Azure Devops Client RemoveAllArtifactSourcesAsync fails with 400 (Bad Request) response for some release pipelines

It seems like Release pipelines can get into a state where this call will fail 100% of the time with error:

Artifact alias 'PrimaryArtifact' does not matching any linked artifact aliases for deploy job 'Agent job'. Specify a valid artifact alias and try again.

As a workaround, deleting the primary artifact from the UI unblocks this call, and it will keep working for subsequent attempts.

I’m unsure what causes the pipeline to get in this state, as I saw it in both an old pipeline I kept for testing, and a pipeline I directly cloned from one of the ones that is not in this state.

When inspecting the body for the request we are sending to the API, the failing request has a reference to the ‘PrimaryArtifact’ that is missing from the successful requests once I did the workaround, so that may well be the culprit for the bad requests.

"deployPhases": [{
        "deploymentInput": {
            "parallelExecution": {
                "parallelExecutionType": "none"
            },
            "skipArtifactsDownload": false,
            "artifactsDownloadInput": {
                "downloadInputs": [{
                        "alias": "PrimaryArtifact",
                        "artifactType": "Build",
                        "artifactDownloadMode": "All",
                        "artifactItems": []
                    }
                ]
            },
...

https://dev.azure.com/dnceng/internal/_release?view=all&definitionId=38 is still in this state in case a repro is required to further debug this.

[riarenas]

When this happens, the releases don’t actually fail, We are unable to even start them. Both https://dev.azure.com/dnceng/internal/_release?definitionId=36 and https://dev.azure.com/dnceng/internal/_release?definitionId=39

hadn’t triggered any releases since 4/18 ~ 4:30 PM.

Once I did the workaround of deleting the “Primary Artifact”, all the releases that kept failing to start and were queued started running again.

You will see that for https://dev.azure.com/dnceng/internal/_release?definitionId=39 there is a huge gap between release # 379 and release # 382, which was the next automated release that ran today.

[JohnTortugo]

Got a repro for this:

• The RemoveAllArtifactSourcesAsync method is incomplete. When the AzDO UI removes an artifact source it also patches the downloadInputs node mentioned in the description of this issue. The RemoveAllArtifactSources method doesn’t do that. This is the failure that we observed. I.e., We try to save a pipeline that still refers to the artifact source that we just removed.

• The problem is caused by editions of the pipeline using the AzDO UI. Some changes (i.e., adding a new stage, or some changes in an existing stage) using the UI can silently bring back the downloadInputs configuration referencing the artifact source. When this happen, the next time ReleasePipelineRunner try to remove the artifact sources + save the pipeline the error happens. Removing the artifact source using the UI fixes the problem because of the behavior described in the previous topic.

• Another problem is that the RunAsync method, which, in the presence of an exception, never process a subsequent element of the queue. Therefore, when an exception happens processing an item, the method will keep forever (and ever) trying to process the same faulty entry.

I’ll work on a PR to fix both issues. Let me know if I need to give more details.