[BUG][EXTRACTOR] API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH is always replacing $(Build.ArtifactStagingDirectory) path with $(Build.SourcesDirectory) path

Release version

v2

Describe the bug

When running the pipeline, no artifacts to be published are created in the PublishPipelineArtifact@1 task. Looking at the extractor log, we found that the extractor is overwriting $(Build.ArtifactStagingDirectory)path's with$(Build.SourcesDirectory)` when creating the portal artifacts.

Our fix was to use the CopyFiles task to manually move the files but I think it is not intended for the path to be rewritten like that.

pipeline YAML:

trigger: none

schedules:
- cron: "5 7 * * *"
  displayName: Run daily extraction of APIM
  branches:
    include: [ api ]
  always: true

parameters:
  - name: environment
    type: string
    default: 'dev'
  - name: TARGET_BRANCH_NAME
    type: string
    default: api
    displayName: Branch where APIM configuration should be stored
variables:
  - group: apiops
  - name: System.Debug
    value: true
  - name: RESOURCE_GROUP_NAME
    value: ${{ format('rg-{0}', parameters.environment) }}
  - name: APIM_INSTANCE_NAME
    value: ${{ format('apim-{0}', parameters.environment) }}
  - name: APIM_REPOSITORY_NAME
    value: $(Build.Repository.Name)
  - name: API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH
    value: IaC/Provisioning/APIM/files
  - name: API_SPECIFICATION_FORMAT
    value: OpenAPIV3Yaml

stages:
  - stage: create_artifact_from_portal
    displayName: Create artifact from portal
    jobs:
      - job: create_artifact_from_portal
        displayName: Create artifact from portal
        pool:
          vmImage: ubuntu-latest
        steps:
          - checkout: self
          - task: UseDotNet@2
            displayName: Install .NET Core SDK
            inputs:
              version: 7.x
              includePreviewVersions: true

          - task: Bash@3
            displayName: Create publisher output folder
            inputs:
              targetType: 'inline'
              script: 'mkdir "$PIPELINE_WORKSPACE/extractor"'

          - task: DotNetCoreCLI@2
            displayName: Publish .NET Core application
            inputs:
              command: publish
              publishWebProjects: false
              projects: '**/extractor.csproj'
              arguments: '--output $(Pipeline.Workspace)/extractor --runtime linux-x64 -p:PublishSingleFile=true'
              modifyOutputPath: false
              zipAfterPublish: false

          - task: AzureCLI@2
            displayName: Set extraction variables
            inputs:
              azureSubscription: "$(SERVICE_CONNECTION_NAME)"
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: |
                echo "##vso[task.setvariable issecret=true;variable=AZURE_BEARER_TOKEN]$(az account get-access-token --query "accessToken" --output tsv)"
                echo "##vso[task.setvariable issecret=true;variable=AZURE_CLIENT_ID]$servicePrincipalId"
                echo "##vso[task.setvariable issecret=true;variable=AZURE_CLIENT_SECRET]$servicePrincipalKey"
                echo "##vso[task.setvariable issecret=true;variable=AZURE_TENANT_ID]$tenantId"
                echo "##vso[task.setvariable issecret=true;variable=AZURE_SUBSCRIPTION_ID]$(az account show --query "id" --output tsv)"
              addSpnToEnvironment: true
              failOnStandardError: true
          - task: AzureCLI@2
            displayName: Run extractor
            inputs:
              azureSubscription: "$(SERVICE_CONNECTION_NAME)"
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: |
                chmod u+x $(Pipeline.Workspace)/extractor/extractor
                $(Pipeline.Workspace)/extractor/extractor
                result=$?
                echo "Exit code is $result"
                exit $result
              addSpnToEnvironment: true
              failOnStandardError: true
            env:
              AZURE_BEARER_TOKEN: $(AZURE_BEARER_TOKEN)
              AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
              AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
              AZURE_TENANT_ID: $(AZURE_TENANT_ID)
              AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
              AZURE_RESOURCE_GROUP_NAME: $(RESOURCE_GROUP_NAME)
              API_MANAGEMENT_SERVICE_NAME: $(APIM_INSTANCE_NAME)
              API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: $(Build.StagingDirectory)/IaC/Provisioning/APIM/files
              API_SPECIFICATION_FORMAT:  $(API_SPECIFICATION_FORMAT)
          - script: ls -lha $(Build.StagingDirectory)/IaC/Provisioning/APIM/files
            displayName: print content of staging path
          - task: PublishPipelineArtifact@1
            displayName: Publish pipeline artifact
            inputs:
              targetPath: '$(Build.StagingDirectory)'
              artifact: 'artifacts-from-portal'
              publishLocation: 'pipeline'
              
  - stage: create_template_branch
    displayName: Create template branch
    jobs:
      - job: create_artifacts_pull_request
        displayName: Create artifacts pull request
        pool:
          vmImage: ubuntu-latest
        steps:
          - checkout: self
          - task: DownloadPipelineArtifact@2
            displayName: Download pipeline artifact
            inputs:
              source: current
              artifactName: artifacts-from-portal
              targetPath: $(Pipeline.Workspace)/artifacts-from-portal
          - task: AzureCLI@2
            displayName: Create pull request with artifacts
            inputs:
              azureSubscription: "$(SERVICE_CONNECTION_NAME)"
              scriptType: bash
              scriptLocation: scriptPath
              scriptPath: $(Build.SourcesDirectory)/IaC/Provisioning/APIM/tools/utils/create_pull_request.sh
              arguments: --organization-url "$(System.TeamFoundationCollectionUri)" --project-name "$(System.TeamProject)" --repository-name "$(APIM_REPOSITORY_NAME)" --pull-request-title "Merging artifacts from portal (Build $(Build.BuildId))" --branch-name "${{ parameters.TARGET_BRANCH_NAME }}" --source-folder-path "$(Pipeline.Workspace)/artifacts-from-portal" --temporary-branch-name "artifacts-from-portal-build-$(Build.BuildId)" --temporary-folder-path "$(Agent.TempDirectory)/artifacts-from-portal" --overwrite-subfolder "$(API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH)" 
            env:
              AZURE_DEVOPS_EXT_PAT: "$(System.AccessToken)"

Example of extractor output explicitly creating artifact in source code (/s) folder:

2022-11-19T08:14:59.5961050Z       Exporting service policies...
2022-11-19T08:14:59.9543653Z info: Extractor[0]
2022-11-19T08:14:59.9544176Z Writing service policy file /home/vsts/work/1/s/IaC/Provisioning/APIM/files/policy.xml...

Expected behavior

Passing $(Build.ArtifactStagingDirectory) to API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH is resulting in extracting artifacts inside /home/vsts/work/1/a/ (Ubuntu MS host example)

Actual behavior

Artifacts are extracted to /home/vsts/work/1/s/ when using $(Build.ArtifactStagingDirectory) as the path prefix.

Reproduction Steps

The pipeline above should technically work.

[Gabriel123N]

Closing as this is confirmed to be an ADO issue.

[Gabriel123N]

@waelkdouh Yes it does as I already know the various workarounds.

The purpose of this issue is to know why the extractor is creating files under /home/vsts/work/1/s/ when it is clear that azure pipeline resolves $(Build.ArtifactStagingDirectory to /home/vsts/work/1/a/ and I am passing API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: $(Build.ArtifactStagingDirectory)/$(API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH as an environment variable.

If the issue appears to come from ADO or even the .NET I’ll close the issue and open a ticket in the appropriate places. However as of now, there isn’t much that leads to think the issue come from them.