[BUG][EXTRACTOR] API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH is always replacing $(Build.ArtifactStagingDirectory) path with $(Build.SourcesDirectory) path
See original GitHub issueRelease version
v2
Describe the bug
When running the pipeline, no artifacts to be published are created in the PublishPipelineArtifact@1
task. Looking at the extractor log, we found that the extractor is overwriting $(Build.ArtifactStagingDirectory)path's with
$(Build.SourcesDirectory)` when creating the portal artifacts.
Our fix was to use the CopyFiles
task to manually move the files but I think it is not intended for the path to be rewritten like that.
pipeline YAML:
trigger: none
schedules:
- cron: "5 7 * * *"
displayName: Run daily extraction of APIM
branches:
include: [ api ]
always: true
parameters:
- name: environment
type: string
default: 'dev'
- name: TARGET_BRANCH_NAME
type: string
default: api
displayName: Branch where APIM configuration should be stored
variables:
- group: apiops
- name: System.Debug
value: true
- name: RESOURCE_GROUP_NAME
value: ${{ format('rg-{0}', parameters.environment) }}
- name: APIM_INSTANCE_NAME
value: ${{ format('apim-{0}', parameters.environment) }}
- name: APIM_REPOSITORY_NAME
value: $(Build.Repository.Name)
- name: API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH
value: IaC/Provisioning/APIM/files
- name: API_SPECIFICATION_FORMAT
value: OpenAPIV3Yaml
stages:
- stage: create_artifact_from_portal
displayName: Create artifact from portal
jobs:
- job: create_artifact_from_portal
displayName: Create artifact from portal
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
- task: UseDotNet@2
displayName: Install .NET Core SDK
inputs:
version: 7.x
includePreviewVersions: true
- task: Bash@3
displayName: Create publisher output folder
inputs:
targetType: 'inline'
script: 'mkdir "$PIPELINE_WORKSPACE/extractor"'
- task: DotNetCoreCLI@2
displayName: Publish .NET Core application
inputs:
command: publish
publishWebProjects: false
projects: '**/extractor.csproj'
arguments: '--output $(Pipeline.Workspace)/extractor --runtime linux-x64 -p:PublishSingleFile=true'
modifyOutputPath: false
zipAfterPublish: false
- task: AzureCLI@2
displayName: Set extraction variables
inputs:
azureSubscription: "$(SERVICE_CONNECTION_NAME)"
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
echo "##vso[task.setvariable issecret=true;variable=AZURE_BEARER_TOKEN]$(az account get-access-token --query "accessToken" --output tsv)"
echo "##vso[task.setvariable issecret=true;variable=AZURE_CLIENT_ID]$servicePrincipalId"
echo "##vso[task.setvariable issecret=true;variable=AZURE_CLIENT_SECRET]$servicePrincipalKey"
echo "##vso[task.setvariable issecret=true;variable=AZURE_TENANT_ID]$tenantId"
echo "##vso[task.setvariable issecret=true;variable=AZURE_SUBSCRIPTION_ID]$(az account show --query "id" --output tsv)"
addSpnToEnvironment: true
failOnStandardError: true
- task: AzureCLI@2
displayName: Run extractor
inputs:
azureSubscription: "$(SERVICE_CONNECTION_NAME)"
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
chmod u+x $(Pipeline.Workspace)/extractor/extractor
$(Pipeline.Workspace)/extractor/extractor
result=$?
echo "Exit code is $result"
exit $result
addSpnToEnvironment: true
failOnStandardError: true
env:
AZURE_BEARER_TOKEN: $(AZURE_BEARER_TOKEN)
AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
AZURE_TENANT_ID: $(AZURE_TENANT_ID)
AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
AZURE_RESOURCE_GROUP_NAME: $(RESOURCE_GROUP_NAME)
API_MANAGEMENT_SERVICE_NAME: $(APIM_INSTANCE_NAME)
API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: $(Build.StagingDirectory)/IaC/Provisioning/APIM/files
API_SPECIFICATION_FORMAT: $(API_SPECIFICATION_FORMAT)
- script: ls -lha $(Build.StagingDirectory)/IaC/Provisioning/APIM/files
displayName: print content of staging path
- task: PublishPipelineArtifact@1
displayName: Publish pipeline artifact
inputs:
targetPath: '$(Build.StagingDirectory)'
artifact: 'artifacts-from-portal'
publishLocation: 'pipeline'
- stage: create_template_branch
displayName: Create template branch
jobs:
- job: create_artifacts_pull_request
displayName: Create artifacts pull request
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
- task: DownloadPipelineArtifact@2
displayName: Download pipeline artifact
inputs:
source: current
artifactName: artifacts-from-portal
targetPath: $(Pipeline.Workspace)/artifacts-from-portal
- task: AzureCLI@2
displayName: Create pull request with artifacts
inputs:
azureSubscription: "$(SERVICE_CONNECTION_NAME)"
scriptType: bash
scriptLocation: scriptPath
scriptPath: $(Build.SourcesDirectory)/IaC/Provisioning/APIM/tools/utils/create_pull_request.sh
arguments: --organization-url "$(System.TeamFoundationCollectionUri)" --project-name "$(System.TeamProject)" --repository-name "$(APIM_REPOSITORY_NAME)" --pull-request-title "Merging artifacts from portal (Build $(Build.BuildId))" --branch-name "${{ parameters.TARGET_BRANCH_NAME }}" --source-folder-path "$(Pipeline.Workspace)/artifacts-from-portal" --temporary-branch-name "artifacts-from-portal-build-$(Build.BuildId)" --temporary-folder-path "$(Agent.TempDirectory)/artifacts-from-portal" --overwrite-subfolder "$(API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH)"
env:
AZURE_DEVOPS_EXT_PAT: "$(System.AccessToken)"
Example of extractor output explicitly creating artifact in source code (/s
) folder:
2022-11-19T08:14:59.5961050Z Exporting service policies...
2022-11-19T08:14:59.9543653Z info: Extractor[0]
2022-11-19T08:14:59.9544176Z Writing service policy file /home/vsts/work/1/s/IaC/Provisioning/APIM/files/policy.xml...
Expected behavior
Passing $(Build.ArtifactStagingDirectory)
to API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH
is resulting in extracting artifacts inside /home/vsts/work/1/a/
(Ubuntu MS host example)
Actual behavior
Artifacts are extracted to /home/vsts/work/1/s/
when using $(Build.ArtifactStagingDirectory)
as the path prefix.
Reproduction Steps
The pipeline above should technically work.
Issue Analytics
- State:
- Created 9 months ago
- Comments:17 (13 by maintainers)
Closing as this is confirmed to be an ADO issue.
@waelkdouh Yes it does as I already know the various workarounds.
The purpose of this issue is to know why the extractor is creating files under
/home/vsts/work/1/s/
when it is clear that azure pipeline resolves$(Build.ArtifactStagingDirectory
to/home/vsts/work/1/a/
and I am passingAPI_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: $(Build.ArtifactStagingDirectory)/$(API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH
as an environment variable.If the issue appears to come from ADO or even the .NET I’ll close the issue and open a ticket in the appropriate places. However as of now, there isn’t much that leads to think the issue come from them.