[BUG] Entity with specified identifier not found while deploying API operation policy
See original GitHub issueRelease version
v4.5.0
Describe the bug
In our source APIM instance, we have an API with 2 operations defined. Both operations have policies to do re-write URL and set backend service ID. These definitions are successfully extracted when running the extractor pipeline. When running the publisher pipeline, the pipeline fails when attempting to deploy policy for either of these operations with the following error message:
System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/{resource_group_name}/providers/Microsoft.ApiManagement/service/{instance_name}/apis/{route}/operations/{operation}/policies/policy?api-version=2022-04-01-preview&format=rawxml failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'.
We have tried removing the policies from both of these operations and the pipeline continues to fail with the same error.
Extractor YAML
parameters:
- name: APIM_INSTANCE_NAME
displayName: APIM instance name
type: string
- name: RESOURCE_GROUP_NAME
displayName: APIM instance resource group name
type: string
- name: APIM_REPOSITORY_NAME
type: string
displayName: APIM repository for pull request
- name: API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH
type: string
displayName: Folder where you want to extract the artifacts
- name: TARGET_BRANCH_NAME
type: string
displayName: Target branch for pull request
default: main
- name: CONFIGURATION_YAML_PATH
type: string
displayName: Optional configuration file
values:
- Extract All
- configuration.extractor.yaml
- name: API_SPECIFICATION_FORMAT
type: string
displayName: API Specification Format
values:
- OpenAPIV3Yaml
- OpenAPIV3Json
- OpenAPIV2Yaml
- OpenAPIV2Json
trigger: none
variables:
- group: sbapim-automation
- name: System.Debug
value: true
stages:
- stage: create_artifact_from_portal
displayName: Create artifact from portal
jobs:
- job: create_artifact_from_portal
displayName: Create artifact from portal
pool:
vmImage: ubuntu-latest
steps:
- task: AzureCLI@2
displayName: Set extraction variables
inputs:
azureSubscription: "$(SERVICE_CONNECTION_NAME)"
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$VerbosePreference = "Continue"
$InformationPreference = "Continue"
Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_BEARER_TOKEN]$(az account get-access-token --query "accessToken" --output tsv)"
Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_CLIENT_ID]$env:servicePrincipalId"
Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_CLIENT_SECRET]$env:servicePrincipalKey"
Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_TENANT_ID]$env:tenantId"
if (-not $env:AZURE_SUBSCRIPTION_ID) {
$subscriptionCount = az account list --query "length([])" --output tsv
if ($subscriptionCount -eq 1) {
$subscriptionId = az account list --query "[0].id" --output tsv
Write-Host "Setting AZURE_SUBSCRIPTION_ID environment variable to: $subscriptionId"
#$env:AZURE_SUBSCRIPTION_ID = $subscriptionId
Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_SUBSCRIPTION_ID]$($subscriptionId)"
}
elseif ($subscriptionCount -gt 1) {
Write-Host "Multiple subscriptions are accessible. Please set the AZURE_SUBSCRIPTION_ID environment variable manually."
exit 1
}
}
else {
Write-Host "AZURE_SUBSCRIPTION_ID is already set to: $env:AZURE_SUBSCRIPTION_ID"
}
addSpnToEnvironment: true
failOnStandardError: true
- task: PowerShell@2
displayName: Fetch extractor
inputs:
targetType: "inline"
script: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$VerbosePreference = "Continue"
$InformationPreference = "Continue"
Write-Information "Downloading extractor..."
$extractorFileName = "$(Agent.OS)" -like "*win*" ? "extractor.win-x64.exe" : "extractor.linux-x64.exe"
$uri = "https://github.com/Azure/apiops/releases/download/$(apiops_release_version)/$extractorFileName"
$destinationFilePath = Join-Path "$(Agent.TempDirectory)" "extractor.exe"
Invoke-WebRequest -Uri "$uri" -OutFile "$destinationFilePath"
if ("$(Agent.OS)" -like "*linux*")
{
Write-Information "Setting file permissions..."
& chmod +x "$destinationFilePath"
if ($LASTEXITCODE -ne 0) { throw "Setting file permissions failed."}
}
Write-Host "##vso[task.setvariable variable=EXTRACTOR_FILE_PATH]$destinationFilePath"
Write-Information "Execution complete."
failOnStderr: true
pwsh: true
- task: PowerShell@2
displayName: Run extractor
inputs:
targetType: "inline"
script: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$VerbosePreference = "Continue"
$InformationPreference = "Continue"
& "$(EXTRACTOR_FILE_PATH)"
if ($LASTEXITCODE -ne 0) { throw "Running extractor failed."}
Write-Information "Execution complete."
failOnStderr: true
pwsh: true
env:
AZURE_BEARER_TOKEN: $(AZURE_BEARER_TOKEN)
AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
AZURE_TENANT_ID: $(AZURE_TENANT_ID)
AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
AZURE_RESOURCE_GROUP_NAME: ${{ parameters.RESOURCE_GROUP_NAME }}
API_MANAGEMENT_SERVICE_NAME: ${{ parameters.APIM_INSTANCE_NAME }}
API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: $(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}
API_SPECIFICATION_FORMAT: ${{ parameters.API_SPECIFICATION_FORMAT }}
${{ if ne( parameters['CONFIGURATION_YAML_PATH'], 'Extract All' ) }}:
CONFIGURATION_YAML_PATH: ${{ parameters.CONFIGURATION_YAML_PATH }}
#Running Super Linting Tool on the API(s) - START
- task: NodeTool@0
inputs:
versionSpec: '12.x'
displayName: 'Install Node.js'
- script: |
npm install -g @stoplight/spectral-cli
displayName: 'Install Spectral'
- script: |
spectral lint --format stylish --format junit --output.junit $(Build.ArtifactStagingDirectory)/spectral-result.xml $(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}/apis/**/specification.{json,yaml,yml} -r https://raw.githubusercontent.com/connectedcircuits/devops-api-linter/main/rules.yaml
displayName: 'Run Spectral Linting'
continueOnError: true
failOnStderr: true
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/spectral-result.xml'
searchFolder: $(Build.ArtifactStagingDirectory)
testRunTitle: 'Linting results for API $(Build.SourceBranchName)'
failTaskOnFailedTests: false
#Running Super Linting Tool on the API(s) - END
- task: PublishPipelineArtifact@1
displayName: Publish pipeline artifact
inputs:
targetPath: "$(Build.ArtifactStagingDirectory)"
artifactType: pipeline
artifactName: artifacts-from-portal
- stage: create_template_branch
displayName: Create template branch
jobs:
- job: create_artifacts_pull_request
displayName: Create artifacts pull request
pool:
vmImage: windows-latest
steps:
- task: DownloadPipelineArtifact@2
displayName: Download pipeline artifact
inputs:
source: current
artifactName: artifacts-from-portal
targetPath: $(Pipeline.Workspace)/artifacts-from-portal
- task: PowerShell@2
displayName: Create pull request
inputs:
targetType: "inline"
script: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$VerbosePreference = "Continue"
$InformationPreference = "Continue"
Write-Information "Installing Azure DevOps extension..."
az extension add --name "azure-devops"
az devops configure --defaults organization="$(System.TeamFoundationCollectionUri)" project="$(System.TeamProject)"
Write-Information "Creating temporary folder..."
$temporaryFolderPath = Join-Path "$(Agent.TempDirectory)" "artifacts-from-portal"
New-Item -Path "$temporaryFolderPath" -ItemType "Directory"
$branchName = "${{ parameters.TARGET_BRANCH_NAME }}"
$temporaryBranchName = "artifacts-from-portal-build-$(Build.BuildId)"
$repositoryName = "${{ parameters.APIM_REPOSITORY_NAME }}"
Write-Information "Cloning branch $branchName in repository $repositoryName..."
$cloneUrl = az repos show --repository "$repositoryName" --query "remoteUrl" --output tsv
Write-Information "Clone URL is $cloneUrl"
git -c http.extraheader="AUTHORIZATION: Bearer $(System.AccessToken)" clone --branch "$branchName" --depth 1 "$cloneUrl" "$temporaryFolderPath"
if ($LASTEXITCODE -ne 0) { throw "Cloning branch $branchName in repository $repositoryName failed." }
Write-Information "Creating temporary branch $temporaryBranchName from $branchName..."
git -C "$temporaryFolderPath" checkout -b "$temporaryBranchName" "$branchName"
if ($LASTEXITCODE -ne 0) { throw "Creating temporary branch $temporaryBranchName from $branchName failed." }
Write-Information "Creating artifacts folder..."
$artifactFolderPath = Join-Path "$temporaryFolderPath" "${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}"
if ((Test-Path -Path "$artifactFolderPath") -eq $false) {
New-Item -Path "$artifactFolderPath" -ItemType "Directory"
}
Write-Information "Synchronizing artifacts..."
$extractorArtifactsFolderPath = Join-Path "$(Pipeline.Workspace)" "artifacts-from-portal" ${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}
if ("$(Agent.OS)" -like "*win*") {
& robocopy "$extractorArtifactsFolderPath" "$artifactFolderPath" /zb /mir /mt
if ($LASTEXITCODE -gt 7) { throw "Setting $artifactFolderPath to contents of $extractorArtifactsFolderPath failed." }
}
else {
& rsync --verbose --archive --delete --force --recursive "$extractorArtifactsFolderPath/" "$artifactFolderPath/"
if ($LASTEXITCODE -ne 0) { throw "Setting $artifactFolderPath to contents of $extractorArtifactsFolderPath failed." }
}
Write-Information "Validating that changes exist to be published..."
$gitStatus = git -C "$temporaryFolderPath" status --porcelain
if ($LASTEXITCODE -ne 0) { throw "Getting git status failed." }
if ([string]::IsNullOrWhiteSpace($gitStatus)) {
Write-Information "No changes exist to be published."
return
}
Write-Information "Setting git user information..."
git config --global user.email "azuredevopsagent@azuredevops.com"
git config --global user.name "Azure Devops agent"
Write-Information "Adding changes..."
git -C "$temporaryFolderPath" add --all
if ($LASTEXITCODE -ne 0) { throw "Adding Git changes failed." }
Write-Information "Committing changes"
$commitOutput = git -C "$temporaryFolderPath" commit --message "Initial commit"
if ($LASTEXITCODE -ne 0) {
if ($commitOutput.Contains("nothing to commit, working tree clean")) {
Write-Information "No changes exist to be published."
return
}
throw "Committing Git changes failed."
}
Write-Information "Pushing changes"
git -C "$temporaryFolderPath" -c http.extraheader="AUTHORIZATION: Bearer $(System.AccessToken)" push --set-upstream origin "$temporaryBranchName"
if ($LASTEXITCODE -ne 0) { throw "Pushing Git changes failed." }
Write-Information "Creating pull request..."
az repos pr create --source-branch "$temporaryBranchName" --target-branch "$branchName" --title "Merging artifacts from portal (Build $(Build.BuildId))" --squash --delete-source-branch "true" --repository "$repositoryName"
if ($LASTEXITCODE -ne 0) { throw "Creating pull request failed." }
Write-Information "Deleting temporary folder contents..."
Remove-Item -Path "$temporaryFolderPath" -Recurse -Force
Write-Information "Execution complete."
pwsh: true
env:
AZURE_DEVOPS_EXT_PAT: "$(System.AccessToken)"
Publisher YAML
parameters:
- name: API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH
type: string
displayName: Folder where the artifacts reside
- name: ENVIRONMENT
type: string
displayName: Environment to display
- name: RESOURCE_GROUP_NAME
type: string
displayName: Resource Group Name
- name: API_MANAGEMENT_SERVICE_NAME
type: string
displayName: APIM Instance Name
default: ""
- name: CONFIGURATION_YAML_PATH
type: string
displayName: Optional configuration file
default: ""
- name: COMMIT_ID
type: string
default: publish-artifacts-in-last-commit
steps:
- script: echo Provided configuration was ${{ parameters.CONFIGURATION_YAML_PATH }}
displayName: Print the name of the yaml configuration file if provided
- script: echo Provided app service name was ${{ parameters.API_MANAGEMENT_SERVICE_NAME }}
displayName: Print the name of the apim service name if provided
- checkout: self
fetchDepth: 0
- task: AzureCLI@2
displayName: Set publishing variables
inputs:
azureSubscription: "$(SERVICE_CONNECTION_NAME)"
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$VerbosePreference = "Continue"
$InformationPreference = "Continue"
Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_BEARER_TOKEN]$(az account get-access-token --query "accessToken" --output tsv)"
Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_CLIENT_ID]$env:servicePrincipalId"
Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_CLIENT_SECRET]$env:servicePrincipalKey"
Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_TENANT_ID]$env:tenantId"
if (-not $env:AZURE_SUBSCRIPTION_ID) {
$subscriptionCount = az account list --query "length([])" --output tsv
if ($subscriptionCount -eq 1) {
$subscriptionId = az account list --query "[0].id" --output tsv
Write-Host "Setting AZURE_SUBSCRIPTION_ID environment variable to: $subscriptionId"
#$env:AZURE_SUBSCRIPTION_ID = $subscriptionId
Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_SUBSCRIPTION_ID]$($subscriptionId)"
}
elseif ($subscriptionCount -gt 1) {
Write-Host "Multiple subscriptions are accessible. Please set the AZURE_SUBSCRIPTION_ID environment variable manually."
exit 1
}
}
else {
Write-Host "AZURE_SUBSCRIPTION_ID is already set to: $env:AZURE_SUBSCRIPTION_ID"
}
addSpnToEnvironment: true
failOnStandardError: true
# replacetokens@3 task will need to be installed to use
- ${{ if ne(parameters.CONFIGURATION_YAML_PATH, '') }}:
- task: qetza.replacetokens.replacetokens-task.replacetokens@3
displayName: "Perform namevalue secret substitution in ${{ parameters.CONFIGURATION_YAML_PATH }}"
inputs:
targetFiles: ${{ parameters.CONFIGURATION_YAML_PATH }}
encoding: "auto"
writeBOM: true
verbosity: "off"
actionOnMissing: "warn"
keepToken: false
tokenPrefix: "{#"
tokenSuffix: "#}"
- task: PowerShell@2
displayName: Fetch publisher
inputs:
targetType: "inline"
script: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$VerbosePreference = "Continue"
$InformationPreference = "Continue"
Write-Information "Downloading publisher..."
$publisherFileName = "$(Agent.OS)" -like "*win*" ? "publisher.win-x64.exe" : "publisher.linux-x64.exe"
$uri = "https://github.com/Azure/apiops/releases/download/$(apiops_release_version)/$publisherFileName"
$destinationFilePath = Join-Path "$(Agent.TempDirectory)" "publisher.exe"
Invoke-WebRequest -Uri "$uri" -OutFile "$destinationFilePath"
if ("$(Agent.OS)" -like "*linux*")
{
Write-Information "Setting file permissions..."
& chmod +x "$destinationFilePath"
if ($LASTEXITCODE -ne 0) { throw "Setting file permissions failed."}
}
Write-Host "##vso[task.setvariable variable=PUBLISHER_FILE_PATH]$destinationFilePath"
Write-Information "Execution complete."
failOnStderr: true
pwsh: true
- task: PowerShell@2
displayName: Run publisher for ${{ parameters.ENVIRONMENT}} environment
inputs:
targetType: "inline"
script: |
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$VerbosePreference = "Continue"
$InformationPreference = "Continue"
& "$(PUBLISHER_FILE_PATH)"
if ($LASTEXITCODE -ne 0) { throw "Running publisher failed."}
Write-Information "Execution complete."
failOnStderr: true
pwsh: true
env:
AZURE_BEARER_TOKEN: $(AZURE_BEARER_TOKEN)
AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
AZURE_TENANT_ID: $(AZURE_TENANT_ID)
AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
AZURE_RESOURCE_GROUP_NAME: ${{ parameters.RESOURCE_GROUP_NAME }}
API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: $(Build.SourcesDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}
${{ if ne( parameters['API_MANAGEMENT_SERVICE_NAME'], '' ) }}:
API_MANAGEMENT_SERVICE_NAME: ${{ parameters.API_MANAGEMENT_SERVICE_NAME }}
${{ if eq( parameters['COMMIT_ID'], 'publish-artifacts-in-last-commit' ) }}:
COMMIT_ID: $(Build.SourceVersion)
${{ if ne( parameters['CONFIGURATION_YAML_PATH'], '' ) }}:
CONFIGURATION_YAML_PATH: ${{ parameters.CONFIGURATION_YAML_PATH }}
Expected behavior
The publisher pipeline should run successfully and should deploy the operation policies without error.
Actual behavior
The pipeline is failing with the error message described in the description of the bug.
Reproduction Steps
- Run extractor pipeline and merge changes into repository
- Run publisher pipeline
- Observe publisher pipeline fails when attempting to deploy operation level policy
Issue Analytics
- State:
- Created 2 months ago
- Comments:6 (3 by maintainers)
We tried exporting as OpenAPI v3 from lower environment and importing to the higher environment via the YAML file. Import complains that there are several spots in the file with invalid syntax:
Looking at the exported YAML , it does indeed contain references like
'/definitions/...'
rather than'./components/...'
. I’m not sure how we could have authored that in the source APIM environment, if APIM won’t accept it. Makes sense though that it’s not accepted as I don’t believe'/definitions...'
is valid for OpenAPI. We manually updated the spec file to use'/components/...'
rather than'/definitions/...'
and that did allow us to successfully import it. I think the problem we’re facing is with the API definitions in the source environment and not an issue with the ApiOps tool. Customer is going to open a support case to understand how they could have invalid spec in the source environment.@christopherhouse anyway you can export the api from apim portal and import it in the apim portal? Then try to manually apply the policy. We are trying to see if there is an issue with the policy itself to understand whether it’s an API opposition in the first place.