ApplicationGatewaySubnetInboundTrafficBlockedByNetworkSecurityGroup on repeated deployments

Describe the bug I use the aks construction set from a module and when I deploy it the first time, it’s all fine. When I don’t change anything and deploy it again, it fails with this error:

{
  "code": "DeploymentFailed",
  "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
  "details": [
    {
      "code": "ApplicationGatewaySubnetInboundTrafficBlockedByNetworkSecurityGroup",
      "message": "Network security group /subscriptions/8b25cfae-660b-4066-9ce5-0858e9ab3eac/resourceGroups/test-raro-us/providers/Microsoft.Network/networkSecurityGroups/nsg-agw-nprd-euw-jk67 blocks incoming internet traffic on ports 65200 - 65535 to subnet /subscriptions/8b25cfae-660b-4066-9ce5-0858e9ab3eac/resourceGroups/test-raro-us/providers/Microsoft.Network/virtualNetworks/vnet-nprd-euw-jk67/subnets/appgw-sn, associated with Application Gateway /subscriptions/8b25cfae-660b-4066-9ce5-0858e9ab3eac/resourceGroups/test-raro-us/providers/Microsoft.Network/applicationGateways/agw-nprd-euw-jk67. This is not permitted for Application Gateways that have V2 Sku."
    }
  ]
}

my aks.bicep that I use to call the aks construction module:

// borrow from: https://github.com/Azure/Aks-Construction
// TODOS:


@description('Unique String for all public names')
@maxLength(4)
@minLength(4)
param salt string = take(uniqueString(resourceGroup().id), 4)

// You should use a bicep file to represent an application deployment
// This "Application Main.Bicep" will then call the AKS Construction module

//------Application General Parameters------

param location string =  resourceGroup().location

@allowed([
  'Production'
  'NonProduction'
])
@description('The performance/size spec of the generated resources')
param environmentScale string = 'NonProduction'

//------------------NAMING-----------------
//- As per CAF naming standards: https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/resource-naming
//Example generated names will be : sql-app-nprd-euw-001, aks-app-nprd-euw-001, log-app-nprd-euw-001

@description('Short name to use for environment resource name')
param environmentName string = 'nprd'

@description('Instance or other differentiator for resource deployment')
param namesuffix string = salt

@description('This json file is used to convert a long location name (eg. UK South) to its short version (eg. UKS)')
var shortLocationNames = json(loadTextContent('./shortLocationNames.json'))

@description('Maps the long name to the short name')
var shortLocationName = '${shortLocationNames[location]}'

@description('Creates a full resource name seed.')
var fullnameseed = '${environmentName}-${shortLocationName}-${namesuffix}'

@description('Should Private Links be created.')
param privateLinks bool = false

//--------------ENVIRONMENT-------------
var environmentConfigurationMap = {
  Production: {
    aks: {
      availabilityZones: [
        '1'
        '2'
        '3'
      ]
      AksPaidSkuForSLA: true
      JustUseSystemPool: false
      privateCluster: true
      networkPolicy: 'azure'
      agentCount:6
      agentCountMax:12
      agentVMSize: 'Standard_D8s_v4'
    }
  }
  NonProduction: {
    aks: {
      availabilityZones: []
      AksPaidSkuForSLA: false
      JustUseSystemPool: true
      privateCluster: false
      agentCount:1
      agentCountMax:3
      networkPolicy: 'azure'
      agentVMSize: 'Standard_DS3_v2'
    }
  }
}


//---------Kubernetes Construction---------
//ref: https://github.com/Azure/Aks-Construction
param k8sVersion string = '1.22.4'

module aksconst './aks-construction/main.bicep' = {
  name: 'aksconstruction'
  params: {
    location : location
    AksDisableLocalAccounts: true
    upgradeChannel: 'stable'
    resourceName: fullnameseed
    kubernetesVersion: k8sVersion
    enablePrivateCluster :  environmentConfigurationMap[environmentScale].aks.privateCluster
    agentCount: environmentConfigurationMap[environmentScale].aks.agentCount
    agentVMSize: environmentConfigurationMap[environmentScale].aks.agentVMSize
    agentCountMax: environmentConfigurationMap[environmentScale].aks.agentCountMax
    enable_aad: true
    enableAzureRBAC : true
    registries_sku: 'Premium' //no acr
    omsagent: true
    retentionInDays: 30
    custom_vnet: true
    ingressApplicationGateway: true
    appGWcount: 0 
	  appGWsku: 'WAF_v2' 
	  appGWmaxCount: 10
    appgwKVIntegration: true
    azureKeyvaultSecretsProvider: true
    createKV: true
    privateLinks: false // something prevents true from working here,  privateLinks
    networkPolicy: environmentConfigurationMap[environmentScale].aks.networkPolicy
    azurepolicy: 'audit' // no policy, lack of permissions
    availabilityZones: environmentConfigurationMap[environmentScale].aks.availabilityZones
    AksPaidSkuForSLA: environmentConfigurationMap[environmentScale].aks.AksPaidSkuForSLA
    JustUseSystemPool: environmentConfigurationMap[environmentScale].aks.JustUseSystemPool
  }
}

output aksClusterName string = aksconst.outputs.aksClusterName
output ApplicationGatewayName string = aksconst.outputs.ApplicationGatewayName

I use the azure cli to deploy it.

To Reproduce Steps to reproduce the behavior:

1. Deploy the AKS Construction with the aforementioned aks.bicep
2. Deploy it again without changing anything
3. See error

Expected behavior I would expect to not get an error and that repeated deployments run fine.

[Gordonby]

This looks like a problem with the Azure Resource Provider, I have raised an issue.

As a short-term workaround, to reduce the impact on Day1 users of this project; I have changed the default parameter value to not deploy the appgw nsg in #207 . For users who require the NSG, it will have to be created outside the scope of the AKS-Construction codebase.

This does not fix the idempotency issue around the NSG, which should come as part of the issue raised.

[Gordonby]

Reproduced issue. Will investigate.

az deployment group create -g innerloop -f .\localdebug\testing-issues\nsg-idempotant\main.bicep -n issuetest1 
az deployment group what-if -g innerloop -f .\localdebug\testing-issues\nsg-idempotant\main.bicep -n issuetest1 

‘int’ object has no attribute ‘append’

az deployment group what-if -g innerloop -f .\localdebug\testing-issues\nsg-idempotant\main.bicep -n issuetest1 --debug

cli.azure.cli.core.azclierror: ‘int’ object has no attribute ‘append’ az_command_data_logger: ‘int’ object has no attribute ‘append’

az deployment group create -g innerloop -f .\localdebug\testing-issues\nsg-idempotant\main.bicep -n issuetest1

{“code”:“DeploymentFailed”,“message”:“At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.”,“details”:[{“code”:“ApplicationGatewaySubnetInboundTrafficBlockedByNetworkSecurityGroup”,“message”:“Network security group blocks incoming internet traffic on ports 65200 - 65535 to associated with Application Gateway. This is not permitted for Application Gateways that have V2 Sku.”}]}