(Custom Script Extension) Script failure causes following ARM template deployment to fail

When a script invoked by the Custom Script extension returns an error code, I’ve observed that the next ARM template deployment will also fail. The real problem is that the VM is the resource that fails. This poses a problem for custom scripts executed in ARM template deployments, because the custom script dependsOn the VM. So, it will never run again and have a chance to correct itself. (We use a timestamp in settings to ensure the script runs on each deployment).

Here’s the error output from New-AzureRmResourceGroupDeployment:

VERBOSE: 12:41:22 AM - Create template deployment 'azuredeploy-####-####'.
VERBOSE: 12:41:30 AM - Resource Microsoft.Network/virtualNetworks '#####' provisioning status is succeeded
VERBOSE: 12:41:33 AM - Resource Microsoft.Storage/storageAccounts '#####' provisioning status is succeeded
VERBOSE: 12:41:33 AM - Resource Microsoft.Network/publicIPAddresses '#####' provisioning status is succeeded
VERBOSE: 12:41:36 AM - Resource Microsoft.Compute/virtualMachines '#####' provisioning status is running
VERBOSE: 12:41:36 AM - Resource Microsoft.Network/networkInterfaces '#####' provisioning status is succeeded
New-AzureRmResourceGroupDeployment : 12:42:12 AM - Resource Microsoft.Compute/virtualMachines '#####' failed with message 'The resource 
operation completed with terminal provisioning state 'Failed'.'
At C:\#####\scripts\Deploy.ps1:65 char:11
+ $deploy = New-AzureRmResourceGroupDeployment -Name ((Get-ChildItem $T ...
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.NewAzureResourceGroupDeploymentCommand

New-AzureRmResourceGroupDeployment : 12:42:12 AM - VM has reported a failure when processing extension 'shexec'. Error message: "Script returned an 
error.
---stdout---
---errout---
(error output)
".
At C:\#####\scripts\Deploy.ps1:65 char:11
+ $deploy = New-AzureRmResourceGroupDeployment -Name ((Get-ChildItem $T ...
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.NewAzureResourceGroupDeploymentCommand

We can work around this problem by invoking custom scripts outside of ARM templates, but it still means we need an extra deploy to fix the issue. Alternatively, I think that we can invoke “/bin/true” after a failed ARM template deployment to get the extension back into a non-error state. However, I can see this issue affecting others, and working around it in this manner is non-intuitive.

[alexandreweiss]

Ok so what can we do when the VMSS is in Failed state and the extension cannot be played again ?

[alexandreweiss]

Just talk to VMSS team. My mistake ! I was running the extension as a top level resource of my template , depending on (using “Depends on” property) the VMSS which was in “Failed” state so execution could never happen. I moved extension to be part of the extensionProfile of the VMSS and now it works as a charm. I am able to trigger the extension many times even if last exit RC is >0. Thanks again !