GCP Instance templates / Instance group manager - error deleting

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you’ve opened one already)

Issue details

When working with GCP and instance templates / instance groups, any failure to the run, corrupts the state without ability to cleanly recover. Say the casual scenario, where you create instance template and use instance group manager in GCP(along with other things). When provisioning, if something else fails during the run, Pulumi marks in the state that instance group should have new template(which was assigned during that run) and then on the next run tries to delete the previous template, but this actually being used as the previous update failed, so it cannot delete it. And this becomes real problem when having multiple instance groups. The only way to recover from it, is to create new templates, switch instance groups to use them and re-run Pulumi, so that it will be able to remove those marked for deletion(refresh does not help in this scenario). Example:

my-app-instance-template (mystack:group$mystack:mygroup$gcp:compute/instanceTemplate:InstanceTemplate)
error: deleting urn:pulumi:myproject::mystack::mystack:group$mystack:mygroup$gcp:compute/instanceTemplate:InstanceTemplate::my-app-instance-template: 1 error occurred:
	* Error waiting for Deleting Instance Template: The instance_template resource 'projects/myproject/global/instanceTemplates/my-app-instance-template-COMMIT_HASH' is already being used by 'projects/myproject/zones/myzone/instanceGroupManagers/my-app'

Steps to reproduce

1. Create multiple compute instance templates and instance groups in GCP, setting instance template name to, say: my-app-instance-template-COMMIT_HASH
2. After provisioning, run Pulumi again with setting instance template to my-app-instance-template-COMMIT_HASH2 and kill the deployment (process kill, network or anything else) in the middle of it, say when it starts creating instance templates or updating instance groups. Build will be marked as failed.
3. Run pulumi refresh/up again
4. If the build was cancelled at the “right” point in time at step 2, the new up command will fail with the above message, that it cannot delete my-app-instance-template-COMMIT_HASH, but it cannot delete because actually the instance group is still using that template(because of failed run), yet Pulumi thinks, that it should be already using my-app-instance-template-COMMIT_HASH2.

Expected: not mark the template for deletion, unless the instance group manager has been actually assigned that template and previous instance was removed from the group. Actual: Pulumi marks that instance group is running on my-app-instance-template-COMMIT_HASH2, while it actually running on my-app-instance-template-COMMIT_HASH and tries to remove my-app-instance-template-COMMIT_HASH unsuccessfully, as it’s still being used by instance group manager.

[amkartashov]

I’ve seen this this issue multiple times, but had not enough details for report. Thanks @stakauskas!

[jdoss]

I ran into this issue today when I was scaling down my instances to zero across three managed groups.

Type                             Name                        Status                   Info
     pulumi:pulumi:Stack              internal-testing            **failed**               1 error
 +-  ├─ gcp:compute:InstanceTemplate  instancetemplate-command    **replacing failed**     1 error
 +-  ├─ gcp:compute:InstanceTemplate  instancetemplate-compute    **replacing failed**     1 error
 +-  └─ gcp:compute:InstanceTemplate  instancetemplate-ephemeral  **replacing failed**     1 error
 
Diagnostics:
  gcp:compute:InstanceTemplate (instancetemplate-compute):
    error: deleting urn:pulumi:testing::internal::gcp:compute/instanceTemplate:InstanceTemplate::instancetemplate-compute: 1 error occurred:
      * Error waiting for Deleting Instance Template: The instance_template resource 'projects/eng-internal/global/instanceTemplates/instancetemplate-compute-dc279c0' is already being used by 'projects/eng-internal/regions/us-central1/instanceGroupManagers/mig-compute-98fe1f2'
 
  gcp:compute:InstanceTemplate (instancetemplate-ephemeral):
    error: deleting urn:pulumi:testing::internal::gcp:compute/instanceTemplate:InstanceTemplate::instancetemplate-ephemeral: 1 error occurred:
      * Error waiting for Deleting Instance Template: The instance_template resource 'projects/eng-internal/global/instanceTemplates/instancetemplate-ephemeral-5fd9833' is already being used by 'projects/eng-internal/regions/us-central1/instanceGroupManagers/mig-ephemeral-e21d83a'
 
  gcp:compute:InstanceTemplate (instancetemplate-command):
    error: deleting urn:pulumi:testing::internal::gcp:compute/instanceTemplate:InstanceTemplate::instancetemplate-command: 1 error occurred:
      * Error waiting for Deleting Instance Template: The instance_template resource 'projects/eng-internal/global/instanceTemplates/instancetemplate-command-340eef8' is already being used by 'projects/eng-internal/regions/us-central1/instanceGroupManagers/mig-command-f9e2c73'

Any chance to get some eyes on this from Pulumi staff? This is the number 1 thumbs up’d bug with the next closest issue having 5 thumbs.

Not being able to reliably use instance templates and groups is a show stopper on adopting Pulumi for advanced deployments on GCP.