CreateChangeSet occurs for every stack on every run, but change sets are never deleted

Every Stacker run executes a Create Change Set API call on every stack to determine if changes exist for specific stack. These change sets are not deleted and there is indeed a max number of change sets allowed. The change sets are cleared only when stacks are updated, when the change set is executed. When the limit is hit, stacker is no longer able to run. The exception thrown is below:

Command executed: stacker build ###.env ./###.yaml -r us-east-1 --replacements-only -i

11:29:10 Traceback (most recent call last):
11:29:10   File "/app/jenkins/workspace/venv/bin/stacker", line 9, in <module>
11:29:10     args.run(args)
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/commands/stacker/build.py", line 46, in run
11:29:10     dump=options.dump)
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/actions/base.py", line 131, in execute
11:29:10     self.run(*args, **kwargs)
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/actions/build.py", line 359, in run
11:29:10     plan.execute()
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/plan.py", line 292, in execute
11:29:10     if not self._single_run():
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/plan.py", line 252, in _single_run
11:29:10     status = step.run()
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/plan.py", line 81, in run
11:29:10     return self._run_func(self.stack, status=self.status)
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/actions/build.py", line 288, in _launch_stack
11:29:10     force_interactive=stack.protected)
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/providers/aws/default.py", line 755, in update_stack
11:29:10     **kwargs)
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/providers/aws/default.py", line 775, in interactive_update_stack
11:29:10     service_role=self.service_role, **kwargs
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/providers/aws/default.py", line 312, in create_change_set
11:29:10     kwargs=args
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/providers/aws/default.py", line 81, in retry_on_throttling
11:29:10     retry_checker=_throttling_checker)
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/stacker/util.py", line 69, in retry_with_backoff
11:29:10     return function(*args, **kwargs)
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/botocore/client.py", line 312, in _api_call
11:29:10     return self._make_api_call(operation_name, kwargs)
11:29:10   File "/app/jenkins/workspace/venv/lib/python2.7/site-packages/botocore/client.py", line 605, in _make_api_call
11:29:10     raise error_class(parsed_response, operation_name)
11:29:10 botocore.errorfactory.LimitExceededException: An error occurred (LimitExceededException) when calling the CreateChangeSet operation: ChangeSet limit exceeded for stack arn:aws:cloudformation:us-east-1:141627368605:stack/#############/14dc3c00-602a-11e7-a567-500c286e1a36
11:29:10 NOTE: FAILURE DETECTED - last stacker run failed, sleeping 10 seconds to back off API and marking job to fail...

[GarisonLotus]

changeset_nuke.py.zip

@phobologic and @russellballestrini - this is our hack job to fix the glitch on previously created change sets. It’s hacky, but works.

[ejholmes]

I just ran into this. When you run stacker in interactive mode, it will always create a change set, and if there’s no changes the change set will be “FAILED” and never removed by stacker. The interactive provider should just delete the changeset if there’s no changes.