question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

(ApplicationLoadBalancedFargateService): Auto scaling won't scale-in after update service with force new deployment option

See original GitHub issue

The ECS(Fargate) auto scaling out is working as expect, but if I update the ECS service with force new deployment option in the AWS web UI, after the deployment process done, the ECS service CPU/MEN usage decreased and the CloudWatch Alarm which named with ‘AlarmLow’ triggered but the Desired count not decreasing.

Reproduction Steps

  1. Deploy a ECS service with ApplicationLoadBalancedFargateService as following.

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-ecs-patterns-readme.html#add-metric-based-auto-scaling-to-an-applicationloadbalancedfargateservice

  1. Trigger auto-scale out with some test traffic
  2. Click update in ECS service panel in AWS UI
  3. Check ‘force new deployment’ box, don’t touch anything else and update the ECS service
  4. waiting for new deployment done and the CPU/MEN usage should below the auto scale alert count.
  5. Check the CloudWatch and the alarm named with ‘AlarmLow’ should in alarm status but ECS Desired Count not decrease

What did you expect to happen?

the CPU/MEN usage is below the alert setting and the ‘AlarmLow’ in alarm status, the desired count should be decreased.

What actually happened?

‘AlarmLow’ in alarm status but the ECS desired count not decreased.

Environment

Other


This is 🐛 Bug Report

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:8

github_iconTop GitHub Comments

1reaction
TrentonAdamscommented, Dec 22, 2021

Oh, I don’t have permission to close. Who can?

1reaction
TrentonAdamscommented, Nov 25, 2021

I’m not sure if this is related, but I’m having a problem with deleting a stack with an ASG for ECS.

There’s a message in the ASG configuration (activity tab) saying Could not scale to desired capacity because all remaining instances are protected from scale-in.. I go disable scale-in protection on the instance, and it continues deleting the stack almost immediately.

I’m guessing the following should default newInstancesProtectedFromScaleIn to false in the properties???

        const autoScalingGroup = new autoscaling.AutoScalingGroup(this, 'ASG', {
            vpc: props.smw_vpc,
            instanceType: new ec2.InstanceType('t2.micro'),
            machineImage: ecs.EcsOptimizedImage.amazonLinux2(),
            minCapacity: 0,
            desiredCapacity: 1,
            maxCapacity: 2,
        });

e.g.

        const autoScalingGroup = new autoscaling.AutoScalingGroup(this, 'ASG', {
            vpc: props.smw_vpc,
            instanceType: new ec2.InstanceType('t2.micro'),
            newInstancesProtectedFromScaleIn: false,
            machineImage: ecs.EcsOptimizedImage.amazonLinux2(),
            minCapacity: 0,
            desiredCapacity: 1,
            maxCapacity: 2,
        });

Should I file a separate issue, or is this related?

To be clear, I think it’s really bizarre for AWS to be putting anything in a “protected” mode without it being explicitly requested.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshoot service auto scaling issues in Amazon ECS
Service auto scaling might not be able to add or remove tasks as expected due to one or more of the following reasons:...
Read more >
Lab 2: Deploy Cluster
This is what we'll have after the deployment is complete: Frontend Application Load Balancer; Rails frontend service with auto-scaling; 3 frontend application ...
Read more >
How do I deploy updated Docker images to Amazon ECS ...
Actually, since/if you have enabled auto scaling on your ECS service, a simple way to do it is to just force a new...
Read more >
Deploy Your Django Apps in AWS With CDK (v2)
It comes with a toolkit that allows creating a new CDK project, synthesizing the code, and deploying it using a chosen account and...
Read more >
ECS Service Auto-Scaling with the CDK - Idan Lupinsky
This is known as a scale-in event. This behavior is achieved by creating a target tracking scaling policy with the Application Auto Scaling...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found