question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

(aws-ecs): ELB TG can't connect to ECS EC2 instances ( healthcheck failed )

See original GitHub issue

ELB TG can’t connect to ECS EC2 instances ( healthcheck failed ) when use cluster.AsgCapacity over cluster.addCapacity .

Reproduction Steps


const.taskDefinition = new ecs.TaskDefinition(this, 'Backend', {
    family: 'someFamily',
    compatibility: ecs.Compatibility.EC2,
    executionRole,
    networkMode: ecs.NetworkMode.BRIDGE,
    taskRole,
});
taskDefinition.addContainer('backend', {
    image: ecs.ContainerImage.fromRegistry('hashicorp/http-echo'),
    memoryLimitMiB: 512,
    command: [
        `-listen=:${containerPort}`,
        '-text="hello world"'
    ],
    environment: {},
    portMappings: [
        {
            containerPort: containerPort,
            protocol: ecs.Protocol.TCP,
        },
    ],
});

const sg = new ec2.SecurityGroup(this, `SG${identifier}`, {
    vpc: this.cluster.vpc,
});

const autoScalingGroup = new autoscaling.AutoScalingGroup(this, `asg${identifier}`, {
    vpc: this.cluster.vpc,
    instanceType: new ec2.InstanceType(instanceType),
    machineImage: ecs.EcsOptimizedImage.amazonLinux2(),
    minCapacity: clusterMinCapacity,
    maxCapacity: clusterMaxCapacity,
    desiredCapacity: clusterDesiredCapacity,
    associatePublicIpAddress: true,
    cooldown: cdk.Duration.minutes(1),
    keyName: clusterKeyName,
    securityGroup: sg,
});

const asgProvider = new ecs.AsgCapacityProvider(this, `AsgProvider${identifier}`, {
    autoScalingGroup,
    canContainersAccessInstanceRole: true,
    enableManagedScaling: false,
    enableManagedTerminationProtection: false,
});

this.cluster.addAsgCapacityProvider(asgProvider);

What did you expect to happen?

I expect aws-ecs library automatically create security group with required inbound rules or have some method to allow connect ELB TG to EC2 instances

I expect method addAsgCapacityProvider add automatically access ELB TG to EC2 instances.

Normal SG created with cluster.addCapacity
image

What actually happened?

Actually EC2 instances create with only my security group inbound rules ( SSH ).
image

How I temporarily fixed this issue. I compared security group where create with cluster.addCapacity() and created SG with ASG provider.

This code fix trouble but I think this code must be default in aws-cdk.
Or I don’t understand from AWS CDK ECS last update and deprecation cluster.addCapacity

this.ecsPatternService.loadBalancer.connections.allowTo(sg, ec2.Port.tcpRange(32768, 65535), `allow ELB TG connect to EC2 ${instanceType}`);

Environment

  • CDK CLI Version : 1.104.0 (build 44d3383)
  • Framework Version : ^1.104.0
  • Node.js Version : v14.16.0
  • OS : Fedora release 33 (Thirty Three)
  • Language (Version) : TypeScript (3.8.3)

Issue Analytics

  • State:open
  • Created 2 years ago
  • Reactions:7
  • Comments:15 (7 by maintainers)

github_iconTop GitHub Comments

2reactions
Insidexacommented, May 24, 2021

@MrArnoldPalmer Sure, I will try to create my first PR to open source 😃

1reaction
robbiecooray1commented, Jul 23, 2022

@rix0rrr still an issue, specifically when upgrading from CDK v1 to v2 where AddAutoScalingGroup is deprecated. Above workaround from @spg works.

With .AddAutoScalingGroup, the following rules are in place. With .AddAsgCapacityProvider, the following diff appears (i.e. the rules get dropped), making the service unavailable for requests.

Tested with cdk v2.33.0.

Group Dir Protocol Peer
- ${alb/SecurityGroup.GroupId} Out TCP 32768 - 65535 ${clusterSG.GroupId}
- ${clusterSG.GroupId} In TCP 32768 - 65535 ${alb/SecurityGroup.GroupId}
Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshoot health check failures for Amazon ECS tasks on ...
Check if the ping port and the health check path are configured correctly. Make sure that your backend database is connected successfully. This ......
Read more >
AWS ECS error: Task failed ELB health checks in Target group
This is resolved. It was the issue with the following points: Docker container port mapping with host port were incorrect; ALB health check...
Read more >
How do I troubleshoot unhealthy ALB health checks in ECS ...
Find more details in the AWS Knowledge Center: https:// aws. amazon.com/premiumsupport... Jieling, an AWS Cloud Support Engineer, ...
Read more >
Gentle Introduction to How AWS ECS Works with Example ...
Use the my-ecs-sg security group that was created. You can get the id of the security group from the EC2 Console / Network...
Read more >
Resource: aws_lb_target_group - hashicorp - Terraform Registry
Instance Target Group. resource "aws_lb_target_group" "test" { name = "tf-example-lb-tg" port = 80 protocol = "HTTP" vpc_id = aws_vpc.main.id } resource ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found