Registering ECS Service as Target of Target Group Not Clear

When registering multiple target groups with different ports on a network load balancer, the registration ignores the target group port and looks for the first port from the container.

It’s easiest to look at the example below…

Reproduction Steps

Take the following code, run cdk synth, and then look at the target-group-stack.template.json in the output.

const vpcStack = new cdk.Stack(app, 'vpc-stack');
const tgStack = new cdk.Stack(app, 'target-group-stack', { 
  tags: { id: 'target-group-stack' }
});

const vpc = new ec2.Vpc(vpcStack, 'Vpc', { maxAzs: 2 });
const cluster = new ecs.Cluster(tgStack, 'cluster', { vpc, clusterName: 'cluster' });
cluster.addCapacity('DefaultAutoScalingGroup', {
  instanceType: ec2.InstanceType.of(ec2.InstanceClass.T2, ec2.InstanceSize.MICRO)
});

// Create Task Definition
const taskDefinition = new ecs.Ec2TaskDefinition(tgStack, 'task-def');
const container = taskDefinition.addContainer('task-container', {
  image: ecs.ContainerImage.fromRegistry('nginx:latest'),
  memoryLimitMiB: 512
});

container.addPortMappings({
  containerPort: 1111,
  protocol: ecs.Protocol.TCP
});

container.addPortMappings({
  containerPort: 2211,
  protocol: ecs.Protocol.TCP
});

const service = new ecs.Ec2Service(tgStack, 'ecs-service', { cluster, taskDefinition });
const lb = new elbv2.NetworkLoadBalancer(tgStack, 'nlb', { vpc });

const listener11 = lb.addListener('listener-11', { port: 1122 });
listener11.addTargets('listener-11-target', {
  targetGroupName: 'listener-11-target',
  port: 1133,
  targets: [service]
});

const listener22 = lb.addListener('listener-22', { port: 2222 });
listener22.addTargets('listener-22-target', {
  targetGroupName: 'listener-22-target',
  port: 2233,
  targets: [service]
});

This example is not runnable (for a lot of reasons), but the bug is visible in the template.json output.

The general idea is that we have a single ECS Service with multiple ports exposed. This ECS Service is running behind a Network Load Balancer and we want to allow traffic to the container on both ports from the Network Load Balancer. Thus, we add two port mappings on the container and two listeners on the load balancer.

In the example, all of the ports are unique to make the issue abundantly clear. The general pattern is that anything beginning with the same two prefix digits (11 / 22) is related to one another.

The problem is that when you call addTargets on a network load balancer listener, with an ECS Service target, the port provided is ignored. Part of the problem is visible in the documentation directly:

/**
 * Properties for adding new network targets to a listener
 */
export interface AddNetworkTargetsProps {
    /**
     * The port on which the listener listens for requests.
     *
     * @default Determined from protocol if known
     */
    readonly port: number;

The issue is that, the port provided to the addTargets method is not the listener port, it is the container port. This is visible in the CloudFormation template which is output:

   ...
    "ecsserviceService49E054F7": {
      "Type": "AWS::ECS::Service",
      "Properties": {
        "TaskDefinition": ...
        "Cluster": ...,
        "DeploymentConfiguration": ...,
        "DesiredCount": 1,
        "EnableECSManagedTags": false,
        "HealthCheckGracePeriodSeconds": 60,
        "LaunchType": "EC2",
        "LoadBalancers": [
          {
            "ContainerName": "task-container",
            "ContainerPort": 1111,
            "TargetGroupArn": {
              "Ref": "nlblistener1listener1targetGroup4A782B4A"
            }
          },
          {
            "ContainerName": "task-container",
            "ContainerPort": 1111,
            "TargetGroupArn": {
              "Ref": "nlblistener2listener2targetGroup274C7DCF"
            }
          }
        ],
        "SchedulingStrategy": "REPLICA"
      },
   ...

In the LoadBalancers section of the Cloud Formation sample, we should have references to the two target groups we added. However, the container reference ignores the provided ports. The expectation is that these values are 1133 and 2233 respectively. (Again this is to make the situation abundantly clear, in reality we would expect the ports to be 1111 and 2211)

The problem appears to be in @aws-cdk/aws-ecs/lib/base/base-service.ts:

  public loadBalancerTarget(options: LoadBalancerTargetOptions): IEcsLoadBalancerTarget {
    const self = this;
    const target = this.taskDefinition._validateTarget(options);
    ...

When the target is resolved from the options, it ignores any port request from the addTargets and it will grab the first containerPort it finds from the container port mappings on the container. Thus, if we change our first container port mapping from 1111 to 3311 the CloudFormation template would look like:

   ...
    "ecsserviceService49E054F7": {
      "Type": "AWS::ECS::Service",
      "Properties": {
        "TaskDefinition": ...
        "Cluster": ...,
        "DeploymentConfiguration": ...,
        "DesiredCount": 1,
        "EnableECSManagedTags": false,
        "HealthCheckGracePeriodSeconds": 60,
        "LaunchType": "EC2",
        "LoadBalancers": [
          {
            "ContainerName": "task-container",
            "ContainerPort": 3311,
            "TargetGroupArn": {
              "Ref": "nlblistener1listener1targetGroup4A782B4A"
            }
          },
          {
            "ContainerName": "task-container",
            "ContainerPort": 3311,
            "TargetGroupArn": {
              "Ref": "nlblistener2listener2targetGroup274C7DCF"
            }
          }
        ],
        "SchedulingStrategy": "REPLICA"
      },
   ...

I think this is an issue for Classic Load Balancers && Application Load Balancers as well. But I have not put together any examples yet.

I think there is a way this can be fixed without any breaking API changes, but I’ll find out this weekend.

Any thoughts shared in-advance of my attempts to fix this issue this weekend would be appreciated! 😄

Environment

• CLI Version : 1.9.0
• Framework Version: 1.9.0
• OS : all
• Language : all

---

This is 🐛 Bug Report

[rix0rrr]

I guess the fact that a Service serves as its own load balancer target (albeit with unconfigurable defaults) is the most confounding factor here. Unfortunately, this is something that we can’t change at the moment without losing backwards compatibility with 90% of people using the CDK. We will have to live with this and rectify for v2.

@iamhopaul123, maybe this can be helped by rewriting the examples in the examples repository as well?

Thanks for the feedback, I will be closing these issues now.

[jd-carroll]

Again, I will start by saying I am happy to have both the issue and the related PR closed. I will also say that I haven’t been clear enough with my own language, and have been conflating a target group and the target of a target group (my apologies).

In an effort to clarify, lets start with the cdk-examples. The advanced load balancing example has the following: (which is the source of all my troubles)

// Attach ALB to ECS Service
listener.addTargets('ECS', {
  port: 80,
  targets: [service],
  // include health check (default is none)
  healthCheck: {
    interval: cdk.Duration.seconds(60),
    path: "/health",
    timeout: cdk.Duration.seconds(5),
  }
});

The above example, taken with the example provided by @rix0rrr plus the registerLoadBalancerTargets represent too many ways to associate listeners/target-groups/targets (IMO).

Ultimately, I think all of my confusion / concerns with the API can be boiled down to these points:

1. If service.registerLoadBalancerTargets() is not recommended, it should be removed / deprecated.
2. The registration of an EcsService as the target of a target group should be done a single way (not multiple ways).

Currently, you can register an EcsService as the target of a target group by:

  targets: [service],

- or -

  targets: [service.loadBalancerTarget({ ... })],

Of the two, I think the latter service.loadBalancerTarget is the correct approach. (And maybe a different method name, although I can’t come up with a better one so loadBalancerTarget is great plus no change to API) This would mean changing the signature of BaseService to:

export abstract class BaseService extends Resource
  implements IService

Now that I have sufficiently beaten this to death 😄, I will go in peace. Only hope is to help avoid confusion for anyone that follows.