(aws_ecs_patterns.ApplicationLoadBalancedServiceBase): ECS service creation in race with EC2 instance creation

Testing ApplicationLoadBalancedEc2Service in CDK version 1.97.0. Here’s simple project that reproduces the problem.

During deployment, I see this in the ECS Cluster Service “Events” tab (see screenshot below):

2021-04-11 12:05:06 -0400 service myWebApp-service has started 1 tasks: task .

2021-04-11 12:04:35 -0400 service myWebApp-service was unable to place a task because no container instance met all of its requirements. Reason: No Container Instances were found in your cluster. For more information, see the Troubleshooting section.

The system does stabilize eventually, but it made me concerned that there might be a (minor) bug in ApplicationLoadBalancedEc2Service.

Is there a race condition or lack of an internal dependency to make sure that the EC2 instance is ready before the service tries to “place a task”?

[jrobbins-LiveData]

@nathanpeck , that is a wonderfully informative answer. I really appreciate the time you took to make the overall “intended use” so clear. I see better that I was overly focused on a “tree” rather than the “forest”, misinterpreting that status message as if it were an error. Thanks for helping me understand that!!

[nathanpeck]

This isn’t really a bug. ECS always continuously monitors services and will launch the tasks when there is capacity for them, once that capacity comes online, so it is perfect safe to create the ECS service first. There is no need to introduce a hard waiting period before creating the ECS service. Ironically that would only make the stack deployment slower because it would introduce an artificial waiting period before ECS even begins looking for capacity. The way it currently works the ECS service can be in place and already there and waiting for the capacity when it comes online. This type of message about being unable to find capacity is not an error, it is just a status message, and you will see it often, for example you see it when scaling out a service before a capacity provider kicks in and scales out the underlying EC2 cluster.

I would not recommend changing behavior here for two reasons:

1. Introducing a hard wait on EC2 capacity to be up first before creating the ECS service would just make the end to end process of stack deployment slower
2. The long term intent of ECS + EC2 integration is that we move away from a model of “you must provision your own EC2 capacity separately from your service” to a model of “you create the ECS service first, it requests cluster capacity, and then a cluster capacity provider fills that request on the fly by launching EC2 instances”.

Of the two reasons the second is the more important reason: in the long run the ideal intended usage of ECS is to start your cluster with zero capacity at all, and then when you create a service it requests capacity from a capacity provider, and the capacity provider fills that request for capacity afterwards.