question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

cdk deploy in endless loop cause of Fargate Service cant fire up task

See original GitHub issue

I am deploying a codepipeline stack with deployment to a fargate service. Problem is, when there is an issue starting the fargate task, the deployment never returns because fargate tries to start the task over and over again (like every minute or so).

Roughly my code is:

public createEcsDeployAction(vpc: Vpc, ecrRepo: ecr.Repository, buildOutput : Artifact): EcsDeployAction {
    return new EcsDeployAction({
      actionName: 'EcsDeployAction',
      service: this.createLoadBalancedFargateService(this, vpc, ecrRepo).service,
      input: buildOutput,
    })
  };


  createLoadBalancedFargateService(scope: Construct, vpc: Vpc, ecrRepository: ecr.Repository) {
    return new ecspatterns.ApplicationLoadBalancedFargateService(scope, 'myLbFargateService', {
      vpc: vpc,
      serviceName: "HelloWorldFargateService",
      memoryLimitMiB: 512,
      cpu: 256,
      taskImageOptions: {
        image: ecs.ContainerImage.fromEcrRepository(ecrRepository, "latest"),
      },
    });
  }

My problem could be that i define an image in the LoadBalancedFargateService which isnt available during deployment of the stack because codePipeline didnt run yet. Dont know for sure.

Question remains if its wise to just never terminate the “cdk deploy” cause of neverending tries to fire up a task in the backend.

Reproduction Steps

hard to reproduce out of context.

Error Log

no error in console on cdk deploy. Hard to find the real error. Tried it via AWS console without success.

Environment

  • CLI Version : aws-cli/2.0.10 Python/3.8.2 Darwin/19.4.0 botocore/2.0.0dev14
  • Framework Version: 1.36.1 (build 4df7dac)
  • OS : Mac OS X

This is 🐛 Bug Report

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:14
  • Comments:20 (3 by maintainers)

github_iconTop GitHub Comments

5reactions
skinny85commented, May 4, 2020

Hey @logemann ,

yes, this is an issue. Basically, the problem is that we’re missing a concept in the CDK currently, that represent “an image that doesn’t exist yet, but will be created when the CodePipeline runs”.

In a demo project we’ve done a long time ago, we have a class that represents exactly that. This is how it is used: [1], [2].

Would adding this class to the main CDK project solve your issue @logemann ? If so, I will convert this issue to a feature request.

Thanks, Adam

5reactions
jonny-rimekcommented, May 2, 2020
		const fargateTask = new ecs.FargateTaskDefinition(this, 'FargateTask', {
			cpu: 256,
			memoryLimitMiB: 512,
		})

		fargateTask.addContainer("GinContainer", {
			image: ecs.ContainerImage.fromAsset('services/api')
		})

		const cluster = new ecs.Cluster(this, 'Cluster', {
			containerInsights: true,
			vpc
		})

		const fargateService = new ecs.FargateService(this, 'FargateService', {
			cluster,
			taskDefinition: fargateTask,
			desiredCount: 1,
			assignPublicIp: true,
			platformVersion: ecs.FargatePlatformVersion.VERSION1_4
		})

I dropped the ecs pattern and did everything from scratch and the deployment works just fine(no ALB yet)

If I remove assignPublicIp I get the following error message Stopped reason ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 1 time(s): RequestError: send request failed caused by: Post https://api.ecr.... and the deployment is back to being stuck

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshoot Amazon ECS tasks for Fargate that are stuck in ...
My Amazon Elastic Container Service (Amazon ECS) task that's running on AWS Fargate is stuck in the PENDING state.
Read more >
Aws Cdk Deploy --All Fails To Create Ecs Service - ADocLib
Ask questionscdk deploy in endless loop cause of Fargate Service cant fire up task. I am deploying a codepipeline stack with deployment to...
Read more >
How to use AWS Fargate and Lambda for long-running ...
The ECS Fargate Task executes the Docker container: that processes the video file to extract thumbnail,; and uploads the thumbnail image to S3....
Read more >
SERVERLESS CONTAINERS with CDK (from 0 to hero)
In this episode you would learn everything you need to know about getting started with AWS Fargate using CDK on.
Read more >
AWS Cloud Development Kit - Infrastructure Is Code (From ...
AWS Cloud Development Kit (CDK) - Infrastructure-Is-Code Provisioning cloud applications can be a challenging process that requires you to perform manual ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found