Many jobs starting at the same time exceeds task definition rate limit and holds it there

If to many tasks start in a short time (as on a busy jenkins) then this plugin starts hitting the rate limit on task definition creation. Once it’s in that state it retries many times and no tasks ever start. It constantly get’s the error:

com.amazonaws.services.ecs.model.ClientException: Too many concurrent attempts to create a new revision of the specified family. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: bb3cafd7-97b2-11e9-9e00-5ff346cf8ee8)

A workaround to get past this is to specify the Task Definition Override option to the name of the most recently created task definition. This stops this plugin from generating new task definitions and get’s past this error.

One idea for a proper fix would be a weighted random back off when this exception is caught.

     com.amazonaws.services.ecs.model.ClientException: Too many concurrent attempts to create a new revision of the specified family. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: bb3cNNNNN8ee8)
             at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1695)
             at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1350)
             at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1101)
             at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758)
             at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732)
             at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714)
             at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674)
             at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656)
             at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520)
             at com.amazonaws.services.ecs.AmazonECSClient.doInvoke(AmazonECSClient.java:3265)
             at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:3232)
             at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:3221)
             at com.amazonaws.services.ecs.AmazonECSClient.executeRegisterTaskDefinition(AmazonECSClient.java:2284)
             at com.amazonaws.services.ecs.AmazonECSClient.registerTaskDefinition(AmazonECSClient.java:2255)
             at com.cloudbees.jenkins.plugins.amazonecs.ECSService.registerTemplate(ECSService.java:285)
             at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.getTaskDefinition(ECSLauncher.java:200)
             at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:106)
             at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
             at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
             at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
             at java.util.concurrent.FutureTask.run(FutureTask.java:266)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
             at java.lang.Thread.run(Thread.java:748)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.