Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

"Rate exceeded" error on FargateCluster

See original GitHub issue

from dask_cloudprovider import FargateCluster
cluster = FargateCluster(n_workers=1, image='rsignell/pangeo-worker:2020-01-23c')
client = Client(cluster)

worked on my SageMaker instance on uswest-2, but failed my useast-1 instance with:

Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /home/ec2-user/SageMaker/myenvs/pangeo/lib/python3.6/asyncio/tasks.py:530> exception=ClientError('An error occurred (ThrottlingException) when calling the DescribeTasks operation (reached max retries: 4): Rate exceeded',)>
Traceback (most recent call last):
  File "/home/ec2-user/SageMaker/myenvs/pangeo/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/home/ec2-user/SageMaker/myenvs/pangeo/lib/python3.6/site-packages/dask_cloudprovider/providers/aws/ecs.py", line 130, in _
    await self.start()
  File "/home/ec2-user/SageMaker/myenvs/pangeo/lib/python3.6/site-packages/dask_cloudprovider/providers/aws/ecs.py", line 240, in start
    await self._update_task()
  File "/home/ec2-user/SageMaker/myenvs/pangeo/lib/python3.6/site-packages/dask_cloudprovider/providers/aws/ecs.py", line 158, in _update_task
    cluster=self.cluster_arn, tasks=[self.task_arn]
  File "/home/ec2-user/SageMaker/myenvs/pangeo/lib/python3.6/site-packages/aiobotocore/client.py", line 102, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeTasks operation (reached max retries: 4): Rate exceeded

It looks like the PR https://github.com/dask/dask-cloudprovider/pull/44 was designed to address at these problems, but it seems I’m still having them despite running v0.1.1 which includes that PR. I’m wondering whether others are still experiencing this?

Issue Analytics

State:
Created 4 years ago
Reactions:5
Comments:24 (10 by maintainers)

Top GitHub Comments

2reactions

lukeorlandcommented, Aug 14, 2020

I’m having a similar issue, except for me it’s botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DeregisterTaskDefinition operation (reached max retries: 4): Rate exceeded

1reaction

michaellee1commented, Jul 26, 2022

Hi - I’m getting the same error as a previous user in this thread.

Error retrieving security group information for [sg-0176ba5002edd59c3]: Request limit exceeded. (ErrorCode: RequestLimitExceeded)

To my knowledge, there isn’t anything I can do to increase this from Service Quotas. This only happens when I send off a large Batch array job on Fargate.

Top Results From Across the Web

How do I prevent "Rate exceeded" errors in CloudFormation?

The Rate exceeded error occurs when API calls to an AWS service exceed the maximum allowed API requests, and the API calls are...

Key Metrics for Monitoring AWS Fargate - Datadog

Learn the key metrics you'll need to monitor when you're running ECS and EKS workloads on AWS Fargate.

Deploy Microservices Using AWS ECS Fargate and API ...

ECS Fargate cluster is created. Isn't easy? You can take a look at your cluster.We have not deployed anything there so, you will...

Autoscaling GitLab CI on AWS Fargate

ECS Fargate cluster created in step 5. ECS task definition created in step 6. Troubleshooting. No Container Instances were found in your cluster...

dask_cloudprovider.aws.ecs - Dask Cloud Provider

... ClientError as e: if e.response["Error"]["Code"] == "ThrottlingException": warnings.warn( "get_log_events rate limit exceeded, retrying after delay.