botocore.exceptions.WaiterError: Waiter LoadBalancerAvailable failed: Max attempts exceeded. Previously accepted state: For expression "LoadBalancers[].State.Code" we matched expected path: "provisioning" at least once

Context

Error when trying to deploy. botocore.exceptions.WaiterError: Waiter LoadBalancerAvailable failed: Max attempts exceeded. Previously accepted state: For expression "LoadBalancers[].State.Code" we matched expected path: "provisioning" at least once

(****-zappa) john@john-work:~/Code/projects/****/code/src$ zappa deploy
Calling deploy for stage dev..
Downloading and installing dependencies..
 - pyyaml==5.4.1: Downloading
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 662k/662k [00:00<00:00, 7.57MB/s]
Packaging project as zip.
Uploading ***-dev-1628189008.zip (25.4MiB)..
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26.7M/26.7M [00:06<00:00, 3.83MB/s]
Scheduling..
Scheduled ****-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Deploying ALB infrastructure...
Waiting for load balancer [arn:aws:elasticloadbalancing:us-east-1:****:loadbalancer/app/***-dev/e6d96ee8829e55df] to become active..
Oh no! An error occurred! :(

==============

Traceback (most recent call last):
  File "/home/john/.pyenv/versions/3.8.11/envs/****-zappa/lib/python3.8/site-packages/zappa/cli.py", line 3422, in handle
    sys.exit(cli.handle())
  File "/home/john/.pyenv/versions/3.8.11/envs/****-zappa/lib/python3.8/site-packages/zappa/cli.py", line 588, in handle
    self.dispatch_command(self.command, stage)
  File "/home/john/.pyenv/versions/3.8.11/envs/****-zappa/lib/python3.8/site-packages/zappa/cli.py", line 630, in dispatch_command
    self.deploy(self.vargs["zip"], self.vargs["docker_image_uri"])
  File "/home/john/.pyenv/versions/3.8.11/envs/****-zappa/lib/python3.8/site-packages/zappa/cli.py", line 947, in deploy
    self.zappa.deploy_lambda_alb(**kwargs)
  File "/home/john/.pyenv/versions/3.8.11/envs/****-zappa/lib/python3.8/site-packages/zappa/core.py", line 1609, in deploy_lambda_alb
    waiter.wait(LoadBalancerArns=[load_balancer_arn], WaiterConfig={"Delay": 3})
  File "/home/john/.pyenv/versions/3.8.11/envs/****-zappa/lib/python3.8/site-packages/botocore/waiter.py", line 53, in wait
    Waiter.wait(self, **kwargs)
  File "/home/john/.pyenv/versions/3.8.11/envs/****-zappa/lib/python3.8/site-packages/botocore/waiter.py", line 362, in wait
    raise WaiterError(
botocore.exceptions.WaiterError: Waiter LoadBalancerAvailable failed: Max attempts exceeded. Previously accepted state: For expression "LoadBalancers[].State.Code" we matched expected path: "provisioning" at least once

==============

Your Environment

• Zappa version used: zappa==0.53.0
• Operating System and Python version: Ubuntu 21.04 - Python3.8.11
• The output of pip freeze:

argcomplete==1.12.3
asgiref==3.4.1
boto3==1.18.13
botocore==1.21.13
certifi==2021.5.30
cfn-flip==1.2.3
charset-normalizer==2.0.4
click==8.0.1
Django==3.2.6
django-localflavor==3.1
django-phonenumber-field==5.2.0
django-storages==1.11.1
durationpy==0.5
future==0.18.2
hjson==3.0.2
idna==3.2
jmespath==0.10.0
kappa==0.6.0
pep517==0.11.0
phonenumberslite==8.12.28
pip-tools==6.2.0
placebo==0.9.0
psycopg2-binary==2.9.1
python-dateutil==2.8.2
python-slugify==5.0.2
python-stdnum==1.16
pytz==2021.1
PyYAML==5.4.1
requests==2.26.0
s3transfer==0.5.0
six==1.16.0
sqlparse==0.4.1
text-unidecode==1.3
toml==0.10.2
tomli==1.2.0
tqdm==4.62.0
troposphere==3.0.2
urllib3==1.26.6
Werkzeug==0.16.1
wsgi-request-logger==0.4.6
zappa==0.53.0

• Link to your project (optional):
• Your zappa_settings.json:

{
    "dev": {
        "alb_enabled": true,
          "alb_vpc_config": {
              "CertificateArn": "****",
              "SubnetIds": [
                "****",
                "****"
              ],
              "SecurityGroupIds": [
                 "****"
              ]
          },
        "apigateway_enabled": false,
        "aws_region": "us-east-1",
        "certificate_arn": "****",
        "domain": "****",
        "django_settings": "****.settings",
        "profile_name": "default",
        "project_name": "****",
        "runtime": "python3.8",
        "s3_bucket": "zappa-****",
        "vpc_config" : {
            "SubnetIds": [
                "****,
                "****"
            ],
            "SecurityGroupIds": [
                "****"
            ]
        }
    }
}

[klowe0100]

From what I can tell so far, it seems pretty isolated to boto’s alb provisioning call. Perhaps, provisioning an ALB just takes longer than previously expected. The wait logic in botocore seems to check-out.

Interestingly, the elbv2 client config had this:

data/elbv2/2015-12-01/waiters-2.json

"LoadBalancerAvailable": {                     
  "delay": 15,                                 
  "operation": "DescribeLoadBalancers",        
  "maxAttempts": 40,                           
  "acceptors": [                               
    {                                          
      "state": "success",                      
      "matcher": "pathAll",                    
      "argument": "LoadBalancers[].State.Code",
      "expected": "active"                     
    },                                         
    {                                          
      "state": "retry",                        
      "matcher": "pathAny",                    
      "argument": "LoadBalancers[].State.Code",
      "expected": "provisioning"               
    },                                         
    {                                          
      "state": "retry",                        
      "matcher": "error",                      
      "expected": "LoadBalancerNotFound"       
    }                                          
  ]                                            
},                                             

That makes me think their default wait is 15x40.

Since everyone’s provisioning time will be somewhat unknown, that makes me lean away from a config param in zappa. If there’s a reasonable default that cover 90-100% of cases I would go with that for simplicity. However if alb provisioning times for everyone fluctuate wildly, then we might need a config variable with some a nice error message.

Anyone out there have an idea of how consistent alb provisioning times are?

May be moot topic if we fix the timeout, but reaching this error pretty much leaves everything in a half-deployed state that needs to be removed and redployed. Is there something more graceful we can do with the error?

Hope this info helps!

[john-parton]

I worked around the issue by increasing the Delay to 30

line 1609 in core.py

 waiter.wait(LoadBalancerArns=[load_balancer_arn], WaiterConfig={"Delay": 30})