question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

AWS CLI calls broken during change from 2.169.1 to 2.262.1

See original GitHub issue

Describe the bug To preface, I know this seems like an AWS bug, but the issue has only occurred when GitHub Actions upgraded its runner version, while the AWS CLI version has stayed the same. The only thing that I have been able to find different between passing and failing runs was the runner version.

Calls to the AWS CLI (namely s3 sync, installed in a GitHub Action in ubuntu-latest (currently 18.04) previously worked with the runner version 2.169.1, but fail with the error code 255 beginning in version 2.262.1.

When reproducing the steps one by one in a EC2 instance, the process works as it did previously in GitHub Actions without any errors.

Please see the following actions for examples: https://github.com/NateSeymour/studymandar.in/blob/master/.github/workflows/deploy.yml https://github.com/Simonwep/intl-demo/blob/master/.github/workflows/deploy.yml

To Reproduce Steps to reproduce the behavior:

  1. Use a call to the AWS CLI inside of a GitHub Action
  2. Error will appear in the Action with a status of 255

Expected behavior The calls to the AWS CLI should perform as normally, just like they do in a EC2 VM running the same OS.

Runner Version and Platform

Version of your runner? 2.262.1

OS of the machine running the runner? GitHub Actions

What’s not working?

The AWS s3 sync command fails with:

<botocore.awsrequest.AWSRequest object at 0x7f34780fd390>
134
##[error]Process completed with exit code 255.

Job Log Output

Current runner version: '2.262.1'
Operating System
  Ubuntu
  18.04.4
  LTS
Virtual Environment
  Environment: ubuntu-18.04
  Version: 20200518.1
  Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20200518.1/images/linux/Ubuntu1804-README.md
Prepare workflow directory
Prepare all required actions
Download action repository 'actions/checkout@v2'
Download action repository 'actions/setup-node@v1'

Runner and Worker’s Diagnostic Logs

More relevant information can be found here:

Passing Runs: https://github.com/Simonwep/intl-demo/runs/531009403?check_suite_focus=true

Failing runs: https://github.com/NateSeymour/studymandar.in/runs/703639126?check_suite_focus=true https://github.com/Simonwep/intl-demo/runs/703653450?

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:2
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
jonstackscommented, Mar 24, 2021

I ran the aws cli command with --debug on our failing workflow and was able to get the following output:

2021-03-24 09:11:07,948 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-03-24 09:11:07,950 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2021-03-24 09:11:07,954 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 400 11
2021-03-24 09:11:07,955 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 457, in main
  File "awscli/customizations/commands.py", line 197, in __call__
  File "awscli/customizations/commands.py", line 191, in __call__
  File "awscli/customizations/s3/subcommands.py", line 709, in _run_main
  File "awscli/customizations/s3/subcommands.py", line 478, in _run_main
  File "awscli/customizations/s3/subcommands.py", line 471, in get_client
  File "botocore/session.py", line 783, in create_client
  File "botocore/session.py", line 832, in _resolve_region_name
  File "botocore/session.py", line 236, in get_config_variable
  File "botocore/configprovider.py", line 301, in get_config_variable
  File "botocore/configprovider.py", line 398, in provide
  File "awscli/utils.py", line 60, in provide
  File "awscli/utils.py", line 65, in _get_instance_metadata_region
  File "awscli/utils.py", line 109, in retrieve_region
  File "awscli/utils.py", line 118, in _get_region
  File "botocore/utils.py", line 307, in _fetch_metadata_token
botocore.utils.BadIMDSRequestError: <botocore.awsrequest.AWSRequest object at 0x7f524d027220>

<botocore.awsrequest.AWSRequest object at 0x7f524d027220>

Running on ubuntu-latest(20.04.2 LTS) with runner version 2.277.1, we were able to get aws cli s3 uploads to work by setting:

env:
  AWS_EC2_METADATA_DISABLED: "true"

as suggested here https://github.com/aws/aws-cli/issues/5262#issuecomment-705832151

Hope that helps

0reactions
TingluoHuangcommented, Apr 22, 2021

Closing this as it’s not a runner issue and seems to work now.

Read more comments on GitHub >

github_iconTop Results From Across the Web

New features and changes in AWS CLI version 2
Learn about new features and changes in behavior between AWS CLI version 1 and AWS CLI ... Breaking changes between AWS CLI version...
Read more >
Migrate from AWS CLI version 1 to version 2
This section contains instructions for updating the AWS CLI version 1 to AWS CLI version 2. You can also learn about the differences...
Read more >
Install or upgrade and then configure the AWS CLI
To check the version, use the aws --version command. To install and configure the AWS CLI. Install or upgrade the AWS CLI with...
Read more >
modify-custom-db-engine-version
Modifies the status of a custom engine version (CEV). You can find CEVs to modify by calling DescribeDBEngineVersions . Note. The MediaImport service...
Read more >
update-action-target — AWS CLI 1.28.0 Command ...
A JMESPath query to use in filtering the response data. --profile (string). Use a specific profile from your credential file. --region (string).
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found