Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

botocore 1.28.0 and higher fail to generate valid S3 presigned URLs

See original GitHub issue

Describe the bug

Since upgrading the botocore 1.28.0 presign requests to S3 generate links like this:

https://s3.amazonaws.com/tiendeo.data.temporary.lab/DataReporting/Process.CommercialReport.CommercialReport/4D370376-2F63-48DF-B150-DA51507BC73F/C35C5CEA-2D51-42ED-A857-F3F86FE6178E.png?AWSAccessKeyId=AKIAJY4KR3BOOATC5GEA&Signature=AFIjb6Idelu5UdQwRac4as6BHNA%3D&Expires=1666883364

instead of this:

https://s3.eu-west-1.amazonaws.com/tiendeo.data.temporary.lab/DataReporting/Process.CommercialReport.CommercialReport/4D370376-2F63-48DF-B150-DA51507BC73F/C35C5CEA-2D51-42ED-A857-F3F86FE6178E.png?AWSAccessKeyId=AKIAJY4KR3BOOATC5GEA&Signature=AFIjb6Idelu5UdQwRac4as6BHNA%3D&Expires=1666883364

As they do not include the region the link fails with the following response:

<Error>
<Code>PermanentRedirect</Code>
<Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
<Endpoint>tiendeo.data.temporary.lab.s3.amazonaws.com</Endpoint>
<Bucket>tiendeo.data.temporary.lab</Bucket>
<RequestId>WKMQ1WVE9XTWNXR2</RequestId>
<HostId>FIefmbEnBZZd4Mrt5SP0IDVC5r6eTPRRzIkFwSSAF+xujig0IVj33FIRQOoRkt8eVCPb24OP7+M=</HostId>
</Error>

Expected Behavior

The presinged link should be valid.

Current Behavior

The presinged link are invalid.

Reproduction Steps

$ pip install -U botocore awscli
$ aws s3 presign s3://bucket/key.png
# invalid link
$ pip install -U botocore==1.27.96
$ aws s3 presign s3://bucket/key.png
# valid link

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.28.2

Environment details (OS name and version, etc.)

Tested on macOS, Linux and Windows

Issue Analytics

State:
Created a year ago
Reactions:1
Comments:16 (6 by maintainers)

Top GitHub Comments

4reactions

nateprewittcommented, Oct 27, 2022

Hi @pvieito, @benedikt-bartscher, thanks for reporting this. We’ve identified the regression and it appears to be localized specifically to bucket names that aren’t valid host labels as defined in RFC 1123. The issue with the first reported bucket is specifically the use of . in the bucket name. These names require different formatting for the presigned URL which is no longer occurring.

We have a fix in the works that we’re currently validating and intend to release as soon as possible. We’ll provide an update here once that’s finished. As a short term mitigation, you may need to continue using 1.24.96 until this patch is available. Thanks for your patience!

1reaction

slimm609commented, Oct 28, 2022

Can 1.28.4 be released to resolve this? Also hitting this issue.