question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

URLencoding breakes S3 key

See original GitHub issue

When uploading a file to S3 like this:

>>> from s3path import S3Path
>>> s=S3Path('/my-bucket/tmp/foo=bar/objname')
>>> s.write_text("foobar")
/path/to/my/venv/lib64/python3.8/site-packages/smart_open/s3.py:220: UserWarning: ignoring the following deprecated transport parameters: ['multipart_upload_kwargs', 'object_kwargs', 'resource_kwargs', 'session']. See <https://github.com/RaRe-Technologies/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.rst> for details
  warnings.warn(message, UserWarning)
6

The location is transformed and the = is urlencoded by a as_uri() call. In S3 I now have a tmp/foo%3Dbar/ prefix. But I expected an tmp/foo=bar/ prefix.

The as_uri() call in s3path.py leads to unexpected behaviour and breakes the behaviour of s3path <=v0.3

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:2
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
JohnHBrockcommented, Jun 29, 2021

I think this is OK actually, sorry for the false alarm. We had a workaround in place for the URL encoding issue, and that workaround is subtly broken by #78, but that’s fine because #78 lets us remove the workaround. 🎉

1reaction
liormizrcommented, Jun 28, 2021

Deployed Version: 0.3.02

Read more comments on GitHub >

github_iconTop Results From Across the Web

Amazon s3 URL + being encoded to %2? - Stack Overflow
If I understand your question correctly, then no, there is no way to really change this. The cause appears to be an unfortunate...
Read more >
File not found if S3 key contains special characters · Issue #279
Solution: The S3 key need to be URL encoded before download. ... However, it breaks another test cases using wildcard.
Read more >
Handling URL-Encoded S3 File Names - Medium
I was naming the file properly, so why was this happening? Turns out that our wrapper around the AWS SDK automatically URL-encodes the...
Read more >
Url-Encoding Amazon S3 Resource Keys For Pre-Signed Urls ...
Ben Nadel looks at which characters need to be encoded when generating pre-signed URLs for Amazon S3 objects; and, how to use ColdFusion's ......
Read more >
[#HADOOP-3733] "s3:" URLs break when Secret Key contains ...
"s3:" URLs break when Secret Key contains a slash, even if encoded. Status: Assignee: Priority: Resolution: Resolved. Steve Loughran.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found