Can't open S3 files with ?-like characters in name

obj = smart_open.open(''s3://bucket-name/folder/picture #1.jpg', 'rb')

results to the s3 key 'folder/picture ' does not exist, or is forbidden for access.

At the same time

s3_session.resource('s3').Bucket('bucket-name').download_file("folder/picture #1.jpg", "/tmp/test.jpg")

downloads the file successfully.

Some investigation

Internally, smart_open takes unquoted value (folder/picture #1.jpg) and after parsing, it thinks that the path is older/picture cutting the rest of the key.

I can try to feed it the quoted value (folder/picture%20%231.jpg) - in that case it will parse it correctly and will think that the whole thing is the key, but after that AWS won’t understand this, as it’s client (boto3?) requires values ‘as is’.

Same problem with having “?” in the file name. I didn’t check other characters.

[hexvolt]

Hm, yes you’re right, just debugged this example again and # appeared to work ok. Maybe I mixed it up with urlparse in requests when I was trying to find a culprit and experimented with both requests and smart_open. So the issue is with question mark only indeed. Thanks for addressing this.

[kn000x]

Could you please check: Temporarily replacing ? with \n in safe_urlsplit is conflicting with the handling of _UNSAFE_URL_BYTES_TO_REMOVE within newer urlsplit versions:

https://github.com/python/cpython/commit/8a595744e696a0fb92dccc5d4e45da41571270a1#diff-b3712475a413ec972134c0260c8f1eb1deefb66184f740ef00c37b4487ef873e