urlparse can not parse s3 url correctly
See original GitHub issueAfter setting s3 bucket, an error occurs:
S3 datastore operation _put_s3_object failed (Parameter validation failed:
Invalid bucket name "": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:s3:[a-z\-0-9]+:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-]{1,63}$"). Retrying 7 more times..
This is due to in metaflow/datastore/s3.py
urlparse
can not figure out schema and netloc
correctly.
A simple workaround is replacing the following content
try:
# python2
from urlparse import urlparse
import cStringIO
BytesIO = cStringIO.StringIO
except:
# python3
from urllib.parse import urlparse as offcial_urlparse
import io
BytesIO = io.BytesIO
with
try:
# python2
from urlparse import urlparse as offcial_urlparse
import cStringIO
BytesIO = cStringIO.StringIO
except:
# python3
import io
from urllib.parse import urlparse as offcial_urlparse
BytesIO = io.BytesIO
# modified by Kevin
# 07/12/2019
def urlparse(path):
return offcial_urlparse(path if path.startswith('s3:') else 's3://' + path)
There should be a more elegant way to fix the issue
Issue Analytics
- State:
- Created 4 years ago
- Comments:5
Top Results From Across the Web
urllib.parse — Parse URLs into components — Python 3.11.1 ...
URL Parsing. The URL parsing functions focus on splitting a URL string into its components, or on combining URL components into a URL...
Read more >Parsing S3 URI into bucket and key in Go - Stack Overflow
An SDK function provides a clear authority on how to properly parse S3 addresses and deters worry over subtle cases that url. Parse()...
Read more >AmazonS3URI (AWS SDK for Java - 1.12.369)
Creates a new AmazonS3URI by parsing the given string. String will be URL encoded before generating the URI. Parameters: str - the URI...
Read more >urllib.parse.url parse Code Example - Code Grepper
urllib.parse.urlparse("http://example.com/pa/th;param1=foo ... “urllib.parse.url parse” Code Answer. how does urllib.parse.urlsplit work in python.
Read more >Easily Parse URLs in JavaScript with parse-url - Stack Abuse
Parsing URLs is a common task to perform during web development, and also one that seems to be simple but can get complex....
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
It will work if you remove the
s3://
prefix. We will make a small patch to handle this in the next release.Thanks, mate. It helps