question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

urlparse can not parse s3 url correctly

See original GitHub issue

After setting s3 bucket, an error occurs:

S3 datastore operation _put_s3_object failed (Parameter validation failed:
Invalid bucket name "": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:s3:[a-z\-0-9]+:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-]{1,63}$"). Retrying 7 more times..

This is due to in metaflow/datastore/s3.py urlparse can not figure out schema and netloc correctly.

A simple workaround is replacing the following content

try:
    # python2
    from urlparse import urlparse
    import cStringIO
    BytesIO = cStringIO.StringIO
except:
    # python3
    from urllib.parse import urlparse as offcial_urlparse
    import io
    BytesIO = io.BytesIO

with

try:
    # python2
    from urlparse import urlparse as offcial_urlparse
    import cStringIO
    BytesIO = cStringIO.StringIO
except:
    # python3
    import io
    from urllib.parse import urlparse as offcial_urlparse
    BytesIO = io.BytesIO
# modified by Kevin
# 07/12/2019
def urlparse(path):
    return offcial_urlparse(path if path.startswith('s3:') else 's3://' + path)

There should be a more elegant way to fix the issue

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5

github_iconTop GitHub Comments

1reaction
savingoyalcommented, Dec 11, 2019

It will work if you remove the s3:// prefix. We will make a small patch to handle this in the next release.

0reactions
xiaochuan-ducommented, Dec 11, 2019

It will work if you remove the s3:// prefix. We will make a small patch to handle this in the next release.

Thanks, mate. It helps

Read more comments on GitHub >

github_iconTop Results From Across the Web

urllib.parse — Parse URLs into components — Python 3.11.1 ...
URL Parsing. The URL parsing functions focus on splitting a URL string into its components, or on combining URL components into a URL...
Read more >
Parsing S3 URI into bucket and key in Go - Stack Overflow
An SDK function provides a clear authority on how to properly parse S3 addresses and deters worry over subtle cases that url. Parse()...
Read more >
AmazonS3URI (AWS SDK for Java - 1.12.369)
Creates a new AmazonS3URI by parsing the given string. String will be URL encoded before generating the URI. Parameters: str - the URI...
Read more >
urllib.parse.url parse Code Example - Code Grepper
urllib.parse.urlparse("http://example.com/pa/th;param1=foo ... “urllib.parse.url parse” Code Answer. how does urllib.parse.urlsplit work in python.
Read more >
Easily Parse URLs in JavaScript with parse-url - Stack Abuse
Parsing URLs is a common task to perform during web development, and also one that seems to be simple but can get complex....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found