Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

urllib3 does not urlencode redirect targets on redirect.

See original GitHub issue

I am crawling pages and I’m getting UnicodeEncodeError.

The problematic url is : http://www.efind.co.il/detailed/52133.html I’ve isolated my code to this: url = "http://www.efind.co.il/detailed/52133.html" manager = PoolManager() response = manager.request('GET', url)

This url is redirecting to this url: articles.efind.co.il/info/דגי-נוי-בבריכה-מאמר which is not ‘ascii’.

Because this is auto redirect i can’t do anything about it. I am quoting the first url and giving it to the urllib, I don’t have control over the redirected urls.

Anything i can do?

Issue Analytics

State:
Created 7 years ago
Comments:9 (7 by maintainers)

Top GitHub Comments

1reaction

Lukasacommented, Aug 19, 2016

So this looks potentially like it’s a bit of a urllib3 bug. It’s passing the redirect URL straight back to httplib, but it needs to be urlencoded first, which is apparently not something urllib3 has ever done.

So yes, this is a bug.

0reactions

Lukasacommented, Aug 19, 2016

I am totally up for urllib3 using rfc3986.

Top Results From Across the Web

urllib2: submitting a form and then redirecting - Stack Overflow

So, I POST something to an URL, the server sets a cookie in the response and doesn't redirect. Now, I set the very...

urllib3 Documentation

By default, urllib3 will retry requests 3 times and follow up to. 3 redirects. To change the number of retries just specify an...

#2320 (Inconsistent URL decoding with X-Accel-Redirect ...

This is a problem of course, because it means I cannot reliable know whether I need to URL-encode the path included in my...

urlencode - Manual - PHP

This function is convenient when encoding a string to be used in a query part of a URL, as a convenient way to...

urllib.parse — Parse URLs into components — Python 3.11.1 ...

The components are not broken up into smaller parts (for example, ... Use the urllib.parse.urlencode() function (with the doseq parameter set to True...