Quoted url Location header fails to redirect

Hello,

I’m trying to crawl a website that returns a Location url in the header that is quoted. When that happens the downloadermiddleware_redirect fails to build the correct url.

This little test helps to explain:

    def test_quoted_location(self):
        req = Request('http://scrapytest.org/first')
        utf8_location = u'http%3A//scrapytest.org/ação'.encode('utf-8')  # header using quoted UTF-8 encoding
        resp = Response('http://scrapytest.org/first', headers={'Location': utf8_location}, status=302)
        req_result = self.mw.process_response(req, resp, self.spider)
        perc_encoded_utf8_url = 'http://scrapytest.org/a%C3%A7%C3%A3o'
        self.assertEquals(perc_encoded_utf8_url, req_result.url)

it fails:

===================================================================== FAILURES =====================================================================
___________________________________________________ RedirectMiddlewareTest.test_quoted_location ____________________________________________________

self = <tests.test_downloadermiddleware_redirect.RedirectMiddlewareTest testMethod=test_quoted_location>

    def test_quoted_location(self):
        req = Request('http://scrapytest.org/first')
        utf8_location = u'http%3A//scrapytest.org/ação'.encode('utf-8')  # header using UTF-8 encoding
        resp = Response('http://scrapytest.org/first', headers={'Location': utf8_location}, status=302)
        req_result = self.mw.process_response(req, resp, self.spider)
        perc_encoded_utf8_url = 'http://scrapytest.org/a%C3%A7%C3%A3o'
>       self.assertEquals(perc_encoded_utf8_url, req_result.url)
E       AssertionError: 'http://scrapytest.org/a%C3%A7%C3%A3o' != 'http://scrapytest.org/http%3A//scrapytest.org/a%C3%A7%C3%A3o'

/home/aurumdev/repos/scrapy/tests/test_downloadermiddleware_redirect.py:177: AssertionError
======================================================= 1 failed, 19 passed in 0.29 seconds ========================================================

The quoted url comes from the website.

Is that a bug?

Thank you

[redapple]

@Tarliton , I’m closing this issue as you found a workaround for the server sending invalid referer values.

[Tarliton]

sorry, I was wrong at the first post. I used:

urllib.quote('http://scrapytest.org/ação')
'http%3A//scrapytest.org/a%C3%A7%C3%A3o'

instead of:

urllib.urlencode({'url':'http://scrapytest.org/ação'})
'url=http%3A%2F%2Fscrapytest.org%2Fa%C3%A7%C3%A3o'

this:

http%3A%2F%2Fscrapytest.org%2Fa%C3%A7%C3%A3o

is different than this:

http%253A%2F%2Fscrapytest.org%2Fa%25C3%25A7%25C3%25A3o

although, I still can’t explain what’s going on on that website.