Scrapy should handle "invalid" relative URLs better

Currently Scrapy can’t extract links from http://scrapy.org/ page correctly because urls in page header are relative to a non-existing parent: ../download/, ../doc/, etc. Browsers resolve these links as http://scrapy.org/download/ and http://scrapy.org/doc/, while response.urljoin, urlparse.urljoin and our ink extractors resolve them as http://scrapy.org/../download/, etc. This results in 400 Bad Request responses.

urlparse.urljoin is not correct (or not modern) here. In the URL Living Standard for browsers it is said:

If buffer is “…”, remove url’s path’s last entry, if any, and then if c is neither “/” nor "", append the empty string to url’s path.

[kmike]

The most evil spider ever: looks innocent, but doesn’t work for multiple reasons

import scrapy

class ScrapySpider(scrapy.Spider):
    name = 'scrapyspider'

    def start_requests(self):
        yield scrapy.Request("http://scrapy.org", self.parse_main)

    def parse_main(self, response):
        for href in response.xpath("//a/@href").extract():
            yield scrapy.Request(response.urljoin(href), self.parse_link)

    def parse_link(self, response):
        print(response.url)

[nirvana-msu]

I am also having issues due to urljoin behaving differently to modern browsers.

For example, all modern browsers tolerate extra slashes after http (see e.g. this discussion in curl).

Using urljoin on most recent Python 3.8 I get:

> urljoin('http://base.com', 'http:////other.com')
http://base.com//other.com

whereas browsers would load http://other.com. Would be great if it was possible to replicate the URL-handling of browsers.

UPDATE:

I’ve just tried scurl and, as you would expect, it handles this case correctly (i.e. same as Chrome). Would be great if Scrapy could adopt it, or any other library that wraps actual browser implementation.