allowed_domains bug/undesired behaviour

Assume crawler have set allowed_domains to below list: self.allowed_domains = ['albert.zgora.pl'] Scrapy shouldn’t go beyond ‘albert.zgora.pl’ domain.

But it goes to: https://www.tumblr.com/widgets/share/tool/preview?shareSource=legacy&canonicalUrl=&url=http%3A%2F%2Falbert.zgora.pl%2F2014%2F08%2Fbierzmowanie%2F&title=Bierzmowanie

This is just one real life example, but there are many more (and I can give them here if you want) where domain string appears somewhere in url e.g. in &url= parameter.

[kmike]

see also: https://github.com/scrapy/scrapy/issues/2241

[oprypin]

I think this problem usually happens due to RedirectMiddleware.

Full repro:

import scrapy.crawler
import scrapy.spiders

class MySpider(scrapy.spiders.Spider):
  name = 'review.webrtc.org'
  allowed_domains = ['review.webrtc.org']
  start_urls = ['https://review.webrtc.org/']

  def parse(self, response):
    yield scrapy.http.Request('https://review.webrtc.org/17839004/')

process = scrapy.crawler.CrawlerProcess()
process.crawl(MySpider)
process.start()

Actual:

[scrapy.core.engine] DEBUG: Crawled (200) <GET https://accounts.google.com/ServiceLogin?service=ah&passive=true&continue=https://appengine.google.com/_ah/conflogin%3Fcontinue%3Dhttps://review.webrtc.org/17839004/> (referer: https://review.webrtc.org/)

Expected:

[scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'accounts.google.com': <GET https://accounts.google.com/ServiceLogin?service=ah&passive=true&continue=https://appengine.google.com/_ah/conflogin%3Fcontinue%3Dhttps://review.webrtc.org/17839004/>

I thought, like in the first post, that it’s due to

domain string appears somewhere in url e.g. in &url= parameter.

(because that also happens to be true in my case) but it’s just a coincidence. I’m quite sure that the logic is sound.

The real issue probably is that the check whether the download is allowed happens without any involvement of RedirectMiddleware but the actual download is of the redirected URL.