[question] Why adding `dont_filter=True` for requests made from `start_urls`

It’s observed that currently (as of b364d27) in scrapy.Spider.start_requests the generated requests have dont_filter=True. (related line of code: link)

As I’ve had a quick look through the history, this behavior of adding dont_filter=True to initial requests seems to have been introduced in the very initial commit of 83dcf8a (related line of code: link).

At least from my personal perspective of view, there seems to be no reasonable purpose for this specific behavior.

Could anyone help explaining the design and resolving my confusion? Or, if there’s no such reason (or the previous reasons do not hold meaningful now), what about removing the dont_filter=True attribute for initial requests?

[kmike]

Just re-read my old comment 😃 The problem with this change is that if we remove dont_filter=True from start_requests and do nothing else, spiders like this would stop working:

import scrapy

class MySpider(scrapy.Spider):
    start_urls = ['http://scrapy.org']

    def parse(self, response):
        # ... parse is never called!

(haven’t re-checked it, but all the related issues are still open)

[MartiONE]

Also in favor of this, specially on imported URL’s from a file or a huge set of them with potential duplicates.