dupefilter skips a request when a page is redirected to itself

Hit this when trying to run a spider against scrapinghub.com: sometimes it responds with 302 moved permanently to scrapinghub.com. Scheduler agrees and tries to schedule another request for scrapinghub.com, but fails because dupefilter already considers it visited.

Maybe dupefilter should only add hosts when the response is not a redirect? And when it is, the scheduler should probably remember the original address too, so that all the redirection chain can be marked as visited.

[victor-torres]

I’ve faced this problem today while writing a spider like this:

• spider has a list of credentials
• submit a FormRequest to website.com/login for each credential with dont_filter=True
• requests are redirected from website.com/login to website.com/profile
• only one profile is fetched because of the default dupefilter

Workaround:

Since this Spider was very simple, I’ve just disabled my dupefilter with:

class WebsiteSpider(scrapy.Spider):

    ...

    custom_settings = {
        "DUPEFILTER_CLASS": "scrapy.dupefilters.BaseDupeFilter",
    }

    ....

But it would be very interesting to have a way to disable this behavior like what’s being proposed on #4314 or something like dont_filter_redirects=True.

[kingname]

Surely I can and I have done. But I think it is a bug and you should solve it. Why don’t you solve it for almost 3 years?

I know in some situations, the anti-spider system will always redirect the spider to one page to protect their data, so this duplicate filter is reasonable. But are there some smarter method to solve my problem?