Raising CloseSpider from DownloaderMiddleware doesn't work

Behavior: Raising a CloseSpider exception from a DownloaderMiddleware’s process_response doesn’t close the spider. Instead, the scraper only outputs CloseSpider to stdout.

Expected behavior: Spider should shut down.

Workarounds:

• Use crawler.stop:

self.crawler.stop()
return None

which requires to make self.crawler available in the DownloaderMiddleware via:

    def __init__(self, crawler):
        self.crawler = crawler

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

• Use crawler._signal_shutdown() (doesn’t work for me)
• Return None from process_response (doesn’t work for me)

[kmike]

As documented, CloseSpider only works in spider callbacks, so this is not a bug:

This exception can be raised from a spider callback to request the spider to be closed/stopped.

But supporting CloseSpider in middlewares and extensions sounds reasonable.

Built-in scrapy.extensions.closespider.CloseSpider extension uses self.crawler.engine.close_spider(spider, 'reason') to stop crawling on some condition. But this API is poorly documented (if documented at all).

[kmike]

Hey @raphapassini! Currently the only place CloseSpider is caught is https://github.com/scrapy/scrapy/blob/e45ef7dcd987f00b94e94d71593f6b3664ceb89f/scrapy/core/scraper.py#L151

It seems it needs to be caught in middleware managers as well: spider middleware manager, downloader middleware manager, ExtensionManager - maybe even in the base MiddlewareManager class, I’m not sure.

If we implement this feature, we’d need to define how this works together with process_exception methods, i.e. do these methods get a chance to catch CloseSpider exception or not (I think it’d be nice to allow that, though I haven’t looked in detail).