raise CloseSpider("Error Message") is not saving error message

When I run the code raise CloseSpider("Error Message") in the Spider parse function, the reason gets saved in the finish_reason stats string.

When it is run in the start_requests function, the spider properly closes but the finish_reason does not have the message it in. The finish_reason value is “finished” instead of “Error Message” as set in the CloseSpider exception.

Is this intentional or a bug? thanks!

[noviluni]

If someone needs a workaround to raise a CloseSpider with a specific message in the start_requests method before the fix is released, we did something like this:

def start_requests(self):
    try:
        # whatever you need to do 
    except:
        yield Request(url='file:///dev/null', callback=self.force_fail, errback=self.force_fail)
        return


def force_fail(self, response):
    raise CloseSpider(reason='failed')

Of course, you can change the try/except by a if/else or whatever you want and the “url” by any other URL. The “magic” here occurs when calling the callback, as the reason of the CloseSpider will be respected.

[elacuesta]

@Gallaecio I beg to differ, but I don’t think this is a bug, since the docs do say the exception “can be raised from a spider callback”. Any change to allow raising the exception from other components should be considered an enhancement IMHO.