Promote a new CrawlSpider that allows overriding `parse`

I see a lot of StackOverflow questions and problems with CrawlSpider and overriden parse methods. (e.g. https://stackoverflow.com/questions/23511230)

I’d like to see a new implementation, called CrawlSpider2 or CrawlingSpider or something, that uses another internal method with another name than parse, so that user could define their own parse method.

Then, the question is if users will expect this parse method to be used by default for each downloaded page (is addition to being parsed for links with Rules), or if the reference to parse should be explicit.

Thoughts?

[nyov]

Closing #712 in favor of #712 …nice sleight of hand there. 😃

[nyov]

When you haven’t read the source and understand what’s going on, it’s easy to forget you should, or even need to call super(). Yes, the warning in the docs is also misleading, but someone reading it will know how to fix it (not using parse).

This issue is so common, it’s becoming a “useability bug” - not a bug in itself, but creating unnecessary headaches, not intuitive.

I propose having another internal _parse method called by the Scraper instead of parse, and using that in spiders who want internal “pre-processing”, then exposing parse as the public, documented, no-baggage method to implement/override from a spider.