scrapy shell error: Deferred.callback is not a method of DefaultSpider

I’m trying to just run a basic scrapy shell command, yet am somehow met with a weird error, see output below:

$ scrapy shell 'http://spu.taobao.com/spu/spulist.htm?cat=1801'
2015-04-06 10:35:41+0000 [scrapy] INFO: Scrapy 0.25.1 started (bot: taobao)
2015-04-06 10:35:41+0000 [scrapy] INFO: Optional features available: ssl, http11
2015-04-06 10:35:41+0000 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'taobao.spiders', 'SPIDER_MODULES': ['taobao.spiders'], 'LOGSTATS_INTERVAL': 0, 'SCHEDULER': 'scrapy_redis.scheduler.Scheduler', 'BOT_NAME': 'taobao'}
2015-04-06 10:35:41+0000 [scrapy] INFO: Enabled extensions: TelnetConsole, CloseSpider, CoreStats, SpiderState
2015-04-06 10:35:41+0000 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-04-06 10:35:41+0000 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-04-06 10:35:41+0000 [scrapy] INFO: Enabled item pipelines: TaobaoPipeline, RedisPipeline
2015-04-06 10:35:41+0000 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-04-06 10:35:41+0000 [default] INFO: Spider opened
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 9, in <module>
    load_entry_point('Scrapy==0.25.1', 'console_scripts', 'scrapy')()
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 143, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 89, in _run_print_help
    func(*a, **kw)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 150, in _run_command
    cmd.run(args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/shell.py", line 65, in run
    shell.start(url=url)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/shell.py", line 44, in start
    self.fetch(url, spider)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/shell.py", line 87, in fetch
    reactor, self._schedule, request, spider)
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "<string>", line 2, in raiseException
ValueError: Function <bound method Deferred.callback of <Deferred at 0x3765f38>> is not a method of: <DefaultSpider 'default' at 0x378aa90>
vagrant@precise64:/vagrant/taobao/scrapy-redis/example-project/taobao$ cat /usr/local/lib/python2.7/dist-packages/twisted/internet/threads.py

So I understand this is about Twisted’s deferred callbacks as used in Scrapy. As I understand it though, in Scrapy it basically hooks into _schedule(), which essentially just returns the spider itself. I’m having a lot of trouble finding any hits on google for similar errors though… might anyone have some insight here? My subclassed spider does have parse() defined.

[momosc]

同样的问题在这里。也可以。谢谢。 有什么暗示它为什么在一个沙哑的项目中失败的原因吗？

同样的问题

scrapy_redis 问题 导致shell不能用

[KiaraGrouwstra]

Thank you!

I followed your advice and retried in a generic spider directory. That worked fine, confirming the issue probably lies with scrapy-redis (which had not actually claimed compatibility with recent versions of scrapy in the first place – mine is 0.25.1).

I guess I was put off by the fact the error trace mentioned only scrapy code, but thank you for putting me back on track then. I’ll try to experiment a bit more to figure out what exactly is up then. As the issue appears not to lie with scrapy in any case, consider my issue resolved. 😃