scrapy shell error: Deferred.callback is not a method of DefaultSpider
See original GitHub issueI’m trying to just run a basic scrapy shell command, yet am somehow met with a weird error, see output below:
$ scrapy shell 'http://spu.taobao.com/spu/spulist.htm?cat=1801'
2015-04-06 10:35:41+0000 [scrapy] INFO: Scrapy 0.25.1 started (bot: taobao)
2015-04-06 10:35:41+0000 [scrapy] INFO: Optional features available: ssl, http11
2015-04-06 10:35:41+0000 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'taobao.spiders', 'SPIDER_MODULES': ['taobao.spiders'], 'LOGSTATS_INTERVAL': 0, 'SCHEDULER': 'scrapy_redis.scheduler.Scheduler', 'BOT_NAME': 'taobao'}
2015-04-06 10:35:41+0000 [scrapy] INFO: Enabled extensions: TelnetConsole, CloseSpider, CoreStats, SpiderState
2015-04-06 10:35:41+0000 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-04-06 10:35:41+0000 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-04-06 10:35:41+0000 [scrapy] INFO: Enabled item pipelines: TaobaoPipeline, RedisPipeline
2015-04-06 10:35:41+0000 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-04-06 10:35:41+0000 [default] INFO: Spider opened
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 9, in <module>
load_entry_point('Scrapy==0.25.1', 'console_scripts', 'scrapy')()
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 89, in _run_print_help
func(*a, **kw)
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 150, in _run_command
cmd.run(args, opts)
File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/shell.py", line 65, in run
shell.start(url=url)
File "/usr/local/lib/python2.7/dist-packages/scrapy/shell.py", line 44, in start
self.fetch(url, spider)
File "/usr/local/lib/python2.7/dist-packages/scrapy/shell.py", line 87, in fetch
reactor, self._schedule, request, spider)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
result.raiseException()
File "<string>", line 2, in raiseException
ValueError: Function <bound method Deferred.callback of <Deferred at 0x3765f38>> is not a method of: <DefaultSpider 'default' at 0x378aa90>
vagrant@precise64:/vagrant/taobao/scrapy-redis/example-project/taobao$ cat /usr/local/lib/python2.7/dist-packages/twisted/internet/threads.py
So I understand this is about Twisted’s deferred callbacks as used in Scrapy. As I understand it though, in Scrapy it basically hooks into _schedule(), which essentially just returns the spider itself. I’m having a lot of trouble finding any hits on google for similar errors though… might anyone have some insight here? My subclassed spider does have parse()
defined.
Issue Analytics
- State:
- Created 8 years ago
- Comments:7 (1 by maintainers)
Top Results From Across the Web
Scrapy Contracts - Unhandled error in Deferred - Stack Overflow
I expect poor exception reporting here is a scrapy bug. Contracts are still considered a new feature and are also pretty limited.
Read more >Source code for scrapy.utils.defer
Helper functions for dealing with Twisted deferreds """ import asyncio import inspect from asyncio import Future from functools import wraps ...
Read more >Release notes — Scrapy 2.7.1 documentation
The parse command now supports asynchronous callbacks (issue 5424, issue 5577). When using the parse command with a URL for which there is...
Read more >Scrapy shell — Scrapy 2.7.1 documentation
The Scrapy shell is an interactive shell where you can try and debug your scraping code very quickly, without having to run the...
Read more >master PDF - Scrapy Documentation
This happens because parse() is Scrapy's default callback method, ... want it to be resilient to errors due to things not being found...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
scrapy_redis 问题 导致shell不能用
Thank you!
I followed your advice and retried in a generic spider directory. That worked fine, confirming the issue probably lies with scrapy-redis (which had not actually claimed compatibility with recent versions of scrapy in the first place – mine is 0.25.1).
I guess I was put off by the fact the error trace mentioned only scrapy code, but thank you for putting me back on track then. I’ll try to experiment a bit more to figure out what exactly is up then. As the issue appears not to lie with scrapy in any case, consider my issue resolved. 😃