Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Scrapyd NOT Enable Item Pipelines
See original GitHub issueHello, I’m having problem using Scrapyd service. I use MongoDB Pipeline to write scraped items to database. It works fine when I directly run spider on command line :
scrapy crawl digi
2018-01-22 13:25:18 [scrapy.middleware] INFO: Enabled item pipelines: ['tutorial.pipelines.MongoPipeline']
When I send curl request to scrapyd server, the spider still runs. However, the item pipelines are not enabled:
2018-01-22 13:29:32 [scrapy.middleware] INFO: Enabled item pipelines: []
Could you suggest any solutions for this?
Issue Analytics
- State:
- Created 6 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Make sure the item pipeline is activated as described here: https://docs.scrapy.org/en/latest/topics/item-pipeline.html#activating-an-item-pipeline-component
Make sure the file containing that setting is discoverable using entry_points like at https://github.com/scrapy/scrapyd-client#including-static-files
hi , i have the same problem :
scrapy crawl digi2018-01-22 13:25:18 [scrapy.middleware] INFO: Enabled item pipelines: ['tutorial.pipelines.MongoPipeline']When I send curl request to scrapyd server, the spider still runs. However, the item pipelines are not enabled: 2018-01-22 13:29:32 [scrapy.middleware] INFO:Enabled item pipelines: []how did you solve it ?