Having an option to define an Items output file in the `scrapy parse` commnad line
See original GitHub issueI use scrapy parse
command to check the Items output for given URL, and I use it a lot.
Sometimes, I need to latter explore the Items extracted by scrapy parse
, e.g . to check that something I want to fix was fixed for that specific URL without having to touch the source code.
Proposed solution:
Add a new parameter -o
or --output
in the parse command line. Allowing the client to set a desired output file for the items.
I could also |
the result of scrapy parse
, filter out want I want and create a json
file out of it, but would be much easier if I could simple write scrapy parse {url} --callback=fixed_function -o items.json
Issue Analytics
- State:
- Created 4 years ago
- Reactions:1
- Comments:11 (11 by maintainers)
Top Results From Across the Web
Command line tool — Scrapy 2.7.1 documentation
The Scrapy tool provides several commands, for multiple purposes, and each one accepts a different set of arguments and options. (The scrapy deploy...
Read more >Scrapy Tutorial — Scrapy 2.7.1 documentation
The parse() method usually parses the response, extracting the scraped data as dicts and also finding new URLs to follow and creating new ......
Read more >Feed exports — Scrapy 2.7.1 documentation
To specify columns to export, their order and their column names, use FEED_EXPORT_FIELDS . Other feed exporters can also use this option, ...
Read more >Requests and Responses — Scrapy 2.7.1 documentation
Scrapy uses Request and Response objects for crawling web sites. Typically, Request objects ... def parse(self, response): request = scrapy.
Read more >Debugging Spiders — Scrapy 2.7.1 documentation
The most basic way of checking the output of your spider is to use the parse command. It allows to check the behaviour...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@AKSHAYSHARMAJS you should be able to install Scrapy in “development mode” by executing
pip install -e .
Hey @adityaa30 this is a long waited one, thanks for working on this 💃