question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Having an option to define an Items output file in the `scrapy parse` commnad line

See original GitHub issue

I use scrapy parse command to check the Items output for given URL, and I use it a lot.

Sometimes, I need to latter explore the Items extracted by scrapy parse, e.g . to check that something I want to fix was fixed for that specific URL without having to touch the source code.

Proposed solution:

Add a new parameter -o or --output in the parse command line. Allowing the client to set a desired output file for the items.

I could also | the result of scrapy parse, filter out want I want and create a json file out of it, but would be much easier if I could simple write scrapy parse {url} --callback=fixed_function -o items.json

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:1
  • Comments:11 (11 by maintainers)

github_iconTop GitHub Comments

1reaction
elacuestacommented, Feb 23, 2020

@AKSHAYSHARMAJS you should be able to install Scrapy in “development mode” by executing pip install -e .

0reactions
raphapassinicommented, Mar 30, 2020

Hey @adityaa30 this is a long waited one, thanks for working on this 💃

Read more comments on GitHub >

github_iconTop Results From Across the Web

Command line tool — Scrapy 2.7.1 documentation
The Scrapy tool provides several commands, for multiple purposes, and each one accepts a different set of arguments and options. (The scrapy deploy...
Read more >
Scrapy Tutorial — Scrapy 2.7.1 documentation
The parse() method usually parses the response, extracting the scraped data as dicts and also finding new URLs to follow and creating new ......
Read more >
Feed exports — Scrapy 2.7.1 documentation
To specify columns to export, their order and their column names, use FEED_EXPORT_FIELDS . Other feed exporters can also use this option, ...
Read more >
Requests and Responses — Scrapy 2.7.1 documentation
Scrapy uses Request and Response objects for crawling web sites. Typically, Request objects ... def parse(self, response): request = scrapy.
Read more >
Debugging Spiders — Scrapy 2.7.1 documentation
The most basic way of checking the output of your spider is to use the parse command. It allows to check the behaviour...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found