question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How do I save scraped items to multiple .jl files?

See original GitHub issue

I can’t seem to find this in the docs, so here goes…

I want to run a scraper that saves different types of items to separate JSON Lines files.

My settings.py has this:

ITEM_PIPELINES = {
   'permits.pipelines.PermitTypePipeline': 300,
   'permits.pipelines.PermitNumberPipeline': 301,
}

My pipelines.py has this:

class PermitTypePipeline(object):
	def process_item(self, item, spider):
		return item

class PermitNumberPipeline(object):
	def process_item(self, item, spider):
		return item

items.py has this:

class PermitType(scrapy.Item):
	permitWebCode=	scrapy.Field()

class PermitNumber(scrapy.Item):
	permitNumber=	scrapy.Field()

my_spider.py has this:

def parse(self,response):
  ## Some scraper code here ... 
  yield PermitType(permitWebCode=someScrapedVariable)
  yield PermitNumber(permitNumber=anotherScrapedVariable)

How do I save the yielded PermitType object to permit_types.jl and the PermitNumber object to permit_number.jl?

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:6 (5 by maintainers)

github_iconTop GitHub Comments

2reactions
VMRuizcommented, Apr 6, 2018

I solved this by using an custom feed exporter

It could be improved because you still need to manualy define the list of item that you want to export but it will work much better than creating a pipeline, because you will be able to use it with any of the already implemented exporters like csv, json, jl…

Also, with this solution you need to include %(item)s in the FEED_URI so it generates a different file name based on each item name.

0reactions
Gallaeciocommented, Nov 7, 2022

I believe FEEDS and https://github.com/scrapy/scrapy/pull/5178 addressed this.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How do I save scraped items to multiple .jl files? - Stack Overflow
1 Answer 1 ; import ClassB class ; object): def ; self, spider): self.file.close() def ; self, item, spider): if ; is ClassA:...
Read more >
How do I save scraped items to multiple .jl files? : r/scrapy
I want to run a scraper that saves different types of items to separate JSON Lines files. My settings.py has this:
Read more >
Saving scraped items to JSON and CSV file using Scrapy
We will scrape data from a webpage, using a Scrapy spider, and export the same to two different file formats.
Read more >
Feed exports — Scrapy 2.7.1 documentation
For serializing the scraped data, the feed exports use the Item ... the output items in multiple files, with the specified maximum item...
Read more >
Scraping web pages with Julia HTTP & Gumbo: Tutorial
There is another Julia package named Cascadia.jl which allows you to scrape HTML elements by CSS class or id, but that's out of...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found