Make FeedExporter customizable for export conditions
See original GitHub issueSummary
The new FeedExporter class (or rather FEEDS
setting) doesn’t allow to create custom conditions on whether to export an item or not.
Motivation
Like in case with #4575, I need to customize the condition under which an item should/shouldn’t be exported. It would be nice to be able to do it like this or something in a similar fashion:
FEEDS = {
uri: {'format': 'jsonlines', 'slot_kwargs': {'param': }}
}
class CustomFeedExporter(FeedExporter):
def item_scraped(self, item, spider):
for slot in self.slots:
if item.param == slot.param:
slot.start_exporting()
slot.exporter.export_item(item)
slot.itemcount += 1
Describe alternatives you’ve considered
I have to create a new FeedExporter class and override open_spider
method to set custom attributes to a slot and then use it in item_scraped
the way I’ve shown above.
Issue Analytics
- State:
- Created 3 years ago
- Comments:9 (6 by maintainers)
Top Results From Across the Web
Exporting a Motor Vehicle | U.S. Customs and Border Protection
The certificate of title is the core requirement in the Customs export process, regardless of the vehicle's economic value, physical condition, ...
Read more >Feed exports — Scrapy 2.7.1 documentation
For serializing the scraped data, the feed exports use the Item exporters ... You can create your own custom filtering class by implementing ......
Read more >Export Solutions - International Trade Administration
Export Solutions provides practical advice and business tools to help U.S. companies expand in global markets.
Read more >Importing and Exporting | USAGov
Get an Import License or Permit · Check the requirements of federal agencies. · Contact the local port of entry you'll use to...
Read more >Package exports - webpack
Conditions might be nested to create a logical AND. ... field is preferred over other package entry fields like main , module ,...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I think this issue can be marked as done because an item filtering feature has been implemented here in the meantime: https://github.com/scrapy/scrapy/commit/4ddc9d6b55fa708becfd70812ea44eb0c6837638 This feature will help us a lot.
A pipeline works if you want to do this for all entries in
FEEDS
. But this enhancement would allow to filter differently for differentFEEDS
entries, which would be nice.