question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Make FeedExporter customizable for export conditions

See original GitHub issue

Summary

The new FeedExporter class (or rather FEEDS setting) doesn’t allow to create custom conditions on whether to export an item or not.

Motivation

Like in case with #4575, I need to customize the condition under which an item should/shouldn’t be exported. It would be nice to be able to do it like this or something in a similar fashion:

FEEDS = {
    uri: {'format': 'jsonlines', 'slot_kwargs': {'param': }}
}

class CustomFeedExporter(FeedExporter):

    def item_scraped(self, item, spider):
        for slot in self.slots:
            if item.param == slot.param:
                slot.start_exporting()
                slot.exporter.export_item(item)
                slot.itemcount += 1

Describe alternatives you’ve considered

I have to create a new FeedExporter class and override open_spider method to set custom attributes to a slot and then use it in item_scraped the way I’ve shown above.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:9 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
wohocommented, Mar 29, 2022

I think this issue can be marked as done because an item filtering feature has been implemented here in the meantime: https://github.com/scrapy/scrapy/commit/4ddc9d6b55fa708becfd70812ea44eb0c6837638 This feature will help us a lot.

1reaction
Gallaeciocommented, Jan 26, 2021

A pipeline works if you want to do this for all entries in FEEDS. But this enhancement would allow to filter differently for different FEEDS entries, which would be nice.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Exporting a Motor Vehicle | U.S. Customs and Border Protection
The certificate of title is the core requirement in the Customs export process, regardless of the vehicle's economic value, physical condition, ...
Read more >
Feed exports — Scrapy 2.7.1 documentation
For serializing the scraped data, the feed exports use the Item exporters ... You can create your own custom filtering class by implementing ......
Read more >
Export Solutions - International Trade Administration
Export Solutions provides practical advice and business tools to help U.S. companies expand in global markets.
Read more >
Importing and Exporting | USAGov
Get an Import License or Permit · Check the requirements of federal agencies. · Contact the local port of entry you'll use to...
Read more >
Package exports - webpack
Conditions might be nested to create a logical AND. ... field is preferred over other package entry fields like main , module ,...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found