question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

CSV item exporter does not flush on close

See original GitHub issue

Description

The csv item exporter does not flush on close.

Steps to Reproduce

  1. Create pipeline with an CSV item exporter
  2. only export few items (one or two)
  3. Let spider close/finish

Expected behavior: All exported items are in the CSV Files

Actual behavior: Files with few items are empty (not even a header)

Reproduces how often: 100%

Versions

Scrapy : 2.3.0 lxml : 4.5.2.0 libxml2 : 2.9.10 cssselect : 1.1.0 parsel : 1.6.0 w3lib : 1.22.0 Twisted : 20.3.0 Python : 3.8.2 (default, Jul 16 2020, 14:00:26) - [GCC 9.3.0] pyOpenSSL : 19.1.0 (OpenSSL 1.1.1f 31 Mar 2020) cryptography : 2.8 Platform : Linux-5.4.0-7642-generic-x86_64-with-glibc2.29

Example pipline.py

from itemadapter import ItemAdapter
from scrapy.exporters import CsvItemExporter

class CricketcrawlerPipeline:
    def open_spider(self, spider):
        self.name_to_exporter = {}

    def close_spider(self, spider):
        for exporter in self.name_to_exporter.values():
            exporter.finish_exporting()

    def process_item(self, item, spider):
        exporter = self._exporter_for_item(item)
        exporter.export_item(item)
        return item

    def _exporter_for_item(self, item):
        name = item['name']
        if name not in self.name_to_exporter:
            f = open(f'{item.folder}/{name}.csv', 'wb')
            exporter = CsvItemExporter(f)
            exporter.start_exporting()
            self.name_to_exporter[name] = exporter
        return self.name_to_exporter[name]

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:1
  • Comments:9 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
wRARcommented, Oct 30, 2020

Another consideration, the file-like object that is passed to the exporter doesn’t necessarily support the flush method.

0reactions
scientescommented, Oct 16, 2020

Yes i’ve used the docs as basis. Maybe we should Update the documentation to close the files.

Read more comments on GitHub >

github_iconTop Results From Across the Web

CSV Export Response.End Exception Thrown - Stack Overflow
I'm not trying to send multiple files in a single response- just want multiple files to download, each in their own response. –...
Read more >
Exporting Items to CSV via SWIS - Orion SDK
Hi Folks,. I want to export Data to a CSV to compare this Data with another import CSV. I already have the "wrapping"...
Read more >
write.table: Data Output - Rdrr.io
To write a Unix-style file on Windows, use a binary connection e.g. file = file("filename", "wb") . CSV files. By default there is...
Read more >
Troubleshoot CSV Export - AppSheet Help
Export CSV failed because CSV file locale is missing.​​ This error message displays when the expression specified in the CSV file locale property...
Read more >
KB250362: How to export Report Services document to CSV ...
flush(); fos.close();. NOTE: An exception is thrown when running the code against the cache few times. It is necessary to disable the report ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found