CSV item exporter does not flush on close
See original GitHub issueDescription
The csv item exporter does not flush on close.
Steps to Reproduce
- Create pipeline with an CSV item exporter
- only export few items (one or two)
- Let spider close/finish
Expected behavior: All exported items are in the CSV Files
Actual behavior: Files with few items are empty (not even a header)
Reproduces how often: 100%
Versions
Scrapy : 2.3.0 lxml : 4.5.2.0 libxml2 : 2.9.10 cssselect : 1.1.0 parsel : 1.6.0 w3lib : 1.22.0 Twisted : 20.3.0 Python : 3.8.2 (default, Jul 16 2020, 14:00:26) - [GCC 9.3.0] pyOpenSSL : 19.1.0 (OpenSSL 1.1.1f 31 Mar 2020) cryptography : 2.8 Platform : Linux-5.4.0-7642-generic-x86_64-with-glibc2.29
Example pipline.py
from itemadapter import ItemAdapter
from scrapy.exporters import CsvItemExporter
class CricketcrawlerPipeline:
def open_spider(self, spider):
self.name_to_exporter = {}
def close_spider(self, spider):
for exporter in self.name_to_exporter.values():
exporter.finish_exporting()
def process_item(self, item, spider):
exporter = self._exporter_for_item(item)
exporter.export_item(item)
return item
def _exporter_for_item(self, item):
name = item['name']
if name not in self.name_to_exporter:
f = open(f'{item.folder}/{name}.csv', 'wb')
exporter = CsvItemExporter(f)
exporter.start_exporting()
self.name_to_exporter[name] = exporter
return self.name_to_exporter[name]
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:9 (6 by maintainers)
Top Results From Across the Web
CSV Export Response.End Exception Thrown - Stack Overflow
I'm not trying to send multiple files in a single response- just want multiple files to download, each in their own response. –...
Read more >Exporting Items to CSV via SWIS - Orion SDK
Hi Folks,. I want to export Data to a CSV to compare this Data with another import CSV. I already have the "wrapping"...
Read more >write.table: Data Output - Rdrr.io
To write a Unix-style file on Windows, use a binary connection e.g. file = file("filename", "wb") . CSV files. By default there is...
Read more >Troubleshoot CSV Export - AppSheet Help
Export CSV failed because CSV file locale is missing. This error message displays when the expression specified in the CSV file locale property...
Read more >KB250362: How to export Report Services document to CSV ...
flush(); fos.close();. NOTE: An exception is thrown when running the code against the cache few times. It is necessary to disable the report ......
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Another consideration, the file-like object that is passed to the exporter doesn’t necessarily support the flush method.
Yes i’ve used the docs as basis. Maybe we should Update the documentation to close the files.