Might be nice to always call `exporter.finish_exporting`
See original GitHub issueDescription
When an exception happens inside Exporter (for whatever reason) - itemcount never gets increased (that’s OK). But then during closing, the code doesn’t get chance to act
This is the place in the code:
https://github.com/scrapy/scrapy/blob/master/scrapy/extensions/feedexport.py#L357
Perhaps the lines 357 and 353 should have been swapped?
Additional context
I’m using slot.storage
to open a database; and an exporter to start a session – store() gets called (to close connection to the database) but session didn’t have chance to commit – yes, there is nothing to commit, but still - given the semantics (also visible from the code – when file
is supposed to be a file descriptor it might be worth consideration to always call finish_exporting
to give the file descriptor chance to react before it is closed by the parent object)
necessary caveat: I might be mis-using the API and it is a minor detail, so please forgive if this looks silly (and feature not a bug)
Issue Analytics
- State:
- Created a year ago
- Comments:9 (4 by maintainers)
Top GitHub Comments
If you mean a test case for making sure it’s ohly called when start_exporting() was called it would just be a test spider with some exporter enabled and some different edge cases like zero items, maybe an exception happening at some early point etc.
Yes, this sounds correct.
I don’t have one handy, but it looks like tests/test_exporters.py is a place with a lot of tests of the related code.