Warning messages to add in the CSV exporter when there's any field dropped
See original GitHub issueSummary
A warning message shall be generated when there’re some fields dropped within CsvItemExporter
(or also in other exporters?) during exporting.
We may have this kind of warning messages emitted only for the first occurrence to avoid generating too much noise, similar to the warning messages when there’s possible data loss (related code).
Motivation
Due to nature of the CSV format, the CsvItemExporter
needs to determine the list of fields to write before writing the 1st record (related code), which would in some occasions cause unintentional loss of data.
Unfortunately I’ve yet observed this kind of mistakenly introduced data loss issue from different developers, where the common cause may be:
- Returning a
dict
as the 1st item that doesn’t have all fields included. - Returning more than one type of items that have different fields.
(both when not having FEED_EXPORT_FIELDS
properly configured)
Describe alternatives you’ve considered
Properly configuring the FEED_EXPORT_FIELDS
setting entry would surely have the issue resolved. However that’s not always guaranteed (a developer may just forget to, etc.).
Issue Analytics
- State:
- Created 4 years ago
- Reactions:1
- Comments:12 (11 by maintainers)
Top GitHub Comments
But we only want such a warning logged if there is actual data loss, I think.
Has this issue been resolved? If so, then maybe this issue can be closed.