CSV files with too many values in a row cause errors
See original GitHub issueOriginal title: csv.DictReader can have None as key
In some cases, csv.DictReader
can have None
as key for unnamed columns, and a list of values as value.
sqlite_utils.utils.rows_from_file
cannot handle that:
url="https://artsdatabanken.no/Fab2018/api/export/csv"
db = sqlite_utils.Database(":memory")
with urlopen(url) as fab:
reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le")
db["fab2018"].insert_all(reader, pk="Id")
Result:
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 2924, in insert_all
chunk = list(chunk)
File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in fix_square_braces
if any("[" in key or "]" in key for key in record.keys()):
File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in <genexpr>
if any("[" in key or "]" in key for key in record.keys()):
TypeError: argument of type 'NoneType' is not iterable
sqlite-utils insert
from command line is not affected by this issue.
Issue Analytics
- State:
- Created a year ago
- Comments:20 (20 by maintainers)
Top Results From Across the Web
6 Common CSV Import Errors and How to Fix Them - Flatfile
One of the most common CSV import errors is that the file is simply too large. That can be caused by too many...
Read more >ValueError: too many values to unpack dealing with csv file
The error says you have too many values to unpack. Each line has up to eight columns. You are trying to unpack each...
Read more >Common CSV Template Error Messages and How to Fix Them
When creating CSV Templates, you will undoubtedly encounter errors that need to be fixed. These can be caused by problems or edge cases......
Read more >What to do if a data set is too large for the Excel grid
Go to the Data tab > From Text/CSV > find the file and select Import. In the preview dialog box, select Load To......
Read more >Handling Messy CSV Files - Gertjan van den Burg
There's a very simple reason for this problem: a “comma separated value” file is not a standard file format, but is actually more...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file
I forgot to add equivalents of
extras_key=
andignore_extras=
to the CLI tool - will do that in a separate issue.