question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

CSV files with too many values in a row cause errors

See original GitHub issue

Original title: csv.DictReader can have None as key

In some cases, csv.DictReader can have None as key for unnamed columns, and a list of values as value. sqlite_utils.utils.rows_from_file cannot handle that:

url="https://artsdatabanken.no/Fab2018/api/export/csv"
db = sqlite_utils.Database(":memory")

with urlopen(url) as fab:
    reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le")   
    db["fab2018"].insert_all(reader, pk="Id")

Result:

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 2924, in insert_all
    chunk = list(chunk)
  File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in fix_square_braces
    if any("[" in key or "]" in key for key in record.keys()):
  File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in <genexpr>
    if any("[" in key or "]" in key for key in record.keys()):
TypeError: argument of type 'NoneType' is not iterable

Code: https://github.com/simonw/sqlite-utils/blob/59be60c471fd7a2c4be7f75e8911163e618ff5ca/sqlite_utils/db.py#L3454

sqlite-utils insert from command line is not affected by this issue.

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:20 (20 by maintainers)

github_iconTop GitHub Comments

0reactions
simonwcommented, Jun 14, 2022

I forgot to add equivalents of extras_key= and ignore_extras= to the CLI tool - will do that in a separate issue.

Read more comments on GitHub >

github_iconTop Results From Across the Web

6 Common CSV Import Errors and How to Fix Them - Flatfile
One of the most common CSV import errors is that the file is simply too large. That can be caused by too many...
Read more >
ValueError: too many values to unpack dealing with csv file
The error says you have too many values to unpack. Each line has up to eight columns. You are trying to unpack each...
Read more >
Common CSV Template Error Messages and How to Fix Them
When creating CSV Templates, you will undoubtedly encounter errors that need to be fixed. These can be caused by problems or edge cases......
Read more >
What to do if a data set is too large for the Excel grid
Go to the Data tab > From Text/CSV > find the file and select Import. In the preview dialog box, select Load To......
Read more >
Handling Messy CSV Files - Gertjan van den Burg
There's a very simple reason for this problem: a “comma separated value” file is not a standard file format, but is actually more...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found