[Feature Request] On import, allow option for number of fields to match widest row of data (filling missing with NaN)

Given data, such as below in ‘csv’ format:

header 1, header 2 data 1, data 2 data 3, data 4, data 5

On import, Pandas will fail with the error:

ParserError: Error tokenizing data. C error: Expected 2 fields in line 2, saw 3

There are many ways around this error:

1. Loading the file in a spreadsheet program (i.e. Excel), and saving it as a new file
2. Specifying usecols
3. Setting error_bad_lines to False

But there are instances where these workarounds are impractical:

1. When you want to preserve all of your data (usecols and error_bad_lines will cause loss)
2. When using a spreadsheet program is not feasible (i.e. encodings not supported by Excel, or when dealing with a large number of files (or in particular, the combination of the two))

Desired feature

In read_csv and read_table, add an expand_header option, which when set to True will set the width of the DataFrame to that of the widest row. Cells left without data due to this would be blank or NaN.

[gfyoung]

@joshjacobson : Thanks for your reply. I think I would agree with @jorisvandenbossche that your first option is the best way to move forward.

Good to know that you have encountered this issue with your datasets, but it’s important to separate between data issues and read_csv being too strict. In this case, I think it falls more on the former than the latter given your description of the data sources.

Also, I understand that you believe that this problem is prevalent given that you encounter it on so many occasions, but for us as maintainers to agree with that assessment, we generally need to see a good number of users independently bring up this issue.

Option 2 is not generalizable because we don’t want to loosen our requirements on files being correctly formed. The responsibility should be on the user to make sure that their file is formatted correctly to be read into pandas.

Option 3 would likely be best integrated as another parameter, but we have too many parameters already. The same goes for Option 4.

Again, if more people bring up this issue, we can always revisit, but at this point, I would stick with option 1 as your way to go.

[jorisvandenbossche]

There is another workaround that does preserve all data: specify header names manually that are long enough:

In [90]: data = """a,b
    ...: 1,2
    ...: 1,2,3"""

In [92]: pd.read_csv(StringIO(data))
...
ParserError: Error tokenizing data. C error: Expected 2 fields in line 3, saw 3

In [93]: pd.read_csv(StringIO(data), names=['a', 'b', 'c'])
Out[93]: 
   a  b    c
0  a  b  NaN
1  1  2  NaN
2  1  2  3.0