question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

BUG: read_csv(index_col=False) -> auto-generated column headers for variable row length get headers of type int

See original GitHub issue

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from io import StringIO

csv = "1,2\n1,2,3"
io = StringIO(csv)

usecols = lambda header: print(type(header))
pd.read_csv(io, index_col=False, usecols=usecols)

# (..)
# <class 'int'>

Issue Description

When using index_col=False kwarg for read_csv then the number of columns read in is equal to the length of the longest row.

In such scenario, the columns which miss headers get names based on their position, but are assigned integers instead of strings. I think headers should rather be strings in general, but specifically this became a problem when I also used: usecols=lambda header: header.strip() in headers and it raised AttributeError.

Expected Behavior

The auto-generated column name/header is a string.

Installed Versions

INSTALLED VERSIONS

commit : 4bfe3d07b4858144c219b9346329027024102ab6 python : 3.8.10.final.0 python-bits : 64 OS : Linux OS-release : 5.4.0-89-generic Version : #100-Ubuntu SMP Fri Sep 24 14:50:10 UTC 2021 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : pl_PL.UTF-8 LOCALE : pl_PL.UTF-8

pandas : 1.4.2 numpy : 1.22.3 pytz : 2022.1 dateutil : 2.8.2 pip : 20.0.2 setuptools : 45.2.0 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : None IPython : 8.3.0 pandas_datareader: None bs4 : 4.8.2 bottleneck : None brotli : None fastparquet : None fsspec : None gcsfs : None markupsafe : 1.1.0 matplotlib : None numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : None snappy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None zstandard : None

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:6 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
phoflcommented, May 27, 2022

Ah got you, thanks.

0reactions
phoflcommented, May 27, 2022

You are correct, these should be strings, since we interpret all columns as strings anyway

Read more comments on GitHub >

github_iconTop Results From Across the Web

pandas.read_csv — pandas 1.5.2 documentation
Column (s) to use as the row labels of the DataFrame , either given as string name or column index. If a sequence...
Read more >
pandas read_csv index_col=None not working with delimiters ...
Quick Answer. Use index_col=False instead of index_col=None when you have delimiters at the end of each line to turn off index column ......
Read more >
Python | Read csv using pandas.read_csv() - GeeksforGeeks
header : It accepts int, a list of int, row numbers to use as the column names, and the start of the data....
Read more >
Reading - CSV.jl
Bool , e.g. header=false : no column names exist in the data; column names will be auto-generated depending on the # of columns,...
Read more >
apache_beam.dataframe.io module - Apache Beam
header (int, list of int, default 'infer') – Row number(s) to use as the column names, and the start of the data. Default...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found