Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

read_csv() - "skiprows" should ignore "comment"

See original GitHub issue

Code Sample, a copy-pastable example if possible

pandas.read_csv(URL, delimiter='|', comment='#', skiprows=4)

Problem description

ParserError when reading CSV files with comments at the beginning of the file. Line count of comments varies.

EDIT: ParseError is due to the fact that the column count of the first 4 rows is different from the column count of the succeeding rows

Objective:

Ignore comments
Ignore the first 4 lines that are not comments

This happens when reading a CSV file similar to the following

# Comment start
# ..... Comment count varies
# Comment end
first|line
second|line
third|line
fourth|line
first|data|to|read
second|data|to|read

Expected Output

The comment lines should not be included on the skiprows

Output of `pd.show_versions()`

[paste the output of ``pd.show_versions()`` here below this line] commit: None python: 3.6.6.final.0 python-bits: 32 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 142 Stepping 9, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None pandas: 0.23.4 pytest: None pip: 18.1 setuptools: 40.4.3 Cython: None numpy: 1.15.2 scipy: None pyarrow: None xarray: None IPython: None sphinx: None patsy: None dateutil: 2.7.3 pytz: 2018.5 blosc: None bottleneck: None tables: None numexpr: None feather: None matplotlib: None openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: None sqlalchemy: None pymysql: None psycopg2: None jinja2: None s3fs: None fastparquet: None pandas_gbq: None pandas_datareader: None

Issue Analytics

State:
Created 5 years ago
Reactions:2
Comments:11 (5 by maintainers)

Top GitHub Comments

2reactions

WillAydcommented, Oct 11, 2018

Just an FYI this is documented behavior:

fully commented lines are ignored by the parameter header but not by skiprows

Definitely a corner case here but I don’t have anything against the feature, given we already account for it with the header parameter. Investigation and PRs welcome

1reaction

JustinZhengBCcommented, Oct 10, 2018

If the data is clean, then you could always do df = pd.read_csv(URL, comment='#')[n:] to skip the first n rows. That doesn’t necessarily work in this case due to the rows having an uneven number of elements, but that’s a whole other issue.

Top Results From Across the Web

pandas.read_csv: how to skip comment lines - Stack Overflow

One workaround is to specify skiprows to ignore the first few entries: ... Otherwise with pd.read_csv(filename, comment='#') I get. pandas.parser.

How to skip rows while reading csv file using Pandas?

Here, we will discuss how to skip rows while reading csv file. We will use read_csv() method of Pandas library for this task....

Pandas: How to Skip Rows when Reading CSV File - Statology

You can use the following methods to skip rows when reading a CSV file into a pandas DataFrame: Method 1: Skip One Specific...

pandas.read_csv — pandas 1.5.2 documentation

This parameter must be a single character. Like empty lines (as long as skip_blank_lines=True ), fully commented lines are ignored by the parameter...

Pandas : skip rows while reading csv file to a Dataframe using ...

While calling pandas.read_csv() if we pass skiprows argument as a list of ints, then it will skip the rows from csv at specified...