question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

File mode in `to_csv` is ignored, when passing a file object instead of a path

See original GitHub issue

Code Sample, a copy-pastable example if possible

>>> import pandas as pd
>>> df = pd.read_csv("example.csv")
>>> df.head()
   just  a  file
0     1  2     3
1     4  5     6
2     7  8     9
>>> with open("someother.csv", "wb") as f:
...     df.to_csv(f, mode="wb")
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3.6/site-packages/pandas/core/frame.py", line 1524, in to_csv
    formatter.save()
  File "/usr/lib/python3.6/site-packages/pandas/io/formats/format.py", line 1652, in save
    self._save()
  File "/usr/lib/python3.6/site-packages/pandas/io/formats/format.py", line 1740, in _save
    self._save_header()
  File "/usr/lib/python3.6/site-packages/pandas/io/formats/format.py", line 1708, in _save_header
    writer.writerow(encoded_labels)
TypeError: a bytes-like object is required, not 'str'

Problem description

When passing a file opened in binary mode to df.to_csv and also passing mode='wb', this mode is ignored. I think it’s because of these lines: https://github.com/pandas-dev/pandas/blob/master/pandas/io/common.py#L407-L411 and these ones: https://github.com/pandas-dev/pandas/blob/master/pandas/io/formats/format.py#L1660-L1662

It seems that is_text isn’t passed, and so it assumes the default value of True

Expected Output

A file should just be written.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None python: 3.6.4.final.0 python-bits: 64 OS: Linux OS-release: 4.14.20-1-lts machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.22.0 pytest: 3.4.0 pip: 9.0.1 setuptools: 38.5.1 Cython: 0.27.3 numpy: 1.14.0 scipy: 1.0.0 pyarrow: None xarray: None IPython: 6.2.1 sphinx: 1.7.0 patsy: 0.4.1 dateutil: 2.6.1 pytz: 2018.3 blosc: None bottleneck: None tables: None numexpr: None feather: None matplotlib: 2.1.2 openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: 4.6.0 html5lib: 1.0b10 sqlalchemy: 1.2.3 pymysql: None psycopg2: None jinja2: 2.10 s3fs: 0.1.2 fastparquet: 0.1.3 pandas_gbq: None pandas_datareader: None

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Reactions:1
  • Comments:21 (9 by maintainers)

github_iconTop GitHub Comments

1reaction
Huitecommented, Jul 3, 2018

I’ve downgraded to 0.23.0 to check, and 0.23.0 works as expected. It appears to be specific to 0.23.1.

0reactions
remram44commented, Jun 9, 2020

I’m very confused about this, because passing a io.BytesIO to pandas will not work (so whatever detection you do isn’t very good) but passing a simple file-like object with a write() method will make to_csv() write bytes to it for some reason. Why to_csv() puts bytes into anything is beyond me.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Writing a pandas DataFrame to CSV file - Stack Overflow
When you are storing a DataFrame object into a csv file using the to_csv method, ... instead of (the case when the default...
Read more >
Support Binary File Objects with pandas.DataFrame.to_csv
DataFrame.to_csv does not support writing to binary file objects 1. ... mode in to_csv is ignored, when passing a file object instead of...
Read more >
pandas.DataFrame.to_csv — pandas 1.5.2 documentation
A string representing the encoding to use in the output file, defaults to 'utf-8'. encoding is not supported if path_or_buf is a non-binary...
Read more >
Pandas to_csv() - Convert DataFrame to CSV - DigitalOcean
We can pass a file object to write the CSV data into a file. Otherwise, the CSV data is returned in the string...
Read more >
Pandas: How to Read and Write Files - Real Python
For instance, if you have a file with one data column and want to get a Series object instead of a DataFrame ,...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found