df.to_csv() ignores encoding when given a file object or any other filelike object.

Code Sample, a copy-pastable example if possible

import pandas as pd
import io

# !! NOTE
# This example uses `io.BytesIO`, however this also applies to file buffers that are
# returned by `io.open` (the `open` function) when opened in binary mode.
buf = io.BytesIO('a, b, 🐟\n1, 2, 3\n4, 5, 6'.encode('utf-8'))
df = pd.read_csv(buf)   # reads in fine using default encoding (utf-8)
buf = io.BytesIO()

df.to_csv(buf, encoding='utf-8')  # this should work, but doesn't.
# TypeError: a bytes-like object is required, not 'str'

buf = io.StringIO()
df.to_csv(buf, encoding='utf-8')  # this 'works', but should fail.  Data is passed in without encoding.
buf.getvalue()
# ',a, b, 🐟\n0,1,2,3\n1,4,5,6\n'

Problem description

Currently, the ‘encoding’ parameter is accepted and doesn’t do anything when dealing with an in-memory object. This is deceptive, and can introduce encoding flaws.

I presume that pandas just sets the encoding on the file it opens. In the case of receiving an already-open filelike object, pandas should encode the string and attempt to write the bytes into the file. If it fails, that’s a valid and appropriate failure, and that failure should be raised.

However, in the interest of backwards compatibility, if it fails, it should probably try to write the unencoded string into the file, and perhaps display a warning.

I’m on Pandas 0.23.4. https://pandas-docs.github.io/pandas-docs-travis/

Expected Output

# a buffer that contains the following:
b',a, b, \xf0\x9f\x90\x9f\n0,1,2,3\n1,4,5,6\n'

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None python: 3.6.7.final.0 python-bits: 64 OS: Linux OS-release: 4.19.3-041903-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.23.4 pytest: 4.0.0 pip: 9.0.1 setuptools: 39.0.1 Cython: None numpy: 1.15.4 scipy: None pyarrow: 0.11.1 xarray: None IPython: 7.1.1 sphinx: None patsy: None dateutil: 2.7.5 pytz: 2018.7 blosc: None bottleneck: None tables: None numexpr: None feather: None matplotlib: None openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: 0.999999999 sqlalchemy: None pymysql: None psycopg2: None jinja2: None s3fs: None fastparquet: None pandas_gbq: None pandas_datareader: None

[Enteee]

Hi folks, I wrote an article on my blog on how to Support Binary File Objects with pandas.DataFrame.to_csv. At the end of the article I added a monkey patch I think can also be used as a work around for this problem. Hope this helps until this is resolved in pandas.

[gfyoung]

foo is just not an accepted use case

Agreed. That could be a first step by updating the docs to reflect that.

That being said, a fix to actual enhance to_csv with the functionality would be a good long-term fix.