to_csv encoding problem?

I’m getting a UnicodeEncodeError when I try to send a table to_csv. The table was made with a join of two other tables, both loaded from csv using utf-8 encoding. The character that seems to have caused the error is apparently just an en-dash – nothing too exotic. Not sure if I’m doing something wrong or if something worse is going on.

The traceback:

Traceback (most recent call last):
  File "path/goes/here/lobbyists.py", line 57, in <module>
    combo.to_csv('combo.csv')
  File "/Users/theofrancis/miniconda3/lib/python3.5/site-packages/agate/table/__init__.py", line 408, in to_csv
    writer.writerow(tuple(csv_funcs[i](d) for i, d in enumerate(row)))
  File "/Users/theofrancis/miniconda3/lib/python3.5/site-packages/agate/csv_py3.py", line 91, in writerow
    self.writer.writerow(row)
UnicodeEncodeError: 'ascii' codec can't encode character '\u2013' in position 282: ordinal not in range(128)

[bitlather]

This was among the top results when I was looking up the error:

UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\xe9’ in position 0: ordinal not in range(128)

I thought I’d show how I fixed my particular issue for anyone else who ends up here.

Replicating The Problem

Consider this script:

import csv

data = [["a", "b", u'\xe9']]

with open("output.csv", "w") as csv_file:
    writer = csv.writer(csv_file, quoting=csv.QUOTE_ALL)
    writer.writerows(data)

When I run it, I get:

$ python tmp_csv_script.py 
Traceback (most recent call last):
  File "tmp_csv_script.py", line 7, in <module>
    writer.writerows(data)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0: ordinal not in range(128)

Solution

Set default encoding to utf8.

import csv

import sys
reload(sys)
sys.setdefaultencoding('utf8')

data = [["a", "b", u'\xe9']]

with open("output.csv", "w") as csv_file:
    writer = csv.writer(csv_file, quoting=csv.QUOTE_ALL)
    writer.writerows(data)

Running this was successful for me, and the output looked like this:

"a","b","é"

Further Reading

There is an excellent blog article that might be able to help you out more, if you don’t want to set the default encoding:

Python: UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\xfc’ in position 11: ordinal not in range(128)

[jpmckinney]

Noting that this might help solve some of the encoding issues on Windows prior to Python 3.8 (currently skipped in CI).