UnicodeDecodeError when running the tests on Debian
See original GitHub issueIn my attempt to build a Debian package from agate v1.6.0, I am getting the following errors at the testing stage:
======================================================================
ERROR: test_from_json (tests.test_from_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_from_json.py", line 28, in test_from_json
table2 = Table.from_json('examples/test.json')
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/from_json.py", line 53, in from_json
js = json.load(f, object_pairs_hook=OrderedDict, parse_float=Decimal, **kwargs)
File "/usr/lib/python3.5/json/__init__.py", line 265, in load
return loads(fp.read(),
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 232: ordinal not in range(128)
======================================================================
ERROR: test_from_json_file_like_object (tests.test_from_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_from_json.py", line 38, in test_from_json_file_like_object
table2 = Table.from_json(f)
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/from_json.py", line 50, in from_json
js = json.load(path, object_pairs_hook=OrderedDict, parse_float=Decimal, **kwargs)
File "/usr/lib/python3.5/json/__init__.py", line 265, in load
return loads(fp.read(),
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 232: ordinal not in range(128)
======================================================================
ERROR: test_from_json_mixed_keys (tests.test_from_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_from_json.py", line 53, in test_from_json_mixed_keys
table = Table.from_json('examples/test_mixed.json')
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/from_json.py", line 53, in from_json
js = json.load(f, object_pairs_hook=OrderedDict, parse_float=Decimal, **kwargs)
File "/usr/lib/python3.5/json/__init__.py", line 265, in load
return loads(fp.read(),
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 226: ordinal not in range(128)
======================================================================
ERROR: test_from_json_newline_delimited (tests.test_from_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_from_json.py", line 75, in test_from_json_newline_delimited
table2 = Table.from_json('examples/test_newline.json', newline=True)
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/from_json.py", line 46, in from_json
for line in f:
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 153: ordinal not in range(128)
======================================================================
ERROR: test_from_json_no_type_tester (tests.test_from_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_from_json.py", line 84, in test_from_json_no_type_tester
table = Table.from_json('examples/test.json', column_types=tester)
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/from_json.py", line 53, in from_json
js = json.load(f, object_pairs_hook=OrderedDict, parse_float=Decimal, **kwargs)
File "/usr/lib/python3.5/json/__init__.py", line 265, in load
return loads(fp.read(),
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 232: ordinal not in range(128)
======================================================================
ERROR: test_from_json_with_key (tests.test_from_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_from_json.py", line 46, in test_from_json_with_key
table2 = Table.from_json('examples/test_key.json', key='data')
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/from_json.py", line 53, in from_json
js = json.load(f, object_pairs_hook=OrderedDict, parse_float=Decimal, **kwargs)
File "/usr/lib/python3.5/json/__init__.py", line 265, in load
return loads(fp.read(),
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 290: ordinal not in range(128)
======================================================================
ERROR: test_reader (tests.test_py3.TestDictReader)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_py3.py", line 176, in test_reader
self.assertEqual(next(reader), dict(zip(self.rows[0], self.rows[1])))
File "/usr/lib/python3.5/csv.py", line 109, in __next__
self.fieldnames
File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
self._fieldnames = next(self.reader)
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 94: ordinal not in range(128)
======================================================================
ERROR: test_reader_alias (tests.test_py3.TestDictReader)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_py3.py", line 181, in test_reader_alias
self.assertEqual(next(reader), dict(zip(self.rows[0], self.rows[1])))
File "/usr/lib/python3.5/csv.py", line 109, in __next__
self.fieldnames
File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
self._fieldnames = next(self.reader)
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 94: ordinal not in range(128)
======================================================================
ERROR: test_line_numbers (tests.test_py3.TestReader)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_py3.py", line 54, in test_line_numbers
rows = list(csv_py3.Reader(f, line_numbers=True))
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/csv_py3.py", line 34, in __next__
row = next(self.reader)
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 94: ordinal not in range(128)
======================================================================
ERROR: test_sniffer (tests.test_py3.TestSniffer)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_py3.py", line 252, in test_sniffer
contents = f.read()
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 94: ordinal not in range(128)
======================================================================
ERROR: test_pivot_compute_pivots (tests.test_table.test_pivot.TestPivot)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_pivot.py", line 211, in test_pivot_compute_pivots
pivot_table.print_table(output=sys.stdout)
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/print_table.py", line 156, in print_table
write_row(formatted_row)
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/print_table.py", line 140, in write_row
write('%s%s%s' % (v_line, text, v_line))
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/print_table.py", line 121, in write
output.write(line + '\n')
UnicodeEncodeError: 'ascii' codec can't encode character '\u2026' in position 17: ordinal not in range(128)
======================================================================
ERROR: test_print_bars_domain (tests.test_table.test_print_bars.TestPrintBars)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_print_bars.py", line 59, in test_print_bars_domain
table.print_bars('three', 'one', domain=(0, 300))
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/print_bars.py", line 222, in print_bars
write('%s %s %s' % (label_text, value_text, bar))
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/print_bars.py", line 184, in write
output.write(line + '\n')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 12-13: ordinal not in range(128)
======================================================================
ERROR: test_print_bars_mixed_signs (tests.test_table.test_print_bars.TestPrintBars)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_print_bars.py", line 85, in test_print_bars_mixed_signs
table.print_bars('three', 'one')
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/print_bars.py", line 222, in print_bars
write('%s %s %s' % (label_text, value_text, bar))
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/print_bars.py", line 184, in write
output.write(line + '\n')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 42-48: ordinal not in range(128)
======================================================================
ERROR: test_print_bars_negative (tests.test_table.test_print_bars.TestPrintBars)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_print_bars.py", line 75, in test_print_bars_negative
table.print_bars('three', 'one')
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/print_bars.py", line 222, in print_bars
write('%s %s %s' % (label_text, value_text, bar))
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/print_bars.py", line 184, in write
output.write(line + '\n')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 110-119: ordinal not in range(128)
======================================================================
ERROR: test_print_csv (tests.test_table.test_to_csv.TestToCSV)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_csv.py", line 105, in test_print_csv
contents2 = f.read()
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 94: ordinal not in range(128)
======================================================================
ERROR: test_to_csv (tests.test_table.test_to_csv.TestToCSV)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_csv.py", line 33, in test_to_csv
table.to_csv('.test.csv')
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/to_csv.py", line 43, in to_csv
writer.writerow(tuple(csv_funcs[i](d) for i, d in enumerate(row)))
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/csv_py3.py", line 91, in writerow
self.writer.writerow(row)
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f44d' in position 2: ordinal not in range(128)
======================================================================
ERROR: test_to_csv_file_like_object (tests.test_table.test_to_csv.TestToCSV)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_csv.py", line 49, in test_to_csv_file_like_object
table.to_csv(f)
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/to_csv.py", line 43, in to_csv
writer.writerow(tuple(csv_funcs[i](d) for i, d in enumerate(row)))
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/csv_py3.py", line 91, in writerow
self.writer.writerow(row)
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f44d' in position 2: ordinal not in range(128)
======================================================================
ERROR: test_to_csv_make_dir (tests.test_table.test_to_csv.TestToCSV)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_csv.py", line 80, in test_to_csv_make_dir
table.to_csv('newdir/test.csv')
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/to_csv.py", line 43, in to_csv
writer.writerow(tuple(csv_funcs[i](d) for i, d in enumerate(row)))
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/csv_py3.py", line 91, in writerow
self.writer.writerow(row)
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f44d' in position 2: ordinal not in range(128)
======================================================================
ERROR: test_to_csv_to_stdout (tests.test_table.test_to_csv.TestToCSV)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_csv.py", line 73, in test_to_csv_to_stdout
contents2 = f.read()
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 94: ordinal not in range(128)
======================================================================
ERROR: test_print_json (tests.test_table.test_to_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_json.py", line 141, in test_print_json
js2 = json.load(f)
File "/usr/lib/python3.5/json/__init__.py", line 265, in load
return loads(fp.read(),
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 232: ordinal not in range(128)
======================================================================
ERROR: test_to_json (tests.test_table.test_to_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_json.py", line 39, in test_to_json
js2 = json.load(f)
File "/usr/lib/python3.5/json/__init__.py", line 265, in load
return loads(fp.read(),
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 232: ordinal not in range(128)
======================================================================
ERROR: test_to_json_file_output (tests.test_table.test_to_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_json.py", line 101, in test_to_json_file_output
table.to_json('.test.json')
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/to_json.py", line 102, in to_json
dump_json(output)
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/to_json.py", line 68, in dump_json
json.dump(data, f, **json_kwargs)
File "/usr/lib/python3.5/json/__init__.py", line 179, in dump
fp.write(chunk)
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f44d' in position 1: ordinal not in range(128)
======================================================================
ERROR: test_to_json_key (tests.test_table.test_to_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_json.py", line 52, in test_to_json_key
js2 = json.load(f)
File "/usr/lib/python3.5/json/__init__.py", line 265, in load
return loads(fp.read(),
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 198: ordinal not in range(128)
======================================================================
ERROR: test_to_json_key_func (tests.test_table.test_to_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_json.py", line 65, in test_to_json_key_func
js2 = json.load(f)
File "/usr/lib/python3.5/json/__init__.py", line 265, in load
return loads(fp.read(),
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 198: ordinal not in range(128)
======================================================================
ERROR: test_to_json_make_dir (tests.test_table.test_to_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_json.py", line 116, in test_to_json_make_dir
table.to_json('newdir/test.json')
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/to_json.py", line 102, in to_json
dump_json(output)
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/agate/table/to_json.py", line 68, in dump_json
json.dump(data, f, **json_kwargs)
File "/usr/lib/python3.5/json/__init__.py", line 179, in dump
fp.write(chunk)
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f44d' in position 1: ordinal not in range(128)
======================================================================
ERROR: test_to_json_newline_delimited (tests.test_table.test_to_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/tests/test_table/test_to_json.py", line 78, in test_to_json_newline_delimited
js2 = json.loads(list(f)[0])
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 153: ordinal not in range(128)
----------------------------------------------------------------------
Ran 449 tests in 0.434s
FAILED (errors=26, skipped=24)
The builds and tests are done with Python 3.5.3 on a clean Debian chroot.
Issue Analytics
- State:
- Created 7 years ago
- Comments:5 (4 by maintainers)
Top Results From Across the Web
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in ...
Debian Bug report logs - #933325 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 749: invalid start byte. version graph.
Read more >error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff ...
Python tries to convert a byte-array (a bytes which it assumes to be a utf-8-encoded string) to a unicode string ( str )....
Read more >Unicode issues in Linux and Unix when running the tests. #78
This happens with scandir 1.5 and Python 2.7.11 in all our Linux, AIX, Solaris, ... Unicode issues in Linux and Unix when running...
Read more >UnicodeDecodeError utf-8 codec can t decode byte in position ...
While I importing the file it shows UnicodeDecodeError: "utf-8" codec can"t decode byte 0xa0 in position ... as pd a ...
Read more >UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in ...
Bug 1397848 - UnicodeDecodeError: 'utf-8' codec can't decode byte ... OS: Linux ... Running transaction test Transaction test succeeded.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
The Debian builders are set with
LANG=C
, and indeed:Please consider hardening your file I/O operations within
agate
by specifying the encoding explicitly where necessary, i.e. in serialization functions (from_json
,from_csv
,to_json
,to_csv
), print functions, and also some of the test setups.Using
io.open
instead of plainopen
should give you a Python>=2.6 compatible way of specifying the encoding without relying onsix
.Thank you for considering this issue.
The tests now set the locale. A release will be made soon.