Python js-beautifier fails when script contains UTF-8, and output is piped

$ cat test.js 
var s = "åäö";
$ js-beautify test.js
var s = "åäö";
$ js-beautify test.js | cat -
'ascii' codec can't encode characters in position 9-11: ordinal not in range(128)
$ js-beautify --version
1.6.4
$ python --version
Python 2.7.12

[fwhite-wsm]

Hey @bitwiseman, did this get fixed by chance? I looked through recent changes here https://github.com/beautify-web/js-beautify/commits/master but didn’t find any relevant ones.

I’m on: EditorConfig (0.12.1) jsbeautifier (1.6.12)

I tested it on Python 2.7.9 thru 2.7.13 on both brew and pkg’s to try to get it to work, only to find that the bug seems to manifest always when redirecting the output, regardless of version.

It works as intended if I do not redirect the output (and the program outputs to stdout), but as soon as I pipe or redirect the output to anything, the program shatters (the output does not get correctly redirected) and emits to stderr a message such as:

‘ascii’ codec can’t encode character u’\u…’ in position 376942: ordinal not in range(128)

This makes using js-beautify an excellent adventure in copy/paste when beautifying larger files.

And just now reading @tobbez’s above proposed change which I somehow failed to read before debugging it, that solution should fix it exactly.

Bump 😃

[tobbez]

Putting it right at the top of the file is fine as long as it’s an application (and was actually intended for that).

Anyway, I didn’t look too closely at the existing code before writing the above, but doing so makes it clear that the above code is much more general than is needed here. That solution is useful when writing to standard out in many places, but seeing as js-beautifier only writes output in a single place, the following should suffice:

diff --git a/python/jsbeautifier/__init__.py b/python/jsbeautifier/__init__.py
index 9eecb59..3697513 100644
--- a/python/jsbeautifier/__init__.py
+++ b/python/jsbeautifier/__init__.py
@@ -7,6 +7,7 @@ import re
 import string
 import errno
 import copy
+import locale
 from jsbeautifier.__version__ import __version__

 #
@@ -2316,7 +2317,16 @@ def main():
                 import msvcrt
                 msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)

-            sys.stdout.write(pretty)
+            if sys.stdout.encoding is None:
+                # sys.stdout.encoding will be None if stdout is redirected (at
+                # least on Python 2), leading to encoding errors when stdout is
+                # redirected and the beautified content contains unicode
+                try:
+                    sys.stdout.buffer.write(pretty.encode(locale.getpreferredencoding()))
+                except:
+                    sys.stdout.write(pretty.encode(locale.getpreferredencoding()))
+            else:
+                sys.stdout.write(pretty)
         else:
             if isFileDifferent(outfile, pretty):
                 mkdir_p(os.path.dirname(outfile))