Example JSON transformer doesn't process escape sequences

The JSON transformer presented in the documentation doesn’t handle the escape sequences in JSON strings. For example, copy the last full source (from Part 5 - Step 1) to a lark-json script and run it:

$ python lark-json /dev/stdin
"foo\nbar"
^D
foo\nbar

The expected output would be foo and bar in two separate lines.

An easy and (I hope) fast way to achieve this is to use the unicode_escape built-in codec, which also handles the \uhhhh sequences:

diff --git a/lark-json.orig b/lark-json
index 3d07bd6..8442239 100644
--- a/lark-json.orig
+++ b/lark-json
@@ -25,7 +25,7 @@ json_grammar = r"""
 
 class TreeToJson(Transformer):
     def string(self, (s,)):
-        return s[1:-1]
+        return s[1:-1].decode('utf-8').decode('unicode_escape')
     def number(self, (n,)):
         return float(n)
 

This will also modify string to return Unicode strings in Python 2. Since JSON is defined in terms of Unicode, this is probably the right thing to do, but if it’s undesirable, the result could be re-encoded to UTF-8.

[goodmami]

Maybe something like this: http://www.json.org/JSON_checker/

See the linked test suite. Note, however, that this test suite expects the top-level element to only be an object or an array, but the spec allows other value types. I think the spec may have changed at some point and the tests were not updated, but it still is a useful set of examples.

[erezsh]

To expand a little on what I mean by real attempt: Lark’s example Python parser can parse every Python file in the built-in library, and a special Python file full of gotchas, made by someone who isn’t me.

If you can get a JSON parser to that level of rigor, I will be happy to include it.