Unicode converting from UTF-16 to UTF-32 then failing on JSON.parse
See original GitHub issueWe’re storing an object via hset
that contains the following unicode character (an emoji) \ud83e\udef6
. When we retrieve the value via hgetall
we get \U0001faf6
back which is the UTF-32
encoding of the original character. This causes the deserializer to fail.
Is there a reason that these characters are getting converted from different encoding types? And what would be the recommended work around here?
Error we receive back from hgetall
:
FetchError: invalid json response body at [URL] reason: Unexpected token U in JSON at position 330
We then fetch the response using /hgetall/
REST API to inspect it - (res.text()
) and it’s telling us the unicode character was transformed to UTF-32
Issue Analytics
- State:
- Created a year ago
- Comments:8
Top Results From Across the Web
Encoding JSON in UTF-16 or UTF-32 - unicode - Stack Overflow
As far as I can tell, yes, you can write the UTF-16 values directly. ... Convert the encoding format to US-ASCII in the...
Read more >FAQ - UTF-8, UTF-16, UTF-32 & BOM - Unicode
The downside of UTF-8 is that without converting into a format that can be displayed on your system, you cannot tell which non-ASCII...
Read more >Decouple encodings from JSON parsing · Issue #6 - GitHub
Way too complicated! The problem is in parsing a JSON String. We have to support both standard Unicode characters and special JSON escapes....
Read more >Problem with JSON encoding - Ivan Sagalaev
You have to decode a byte stream twice: first convert a UTF encoding to Unicode code points and then replace any \uXXXX sequences...
Read more >JSON Parse() raises error after CODEPAGE-CONVERT of ...
When the parameter to the Parse() method is a LONGCHAR, it must be encoded in a Unicode code page. Error Message. Invalid encoding...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Hey! Sorry for not replying - we ended up solving this by with
encodeURI
.Setting -
encodeURI(JSON.stringify(VALUE_TO_STORE_IN_REDIS))
Getting -JSON.parse(decodeURI(VALUE_RETREIVED_FROM_REDIS)
- this returns the original unicode charactersThanks for looking into this!
Thanks, I will try to debug this asap, but I can’t promise you anything until the end of next week.