question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unicode converting from UTF-16 to UTF-32 then failing on JSON.parse

See original GitHub issue

We’re storing an object via hset that contains the following unicode character (an emoji) \ud83e\udef6. When we retrieve the value via hgetall we get \U0001faf6 back which is the UTF-32 encoding of the original character. This causes the deserializer to fail.

Is there a reason that these characters are getting converted from different encoding types? And what would be the recommended work around here?

Error we receive back from hgetall:

FetchError: invalid json response body at [URL] reason: Unexpected token U in JSON at position 330

We then fetch the response using /hgetall/ REST API to inspect it - (res.text()) and it’s telling us the unicode character was transformed to UTF-32

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:8

github_iconTop GitHub Comments

1reaction
cathykccommented, Aug 22, 2022

Hey! Sorry for not replying - we ended up solving this by with encodeURI.

Setting - encodeURI(JSON.stringify(VALUE_TO_STORE_IN_REDIS)) Getting -JSON.parse(decodeURI(VALUE_RETREIVED_FROM_REDIS) - this returns the original unicode characters

Thanks for looking into this!

1reaction
chronarkcommented, Jul 7, 2022

Thanks, I will try to debug this asap, but I can’t promise you anything until the end of next week.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Encoding JSON in UTF-16 or UTF-32 - unicode - Stack Overflow
As far as I can tell, yes, you can write the UTF-16 values directly. ... Convert the encoding format to US-ASCII in the...
Read more >
FAQ - UTF-8, UTF-16, UTF-32 & BOM - Unicode
The downside of UTF-8 is that without converting into a format that can be displayed on your system, you cannot tell which non-ASCII...
Read more >
Decouple encodings from JSON parsing · Issue #6 - GitHub
Way too complicated! The problem is in parsing a JSON String. We have to support both standard Unicode characters and special JSON escapes....
Read more >
Problem with JSON encoding - Ivan Sagalaev
You have to decode a byte stream twice: first convert a UTF encoding to Unicode code points and then replace any \uXXXX sequences...
Read more >
JSON Parse() raises error after CODEPAGE-CONVERT of ...
When the parameter to the Parse() method is a LONGCHAR, it must be encoded in a Unicode code page. Error Message. Invalid encoding...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found