Cannot resolve mojibake on emoji

We’re getting JSON from a third party

{
  "title": "\\u011f\\u0178\\ufffd\\u0161 Cooked Rice"
}

This is supposed to correspond to the Cooked Rice emoji: 🍚 https://www.compart.com/en/unicode/U+1F35A

They also send us

{
  "first_name": "Ya\\u00c3\\u00abl"
}

Which could be cleaned by simply going over latin1.

print('Ya\u00c3\u00abl')
'Yaël'
print('Ya\u00c3\u00abl'.encode('latin1').decode('utf-8'))
'Yaël'

Any help here would be amazing.

[rspeer]

Unfortunately, ftfy isn’t going to be able to automatically fix either of these.

The second example is too short to be reliably detected. If you know that latin1 / utf-8 mixups are likely, you should probably just try that encoding and decoding yourself.

The first one makes a fascinating example, because it’s a Windows-1254 / utf-8 mixup, and I hadn’t seen one of those in the wild before. There are two problems now:

• ftfy has never tried to recognize Windows-1254 mojibake
• the third character was an unassigned byte in Windows-1254 that has been replaced by ‘�’, so if we did look for Windows-1254 mojibake patterns, the intended text could still have been any of these characters:

U+1F05A  🁚       [So] DOMINO TILE HORIZONTAL-05-06
U+1F35A  🍚      [So] COOKED RICE
U+1F39A  🎚       [So] LEVEL SLIDER
U+1F3DA  🏚       [So] DERELICT HOUSE BUILDING
U+1F41A  🐚      [So] SPIRAL SHELL
U+1F75A  🝚       [So] ALCHEMICAL SYMBOL FOR POWDERED BRICK
U+1F79A  🞚       [So] WHITE DIAMOND CONTAINING BLACK VERY SMALL DIAMOND

[rspeer]

Thanks for the details. That’s terrifying – I had no idea that the default behavior of requests on a text file is to blindly accept whatever chardet says.

Even though requests provides a better path to decoding JSON, I’d still say that the issue is really on requests’ end, for silently introducing encoding bugs like that.