need help for fixing a text

I have the following text: “WalldÃ?©n”. I know that correctly it is “Walldén” (“Walld\u00e9n”). ftfy.fix_text returns the same text:

s = "Walld�©n"                                                                                                                                                                              
print(ftfy.fix_text(s))     # prints "Walld�©n"

Do you know how to correct this text? Thanks.

[rspeer]

@ArtemBernatskyy That one is a mixup between Windows-1251 and Windows-1252. You’d want 'Ñòðåëà'.encode('windows-1252').decode('windows-1251').

Both of these are single-byte encodings, which makes this the hardest case to try to auto-detect in ftfy – see issue #18.

[rspeer]

@jabbalaci Oh, so it’s the Unicode replacement character �, not a literal question mark. (Both of them can show up in different encoding accidents.)

Unfortunately this is still really lossy, but maybe I can help you with your approach. I can mostly reconstruct the encoding mixup that produced this data – it probably uses Latin-1 instead of the very similar Windows-1252, and then throws out the control characters. This would at least help you to generate the strings you need to replace:

def mangle_text(text):
    text = text.encode('utf-8').decode('latin-1').encode('utf-8').decode('latin-1')
    return re.sub('[\x80-\x9f]', '\ufffd', text)

For example:

>>> mangle_text('ñ')
'�±'

But you’ll probably see why I don’t think even an improvement to ftfy would be able to fix this, which is that different common characters end up with the same representation:

>>> mangle_text('Å')
'��'

>>> mangle_text('Á')
'��'