ability to validate UTF-8 encoding

In iconv:

> Iconv = require('iconv').Iconv
> i = new Iconv('UTF-8', 'UTF-8')
> i.convert(new Buffer([128]))
Error: Illegal character sequence.

In iconv-lite:

> iconv = require('iconv-lite')
> iconv.decode(new Buffer([128]), 'UTF-8')
'�'

With iconv, you know whether the decoding succeeded. With iconv-lite there is no way to know if the decoding succeeded.

[ashtuchkin]

“�” is used for decoding, “?” for encoding (as many encodings cannot represent �).

Alexander Shtuchkin

On Mon, May 25, 2015 at 10:57 PM, Benjamin Pasero [e-mail] wrote:

@ashtuchkin https://github.com/ashtuchkin got it. But in the docs you say “Untranslatable characters are set to � or ?”, so it might also be “?” which I cannot really check for because its a valid character.

— Reply to this email directly or view it on GitHub https://github.com/ashtuchkin/iconv-lite/issues/83#issuecomment-105406513 .

[ashtuchkin]

Yeah, that’s a completely separate issue from “ability to validate UTF-8 encoding” 😃

I have a callback-like mechanism in mind (see #53), where you can either throw exception or define replacements for characters that cannot be represented. Still haven’t got a moment to implement it though.​

Alexander Shtuchkin

On Tue, May 26, 2015 at 10:25 AM, Benjamin Pasero [e-mail] wrote:

In our case, we use encode() to convert an existing file to another encoding. E.g. we have a source UTF-16 file that we want to save as DOS encoding. DOS can not present all encodings (Chinese et. al), so I would like to show an error to the user. Similar to how in Sublime Text you will not be able to save a file in an encoding that cannot represent all characters you have before encoding.

— Reply to this email directly or view it on GitHub https://github.com/ashtuchkin/iconv-lite/issues/83#issuecomment-105611321 .