Non-latin characters get HTML-encoded with decodeEntities=true

Hi,

Consider this code:

var cheerio = require('cheerio')
var ch = cheerio.load('<div>абв</div>', { decodeEntities: true })
console.log(ch('div').html())

It prints абв. If I set decodeEntities to false, the output will be the expected абв.

Two issues here:

1. This is not how decodeEntities is supposed to work. I tested with htmlparser2 directly, and it works as expected both ways (the code is below).
2. Htmlparser recommends always set decodeEntities to true for security reasons.

Test code for htmlparser:

var htmlparser = require("htmlparser2");
var parser = new htmlparser.Parser({
    ontext: function(text){
        console.log("-->", text);
    }
}, {decodeEntities: true});
parser.write("<div>абв</div>");
parser.end();

Output: --> абв (as expected)

P.S. Cheerio version: 0.20.0

[rlidwka]

Did you find a way around this ?

An obvious solution is, undo all the escaping work that dom-serializer does:

var cheerio = require('cheerio')

var cheerio_html = cheerio.prototype.html

cheerio.prototype.html = function wrapped_html() {
  var result = cheerio_html.apply(this, arguments)
 
  if (typeof result === 'string') {
    result = result.replace(/&#x([0-9a-f]{1,6});/ig, function (entity, code) {
      code = parseInt(code, 16)

      // don't unescape ascii characters, assuming that all ascii characters
      // are encoded for a good reason
      if (code < 0x80) return entity

      return String.fromCodePoint(code)
    })
  }

  return result
}

console.log(cheerio.load('<div>абв"&quot;&lt;&gt;</div>').root().html())
// <div>абв&quot;&quot;&lt;&gt;</div>

It modifies cheerio prototype (which might not be desirable in your case), and it could slow down parsing if you’re calling html() a lot. So I’d be happy to know if there’s a better solution out there.

[shelldweller]

This was reported in #565 and for some reason closed. I agree, .html() should not escape non-ASCII characters to entities, but should return Unicode string, like .text() does.