JSON cannot represent some naughty strings

One of my favourite naughty strings is invalid utf-8 - for example, a bare \xff. It’s quite common to get 500s,etc on these as no-one ever bothers to check for unicode decoding errors. However, because JSON requires all strings to be valid utf-8, this example is only able to be included in the txt file.

Would this be something worth including and adding a special case in the script to omit from the json? Or is it too naughty for blns?

EDIT: This HN comment (https://news.ycombinator.com/item?id=10035738) suggested having the JSON file be of b64 encoded strings. This is a good suggestion, and allows arbitrary naughty bytes to be used, at the cost of readability.

[jfinkhaeuser]

Just so that’s clear, JSON does not require UTF-8 any longer, but valid unicode. The UTF-8 requirement is from an older version of JSON.

Doesn’t change much about this topic, but it’s worth noting.

[timmc]

Given that some software environments conflate bytes and strings, or naively assume ASCII (or well-formed UTF-8, or whatever), I think it makes perfect sense to include byte sequences in here that cause decoding issues.

I don’t think that specifying an encoding is necessary, but having a comment explaining what the point of the byte sequence is would be useful—and that could mention the specific encoding that is being targeted.