Better handling of responses without correct content-type charset

Problem

If server doesn’t provide correct content-type charset, we’re defaulting to latin1 because requests underlying does that. This is undesirable for user experience.

Possible solutions

• Default to utf8
• For well-known content-types, handle them as defined in respective RFC (utf8 for application/json, first line or BOM for XML, meta tag in HTML)
• Use chardet to detect from the body

Considerations

• Streaming mode and chunks
• Discrepancies between streaming and display mode: should this only be done for readability when showing to the terminal, or also when piping to other commands? Current decision: only doing this for terminal display. Consider flag for enforcing this for piping as well
• chardet requires downloading the whole body, so it would download a whole video to tell you it’s binary data

[Ousret]

For now, my first choice would be to do it directly using charset-normalizer after HTTPie got the whole body. Or resetting the buffer seek if this is how it is done/checked. (Or maybe storing the body?)

[BoboTiG]

https://github.com/psf/requests/issues/2086 is interesting to follow.