HTTP 1 request headers decoded using default encoding instead of ISO-8859-1

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

headers are decoded here without specifying their encoding:

https://github.com/sanic-org/sanic/blob/ad4e526c775fc3ce950503d6476d9d344492b0dd/sanic/http/http1.py#L205

On my system (osx using python 3.10.8 installed via homebrew) this causes bytes that are valid characters in ISO-8859-1 but not in UTF-8 to be decoded as surrogate escape characters, e.g. b"\x80" becomes "\udf80" instead of "\x80"

Code snippet

No response

Expected Behavior

headers encoded as ISO-8859-1 with no MIME type to be decoded correctly without using UTF-8 surrogate escape characters.

How do you run Sanic?

As a script (app.run or Sanic.serve)

Operating System

linux

Sanic Version

22.9.1

Additional context

this used to work as expected in Sanic<=20.12.7

[Tronic]

@relud After a discussion we are now planning to use ISO-8859-1 for request headers within the Sanic built-in server as well, to make it match ASGI and other frameworks that behave this way, also matching the behavior of older Sanic releases. It is noted that ISO-8859-1 can also be encoded back to original bytes, from which one can obtain UTF-8 or other decoding if needed.

As your bug report was apparently the first we’ve received on this, the issue is probably not affecting many at all, but at least this should make your implementation a bit easier. This is a breaking change, so no promises yet on when it will be released, even if everyone else is using ASCII headers and thus isn’t affected by it.

[Tronic]

Earlier Sanic versions handled headers as ISO-8859-1, which was causing trouble when they actually were in UTF-8 (more common nowadays). I had to put a lot of thought into this while reimplementing the HTTP parser code as leaving them as bytes wouldn’t be practical either. The surrogate escape coding is WTF-8 which indeed is meant for preserving garbage, being able to restore original bytes of what might be ill-formed UTF-8. I’m glad you found use for this detail of Sanic’s implementation, being able to restore those bytes instead of simply showing “replacement character” as a naive implementation might.