Fails to decode a header that requests can handle

I am trying to read this url

https://www.bitstamp.net/api/v2/trading-pairs-info/

It fails (see https://github.com/theelous3/asks/issues/60) with an exception:

....
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/h11/_readers.py", line 85, in maybe_read_from_SEND_RESPONSE_server
    return class_(headers=list(_decode_header_lines(lines[1:])),
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/h11/_readers.py", line 55, in _decode_header_lines
    matches = validate(header_field_re, line)
  File "/Users/michael/.pyenv/versions/3.6.3/lib/python3.6/site-packages/h11/_util.py", line 96, in validate
    raise LocalProtocolError(msg)

Looking closer, this is the header that seems to cause trouble:

bytearray(b'Set-Cookie: ___utmvafIumyLc=kUd\x01UpAt; path=/; Max-Age=900')

My guess is it’s the \x.

[royfielding]

Note that curl allows all sorts of things specifically because it is a tool used for pentesting and verification. It would be nice if it had some sort of validation mode that highlighted spec errors.

[njsmith]

On further thought, I realized my suggestion above wouldn’t actually handle the case that started this, because there the offending byte is inside a cookie, which means that the client has to be able to send it back to the server 😕. So I guess our options are:

• Loosen the header-value regex to accept any character except \0, \r, \n

• Be spec-compliant about what we send and receive in server mode, but use the loosened rule above in client mode. This would be somewhat annoying to implement, though, because of how h11 is structured (the same h11.Request or h11.Response objects are used by both clients and servers, they validate headers, and they don’t know which role we’re playing – if we were only changing the rules for parsing from the wire, that would be easier, because that’s done by the connection object itself).

So I guess I’m now leaning towards the first option.