urllib3 incorrectly folds/concatenates all HTTP headers (except for `set-cookie`), causing issues with HSTS in particular

If a site returns multiple HSTS heads:

Strict-Transport-Security: max-age=31536000
Strict-Transport-Security: max-age=31536000; includeSubDomains

urllib3 folds these into a single header:

Strict-Transport-Security: max-age=31536000, max-age=31536000; includeSubDomains

http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 has:

Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one “field-name: field-value” pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded

Since the field-value for HSTS is not defined as a comma-separated list and since doing this does change the semantics of the message (by resulting in an invalid HSTS header), this concatenation shouldn’t be happening.

https://github.com/mozilla/http-observatory/issues/363#issuecomment-434011250 points to https://github.com/urllib3/urllib3/pull/534/files#diff-01955f24bc4d0d621454698a584ab854R290 as being the problem and proposes some ways to fix this.

[globau]

thanks for your time and for pointing me in the right direction.

[sethmlarson]

I’m going to mark this as closed as there are methods of getting header values without the folding.