'Failed to parse headers' warning logged when getting message/rfc822 content

I’ve been investigating an issue I’ve recently discovered when retrieving objects from S3. I’ve now tracked it to urllib3; this test case (which I’ve added to urllib3/test/with_dummyserver/test_socketlevel.py) demonstrates it:

class TestOkayHeaders(SocketDummyServerTestCase):

    def _test_okay_header_parsing(self, header):
        self.start_response_handler((
           b'HTTP/1.1 200 OK\r\n'
           b'Content-Length: 0\r\n'
           ) + header + b'\r\n\r\n'
        )

        pool = HTTPConnectionPool(self.host, self.port, retries=False)
        self.addCleanup(pool.close)

        with LogRecorder() as logs:
            pool.request('GET', '/')

        for record in logs:
            assert 'Failed to parse headers' not in record.msg

    def test_header_text_plain(self):
        self._test_okay_header_parsing(b'Content-type: text/plain')

    def test_header_message_rfc822(self):
        self._test_okay_header_parsing(b'Content-type: message/rfc822')

The test with text/plain passes, while the test with message/rfc822 fails, and this is logged:

Failed to parse headers (url=http://localhost:36732/): Unknown, unparsed data: [<http.client.HTTPMessage object at 0x7f8fab9373c8>]
Traceback (most recent call last):
  File "/home/user/git/urllib3/src/urllib3/connectionpool.py", line 396, in _make_request
    assert_header_parsing(httplib_response.msg)
  File "/home/user/git/urllib3/src/urllib3/util/response.py", line 68, in assert_header_parsing
    raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)
urllib3.exceptions.HeaderParsingError: Unknown, unparsed data: [<http.client.HTTPMessage object at 0x7f8fab9373c8>]

While retrieving content of type message/rfc822 still works, the warning message being logged is incorrect and unhelpful.

[sethmlarson]

Oh I wasn’t going to put in the fix that’s in the issue, that’s just what I dreamt up to be able to see the unit test pass. 😃 I want to bring full coverage to this function for sure so I’ll have to explore that further and research why this code branch was added in the first place.

Doing a quick google search for unparsed_data python http has yielded a few StackOverflow questions where unparsed_data is byte data which is what I assume the original author of this code branch intended but I haven’t found an example where only the unparsed_data is set and defects aren’t.

[vattikondahemanth]

for me the backend application to which the url is pointing is directly returning the string I changed it to return Response(response=original_message,status=200, content_type=‘application/text’)

in the start i just returned text like

return original_message

I think this answer works only for my case