Override "Connection" in headers will result in two "Connection" in headers.

If I try to override the “Connection” in the headers with either custome_settings

custom_settings = {
        'DEFAULT_REQUEST_HEADERS': {
            'Connection' : 'Keep-Alive'
        }
}

or within the request headers

headers = {}
 headers['Connection'] : 'Keep-Alive'
 headers['User-Agent'] : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'

for url in urls:
     yield scrapy.Request(url, headers=headers, callback = self.parse)

It will result with a request with two "Connection"s in the header.

GET http://example.com HTTP/1.1
Connection: close
Connection: Keep-Alive
User-Agent: Scrapy/1.4.0 (+http://scrapy.org)
Accept-Encoding: gzip,deflate

[cathalgarvey]

Hey folks, I fired up Wireshark and tried to reproduce this, with no success, on Python 3.5 and Scrapy 1.5.0.

If I issue the following:

>>> req = scrapy.Request('http://httpbin.org/headers', headers={'Connection': 'wat'})
>>> fetch(req)

…then I see in Wireshark the following request firing:

GET /headers HTTP/1.1
Accept-Language: en
Accept-Encoding: gzip,deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: wat
User-Agent: Scrapy/1.5.0 (+https://scrapy.org)
Host: httpbin.org

(I did try sending ‘keep-alive’ first, and that also works as expected. I just felt like mixing it up a bit).

Interestingly, and disappointingly, the response from httpbin.org/headers always includes the header ‘Connection: close’, reducing its usefulness for debugging this issue.

Can anyone provide me with a reproducible test case so I can further test this?

[kmike]

@cathalgarvey in a link @tlinhart posted (https://github.com/scrapy/scrapy/issues/2970#issuecomment-360706840) Connection header is set only when body is not None. So it affects POST, PUT and other requests with non-empty bodies; it doesn’t affect usual GET requests, which your Wireshark experiment shows. Though it uses .setdefault, so it shoudn’t override an existing header value…