Scrapy capitalizes headers for request

I’m setting the headers following way

headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'cache-control': 'no-cache',
    ...
}

And calling request like that:

yield scrapy.Request(url='https:/myurl.com/', callback=self.parse, headers=headers, cookies=cookies, meta={'proxy': 'http://localhost:8888'})

And it makes that scrapy capitalizes all these headers and it looks like that (I’m using Charles proxy for debugging):

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cache-Control: no-cache

And this is not working correctly for my case.

If I’m using curl and set headers lowercase

accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
cache-control: no-cache

everything works like a charm.

Is there any way how I can disable this capitalizing behavior in Scrapy? Thanks for any help!

[kalessin]

A fast solution for this issue. In your spider code:

from twisted.web.http_headers import Headers as TwistedHeaders

TwistedHeaders._caseMappings.update({
    b'cache-control': b'cache-control',
    b'accept': b'accept',
})

that prevents twisted to capitalize those headers. The problem is entirely at twisted side. Nothing is required to be done at scrapy side.

[kmike]

I think it’d be good to not capitalize header names by default, and pass them as-is.