HTTP 2 support

I peeked at the docs, at the issues but couldn’t find any info about HTTP 2 support.

Does scrapy support it?

[Glennvd]

Hey @kmike, I’m afraid I disagree. I’m seeing an increase in websites (using a specific vendor) that detect bot traffic based on HTTP2/HTTP1.1 version compared to what’s the expected default from your user agent. While I do understand that that does not make it a priority for the Twisted team, I’m sure it has a significant impact for a lot of scrapy users.

[jseyfert]

Hey @kmike, I’m afraid I disagree. I’m seeing an increase in websites (using a specific vendor) that detect bot traffic based on HTTP2/HTTP1.1 version compared to what’s the expected default from your user agent. While I do understand that that does not make it a priority for the Twisted team, I’m sure it has a significant impact for a lot of scrapy users.

Hi, @Glennvd I have encountered multiple sites that blocked scrapy due to HTTP1.1, but i am finding they are only identifying my spider due to scrapy automatically capitalizing the keys of the headers. I have been able to use a workaround that has worked 100% of the time so far.

example This is what the site expects: accept-encoding: gzip, deflate, br accept-language: en-US,en;q=0.9,hi;q=0.8,pt;q=0.7

And this is what scrapy sends(even if you make them lowercase in the spider): Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,hi;q=0.8,pt;q=0.7

This seems to be a workaround to keep the keys lowercase:

headers={
    "":"accept-encoding: gzip, deflate, br",
    "":"accept-language: en-US,en;q=0.9,hi;q=0.8,pt;q=0.7",
},

I hope this helps someone:)