Warn about br handling if brotlipy is not installed
See original GitHub issueCurrently, the HTTP compression middleware sets the Accept-Encoding
header to gzip,deflate
or gzip,deflate,br
depending on whether or not brotlipy is installed.
However, if the user manually sets the Accept-Encoding
header to a value containing br
, and does not have brotlipy
installed, the server can send a response compressed with br
and Scrapy will silently fail to decompress it, so the response content will not be readable.
I think we should improve a bit the handling of this scenario.
If the user sets br
manually on the Accept-Encoding
header, I think we should:
- Log a warning if
brotlipy
is not installed. - Have the middleware cause an error if the server sends the response br-encoded, and
brotlipy
is not installed. - Log an information message the first time that the user sends a response with a manually-set
Accept-Encoding
header which is identical to the one that the HTTP compression middleware would define. It should not be a warning because the user does not need to do anything about it. It should only be displayed once because there is a chance the user is doing it on purpose (e.g. if the default headers define a different value but the user wishes to override that for some requests with the default value.
The 3rd bullet point may not be that useful, but I think the 1st and 2nd are.
Issue Analytics
- State:
- Created 3 years ago
- Comments:11 (6 by maintainers)
Top Results From Across the Web
python3.8 brotli brotli.error: BrotliDecompress failed?
This is mentioned nowhere in the documentation of requests but once brotli is installed, it is directly handled by Requests.
Read more >brotlipy - PyPI
Resolved a bug where decompress() would return an empty bytestring instead of erroring if the provided bytestring was small enough.
Read more >Garmin Connect Integration - Errors
Hi, my Garmin integration doesn't work and I've got a lot of error like: Error occurred during Garmin Connect get activity request: Error ......
Read more >Scrapy Documentation
If you want to install scrapy with Python 3, install scrapy within a Python ... parse(): a method that will be called to...
Read more >WhiteNoise Documentation
Also, when used with gunicorn (and most other WSGI servers) the actual business of pushing the file down the network interface is handled...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
It’s quite a good start, but it would need some adjustments to get merged. It should definitely make things easier for the next person that comes around.
Alternatively: #4698