question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Warn about br handling if brotlipy is not installed

See original GitHub issue

Currently, the HTTP compression middleware sets the Accept-Encoding header to gzip,deflate or gzip,deflate,br depending on whether or not brotlipy is installed.

However, if the user manually sets the Accept-Encoding header to a value containing br, and does not have brotlipy installed, the server can send a response compressed with br and Scrapy will silently fail to decompress it, so the response content will not be readable.

I think we should improve a bit the handling of this scenario.

If the user sets br manually on the Accept-Encoding header, I think we should:

  • Log a warning if brotlipy is not installed.
  • Have the middleware cause an error if the server sends the response br-encoded, and brotlipy is not installed.
  • Log an information message the first time that the user sends a response with a manually-set Accept-Encoding header which is identical to the one that the HTTP compression middleware would define. It should not be a warning because the user does not need to do anything about it. It should only be displayed once because there is a chance the user is doing it on purpose (e.g. if the default headers define a different value but the user wishes to override that for some requests with the default value.

The 3rd bullet point may not be that useful, but I think the 1st and 2nd are.

Issue Analytics

  • State:open
  • Created 3 years ago
  • Comments:11 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
Gallaeciocommented, Mar 5, 2021

It’s quite a good start, but it would need some adjustments to get merged. It should definitely make things easier for the next person that comes around.

0reactions
wRARcommented, Nov 23, 2022

Alternatively: #4698

Read more comments on GitHub >

github_iconTop Results From Across the Web

python3.8 brotli brotli.error: BrotliDecompress failed?
This is mentioned nowhere in the documentation of requests but once brotli is installed, it is directly handled by Requests.
Read more >
brotlipy - PyPI
Resolved a bug where decompress() would return an empty bytestring instead of erroring if the provided bytestring was small enough.
Read more >
Garmin Connect Integration - Errors
Hi, my Garmin integration doesn't work and I've got a lot of error like: Error occurred during Garmin Connect get activity request: Error ......
Read more >
Scrapy Documentation
If you want to install scrapy with Python 3, install scrapy within a Python ... parse(): a method that will be called to...
Read more >
WhiteNoise Documentation
Also, when used with gunicorn (and most other WSGI servers) the actual business of pushing the file down the network interface is handled...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found