Wrong response.body encoding with http-equiv headers

A Response object doesn’t seem to obey a http-equiv header for Content-Type encoding when it found a HTTP header saying different. So if the http header says ‘utf-8’ but the body content is, say, codepage 1252 and the documents’ http-equiv says 1252, then scrapy appears to still picks utf-8 for decoding body content.

That might be the right decision, but I think it’s wrong. The document itself should know it’s encoding better than a server-wide setting would.

[Gallaecio]

Instead of creating a pull request, I think you could just share the solution here, as people will have to copy-paste it anyway.

[majate]

Thank you for the information and your ideas! Sadly, we have a quite short time frame for our assignment, so we did not have the possibility to change approach after we had started. As of right now, we have implemented a new downloader middleware that is only included in the middleware pipeline if it is enabled in the settings. The processing of the responses is implemented as:

def process_response(self, request, response, spider):

        if isinstance(response, TextResponse) \
                and response._encoding is None \
                and response._headers_encoding() is not None \
                and response._body_declared_encoding() is not None \
                and response._headers_encoding() != response._body_declared_encoding():
            return response.replace(encoding=response._body_declared_encoding())

        return response

This would update the response to obey the encoding defined by the body over the encoding defined in the header, as well as keeping the behaviour of letting an encoding passed in the __init__ method encoding argument remain its priority above both the encoding in the body and the header. Our solution works according to our tests, but there might be some edge cases that we have forgotten to test.

We realise, as you have already mentioned, that it might be better to implement the feature directly in the TextResponse class, instead of needing to process every Response in a middleware. However, this is definitely a quick and easy solution for someone wanting to achieve this behaviour before a better solution is implemented and merged.

Would it be useful for anyone if we created a pull request (just to show our full solution) or would that be unnecessary?