Explicit "content-length" in header leads to incorrect HTTP request

Description

Explicit “content-length” in header leads to incorrect HTTP request - 400 error from target

Steps to Reproduce

Request("https://webhook.site/<YOUR ID>", method='POST', headers={}) >>> OK

Request("https://webhook.site/<YOUR ID>", method='POST', headers={'content-length': '0'}) >>> ERROR: <twisted.python.failure.Failure scrapy.spidermiddlewares.httperror.HttpError: Ignoring non-200 response>

Expected behavior: [What you expect to happen] OK

Actual behavior: [What actually happens] ERROR

Reproduces how often: [What percentage of the time does it reproduce?] 100%

Versions

2.4.1

Additional context

Seems twisted duplicates content-length field in the resulting request. I came across this from standard workflow, when I copy the request from browser (import to curl), test it with curl (ok), implement in scrapy (fails).

[Gallaecio]

I am not sure what the best approach is.

I mean, ideally the user should not set the Content-Length header at all, and let Scrapy (Twisted) do that.

If a user sets the header, we could drop it from the download handlers to avoid this issue. However, doing so silently could be confusing. For example, imagine the user sets the wrong header value. Do we ignore the user-defined value without any feedback?

Maybe we should error out on the client side (i.e. do not let the request reach the target server) on requests with the header, and ask the user to remove the header. And if someone has a valid use case for setting Content-Length to the wrong value, let them explain and defend that use case in a GitHub feature request.

We should agree on the best solution before someone tries and implements the wrong one.

[mmitropoulou]

Hello @wRAR! @marlenachatzigrigoriou and I would like to contribute to this issue. We are new to scrapy so we would like to have your guidance. Thank you!