DownloadLengthMismatchError because requests automatically decodes gzip files

Description of issue or feature request:

If target is a gzip file, served with a Content-Encoding: gzip header, requests will automatically decode the file.

Current behavior:

As a result, the default fetcher for tuf ngclient raises a DownloadLengthMismatchError:

tuf.api.exceptions.DownloadLengthMismatchError: Downloaded x bytes exceeding the maximum allowed length of y

Expected behavior:

From the requests docs:

An important note about using Response.iter_content versus Response.raw. Response.iter_content will automatically decode the gzip and deflate transfer-encodings. Response.raw is a raw stream of bytes – it does not transform the response content. If you really need access to the bytes as they were returned, use Response.raw.

I would expect tuf to access the Reponse.raw instead of Response.iter_content here:

https://github.com/theupdateframework/python-tuf/blob/b5e952cee3edb1c3363a8a7859cb548b13ab71f3/tuf/ngclient/_internal/requests_fetcher.py#L99

Note that Response.raw is a urllib3.response.HTTPResponse, which also decodes gzip encoded content by default, but requests disables this by setting decode_content=False.

[lukpueh]

This is not a bug in the client but in the server. See https://github.com/theupdateframework/python-tuf/pull/1774 and https://github.com/theupdateframework/python-tuf/pull/2048#issuecomment-1188969803 pp. for details.

[dennisvang]

@joshuagl I believe PR #2048 does the job, although I’m not entirely sure my method of testing is acceptable.