Stream response body for large files

Summary

Using response.body is inefficient when downloading large files because it loads the full body in memory. This is not feasible for large zips for instance. We need something like requests’s response.raw to be able to stream the response body to disk without using the whole RAM.

Code example to highligh the difference:

• scrapy

with open(my_file, 'wb') as f:
    f.write(response.body)
    # loads all of response.body in memory then writes to disk

• requests

with open(my_file, 'wb') as f:
    with requests.get(url, stream=True) as r:
          shutil.copyfileobj(r.raw, f)
          # memory efficient, only small chunks loaded in memory

Motivation

To be able to download larges files when automatically scrapping a full website.

Describe alternatives you’ve considered

An alternative is to use requests to download the file in a deferred async process_item but I think this is not efficient as it won’t use scrapy’s cache.

[elacuesta]

See #4205 as well, which will be released shortly. Unfortunately, while this allows you to read the body in chunks, it doesn’t stop the whole body from being stored in memory. I guess some changes could be done so that we can indicate in the handler whether or not we want the chunk to be appended to the existing body, but it’s not a pretty design IMHO.

Now that we have async/await support, I think a nice API would be something like:

def parse(self, response):
    yield Request("https://example.org", callback=self.parse_stream, stream=True)
    # or maybe
    yield StreamRequest("https://example.org", callback=self.parse_stream)

async def parse_stream(self, response):
    assert response.body is None
    async for data in response.stream_body():
        # do something with the data

[wRAR]

Related to #3880 I think.