Feature suggest: add downloaded/uptodate status to information about downloaded media

Currently File/Image pipelines populate files/images fields with dicts containing information about the downloaded files (the downloaded path, the original scraped url, and the file checksum). It would be useful to have downloaded/uptodate status in this dict (motivation).

It goes along with other features requests such as having width/height of images also in the dict output.

[ilias-ant]

I think the first approach (in a previous crawl) is sufficient and well-defined, at the right level of abstractness.

The second approach is probably going to face significant definition challenges in Scrapy projects that utilize more than one Media Pipelines or heavily depend on a specific file expiration policy. Since the whole point of the file expiration policy is to “avoid downloading files that were downloaded recently”, I think to take into account the “current crawl” semantics would really confuse the end-user.

Overall, maintaining consistency with the file_status_count statistics is definitely the way.

I hope I did not miss your point on this distinction 😃

[ilias-ant]

this is a great feature suggestion, that is general enough to have many applications on projects that utilize Scrapy!

Hope I find the time to have a try on this soon!

update: did a bit of research and realized that this enhancement is achievable with minimal source code alterations. Namely:

• FilesPipeline.media_to_download#_onsuccess callback should now return: {'url': request.url, 'path': path, 'checksum': checksum, 'status': 'uptodate'}
• FilesPipeline.media_downloaded should now return: {'url': request.url, 'path': path, 'checksum': checksum, 'status': status}
• the necessary testing and documentation considerations

Any feedback on this will be deeply appreciated and if this is indeed the case, I will certainly open a pull request (if that ok with you) 😃