Batch deliveries for long running crawlers

Summary

Add a new setting FEED_STORAGE_BATCH that will deliver a file whenever item_scraped_count reaches a multiple of that number.

Motivation

For long running jobs (say we are consuming inputs from a working queue) we may want partial results instead of waiting for a long batch to finish.

Describe alternatives you’ve considered

Of course we can stop and restart a spider every now and then. However, a simpler approach is to have it running as long as required, but delivering partial results.

[BroodingKangaroo]

@dipiana I don’t know exactly, but when I finish I will inform you from this thread.

[ejulio]

@dipiana it is not merged yet. But before merging, we always update the project documentation with the proper details. Maybe @BroodingKangaroo can include a simple tutorial once it is finished