Allow consumption of requests

Currently, in the run method of the core Stoq class, a provider can only add payloads to the payload_queue. When a payload is consumed in the _consume method, a Request is created with the request_meta that is provided to the run method. This does not allow me to specify specific request_meta for each payload added to the queue by the provider plugin. I propose that the payload_queue be renamed provider_queue and allow Requests to be added to this queue: Here are some of the changes I would make:

 async def _consume(
        self,
        provider_queue: asyncio.Queue,
        request_meta: Optional[RequestMeta] = None,
        add_start_dispatch: Optional[List[str]] = None,
    ) -> None:
        while True:
            try:
                task = await provider_queue.get()
                # Determine whether the provider has returned a `Payload`, or a task.
                # If it is a task, load the defined archiver plugin to load the
                # `Payload`, otherwise, simply continue on with the scanning.
                if isinstance(task, Payload):
                    request = Request([task], request_meta)
                    await self.scan_request(request, add_start_dispatch)
                elif isinstance(task, Request):
                    await self.scan_request(request, add_start_dispatch)
                else:

To show you the benefit of this, currently the dirmon public plugin sets the source_dir in the payload_meta, when really, I would argue such information belongs in the request_meta. If the framework supported providing Requests, dirmon could be modified as follows:

self.log.info(f'Monitoring {self.source_dir} for newly created files...')
async for changes in awatch(self.source_dir):
    for change in list(changes):
        event = change[0]
        src_path = os.path.abspath(change[1])
        # Only handle Change.added
        if event != 1:
            continue        
        payload_meta = PayloadMeta(
            extra_data={
                'filename': os.path.basename(src_path),
            }
        )
        request_meta = RequestMeta(
            extra_data={
                'source_dir': os.path.dirname(src_path),
            }
        )
        with open(src_path, 'rb') as f:
            payload = Payload(f.read(), payload_meta)
            request = Request([payload], request_meta)
            await queue.put(request)

Let me know if you think this is a worthwhile enhancement.

[ytreister]

I agree that it would have to be documented, I will submit a PR when I have time to add this enhancement.

[brbickel]

Thanks for the reminder @ytreister. Looks good, merged into master.