General Message Queues as Storage for Requests

It is common request to use external message queues as a storage for scrapy requests (e.g. https://github.com/scrapy/scrapy/issues/3723). There are several implementations exists:

• https://github.com/rmax/scrapy-redis
• https://github.com/dfdeshom/scrapy-kafka
• https://github.com/mbriliauskas/scrapy-rabbitmq-link

There is nothing special about these. They just came up as first results on a github for scrapy <message_queue_name> request.

Not so long ago integration of non-disk and non-memory queues into scrapy required separate scheduler. Improvements for scrapy scheduler weren’t shared across these implementations.

After merge of https://github.com/scrapy/scrapy/pull/3884 adding different types of queues is much easier and support requires less maintenance.

There is no predefined list for message queues. Redis is a must to have. Others are optional.

[whalebot-helmsman]

Thanks @AKSHAYSHARMAJS

[faizan2700]

I think it is better to scope out bloom filters. It is not directly related not to request queues nor to redis (or other message queue)

Ok