Extreme performance of the waste when CONCURRENT_ITEMS is large

Description

There is Extreme performance of the waste When CONCURRENT_ITEMS set to a large number, such as 9999.

Some days ago, I wrote a spider with CONCURRENT_ITEMS=9999, and run it. I found my spider use 100% cpu and crawl speed is very very slow （40 pages/mins）, normally it would be 400+pages/min.

Steps to Reproduce

Here is code , you can run directly. You would see cpu usage up to 100%. Wait 1 minute, you will see [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 27 pages/min)

import scrapy


class TestSpider(scrapy.Spider):
    name = 'test'

    custom_settings = {
        "LOG_LEVEL": "INFO",
        "CONCURRENT_ITEMS": 99999,
    }

    def start_requests(self):
        for _ in range(3000):
            yield scrapy.Request(
                url="http://httpbin.org/get",
                dont_filter=True,
            )

    def parse(self, response):
        pass

[Gallaecio]

I suspect that might be to be expected.

I believe the root code for this is https://github.com/scrapy/scrapy/blob/7e23677b52b659b11471a63f3be9905a0bbaf995/scrapy/utils/defer.py#L72 (count comes from CONCURRENT_ITEMS). Quoting the first paragraph of the blog post linked from there as the source of this implementation:

Concurrency can be a great way to speed things up, but what happens when you have too much concurrency? Overloading a system or a network can be detrimental to performance.

So maybe the best way forward here is to treat this as a documentation issue, and document how increasing this number may be counter-productive.

I also wonder if it would make sense to count the number of elements in iterable (assuming they are not infinite) and limit count to the lesser value, i.e. count = min(count, iterable_length). Maybe we could even implement this while supporting an arbitrarily large iterable, by peaking items from iterable up to count.

[vgerak]

@edorado93 's proposal sounds very good, better than my “sane defaults” approach. Withdrawing my PR, looking forward to the implementation!