how to run Multiple spider ** independently ** at the same time ?

when i wrote two simple spider

class Bing(Spider):

    name = "bing"

    def start_requests(self):
        for _ in range(1000):
            yield Request("http://bing.com", dont_filter=True)

    def parse(self, response):
        print(self.name, response.url)

class Sogou(Spider):
    name = "sogou"
    def start_requests(self):
        for _ in range(1000):
            yield Request('http://sogou.com', dont_filter=True)

    def parse(self, response):
        while True:
            print('123')
        print(self.name, response.url)

def run(spider_cls):
    crawler = Crawler(spider_cls)
    crawler.crawl()

run(Bing)
run(Sogou)

reactor.run()

just like this. Use Crawler and reactor to run these spider. Sogou spider was blocked, it was obvious. but Bing spider was blocked too, why ? and how to solve it . use Crawler class make multiple spider run independently at the same time? I used to use twisted.threads like deferToThread , or callInThread , it couldn’t work . Can someone have some idea? Thank you very much!

[grammy-jiang]

hi, @xiaochonzi

Please refer to the section Running multiple spiders in the same process in the document Common Practices — Scrapy 1.5.0 documentation. There is a very clear example for your question.

[grammy-jiang]

Hi, @xiaochonzi

Please, please read the section I mentioned before CAREFULLY. The document provides a very clear way to do what you want.