How to get all the items (via Signal API)

Hello there,

I wrote the following code:

 process = CrawlerProcess()
 results = []

 def crawler_results(parse_result):
      results.append(parse_result)

 # The line stuff are some params, not interesting ;)
 process.crawl(BASpider, self.server, line[0], line[2], line[1])
 for p in process.crawlers:
     p.signals.connect(crawler_results, signal=scrapy.signals.item_dropped)
 process.start()

But the method crawler_results is never triggered and I do not understand why. Is it not supposed to work like this? (I mean, it’s not working with item_scrapped or engine_started either) I just get no events triggered

Else, how would your retrieve the items of a spider with CrawlerProcess? (having a pipe is not something handy because I need to send all the data at once).

[kmike]

Hey @lolobosse! I can’t reproduce the issue. The following script works for me, i.e. it prints results:

# -*- coding: utf-8 -*-
import scrapy
import scrapy.signals
from scrapy.crawler import CrawlerProcess


class BASpider(scrapy.Spider):
    name = 'ba'
    start_urls = ['http://example.com']

    def parse(self, response):
        yield {'url': response.url}


process = CrawlerProcess()
results = []

def crawler_results(item, response, spider):
    results.append(item)
    print(results)

process.crawl(BASpider)
for p in process.crawlers:
    p.signals.connect(crawler_results, signal=scrapy.signals.item_scraped)

process.start()

Maybe the problem is that you’re not using a right signature for a signal handler - see item_dropped argument names here: https://doc.scrapy.org/en/latest/topics/signals.html#item-dropped. Note that you must use the same argument names as in docs, because they are passed as keyword arguments.

[kmike]

@lolobosse can we close this issue? Was it an issue with signal argument names?