Needed a possibility to pass start_urls parameter in constructor

I have a Spider that should get its start_urls from an external source: file system, database, etc. It will be extremely useful to pass it directly in constructor. Especially if I want to parse some similar sites on different URLs.

[kmike]

@Felix-neko it should already work; in your example CrawlerProcess API is used incorrectly: process.crawl expects Spider class, not a spider instance. This works for me:

import scrapy
from scrapy.crawler import CrawlerProcess

class HabraSpider(scrapy.Spider):
    name = 'habra'

    def parse(self, response):
        print(response.url)

process = CrawlerProcess()
process.crawl(HabraSpider, start_urls=["http://example.com", "http://example.org"])
process.start()

[nyov]

You can of course override your Spider’s __init__() method to pass any urls from elsewhere. This is rather a support question though, please see http://scrapy.org/community/ for help channels.

As a quick starter, here’s an example:

class MySpider(scrapy.Spider):

    name = "myspider"

    def __init__(self, *args, **kwargs):
        urls = kwargs.pop('urls', []) 
        if urls:
            self.start_urls = urls.split(',')
        self.logger.info(self.start_urls)
        super(MySpider, self).__init__(*args, **kwargs)

run it as scrapy crawl myspider -a "urls=http://localhost/test/,http://localhost"