Scrapy is slow to start the first time

Hi,
This is not an issue for me, but I’m really curious about why it works this way. I guess this is in general for all python apps, although it’s more prominent in scrapy.
The first time I start a scrapy spider, it took:

real    0m40.704s
user    0m3.547s
sys     0m3.484s

Any subsequent runs took around 6 seconds:

real    0m6.499s
user    0m3.266s
sys     0m1.547s

Is some sort of caching happening here? I’m running an SSD and scrapy has around 80 spiders.

[wRAR]

scrapy crawl foo imports all spider modules because foo is the spider name attribute, not the module name. If importing your modules takes that much time we can’t help with that, but you can try moving imports of heavy modules from the top level to code that only runs when the spider is used.

[elacuesta]

I find it very difficult to give a general answer without any knowledge of the code being run. Off the top of head, I can think about the resuming jobs feature and the HttpCacheMiddleware middleware. I suspect this is more of a support question (see Getting help), but let’s leave it open for now in case we find some bottleneck within the codebase. Could you provide more information?