High CPU

Scrapy uses a lot of CPU on our server, 40% is the minimum (when running 1 spider) and most of the time it goes up to 99% (running multiple spiders at the same time). We’re using a DigitalOcean droplet with 1 CPU.

Is this normal? I tried already several things:

• add sleep(0.1) to Pipelines
• disabling Pipelines
• Using Scrapyd with job persistance.
• set the amount of jobs per cpu with settings like below. scrapy.cfg:

[scrapyd]    
max_proc = 0    
max_proc_per_cpu = 2
Which didn't help either. These are my stats when running a spider:

Nothing helps. These are my stats when I run one scraper:

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 3221 scrapy    20   0  836724  75568   8432 R 40.5  7.4   1:04.31 python
 3231 scrapy    20   0  836724  75568   8432 S  1.0  7.4   0:01.40 python
 3234 scrapy    20   0  836724  75568   8432 S  1.0  7.4   0:01.42 python
 3235 scrapy    20   0  836724  75568   8432 S  1.0  7.4   0:01.44 python
 3228 scrapy    20   0  836724  75568   8432 S  0.7  7.4   0:01.43 python

I found out that if I set the variable CONCURRENT_REQUESTS = 1 the CPU stays below the 40% when running 1 spider, but it takes 3 times as long to finish.

I also tried to turn off every extension, middleware and pipeline, but that didn’t reduce the CPU load either. Any ideas?

[ErikvdVen]

Thanks a lot!!! Guess it’s working:

I added these settings:

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 0.1
AUTOTHROTTLE_MAX_DELAY = 0.5

Result:

[nramirezuy]

@ErikvdVen yw, if you have anymore doubts you can reach us on scrapy-users or #scrapy IRC.