User timeout caused connection failure

Hello, when I send requests at the same time(with dont_filter=True, because some will be error, and the ‘DOWNLOAD_TIMEOUT’: 60), some requests appear the info as following: Retrying <GET http://www.xxx> (failed 1 times): User timeout caused connection failure: Getting http://www.xxx took longer than 1800 seconds.. It doesn’t appear other error, and some log that have crawled XX pages, and XX items. I know the log from https://github.com/scrapy/scrapy/blob/master/scrapy/core/downloader/webclient.py but I don’t know why. Can someone help me?

[nyov]

The error message Retrying <GET http://www.xxx> (failed 1 times): User timeout caused connection failure: Getting http://www.xxx took longer than 1800 seconds.. says:

1. Downloader: “I timed out trying to fetch http://www.xxx in 1800 seconds, sorry”
2. RetryMiddleware: “I am retrying the Request that failed with Error I timed out trying to fetch http://www.xxx in 1800 seconds, sorry”

[nyov]

Yes, if you are using Crawlera with scrapy-crawlera, then there is a CRAWLERA_DOWNLOAD_TIMEOUT that defaults to 1800 seconds. It seems to override the standard DOWNLOAD_TIMEOUT setting when enabled.

But what is your problem, here? Do you want to change the timeout? Do you think there is a bug with the timeout? Or something else?

You can change the timeout by setting CRAWLERA_DOWNLOAD_TIMEOUT = 60 in your settings (or CRAWLERA_DOWNLOAD_TIMEOUT = DOWNLOAD_TIMEOUT). I think setting request.meta['download_timeout'] = 60 on a specific request should also work.