Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

SSL handshake failure

See original GitHub issue

Hi,

I’m getting a handshake error for the sites listed below:

2016-12-03 00:02:19 [scrapy] ERROR: Error downloading <GET https://apnews.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl23_read', 'ssl handshake failure')]>]
2016-12-03 00:03:25 [scrapy] ERROR: Error downloading <GET https://techcrunch.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl23_read', 'ssl handshake failure')]>]
2016-12-03 00:03:53 [scrapy] ERROR: Error downloading <GET https://medium.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]>]
2016-12-03 00:05:08 [scrapy] ERROR: Error downloading <GET https://theintercept.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl23_read', 'ssl handshake failure')]>]
2016-12-03 00:06:32 [scrapy] ERROR: Error downloading <GET https://www.opendemocracy.net/>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]>]
2016-12-03 00:07:55 [scrapy] ERROR: Error downloading <GET https://www.rt.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl23_read', 'ssl handshake failure')]>]
2016-12-03 00:19:53 [scrapy] ERROR: Error downloading <GET https://www.thestar.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]>]
2016-12-03 00:58:42 [scrapy] ERROR: Error downloading <GET https://www.cnet.com/>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]>]

What’s strange is that it works if I try each one of those sites individually using scrapy shell so I might be doing something wrong.

Here’s some information about my environment:

$ scrapy version -v
Scrapy    : 1.2.1
lxml      : 3.6.4.0
libxml2   : 2.9.4
Twisted   : 16.6.0
Python    : 2.7.12 (default, Jul  1 2016, 15:12:24) - [GCC 5.4.0 20160609]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.2g-fips  1 Mar 2016)
Platform  : Linux-3.13.0-52-generic-x86_64-with-Ubuntu-16.04-xenial

Any ideas where I can look to troubleshoot the problem?

Issue Analytics

State:
Created 7 years ago
Reactions:1
Comments:24 (10 by maintainers)

Top GitHub Comments

1reaction

singhalhimanshucommented, Mar 15, 2019

Getting the same error: Website need to crawl: [“https://www.labor.ny.gov/”] Installed scrapy details Scrapy : 1.6.0 lxml : 4.2.5.0 libxml2 : 2.9.8 cssselect : 1.0.3 parsel : 1.5.1 w3lib : 1.20.0 Twisted : 18.7.0 Python : 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)] pyOpenSSL : 19.0.0 (OpenSSL 1.0.2p 14 Aug 2018) cryptography : 2.3.1 Platform : Windows-10-10.0.17763-SP0

ERROR: Retrying <GET https://www.labor.ny.gov> (failed 1 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [(‘SSL routines’, ‘ssl23_read’, ‘ssl handshake failure’)]>]

0reactions

cathalgarveycommented, Feb 15, 2018

Hi @YPersonal - This particular issue has gone stale, so I’ll close it. But the SSL/TLS issue continues in other issues here, and in some cases it’s caused by the evolution of the binary builds of PyOpenSSL and Cryptography for various platforms. Sometimes, ciphers get removed wholesale from these libraries as a security improvement for application-level users, and it affects use-cases where security and confidentiality is not as urgent, such as most web-scraping tasks.

Thank you for your contribution to Scrapy so far!