Invalid DNS pattern

Hello,

We are using https proxies to crawl some website and sometimes i get the following stack trace:

Error during info_callback
Traceback (most recent call last):
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 315, in dataReceived
    self._checkHandshakeStatus()
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 235, in _checkHandshakeStatus
    self._tlsConnection.do_handshake()
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1442, in do_handshake
    result = _lib.SSL_do_handshake(self._ssl)
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 933, in wrapper
    callback(Connection._reverse_mapping[ssl], where, return_code)
--- <exception caught here> ---
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1102, in infoCallback
    return wrapped(connection, where, ret)
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/scrapy/core/downloader/tls.py", line 67, in _identityVerifyingInfoCallback
    verifyHostname(connection, self._hostnameASCII)
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/service_identity/pyopenssl.py", line 44, in verify_hostname
    cert_patterns=extract_ids(connection.get_peer_certificate()),
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/service_identity/pyopenssl.py", line 73, in extract_ids
    ids.append(DNSPattern(n.getComponent().asOctets()))
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/service_identity/_common.py", line 156, in __init__
    "Invalid DNS pattern {0!r}.".format(pattern)
service_identity.exceptions.CertificateError: Invalid DNS pattern '194.167.13.105'.

I think this issue is somewhat similar to https://github.com/scrapy/scrapy/issues/2092 and this ‘invalid DNS pattern’ error should be caught similarly as the ‘Invalid DNS-ID’. What do you think ?

In the meantime, how can i catch it myself and silent it ?

[dangra]

@oximer We are looking forward to release Scrapy 1.6.0 by next week.

But you don’t need git to install Scrapy from master branch, you can include the following line in your requirements.txt instead:

https://github.com/scrapy/scrapy/archive/6f5c39d.zip#egg=Scrapy

6f5c39d is the commit id for the last commit on master branch at the moment.

[jslay88]

It takes about 20s to look at the file structure to realize requirements.txt has been renamed to requirements-py2.txt and requirements-py3.txt accordingly depending on the env you are running.