fails on older TLS stacks with OpenSSL 1.1.1pre9

The upcoming adoption of OpenSSL 1.1 (or more specifically, 1.1.1) by Linux distributions might cause problems for requests when checking old websites.

This was first reported in the Debian bugtracker as bug #907807, originally against the linkchecker program (and forwarded upstream as https://github.com/linkchecker/linkchecker/issues/188), but it was found that the requests library directly suffers from this problem as well.

~This issue will probably block requests from being released with buster, the current “testing” release and upcoming (mid 2019?) stable release unless it is somewhat fixed.~ That assertion was incorrect: the bug is currently marked with a “normal” severity which is not blocking release.

The tested sites were:

• https://get.adobe.com/
• https://caniuse.com
• https://www.nada.kth.se/~snilsson/publications/IP-address-lookup-using-LC-tries/

All sites load correctly in Firefox 60.1.0 and although I haven’t tested that with OpenSSL 1.1.1, I doubt it would be affected as Firefox (and Chromium) have their own TLS library (NSS). Also note that urllib3 seems to have no problem loading those sites itself:

>>> import urllib3
>>> http = urllib3.PoolManager()
>>> r = http.request('GET', 'https://get.adobe.com/')
>>> 

Expected Result

All sites should load correctly, and do load correctly in Debian buster (which still has OpenSSL 1.1.0):

$ python
Python 2.7.15+ (default, Aug 31 2018, 11:56:52) 
[GCC 8.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('https://get.adobe.com')
<Response [200]>

Actual Result

(unstable-amd64-sbuild)anarcat@curie:/$ python
Python 2.7.15+ (default, Aug 31 2018, 11:56:52) 
[GCC 8.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
 >>> requests.get('https://get.adobe.com')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
     return request('get', url, params=params, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='get.adobe.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: WRONG_SIGNATURE_TYPE] wrong signature type (_ssl.c:726)'),))
 >>> requests.get('https://www.nada.kth.se')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
     return request('get', url, params=params, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='www.nada.kth.se', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:726)'),))
 >>> requests.get('https://caniuse.com')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
     return request('get', url, params=params, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='caniuse.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: VERSION_TOO_LOW] version too low (_ssl.c:726)'),))

Reproduction Steps

Run a Debian unstable machine to get the latest 1.1.1~~pre9 package. This can be done with Docker with:

docker run -it debian:unstable

If you do not have access to such an environment, the latest OpenSSL code can be found in their project page.

import requests

for url in ('https://get.adobe.com/', 'https://caniuse.com',
'https://www.nada.kth.se/~snilsson/publications/IP-address-lookup-using-LC-tries/'):
    requests.get(url)

System Information

$ python -m requests.help

(unstable-amd64-sbuild)anarcat@curie:/$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  }, 
  "cryptography": {
    "version": ""
  }, 
  "idna": {
    "version": "2.6"
  }, 
  "implementation": {
    "name": "CPython", 
    "version": "2.7.15+"
  }, 
  "platform": {
    "release": "4.17.0-3-amd64", 
    "system": "Linux"
  }, 
  "pyOpenSSL": {
    "openssl_version": "", 
    "version": null
  }, 
  "requests": {
    "version": "2.18.4"
  }, 
  "system_ssl": {
    "version": "10101009"
  }, 
  "urllib3": {
    "version": "1.22"
  }, 
  "using_pyopenssl": false
}

I am aware that we are one minor version behind upstream in Debian, and this is being worked on. But I have reviewed the requests changelog and there didn’t seem to be any change relevant to this. I am therefore going under the assertion that current versions also suffer from the same problem.

Also note that this might be a regression in OpenSSL itself. I am filing this here to see what your opinion is regarding this issue and whether it is something that belongs in requests or the upstream cryptographic library(ies).

[Matir]

The following works for me:

import requests
from requests import adapters
import ssl
from urllib3 import poolmanager


class TLSAdapter(adapters.HTTPAdapter):

    def init_poolmanager(self, connections, maxsize, block=False):
        """Create and initialize the urllib3 PoolManager."""
        ctx = ssl.create_default_context()
        ctx.set_ciphers('DEFAULT@SECLEVEL=1')
        self.poolmanager = poolmanager.PoolManager(
                num_pools=connections,
                maxsize=maxsize,
                block=block,
                ssl_version=ssl.PROTOCOL_TLS,
                ssl_context=ctx)

session = requests.session()
session.mount('https://', TLSAdapter())
session.get(TARGET)

[kroeckx]

If you want to lower the security setting in an other file than /etc/ssl/openssl.cfg you need to use DEFAULT@SECLEVEL=1