httpie refuses to trust a self-signed certificate

I don’t entirely understand the discussion for issue #480 but I think the root cause may be the same for this issue. I need httpie to trust a self-signed certificate. Technically it is a self-signed intermediate certificate authority, but I can use one of the actual server certificates from that authority and get the same result. This certificate satisfies every other application on my Mac (OS 10.12), including curl. Httpie simply won’t trust it. Sample output:

$ http --debug -j --verify=/usr/local/etc/openssl/certs/intermediate-authority.pem https://www.testdomain.com/robots.txt
HTTPie 0.9.9
Requests 2.12.3
Pygments 2.1.3
Python 3.6.1 (default, Mar 24 2017, 17:14:46)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
/usr/local/Cellar/httpie/0.9.9/libexec/bin/python3.6
Darwin 16.5.0

<Environment {
    "colors": 256,
    "config": {
        "__meta__": {
            "about": "HTTPie configuration file",
            "help": "https://github.com/jkbrzt/httpie#config",
            "httpie": "0.9.4"
        },
        "default_options": "[]"
    },
    "config_dir": "/Users/username/.httpie",
    "is_windows": false,
    "stderr": "<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>",
    "stderr_isatty": true,
    "stdin": "<_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>",
    "stdin_encoding": "UTF-8",
    "stdin_isatty": true,
    "stdout": "<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>",
    "stdout_encoding": "UTF-8",
    "stdout_isatty": true
}>

>>> requests.request(**{
    "allow_redirects": false,
    "auth": "None",
    "cert": "None",
    "data": "",
    "files": {},
    "headers": {
        "Accept": "application/json, */*",
        "Content-Type": "application/json",
        "User-Agent": "HTTPie/0.9.9"
    },
    "method": "get",
    "params": {},
    "proxies": {},
    "stream": true,
    "timeout": 30,
    "url": "https://www.testdomain.com/robots.txt",
    "verify": "/usr/local/etc/openssl/certs/intermediate-authority.pem"
})


http: error: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) while doing GET request to URL: https://www.testdomain.com/robots.txt
Traceback (most recent call last):
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 588, in urlopen
    self._prepare_proxy(conn)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 801, in _prepare_proxy
    conn.connect()
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 323, in connect
    ssl_context=context)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__
    self.do_handshake()
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 624, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/http", line 11, in <module>
    load_entry_point('httpie==0.9.9', 'console_scripts', 'http')()
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/__main__.py", line 11, in main
    sys.exit(main())
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/core.py", line 227, in main
    log_error=log_error,
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/core.py", line 99, in program
    final_response = get_response(args, config_dir=env.config.directory)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/client.py", line 70, in get_response
    response = requests_session.request(**kwargs)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

If I use --verify=no, the connection is accepted. I’ve tried converting the certificate file to DER format instead of PEM with no change.

Httpie was installed via Homebrew, which installed its own Python3 as a dependency.

[alvis]

Trusting a self-signed certificate is never encouraged. It is dangerous if a program accepts it by default. A hacker can simply inject a self-signed one to make the users feeling secure while it is not the case.

However, I would suggest httpie to response the issue of using a self-signed certificate with a clearer message rather than simply fail it. A suggestion to use --verify=no to disable the verification for testing purpose would be great.

[squarism]

SSL is good, probably devs know that. Disabling SSL because of corner cases and language limitations or whatever stinks. I know people just want to get work done. I just wonder when we’ll say SSL is good and have the same gut reaction at saying “disable SSL verification”. If you disable SSL verification, SSL is off. But SSL is good. We all know that right? Is it confusing or common knowledge that SSL verification is really what the whole SSL thing is? If postman has an option to disable SSL verification is that just a fancy way of saying something that is potentially insecure? curl has that same option as a flag called --insecure. Is it just a fancy way that hides exactly what the problem is (it turns off SSL)? I think it’s fine to test in Postman. TIL that Postman has that option. Am I going to turn it back on? Or did I just disable SSL for every current and future project? Am I sending API keys from a coffee shop? 😐

I hope https dev machines, self signed certs and corporate CAs are supplanted by scriptable automatic letsencrypt certs one day. It’s a shame that python makes http less organic at trusting the system keychain / ssl store. It’s a curl replacement except in this area. I guess golang has operating system detection for all this stuff. So a modern curl replacement (like bat) that’s written in go would have pretty output, a nice CLI experience and handle custom CAs or ssl termination (load balancers).