Differences between Windows and Linux when handling HTTPS requests through HTTP proxy

Hi,

Let me explain the problem I was facing and how I end up here. I’ll try to do a brief summary.

I work in a company that protects access to the Internet through a proxy that uses NTLMv2 for authentication. This is not a problem for Windows computers but it is a pain when working with GNU/Linux machines. Anyway, there is a great solution for that: CNTLM. It is possible to create a local proxy that is able to authenticate with the corporate proxy and then, by setting http_proxy, https_proxy and no_proxy env variables and configuring properly some specific tools that do not use this variables (apt, yum, git, docker, etc.), voilà, Internet for everyone! No problems so far with two exceptions: pip and conda.

For those struggling with the same issue, please have a look at this open related issue: Doesn’t work behind proxy in corporate Windows network (NTLM). I was able to use pip with Linux machines by setting up a local Nexus repository and adding pypi as a proxy repo. Yes, Nexus is able to authenticate with the NTLM proxy just fine.

But, why do I think this is a urllib3 issue, too? Sure, some interesting feature to add would be allowing NTLM authentication (see https://github.com/urllib3/urllib3/issues/242), but this is not what I am asking here (sure there are more interesting things to implement before an old authentication method). The problem is when I noticed that pip and conda work just fine in Windows with CNTLM, but not in Linux. Same CNTLM, python and urllib3 versions. And the problem is that urllib3 does not work properly when doing https requests through a proxy in Linux. I will try to have a look at the code, but I write these issue just in case more familiar with urllib3 can help 😃

How to replicate:

1. Install and configure CNTLM^✤
2. Create a virutalenv or conda env with python 3
3. Install urllib3 with pip (I tried version 1.23)
4. Execute the following in Windows and Linux:

import urllib3
proxy = urllib3.ProxyManager()
proxy.request('GET', <any http site>)
proxy.request('GET', <any https site>)

^✤: A similar proxy can be simulated in GNU/Linux with Squid + Samba + NTLMv2 auth. Also, have a look at this comment: it is possible to set it up with Apache.

In Windows both requests works well. I just get the (expected) warning:

InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

On the other hand, Linux http requests works but https one fails:

HTTPSConnectionPool(host=‘<https url>’, port=443): Max retries exceeded with url: / (Caused by ProxyError(‘Cannot connect to proxy.’, OSError(‘Tunnel connection failed: 407 Proxy Authentication Required’,)))

It looks like packet is not properly managed by CNTLM. Of course, this is the same error I get when trying to use pip and conda or when using python requests module.

Any ideas? Do you know any OS dependent feature that may be causing this?

Thanks!

[dopstar]

@YuMan-Tam your snippet worked well. I did this:

• copied and reworked requests-ntlm library into requests-ntlm2 library
  • when requests-ntlm and/or urllib3 finally addresses this I can deprecate requests-ntlm2 and archive the repo
• created requests_ntlm2.connection.VerifiedHTTPSConnection which inherit from urllib3.connection.VerifiedHTTPSConnection and I overridden its _tunnel() method to be like your snippet
• created requests_ntlm2.adapters.HttpNtlmAdapter which is responsible of monkey-patching pool classes in urllib3.poolmanager AND sending ntlm credentials downstream.

repo is here: https://github.com/dopstar/requests-ntlm2

[YuMan-Tam]

I had exactly the same issue and found a hack. It has nothing to do with the OS.

The source of this issue lies in http.client. When an https request is made via an NTLM proxy, the function _tunnel() in the HTTPconnection class is called. The original code proceeds if the return code is 200. For NTLM proxy, the return code is 407 so the following code is called.

if code != http.HTTPStatus.OK:
    self.close()
    raise OSError("Tunnel connection failed: %d %s" % (code,message.strip()))

The key is that, failed or otherwise, the code only sends one request and then returns. Therefore, NTLM would work if you modify the code to perform the dances prior to the above block. This hack works on my work Windows machine. I don’t think it’s a “solution” but works in (and only in) this particular case.