Provide an option to change the default behaviour of using https?

Some background on the chat in irc#pip

08:13 < feniix> is there a way to make pip use http instead of https ? I have an unfriendly proxy that is preventing my use of pip completely
08:15 < jezdez> feniix: you mean accessing pypi?
08:16 < feniix> jezdez: I mean using pip to install packages, when it reaches the https host ( https://pypi.python.org/pypi) it blows up
08:18 < feniix> jezdez: apparently debian and ubuntu link urllib with curl+gnutls and gnutls fails with some content filtering proxies :(
08:18 < jezdez> feniix: can you paste how it blows up?
08:18 < feniix> yep one min
08:19 < feniix> jezdez: https://gist.github.com/feniix/5756751
08:19 < jezdez> feniix: if your proxy provider uses a self signed cert that may happen
08:20 < feniix> jezdez: it is not "my provider" it is my company and the stupid security people.
08:20 < jezdez> feniix: sorry, I didn't know
08:20 < jezdez> but yeah, this is pretty bad
08:20 < jezdez> pip has a --cert parameter, if you get the root certificate the ssl of the proxy was signed with
08:21 < feniix> jezdez: ill try that
08:21 < jezdez> the file needs to be in the same format as the cacert.pem we ship with pip
08:21 < jezdez> which is the mozilla cert file
08:22 < jezdez> that problem is probably that your "security" people intercept the request to pypi, change the cert on the fly to something that doesn't 
                have the correct hostname set
08:22 < feniix> jezdez: where do I find that file in the installation ?
08:23 < jezdez> feniix: https://github.com/pypa/pip/blob/develop/pip/cacert.pem
08:23 < jezdez> you know what, maybe that won't even work
08:23 < jezdez> since the cert won't match the hostname
08:24 < jezdez> I guess your best bet is to set up an internal pypi mirror
08:24 < jezdez> see https://pypi.python.org/pypi/bandersnatch
08:24 < jezdez> you can tell pip to use it instead then with the -i option
08:24 < jezdez> dstufft should probably doublecheck everything I said about that, he's our sec person
08:25 < feniix> jezdez: the other option that is not very nice is change back to pip 1.2.x
08:25 < feniix> right?
08:25 < jezdez> feniix: yeah. we've put those security checks in place for a reason
08:26 < jezdez> I admit it's not a pleasant situation for you
08:26 < feniix> jezdez: it'd be nice to be able to do --no-secure-connection or something 
08:26 < jezdez> maybe dstufft can provide you with some arguments to ask your security people to relax the situation
08:27 < jezdez> feniix: understood, but to be honest I don't think that would be good
08:28 < jezdez> that said, can you open an issue so we can discuss this further?
08:28 < jezdez> maybe we find a better solution
08:28 < feniix> jezdez: my security people is not very reasonable, i have the same issue with github+https, the solution is to rebuild git and pycurl to not 
                link with gnutls
08:28 < feniix> jezdez: and that is not a sensible solution...
08:29 < feniix> jezdez: the solution according to the security guy
08:29 < jezdez> feniix: *facepalm*

So basically when using pip on an ubuntu/debian box, python or the libraries that python uses to access https it fails with this error:

Could not fetch URL https://pypi.python.org/simple/mako/: There was a problem confirming the ssl certificate: <urlopen error [Errno 1] _ssl.c:499: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed>
Will skip URL https://pypi.python.org/simple/mako/ when looking for download links for mako in /usr/lib/python2.7/dist-packages
Could not fetch URL https://pypi.python.org/simple/: There was a problem confirming the ssl certificate: <urlopen error [Errno 1] _ssl.c:499: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed>
Will skip URL https://pypi.python.org/simple/ when looking for download links for mako in /usr/lib/python2.7/dist-packages
Cannot fetch index base URL https://pypi.python.org/simple/
Could not fetch URL https://pypi.python.org/simple/mako/: There was a problem confirming the ssl certificate: <urlopen error [Errno 1] _ssl.c:499: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed>
Will skip URL https://pypi.python.org/simple/mako/ when looking for download links for mako in /usr/lib/python2.7/dist-packages
Could not find any downloads that satisfy the requirement mako in /usr/lib/python2.7/dist-packages
Downloading/unpacking mako
No distributions at all found for mako in /usr/lib/python2.7/dist-packages
Storing complete log in /home/otaeguis/.pip/pip.log

I am in an environment in which all outgoing connections are intercepted and MiTMed by a “transparent” proxy, the MiTM “attack” is performed by getting the pub cert from the destination website and and re-issuing it with the internal CA (which is already part of the keystores of all the machines).

One of the things I tried is to add our CA to the cacerts file in /usr/local/lib/python2.7/dist-packages/pip/cacert.pem, when I do that I get this other error:

sudo pip install --upgrade mako
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 139, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 266, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 960, in prepare_files
    req_to_install, self.upgrade)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 125, in find_requirement
    page = self._get_page(main_index_url, req)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 353, in _get_page
    return HTMLPage.get_page(link, req, cache=self.cache)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 471, in get_page
    resp = urlopen(url)
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 143, in __call__
    response = self.get_opener(scheme=scheme).open(url)
  File "/usr/lib/python2.7/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 123, in https_open
    return self.do_open(self.specialized_conn_class, req)
  File "/usr/lib/python2.7/urllib2.py", line 1165, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib/python2.7/httplib.py", line 955, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 989, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 951, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 811, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 773, in send
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 107, in connect
    match_hostname(self.sock.getpeercert(), self.host)
  File "/usr/local/lib/python2.7/dist-packages/pip/backwardcompat/ssl_match_hostname.py", line 53, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
CertificateError: hostname 'pypi.python.org' doesn't match either of '*.a.ssl.fastly.net', 'fast.wistia.com', 'purge.fastly.net', 'app.fastly.com', 'api.fastly.com', 'www.fastly.com', 'blog.fastly.com', 'mirrors.fastly.net', '*.imgix.net', 'api.metacpan.org', 'www.metacpan.org', 'signin.epek.com', '*.parsecdn.com', '*.fastssl.net', 'voxer.com', 'www.voxer.com', '*.firebase.com', 'sites.yammer.com', 'sites.staging.yammer.com', '*.skimlinks.com', '*.skimresources.com', 'assets.wantful.com', 'cdn.shazam.com', 'cdn.thinglink.me', '*.fitbit.com', 'beta.fastly.com', '*.hosts.fastly.net', 'control.fastly.net', '*.perfectaudience.com', 'api.foursquare.com', '*.wikia-inc.com', '*.wikia.com', 'f.cloud.github.com', '*.digitalscirocco.net', '*.etsy.com', '*.etsystatic.com', '*.addthis.com', '*.addthiscdn.com', 'fast.wistia.net', 'raw.github.com', 'www.userfox.com', 'www.shazam.com', '*.assets-yammer.com', '*.staging.assets-yammer.com', 'assets.huggies-cdn.net', 'beacon.fastly.'

Storing complete log in /home/otaeguis/.pip/pip.log

So the question is, would it be possible to add an option --no-https-transport or something of the likes that forces http instead of https?

[dstufft]

You should be able to specify an index-url that uses http:// either via config file, command line, or requirements file suck as pip install --index-url=http://pypi.python.org/simple/.

Looking at the traceback whoever is running your proxy has it misconfigured. It looks like it has an old version of the Fastly SSL certificate that doesn’t include pypi.python.org in the SAN. Your best option is to contact them and ask them to fix their broken MITM proxy to update their certificate. After that you’ll probably need to tell pip about your root cert.

[Ivoz]

@earonesty pypi now redirects to https always, is the reason for that.