Allow configuration of CA certificate in .pydistutils.cfg

Issue

When setuptools retrieves dependencies specified in setup_requires there is currently no way to specify a CA certificate bundle to use for TLS certificate validation of the PyPI repository. We should allow configuration of the CA certificate bundle in .pydistutils.cfg to avoid tedious workarounds and enable specific easy instructions for users using a PyPI proxy with a TLS certificate signed by a custom CA.

Background

We use Sonatype Nexus as a PyPI proxy and to host internal packages. Nexus is configured with a TLS certificate issued by our internal CA. When a user is on a host that does not have our internal CA cert configured globally then pip install fails for packages using setup_requires when a wheel is not used.

For example, installing pylint:

$ pip install pylint --no-binary :all:
Looking in indexes: https://nexus.company.com/repository/pypi-all/simple
Collecting pylint
.venv/lib/python3.7/site-packages/pip/_vendor/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for nexus.company.com has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
  Using cached https://nexus.company.com/repository/pypi-all/packages/04/1f/1d3929051b45c3e4015178c5fe5bbee735fb4e362e0fc4f0fbf3f68647ad/pylint-2.1.1.tar.gz
    Complete output from command python setup.py egg_info:
    Download error on https://nexus.company.com/repository/pypi-all/simple/pytest-runner/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045) -- Some packages may not be found!
    Couldn't find index page for 'pytest-runner' (maybe misspelled?)
    Download error on https://nexus.company.com/repository/pypi-all/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045) -- Some packages may not be found!
    No local packages or working download links found for pytest-runner
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-r60ksvol/pylint/setup.py", line 177, in <module>
        install()
      File "/tmp/pip-install-r60ksvol/pylint/setup.py", line 174, in install
        **kwargs)
      File ".venv/lib/python3.7/site-packages/setuptools/__init__.py", line 128, in setup
        _install_setup_requires(attrs
      File ".venv/lib/python3.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File ".venv/lib/python3.7/site-packages/setuptools/dist.py", line 513, in fetch_build_eggs
        replace_conflicting=True,
      File ".venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 774, in resolve
        replace_conflicting=replace_conflicting
      File ".venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1057, in best_match
        return self.obtain(req, installer)
      File ".venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1069, in obtain
        return installer(requirement)
      File ".venv/lib/python3.7/site-packages/setuptools/dist.py", line 580, in fetch_build_egg
        return cmd.easy_install(req)
      File ".venv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 667, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pytest-runner')

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-r60ksvol/pylint/

We have the CA certificate configured in ~/.pip/pip.conf so we can workaround this by issuing a pip install pytest-runner then rerunning the above command. But this means that any instructions for installing pylint (or other impacted packages) now need to keep information about these transitive dependencies.

If it was possible to provide a CA cert in .pydistutils.cfg then things would be much simpler. With this change implemented, here would be the basic instructions for a user using a custom PyPI and CA:

1. In ~/.pip/pip.conf put:

[global]
index = https://nexus.company.com/repository/pypi-all/pypi
index-url = https://nexus.company.com/repository/pypi-all/simple
cert = /path/to/cacert.pem

1. In ~/.pydistutils.cfg put:

[easy_install]
index-url = https://nexus.company.com/repository/pypi-all/simple
cert = /path/to/cacert.pem

1. Use twine for uploading packages, passing the certificate to --cert or using TWINE_CERT.

Then everything would “just work” regardless of their system configuration.

[idgserpro]

I will give another approach to this problem, if you can use newer versions of setuptools. From >= 42, pip is used instead of setuptools itself to download your packages:

https://setuptools.readthedocs.io/en/latest/history.html#v42-0-0

Mark the easy_install script and setuptools command as deprecated, and use pip when available to fetch/build wheels for missing setup_requires/tests_require requirements, with the following differences in behavior: pip environment variables are honored (and take precedence over easy_install options)

So, in your situation, I would:

• Upgrade setuptools to at least 42;
• Define PIP_CERT environment variable, pointing to your cert, or define PIP_TRUSTED_HOST adding your pypi proxy url.

It’s wasn’t that straightforward knowing that PIP_CERT and PIP_TRUSTED_HOST existed unless you read pip’s documentation in https://pip.pypa.io/en/stable/user_guide/#environment-variables:

pip’s command line options can be set with environment variables using the format PIP_<UPPER_LONG_NAME> . Dashes (-) have to be replaced with underscores (_).

Maybe newer versions of setuptools will honor pip.conf and setting these environment variables will not be needed, but I’m not sure, would need to read all changelog or @jaraco can answer this.

Just for the sake of documentation, if you’re having this problem and are using buildout as well (really common in the Plone world) and don’t wan’t to mess up with environment variables in your server, you can use an extension to solve your problem:

[buildout]

parts =
    environment
 
[environment]
recipe = collective.recipe.environment
PIP_TRUSTED_HOST = yourproxyurl.domain

If this solves your problem, I believe this issue can be closed.

[jaraco]

Setuptools is dropping support for setup_requires and easy_install, obviating this issue. #2824.