pipenv injects secrets into Pipfile and lock file

Issue description

When running pipenv in an environment which has a PIP_INDEX_URL environment variable defined, and that env var contains credentials to use when authenticating to the package index server, those credentials are injected as plain-text into the Pipfile and Pipefile.lock files.

Expected result

pipenv should not inject index URLs containing credentials verbatim into the config file.

Actual result

pipenv does inject index URLs containing credentials into the config file.

Steps to replicate

We work behind a firewall with a mirror of the global PyPI repository in our lab. This mirror requires authentication to access. To make this work properly / easily with pip we generally expect users to define a custom PIP_INDEX_URL containing the users’ credentials along with the URL for the mirror.

Further, when using pipenv to create a new environment or install a package (ie: pipenv install requests), we end up getting a Pipfile that looks something like this:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
requests = {index = "https://<credentials>@<server_url>/pypi/pypi-virtual/simple",version = "*"}

[requires]
python_version = "3.7"

I debated whether this should be reported as a “bug” or perhaps just a feature request to have this behavior changed, however given the severity of the impact of this default behavior I thought it was better suited as a bug. The tools should never inject secrets into files that may be commited to version control and this behavior could be very easily overlooked by users of the tool, which makes me very hesitant to roll out this tool for widespread use on my team.

---

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/usr/local/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.7'

Python installations found:

• 3.7.2: /usr/local/bin/python3
• 3.7.2: /usr/local/bin/python3.7m
• 3.6.1: /usr/local/bin/pypy3
• 3.5.2: /usr/local/bin/python3.5m
• 3.5.2: /usr/local/bin/python3.5
• 2.7.16: /usr/local/bin/python
• 2.7.16: /usr/local/bin/pythonw
• 2.7.13: /usr/local/bin/pypy
• 2.7.10: /usr/bin/python
• 2.7.10: /usr/bin/pythonw
• 2.7.10: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '18.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT '
                     '2019; root:xnu-4903.251.3~3/RELEASE_X86_64',
 'python_full_version': '3.7.2',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

• JENKINS_KEY
• TERM_PROGRAM
• TERM
• SHELL
• TMPDIR
• Apple_PubSub_Socket_Render
• VAULT_ADDR
• TERM_PROGRAM_VERSION
• PIP_INDEX_URL
• OLDPWD
• TERM_SESSION_ID
• SDKMAN_PLATFORM
• OBJC_DISABLE_INITIALIZE_FORK_SAFETY
• USER
• SDKMAN_CANDIDATES_API
• PIP_INDEX_URL2
• GITLAB_API_TOKEN
• SSH_AUTH_SOCK
• ART_KEY
• LSCOLORS
• PATH
• PWD
• JAVA_HOME
• LANG
• SDKMAN_VERSION
• XPC_FLAGS
• PS1
• XPC_SERVICE_NAME
• HOME
• SHLVL
• LOGNAME
• SDKMAN_DIR
• GROOVY_HOME
• SDKMAN_CANDIDATES_DIR
• JENKINS_SANDBOX_KEY
• DISPLAY
• _
• __CF_USER_TEXT_ENCODING
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/X11R6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/opt/X11/bin:~/Documents/node_modules/.bin
• SHELL: /bin/bash
• LANG: en_CA.UTF-8
• PWD: *******

---

Contents of Pipfile (‘******/Pipfile’):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
requests = {index = "https://<credentials>@<server_url>/pypi/pypi-virtual/simple",version = "*"}

[requires]
python_version = "3.7"

Contents of Pipfile.lock (‘*****/Pipfile.lock’):

{
    "_meta": {
        "hash": {
            "sha256": "343de2d326ba044580f1fbb17c5ecdf44c9002aa99934966d44a6ef43585c754"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:59b7658e26ca9c7339e00f8f4636cdfe59d34fa37b9b04f6f9e9926b3cece1a5",
                "sha256:b26104d6835d1f5e49452a26eb2ff87fe7090b89dfcaee5ea2212697e1e1d7ae"
            ],
            "version": "==2019.3.9"
        },
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "version": "==3.0.4"
        },
        "idna": {
            "hashes": [
                "sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407",
                "sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c"
            ],
            "version": "==2.8"
        },
        "requests": {
            "hashes": [
                "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
                "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
            ],
            "version": "==2.22.0"
        },
        "urllib3": {
            "hashes": [
                "sha256:a53063d8b9210a7bdec15e7b272776b9d42b2fd6816401a0d43006ad2f9902db",
                "sha256:d363e3607d8de0c220d31950a8f38b18d5ba7c0830facd71a1c6b1036b7ce06c"
            ],
            "version": "==1.25.2"
        }
    },
    "develop": {}
}

[TheFriendlyCoder]

Building upon my earlier comments, perhaps one easy solution for this problem could be to simply respect the use of the enviroment variable expansion in this case, and to preserve the Pipfile contents even when the PIP_INDEX_URL env var is found.

To put this another way, if I rename the PIP_INDEX_URL env var to PIP_INDEX_URL2, update the Pipfile to reflect the change, and re-run my test command, I get the expected behavior. Here is a sample Pipfile to illustrate what I mean:

name = "pypi"
url = "${PIP_INDEX_URL2}"
verify_ssl = true

[dev-packages]

[packages]
requests = {index = "pypi",version = "*"}

[requires]
python_version = "3.7"

With this Pipfile, I can run PIP_INDEX_URL2=$PIP_INDEX_URL && unset PIP_INDEX_URL && pipenv install requests and I get the desired results. The package is installed from the PyPI index url provided by PIP_INDEX_URL2, and the Pipfile remains unaffected by the install operation.

NOTE: This is not a scalable solution for general use, however. Our team manages dozens of Python libraries and applications, not all of which would be using pipenv. For those that do not we would still need the PIP_INDEX_URL env var defined so the traditional package tools continue to work. Further, it would not be practical or safe for us to continually define and undefine this env var depending on the project we are working on because accidentally forgetting to undefine the variable could have the adverse effect of having our private credentials being injected into the config files and risk being committed to version control afterwards. This would be too risky and fragile to be considered a viable workaround.

[matteius]

I believe this is resolvable now – but let me know if its still an issue on pipenv==2022.8.19