Avoid exposing secrets on requirements.txt for --[extra-]index-url with credentials

When running

pip-compile requirements.in --extra-index-url https://${PYPI_USERNAME}:${PYPI_PASSWORD}@gitlab.com/api/v4/projects/.../packages/pypi/simple

the resulting requirements is

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --extra-index-url='https://[USERNAME]:****@gitlab.com/api/v4/projects/.../packages/pypi/simple' requirements_test.in
#
--extra-index-url https://[USERNAME]:[PASSWORD]@gitlab.com/api/v4/projects/.../packages/pypi/simple

...==0.2.7  # via -r requirements.in
numpy==1.19.1             # via ...
scipy==1.5.2              # via ...

where both [USERNAME] and [PASSWORD] are shown in plain text. Since the requirements.txt shall end in version control, specially with post-commit hooks, this exfiltrates secrets to the git repository.

Environment Versions

1. OS Type: Mac
2. Python version: Python 3.7.3
3. pip version: pip 20.2.1
4. pip-tools version: pip-compile, version 5.3.1

Steps to replicate

See above.

Expected result

The secrets should not be outputted to the compiled requirements.txt. Instead, they should keep their original names, as pip accepts.

In the example above, IMO we should only output

--extra-index-url https://${PYPI_USERNAME}:${PYPI_PASSWORD}@gitlab.com/api/v4/projects/.../packages/pypi/simple

both on the header and on the actual requirements. It is the responsibility of the deployment team to ensure that they have the credentials to download dependencies from a pypi server.

Actual result

The secrets are outputted to requirements.txt.

[jorgecarleitao]

closing in favor of #966 .

[ericbn]

The issue here is that the shell is expanding the environment variables, so pip-compile will get the --extra-index-url parameter value with the environment variables already expanded. A solution is to use single quotes, to prevent the shell from expanding them, like:

pip-compile requirements.in --extra-index-url 'https://${PYPI_USERNAME}:${PYPI_PASSWORD}@gitlab.com/api/v4/projects/.../packages/pypi/simple'

But that only works partially. See #1367.