Pipenv doesn't support browser auth flow for Azure artifacts-keyring

Background

Trying to install a closed-source python package from my company’s private package feed hosted on Azure DevOps, following these instructions.

The instructions, exactly as presented in that link, work fine using pip install, within a virtual environment (which I entered via pipenv shell). For readers unfamiliar with artifacts-keyring, it causes the pip install process to produce a message on stdout (well, I assume it’s stdout) with a URL to follow and a one-time code to copy, and then wait. The human has to manually browse to the URL, paste the one-time code, then log in to their Microsoft account, then close the browser tab to allow the pip installation process to continue.

But I want to use pipenv install directly if possible!

Notable here as well is that artifacts-keyring is currently in pre-release, which means #3947 is relevant, but I believe I’ve controlled for it below.

Issue description

This would not be directly reproducible for anyone else copy-pasting exactly from below, because the private package feed in question is private. You’ll have to set up your own Azure DevOps Artifacts package feed if you want to completely replicate, I’m afraid. I have anonymized the organization name, package name etc as a precaution, since it wouldn’t work either way.

Start in a virtual environment, with artifacts-keyring already successfully installed by pipenv (using --pre switch because it is a pre-release package).

Try to install my private package:

$ pipenv install mypackage --pre --verbose --skip-lock --pypi-mirror https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/ 

Including --skip-lock works around #3947. Without that flag, I am just reproducing #3947. I don’t think --pre is really necessary here but I threw it in just in case.

Expected result

Successful install. This would have to invoke the browser authentication flow, same as Pip does (provided one uses a totally fresh virtual env for this test; the user access token Pip obtained hangs around within a virtual env, I believe). After doing that I’d expect to get the usual happy messages from pipenv, and to see my Pipfile updated. (But not Pipfile.lock in this case.)

Actual result

The installation process churns for an unusually long time, mostly staying on the [== ] Installing... progress monitor. I don’t see any authentication prompts. Then eventually it spews a lot more output and ends, all in quick succession.

The complete output is attached: pipenv traceback.txt.

Notably, this output clearly shows that multiple attempts were made to carry out the interactive browser authentication flow. But since I couldn’t see them until too late, I couldn’t do them, and there is apparently a 90-second timeout.

So pipenv needs to pass the inner output back up to the user’s stdout much sooner, instead of presumably caching it?

---

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: 'c:\\python38\\lib\\site-packages\\pipenv'

Python location: 'c:\\python38\\python.exe'

Python installations found:

• 3.8.0: C:\Python38\python.exe
• 3.7.4: C:\Python37\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.0',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.18362',
 'python_full_version': '3.8.0',
 'python_version': '3.8',
 'sys_platform': 'win32'}

System environment variables:

• ALLUSERSPROFILE
• APPDATA
• CHOCOLATEYINSTALL
• COMMONPROGRAMFILES
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• COMPUTERNAME
• COMSPEC
• DRIVERDATA
• FPS_BROWSER_APP_PROFILE_STRING
• FPS_BROWSER_USER_PROFILE_STRING
• HOMEDRIVE
• HOMEPATH
• LOCALAPPDATA
• LOGONSERVER
• NUMBER_OF_PROCESSORS
• ONEDRIVE
• ONEDRIVECOMMERCIAL
• OS
• PATH
• PATHEXT
• PIPENV_ACTIVE
• PIP_DISABLE_PIP_VERSION_CHECK
• PIP_PYTHON_PATH
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMDATA
• PROGRAMFILES
• PROGRAMFILES(X86)
• PROGRAMW6432
• PROMPT
• PSMODULEPATH
• PUBLIC
• PYTHONDONTWRITEBYTECODE
• SESSIONNAME
• SNC_LIB
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TMP
• UATDATA
• USERDNSDOMAIN
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• VIRTUAL_ENV
• WINDIR
• PIP_SHIMS_BASE_MODULE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv├╗specific environment variables:

• PIPENV_ACTIVE: 1

---

Contents of Pipfile (‘C:\src\python\testcreds\Pipfile’):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
url = "https://pkgs.dev.azure.com/OrgName/Main/_packaging/OrgNamePyPI/pypi/simple/"
verify_ssl = true
name = "drpypi"

[pipenv]
allow_prereleases = true

[dev-packages]

[packages]
artifacts-keyring = "*"
mypackage = "*"

[requires]
python_version = "3.8"

Contents of Pipfile.lock (‘C:\src\python\testcreds\Pipfile.lock’): Omitted, not relevant, lockfile is outdated due to other issue.

[pataraco]

combining work around here and in the other related issue, i got this to work:

# get a TOKEN
$ export ADO_ARTIFACTS_TOKEN=$(pipenv run keyring get -b artifacts_keyring.ArtifactsKeyringBackend https://pkgs.dev.azure.com/COMPANY/PROJECT/_packaging/FEED/pypi/simple/ VssSessionToken 2> /dev/null)

# Pipfile: set up the index URL to use the token
[[source]]
name = "pypi-ado"
url = "https://azure:${ADO_ARTIFACTS_TOKEN}@pkgs.dev.azure.com/COMPANY/PROJECT/_packaging/FEED/pypi/simple"
verify_ssl = true

# Install the package
$ pipenv install PACKAGE[==VERSION] -i pypi-ado [--verbose]

[rgammans]

The problem there is Microsoft’s documentation is overly simplistic. Actually setting those variable means you are following the instructions in the docs. The only actually variable needed is the ‘VIRTUALENV_SEEDER=azdo-pip’ and the azure_devops_artifacts_helpers package, in the base or outer environment.

What this package and this environment variable do is make sure that the ‘artifacts-keyring’ package is installed when the new virtual environment is created. The Microsoft docs describe a manual process with pip and virtualenv, but pipenv automates the process so we need the environment variables to provide hints/instructions to virtualenv to do the additional steps required. This should be documented for Pipenv, but nowhere does Microsoft seem to document using Pipenv with Python.